We used RE-AIM (Table 1), a framework designed to address issues related to the implementation of health services and clinical informatics research [26], to identify research questions to evaluate the feasibility and acceptability of the PDTK (Table 2) in a real-world clinical setting. Specifically, we employed a variety of quantitative and qualitative methods to assess the Reach and potential for Adoption, while identifying barriers to Implementation and strategies to Maintain the PDTK for hospitalized general medicine patients. Effectiveness of the intervention on outcomes will be evaluated in future studies.

The PDTK study (Clinicaltrials.gov NCT03116074) was approved by the Partners’ institutional review board and was conducted at Brigham and Women’s Hospital in Boston, MA. The study was conducted on three 30-bed general medicine units from December 2017 through July 2018 in parallel with our AHRQ-funded Patient Safety Learning Laboratory (PSLL; Clinicaltrials.gov NCT02969343) reported elsewhere [27-30]. As part of the PSLL, we integrated a bedside display for patients and clinicians, a patient portal for patients and caregivers, and a safety dashboard for clinicians into our EHR environment (Epic Systems, Inc, Verona, WI) [29,30]. These applications used enterprise data services to obtain clinical data from the EHR in real time [19,30-33]. This EHR-integrated digital health infrastructure (Figure 1) served as a platform on which to incorporate enhancements for the independently funded PDTK study. The key focus of the PSLL was to use systems engineering and human factors methods to design, develop, and implement tools to prevent harm in the hospital (eg, falls, catheter-associated urinary tract infections) [29]. The goal of the PDTK study was to improve safety during transitions out of the hospital by designing, developing, and implementing enhancements to the PSLL infrastructure (Table 2) based on clinician end-user requirements and organizational priorities.

The PDTK comprised enhancements to each of the 3 components of the PSLL technical infrastructure: bedside display, patient portal, and safety dashboard (see Multimedia Appendix 1).

Each of the 3 study units was codirected by a physician and nurse pair and staffed by its own group of nurses. Patients admitted to each unit were cared for by 1 of 2 geographically localized general medicine teams comprising residents or physician assistants and a supervising hospitalist attending physician [34]. These medical teams rotated approximately every 2 weeks. A few off-service patients (ie, admitted to a service other than general medicine) were admitted to these units under the care of a different attending physician.

Any English- and Spanish-speaking patient admitted to these units under the general medicine service was eligible to participate by using any of the patient-facing PDTK components (checklist, video, and secure messaging). For patients who did not have the capacity to consent (as determined by a member of the care team), a caregiver (a designated health care proxy) could participate on their behalf. Patient or caregiver participants were offered access to an acute care patient portal on either personal devices or study-issued mobile devices (iPad Air, Apple, Inc, Cupertino, CA) as part of the concurrent PSLL study [19]. Any clinician (nurse, resident, physician assistant, and attending) caring for a general medicine patient admitted to these units was eligible to participate: all clinicians had access to EHR-integrated digital health infrastructure as part of the concurrent PSLL study and could therefore access the clinician-facing PDTK components (safety dashboard and secure messaging).

In previous work, we engaged patient advisors, clinical stakeholders, information system professionals, and quality and safety leaders to identify gaps in discharge processes [19,35]. For this study, during the design and development phase, we conducted informal workflow observations on study units and interviews with stakeholders to identify end-user requirements for addressing these gaps by engaging patients and clinicians in discharge preparation while aligning with key organizational priorities: engaging patients to improve patient satisfaction, improving expected discharge date (EDD) documentation in the EHR, and reducing 30-day hospital readmissions [31]. For example, improving EDD accuracy—defined by our institution as the percent of final EDD entries equal to the actual discharge date (ADD)—was an organizational priority for improving operational throughput. Thus, to ensure timeliness of checklist completion and review of checklist responses by the care team, we enhanced the EHR-integrated patient portal, bedside display, and safety dashboard to improve the visibility of the EDD for both patients and clinicians (Figure 1). We presumed that the likelihood of checklist submission by patients and review by clinicians would be dependent on where patients were in their hospital course as well as their currently documented EDD.

As in the concurrent PSLL study [27,29,36], we applied user-centered design principles to refine patient- and clinician-facing intervention components (Figure 1) to ensure that we addressed end-user needs [31]. For the discharge checklist (Figure 2, left), our goal was to improve structure and organization, validate content, and clarify wording and utility. Key refinements were identified through multiple iterations of the original checklist within our research team (in part based on our experience with a transitions study funded by the Patient-Centered Outcomes Research Institute [37]), 2 sessions with our hospital’s patient and family advisory council, and a short pilot in which we administered a paper-based prototype to a convenience sample of 10 hospitalized patients and requested feedback. On the basis of the feedback from unit nurses and patient advisors, we also created a video to help hospitalized patients understand the purpose of completing the checklist to prepare for discharge. To develop the discharge video (Figure 2, right), we adapted a method previously demonstrated to improve patients’ understanding of their medical condition and care plan [38]. Finally, we determined that patients would need to watch the video and complete the checklist via one of several workflows: using the patient portal on a hospital-issued mobile device, using their own mobile device, or having research staff coach patients or caregivers to complete the checklist and then submit responses on their behalf.

Next, we made discharge process–based enhancements to the PSLL technical infrastructure integrated with our EHR to ensure that patient-reported information from the checklist would be communicated to the care team as the EDD approached. On the basis of the feedback from patients and clinical unit leadership, we confirmed that patients would want their care team to have access to the checklist responses sufficiently before actual discharge to allow time for the care team to review and address any issues (see Enrollment below). Thus, we developed a checklist submission and review process (Figure 3) to ensure that checklist responses submitted by patients would be visible for clinicians to review in the EHR in real time.

The discharge checklist (Figure 2, left), originally created by Coleman [24], was adapted for our institution [4], and further refined into a four-domain, 16-item discharge checklist. Questions were simplified, reordered, and separated into four domains: My Understanding, My Medications, My Self-Care Management, and My Follow-up. Dichotomous checklist responses (Yes/No) were determined to be too strict (ie, patients were uncertain of their response and did not want to check yes or no); thus, a third option (Unsure) and a box for a free-text response was added. A caregiver version to be completed by health care proxies, as well as a Spanish version (approved by our hospital’s interpreter services) was created to make the checklist more inclusive of patients who lacked capacity or who did not speak English as their primary language (Multimedia Appendix 2).

The discharge video (Figure 2, right) was developed by creating an English version of a script in which a clinician guided the viewer through each checklist domain. This script was translated into Spanish and approved by our hospital’s interpreter services. A mobile device (iPad Air, Apple, Inc) was used to film English and Spanish versions. Video editing software (iMovie, Apple, Inc) was used to produce the videos. The videos were uploaded onto a video-hosting site (YouTube, LLC); hyperlinks to these videos were incorporated into the patient portal and REDCap (Research Electronic Data Capture, Nashville, TN; see Checklist Submission section).

The two options available for patients to complete and submit the checklist were as follows: the patient portal or a Web-based REDCap survey [19,39]. REDCap submissions could be completed by patients on a mobile device (via a hyperlink emailed to the patient), or by study staff who would submit the responses on their behalf. Checklist responses submitted via the patient portal were made visible in the safety dashboard via a shared database. Checklist responses submitted via REDCap were routed to the safety dashboard using the REDCap application programming interface and matched to the corresponding patient using key identifiers (medical record number and admission date). In either case, checklist responses were displayed on the safety dashboard for clinicians within 5 min of submission.

A new discharge column in the safety dashboard displayed key data elements (medical and nonmedical barriers, EDD) from the EHR (Epic Systems, Inc, Verona, WI) using enterprise Web services [27]. EHR data were transformed into clinical decision support using color-coded flags: red=action needed; yellow=risky state; green=guideline compliant; gray=not applicable. For example, a red flag appeared if the current date was past the EDD, no EDD was entered, or the current date was within 24 hours of the EDD, but no checklist had been submitted. A checklist icon appeared on the safety dashboard on successful submission. Checklist items with a no or unsure response were briefly summarized as a keyword on the patient-detail view of the safety dashboard (eg, Meds: Access/Adherence) with a hover-over displaying the specific item answered by the patient (eg, I understand how to get my medications and will take them as prescribed after I leave the hospital). Patients were permitted to enter their mobile phone number during checklist submission to request secure messaging (Imprivata, Lexington, MA) with a hospital physician (attending or senior resident) for up to 7 days after discharge [19,27,40]. A link appeared in the safety dashboard for the clinician to initiate a message thread with the patient.

The intervention went live in December 2017, starting with a 1-month wash-in period in which we debugged various technical components. During business days, recruitment lists were created in which the care unit and patient approach order were randomized to minimize confounding (eg, always starting enrollment on the top floor of the hospital and ending at the bottom). All patients admitted to intervention units for at least 24 hours with EDDs within 24-48 hours were eligible. Research assistants asked the nurses caring for these patients if they were appropriate to approach, temporarily excluding any patient who was nonverbal, incapacitated, or behaviorally not safe. When an eligible patient was confirmed as not capable of participating, the research assistant attempted to identify a caregiver (a designated health care proxy). Patients and caregivers who were not available for recruitment (eg, off unit getting a test) were reapproached later that day or on subsequent days if they still met eligibility criteria. Research assistants then asked eligible patients (or caregivers) to watch the video and complete the checklist (including entering their mobile phone number to request postdischarge messaging) via the patient portal or REDCap workflow. A patient was considered enrolled on successful submission of the checklist.

We used mixed methods to evaluate our implementation experience per each RE-AIM dimension (Table 1). Specifically, we used quantitative methods to measure usage and perceived utility of intervention components (Reach, Adoption, and Implementation). We used qualitative methods to assess barriers to and facilitators of Implementation and Maintenance from the patient and clinician perspective.

After the study was completed, we conducted focus groups with physicians, physician assistants, and nurses to assess implementation barriers and facilitators until thematic saturation. Using a structured guide, we asked about EDD entry via the EHR, perception of the checklist workflow, usage of the discharge column of the safety dashboard to review checklist responses and initiate postdischarge messaging, and awareness of patient-facing components (bedside display and patient portal). Focus group discussions were digitally recorded, transcribed, and reviewed for accuracy.

Descriptive statistics were used to report patient demographic and administrative data, quantify patient-reported concerns, calculate the frequency of tool use by patients and clinicians, and quantify survey data. We calculated the proportion of patients (or caregivers) completing the checklist via the patient portal or REDCap. All qualitative data collected from patient and clinician participants were transcribed, openly coded, and analyzed using the constant comparative method [42]. Two researchers (DP and NP) independently coded all transcripts line by line using Word (Microsoft, Inc, Redmond, WA). The code structure was revised as needed to capture novel concepts, adapt, and merge existing concepts; transcripts were coded with each iteration of the codebook, with any discrepancies resolved during consensus meetings. This process was repeated until no novel concepts were identified, at which point, the 2 researchers again independently applied the final code structure to all transcripts. Key themes were identified in a final group consensus meeting with the study staff. Key implementation barriers were identified by study staff via a group consensus approach based on quantitative and qualitative data.

We used the RE-AIM framework to evaluate the feasibility and acceptability of a suite of EHR-integrated digital health tools to engage patients, caregivers, and clinicians in discharge preparation. Most patients agreed to watch the discharge video and complete the checklist to self-assess discharge preparedness when coached, and we did not encounter significant technical difficulty in our approach. The patient-facing tools were perceived to be valuable by both patients and clinicians, and most patient-reported concerns submitted via the checklist related to medications and follow-up. Clinician use of the safety dashboard discharge column to view these concerns was modest, mostly due to workflow challenges. A large percentage of patients requested postdischarge messaging, but very few clinicians opted in. Themes identified from our qualitative analysis suggest that timing of administration, additional workflow integration, optimization, and leadership are necessary to promote a more robust adoption of these tools.

We attribute the high rate of patient participation to flexible Web-based workflows and facilitation by research assistants. First, the iterative process to develop and refine the checklist and video incorporated feedback from patient advisers and institutional stakeholders, resulting in a product that was relevant and understandable to patients. Next, research assistants functioned as discharge advocates for study participants, guiding them through the process of viewing the video, and completing the checklist. This encouraged participation from patients who might otherwise have not been comfortable or motivated to do so independently. In addition, we included a video component and incorporated a checklist submission process that did not depend on the acute care patient portal: REDCap’s Web-based workflow was useful as a mobile app prototype and also circumvented key barriers to patient portal enrollment and use during hospitalization, such as patients’ dislike of having a separate log-in for the acute care patient portal [19,36]. In contrast, the direct hyperlink to the discharge checklist in REDCap offered more streamlined access (eg, no log-in was needed) for patients, and facilitation by research staff mitigated the perceived burden of submitting the checklist electronically on their own.

Although most patient participants perceived that the intervention would facilitate communication regarding discharge concerns to their care team, often this did not occur because of low uptake of the intervention by clinicians; we attribute this to low awareness of the intervention, inconsistent understanding of its purpose and how to use it, and lack of specificity of patient-reported concerns viewable on the safety dashboard. Specifically, although the safety dashboard discharge column logic was vetted by institutional stakeholders to align with hospital priorities (improving EDD accuracy), clinicians often misinterpreted safety dashboard flag colors (eg, green did not signify safe for discharge) and had a different understanding of responsibility for updating the EDD in the EHR because of inconsistent processes. In addition, if clinicians did not access the detailed view for the patient or click on the flag, they did not see the full text of the flag and what it meant, only its color [43]. Finally, variable use of the discharge column by physicians likely led to poor awareness of the link to initiate a secure messaging thread when requested by patients, and many were resistant to using this feature altogether.

As one of the first reported attempts at engaging patients, caregivers, and clinicians in discharge preparation using a suite of EHR-integrated digital health tools, findings from our implementation study, guided by the RE-AIM framework, offer several instructive lessons (Table 6). First, the optimal timing of when to complete the checklist is paramount. If completed too early, patients perceive less utility, anticipating that the care team would eventually address their concerns. If completed too late, concerns identified from the patients’ perspective are less likely to be communicated to clinicians, leading to potential deficiencies or delays in addressing them. Second, dashboard flag changes were not linked to relevant EHR data elements because of competing workflows (eg, a newly created process for documenting and escalating discharge barriers to hospital leadership) and technical limitations (eg, lack of a Web service to retrieve readmission risk scores from the EHR). Rather than linking flag changes to EDD documentation and timing, using the safety dashboard to review patient-reported concerns about discharge might be more clinically meaningful in the context of patient-specific readmission risk scores; this would be more consistent with the overall intent of the safety dashboard as a tool for proactively identifying patients at risk for harm [27,44], Third, improving EDD accuracy via this type of intervention is more likely to be achieved if the responsibility for updating EDD resides with clinicians rather than unit clerks and is clear to all parties. Patients and clinicians will then have more confidence in the EDD displayed on the patient portal, bedside display, and safety dashboard. Still, it is noteworthy that EDD accuracy for enrolled patients (307/510, 60.2%) was marginally higher than the EDD accuracy rate for general medicine (965/1702, 56.70%) as a whole. Finally, secure messaging after discharge clearly requires a better understanding of factors predicting whether clinicians will use this feature and how to incentivize its use.

Our study has several limitations. First, it was conducted for general medicine patients at a single institution without a control group; therefore, we could not evaluate the impact of our intervention on clinical outcomes. Similarly, our qualitative analyses were performed on convenience samples of participants, which may limit the generalizability of our findings. Second, we used research assistants to coach patients in submitting the checklist; although dedicated discharge advocates are becoming increasingly common, many institutions lack sufficient personnel [45]. In most hospitals, nursing staff could serve patients in this capacity as they are often the first to identify concerns reported by patients preparing for discharge. Third, we identified disparities among those who submitted and those who did not submit a checklist. Clearly, additional work is needed to address disparities in underrepresented groups to fully evaluate the utility of this intervention in a broader population. Finally, this was a hospital-centric intervention—we did not engage primary care physicians. Although our efforts at postdischarge messaging attempted to bridge the transition from inpatient to ambulatory care, the secure messaging vendor used in this study did not offer the ability to communicate with multiple care team members simultaneously, as we previously described [40]. Nonetheless, seamless communication with key ambulatory clinicians is important during the immediate postdischarge period [46,47].

We believe that EHR-integrated digital health tools such as those we described will become increasingly useful as part of an institutional strategy to engage patients, caregivers, and clinicians in improving discharge safety if they simultaneously address key hospital priorities (eg, improving EDD accuracy and mitigating readmission risk). Currently, we are making further enhancements to the intervention components and their implementation. For example, we are reconfiguring the dashboard discharge column logic to more clearly identify patients at high risk for readmission, which should provide context for the types of concerns patients report after completing the checklist. Exploratory features, such as secure messaging with patients, clearly require further investigation to better characterize patient and clinician perceptions of its value and appropriate use after discharge. However, we believe that many of these features will become increasingly utilized to comply with new regulations (eg, Caregiver Advise, Record, Enable [CARE] Act) [48]. Finally, we plan to conduct rigorously designed studies to evaluate the impact of the PDTK on key outcomes during transitions, such as patient activation at discharge, postdischarge health care resource utilization, and hospital readmissions.