++++
Editor’s note: you might find the following links helpful in reading the previous introductory material and chapters in Dr. Kumar’s doctoral thesis:
- Introductory material and Chapter One: http://www.joelsolkoff.com/dr-sonali-kumars-thesis-on-virtual-reality-modeling/
- Chapter Two: https://joelsolkoff.com/chapter-2-dr-kumars-thesis-on-virtual-reality-modeling/
- Chapter Three: https://joelsolkoff.com/chapter-3-dr-kumars-thesis-on-virtual-reality-modeling/
- Chapter Four: https://joelsolkoff.com/chapter-4-dr-kumars-thesis-on-virtual-reality-modeling/
- Chapter Five: https://joelsolkoff.com/chapter-5-dr-kumars-thesis-on-virtual-reality-modeling/
++++
Chapter Six: EVALUATION OF EVPS
The Experience-based virtual prototyping system (EVPS) is developed with added interactivity that allows end users to perform specific health care related tasks while reviewing a virtual prototype. Before embarking on an evaluation of the experience-based virtual prototyping system, it was important to establish the approach for testing the effectiveness of task-based scenarios embedded in virtual prototypes. The first step was to identify and examine prior studies that have performed similar evaluation studies to help define and design the approach for evaluation.
Previous studies have tested the use of virtual prototypes for design review of facilities in various ways. While some studies have compared the use of virtual prototypes with traditional modes of design review (Carvajal 2005; Pinet 1997), other studies have employed both quantitative and qualitative approaches to ascertain the value of using virtual prototypes in design review (Bullinger et al. 2010; Dunston et al. 2007; Patel et al. 2002; Whalström et al. 2009).
One of the first studies by Pinet (1997) evaluated use of virtual prototypes for design review of a waiting room by comparing them to physical scaled models. The results showed that virtual prototypes provide more realism with colors, lighting and realistic point of view as compared to scaled physical models made by materials such as foam core. Another study compared design review of student accommodations using two-dimensional (2-D) drawing and three-dimensional (3-D) models with student participants (Carvajal 2005). An experimental approach was employed using three different design proposals and two visual methods to quantitatively and qualitatively evaluate the spatial cognition of end users identified as non-spatial specialists. The results reveal that increase in understanding is achieved when 3-D models are used rather than traditional 2-D drawings and the study suggests that 3-D models identified an increase in awareness of particular issues.
In another study, Patel et al. (2002) effectively used virtual prototypes during client design review of residential facilities. Their study used qualitative and quantitative evaluation to test spatial cognition – participant’s ability to judge scale and spatial qualities of design, immersion or involvement of participants with the design, richness of view and interaction with features of the technology. The study revealed that appropriate visualization tools have potential to allow end users to perceive and experience architectural space, conveying spatial dimension, contextual information and realism to confirm that the architectural design meets with requirements.
Other studies have used immersive virtual environments such as a CAVE to test the effectiveness of virtual prototypes for design review of patient rooms with end users like patients and nurses using primarily qualitative modes for data collection.
Dunston et al. (2007, 2010) compared physical mock-ups (PMU) with virtual prototypes within the context of health care facilities using a CAVE-like VR system setup. Their study revealed that adding realistic interactive features in virtual prototypes helped the health care practitioner users, reviewing a bariatric patient room model, view the design from unique perspectives and identify safety implications for both patient and care provider.
Whalström et al. (2010) used a qualitative end-user opinion contrasting method in their study comparing the feedback on real environment with a model displayed using an immersive virtual reality system. Using semi-structured interviews, the study focused on determining what issues could be evaluated and exploring end user perceptions about the virtual environment as a design review tool. Their study revealed that the CAVE system used was convenient for evaluating certain design issues such as aesthetics, location of equipment, supplies and materials and positions and sizes of component in patient rooms.
Based on the methods employed in previous studies the next step was to clarify the objectives for evaluation of the EVPS and identify the appropriate approach of evaluation. The objective of the evaluation study was to evaluate effect of embedding task-based scenarios in virtual prototypes on design review feedback generated by health care facility end-users.
6.1 EVALUATION STUDY DESIGN
The evaluation study explored the nature of end user feedback obtained during design reviews by adding task-based scenarios. While previous studies have looked at the use of virtual prototypes for design review as a whole, this evaluation seeks to differentiate between the type of design review that occurs when end users either walk-through the virtual prototype or perform specific task-based scenarios.
Since the study aimed at using actual nurses of the Children’s Hospital for evaluation, it was important to design the study and plan logistics such that the participant’s regular work routine of providing health care to patients was not disrupted. For this reason it was decided to conduct the tests at close proximity to the hospital and schedule the tests on days that were most convenient to the staff. The researcher recognized that it was not possible to achieve a fully controlled setting for the test. According to Cook et al. (1979), a true experiment is not possible due to the lack of control of all aspects. Therefore, a quasi-experimental approach (Cook et al. 1979) was adopted for a user study to evaluate the effectiveness of embedding interactive scenarios in virtual prototypes. The following steps were taken to design the evaluation study:
- Develop stimulus with task-based scenarios and embed faults;
- Operationalize and develop metrics for variable of interest – design feedback;
- Design and develop questionnaire;
- Validate with domain experts and pilot study with researchers; and
- Test experience-based tasks in virtual prototypes for effective design review
To compare task-based scenarios with a generic walk through with limited interactivity, two stimuli of EVPS developed in prior research efforts were intended for use in the study. The difference designed in the two stimuli reflected the varied levels of interactivity and type of scenarios that the participants could perform within the prototypes.
6.1.1 Metrics for Effective Design Review
As a comparison study, the feedback obtained from design reviews was identified as the variable of interest. Metrics for design review feedback were operationalized as the amount of faults that end users could identify with the design and categories or type of feedback that they would provide. To measure the amount of feedback, the study planned to introduce intentional design faults in the virtual prototype. The study used qualitative methods to determine the type of feedback obtained and categorize it based on functional, aesthetic or process-oriented issues as well as explore other categories that could emerge during analysis.
6.1.2 Stimulus Design
The design errors introduced in hospital patient rooms were validated through discussions with the nurse manager of the Hershey Children’s Hospital and by reviewing the documents provided by the architect’s and project manager’s team of the physical mock-up design review conducted with end users the previous year. Design changes recommended during a physical mock-up review were noted for incorporation in the virtual prototype.
Initially the plan was to use the fourth floor virtual prototype of the new Children’s Hospital developed after focus group studies. The stimulus design comprised the nurse’s documentation center, a few patient rooms and the hallway area at the Hershey Children’s Hospital. However, after discussions and validation of the scenarios and model with the Nurse Manager, it was decided to use a generic patient room prototype instead of the new Children Hospital prototype (Figure 6-1). Discussions revealed that reintroducing design faults in the new under-construction hospital could add unnecessary anxiety and confusion in the nurses (end users) as to the actual facility design so late in the construction phase. It was decided to use a generic hospital floor model instead to develop EVPS stimuli for use in the study.
Figure 6‑1. Development of evaluation study stimuli in Unity game engine. |
Seven rooms were modeled around the hallway to depict a typical patient room floor. Charting stations were added in the hallways with windows aligned to look into the patient rooms. Furniture including a chair for family use and storage table with drawers was added along with two different types of patient beds. To add detail and realism, different types of medical equipment were also included in the virtual prototypes. The equipment comprised of IV poles, defibrillators, crash carts, thermometers, etc. Finally, headwalls were added next to the beds to depict outlets for oxygen, air, suction and electricity.
The next step in the stimulus design was to add interactivity and embed task-based scenarios for the test condition stimulus. A mini-map was added to the prototype and could be used by both conditions. The mini-map had a tracker to display the position as well as number marking rooms from 1 to 7. The medical equipment such as carts, IV poles had embedded interactivity that allowed them to be moved or dragged within the prototype using simple mouse clicks. Next, scenarios of patient care and specific activities that the nurses may perform within these patient rooms were discussed with the nurse manager and owner representatives of the project.
After discussion of several possible scenarios that included blood glucose measure of patients, central line insertion by physicians, getting the bed and room ready for new patients, it was finally decided to embed the “patient needs dialysis” scenario. Figure 6-2 shows the development of stimuli for evaluation. The highlighted light green air frames depict the space trigger objects that were used for the dialysis scenario and are explained further in the next section.
The following stimuli were developed:
Control Group (Condition A): Virtual prototype with walk through scenario
Test Condition (Condition B): Virtual prototype with task-based scenario – patient dialysis
The stimuli were exactly the same except that stimulus for task-based scenario condition (B) had greater interactivity as it allowed participants to move objects, monitor progress in doing multiple steps of a scenario as compared to only walking through in the walk through condition (A).
6.1.3 Scenario Development
To embed the dialysis scenario within the prototype, the skeletal framework for scenarios, discussed in Chapter 4, was used with further development. The script for the scenario menu was modified to incorporate the dialysis patient scenario. The dialysis scenario was broken down into steps of tasks that need to be done based on feedback from nurse manager. Each task was then assigned a specific object or action, the relevant script with conditions that would allow user to progress to next step and text that the user could see in the heads-up display (HUD). For example, the dialysis machine geometry with realistic textures was placed in the prototype and DragRigidBody script was added to enable the users to move the machine through rooms. For the scenario to progress to the next step, TriggerClick script was added such that once the user clicked on the machine, the scenario menu would indicate that the first step is completed and prompt the user to do the next step by displaying text on the screen through HUD script. The task-based scenario used was the dialysis scenario and had the following sub-tasks as shown in Table 6-1:
6.1.4 Addition of Design Inaccuracies
Based on discussion and outcomes of the physical mock up design review, a list of intentional accuracies that could be incorporated in the virtual prototype was created (Table 6-2).
The generic virtual prototype was modeled and developed to reflect these design anomalies. Using a generic model allowed the faults to be exaggerated. For example, in Rooms 3 and 4, the beds were placed against the wall and outlets in some rooms were inconspicuously missing. Figure 6-3 shows snapshots from EVPS depicting the faults included in the stimuli.
Figure 6‑3. Snapshots of EVPS stimuli with intentional faults: lack of outlets and overcrowding
Table 6-2. List of design inaccuracies added in stimuliDesign inaccuracies in the stimuli for Condition A and B |
1. No bathrooms with any rooms |
2. Inadequate outlets (Rooms 2, 3 and 4) |
3. Incorrect orientation and location of beds (Rooms 3 and 4) |
4. Layout of furniture and equipment in rooms and hallway |
5. Missing glove boxes in certain rooms |
6. Cart placed in the middle of Room 5 |
6.2 DATA COLLECTION PROCEDURE
The data collection was done over four days in a two-week span at the Hershey Hospital. The study was conducted in a conference room located on the seventh floor, where the children’s patient rooms were also located. The proximity of the conference room allowed nurse participants to take part in the study for the duration of 30-minutes and then resume their daily responsibilities. The size of the conference room allowed for two simultaneous tests to take place. According to Taylor et al. (2011), multiple individuals performing the data collection can help meet the clear and convincing evidence standard in field and case study research settings. Apart from the researcher, four other researchers helped conduct the tests, observe the participants and aid in the overall data collection protocol.
The study protocol required one researcher and one nurse participant per laptop. Participants were randomly assigned to either conditions of walk-through (A) and task-based scenario (B). Dell Latitude E6250 laptops with 2.40 GHz processor and 8.0GB RAM and 15.6 HD display were used to conduct the test. The rationale for testing with laptops was that they provided greater flexibility and could be taken to the hospital for running tests with the nurse participants without causing too much disruption in their work routine. Using standard off-the shelf commonly available laptops also proved the scalability of the developed EVPS through the ability to be displayed via different media.
The laptops were preloaded with Unity executable files of the EVPS for both Condition A and B. Two laptops were set on desks as far away as possible within the conference room to ensure that conversations could not be overheard. This was an important precaution as participants were going through different conditions simultaneously. Video cameras were set on tripods behind each desk for recording and TechSmith’s (http://www.techsmith.com/products.html) Camtasia software was used on the laptops for video screen capture of the test sessions. Although redundant, each test session therefore had a backup of two recordings that included video and screen capture (Figure 6-4).
Figure 6‑4. Data collection and video recording setup. |
Once participants were ready to begin the test session, it was clarified that they were going to view a generic hospital model and were asked to comment while exploring the model. The instructions were exactly similar regardless of the condition except that Condition B had more interactive features and a dialysis scenario that the participants were asked to perform. The participants had 10-15 minutes to review the virtual prototype and could ask questions and give feedback to the researcher throughout the session. After completing the session, participants had to fill out another post-test questionnaire. The entire duration of the study was not expected to exceed thirty minutes, and since the participants were nurses, they were assured that they could stop at any time if they had to attend to their patients or any other work. The following are the various modes of data collection employed during the study:Data collection was also done through a questionnaire that was filled out by participants before and after they explored the EVPS during the test session. A script was prepared (Appendix D) and given to researchers that outlined the exact procedure and briefing to be given to the participant. The scripts ensured that researchers followed the study protocol strictly to avoid any digressions while conducting the study. Once the researchers had welcomed the participants, they were briefly introduced to the study and asked to fill out the pre-test questionnaire. Prior to test session, the questionnaire was used to obtain basic demographic information on the participants as well as to test their familiarity with design reviews and virtual prototypes. After the pre-test, participants were asked to practice exploring the existing 4th floor EVPS of the new Hershey Children’s Hospital. The participants were trained on how to move within the virtual prototype using the keyboard and mouse and were allowed to practice until they felt comfortable.
6.2.1 Pre-test questions
The participants were asked to fill out a set of demographic related questions that asked for their gender and age range. Job related questions asked their role and the department they worked in as well as the number of years of experience they have in nursing. The next set of questions sought to ask about their frequency of computer use outside of work, their comfort level with computers and if they play any computer games. The final set of questions focused on participants’ prior experience if any with design review and tools they may be familiar with for design review including virtual prototypes. Lastly participants were asked if they were interested in exploring virtual prototype of health care facility. Appendix E shows a sample of the questionnaire that participants filled out during the study.
6.2.2 Think aloud protocol
Think aloud protocol and open-ended questions were adopted as the method of data collection during the actual test session. According to van Someren et al. (1994), observations are unstructured techniques in the sense that they don’t constrain subject behavior. Therefore, during the test, the think aloud method, that has its roots in psychological research for problem-solving and knowledge acquisition, was adopted for data collection. Following the think-aloud protocol technique (Van Someren et al. 1994), participants were prompted by the researchers to continue talking and speaking aloud whatever thoughts came into their mind while navigating through the EVPS. The researchers were instructed to prompt the participants to talk about first, any inaccuracies, errors, flaws or faults in the design of the facility, second, any missing features or omissions in the facility model and lastly, any recommendations or design changes they may want in the health care facility.
- Please list some of the design errors (flaws, faults or inaccuracies) you recognized in the health care facility model.
- Please list any missing features (omissions) in the health care facility model you explored.
- Please provide some recommendations or design changes to the health care facility model.
6.2.3 Post-test questions
The post-test questions focused on asking questions related to ease of using the EVPS, feedback on design of the spaces, ability to locate equipment, specific feedback on aesthetics and model content and finally the overall experience of using the EVPS. There were a total of 18 post test questions and each question was phrased on a Likert-type five-point scale. To review the actual questions, please refer to Appendix E. Following are the broad categories of the questions:
6.2.3.1 Ease of use
The first five questions were based on studies related to evaluating usability of VR systems (Kalawsky 1999) such as first person controller mobility as well as ability to move objects and interact within the EVPS. Other questions focused on perceived spatial cognition, which included being able to understand layout, envision various spaces, and tasks that can be performed within the virtual prototype of the healthcare facility.
6.2.3.2 Design review
The next three questions were based on gaining overall design feedback on space layout and equipment layout. The third question was broken down into design feedback based on the following criteria- ability to perform tasks in the spaces, the influence of design on patient falls, confidentiality and privacy of patients, and design’s ability to allow for family-centered care. These questions were adapted from post-occupancy evaluation studies (Becker 1982) and studies evaluating CAVE environments for health care facility design (Whalström et al. 2009).
6.2.3.3 Location of equipment
A set of four questions was specifically framed keeping the developed EVPS stimulus in mind, regarding location of equipment. These questions dealt with the location of dialysis machine, which was used in the task-based scenario condition (B), along with convenience of bed, carts, glove boxes and sanitizer’s locations.
6.2.3.4 Aesthetics and model content
Two aesthetic questions were related to the feedback on the color of walls and overall appearance of the design. The next two questions were model content related that asked if the virtual prototype was detailed and realistic enough.
6.2.3.5 Overall Experience
The last four questions on overall experience asked if the virtual prototype was easy to use, interactive, informative and exciting.
6.2.4 Participants
Study participants were comprised of nurses from the Hershey Children’s Hospital floor currently located on the seventh floor of the facility. There were a total of 33 participants that took part in the study. While there were 15 females in Condition A (walk through scenario), there were 16 females in Condition B (task-based scenario), both Condition A and B had one male each.
Regarding the nurse participants’ specialty, there were mostly charge nurses, few clinical head nurses and other health care practitioners including family centered care, environmental health, nursing education and dietitians. Maximum number of participants were in the 25-30 (30%) and 31-40 (30%) age range, followed by 18-25 and 51-60 age range both at 15% each, with the lowest number in 41-50 (9 %) age range (Figure 6-5). The years of nursing experience of the participants ranged from less than 1 year to over 35 years (Mean = 10.3 years of experience).
Figure 6‑5. Age range of participants.
The average time to explore in both conditions was 711.9 seconds. The average time to explore in Condition A (static/ walk-through scenario) was 713.3 seconds and the average time to explore in Condition B (task-based scenario) was 710.7 seconds (Table 6-3). Figure 6-6 shows a plot of time taken to explore the stimuli by participants in both conditions.Table 6‑3.Time taken by participants to explore the stimuli.
6.3 QUALITATIVE DATA ANALYSIS
Apart from the questionnaire, most of the data were collected through video recording of the participants using the Talk-Aloud Protocol (TAP). This method originated in psychological research on problem solving. Participants are instructed to speak their thoughts as they work on something, usually problems, and speak as though they are speaking to themselves (Crandall et al. 2006). During the test session, participants were encouraged to explicate on what they were doing at the moment and in addition to talking aloud, participants were also probed with questions and provided clarifications by the researcher.
For analysis, data from TAP consists largely of the participant’s verbalizations in the form of audio or video recordings. The video recordings obtained during data collection were transcribed and then coded through a procedure that is referred to as protocol analysis (PA). In traditional protocol analysis, every statement is coded according to some sort of a priori scheme that reflects the goal of the research (Crandall et al. 2006). The coding scheme of the analysis was aligned with the purpose of evaluation, which was to understand the effect on design review feedback. A priori analysis scheme for feedback was to study the amount and type of feedback obtained from both stimuli and to compare and contrast between them.
6.3.1 Analysis Process
The analysis process consisted of four stages of data preparation, data structuring, discovering meaning and representing findings. The first step of data preparation involved transcription of video recordings. The videos were transcribed with the help of QSR international’s (https://www.qsrinternational.com) Nvivo software. Apart from video transcription, the Nvivo software also allowed for content analysis of Think Aloud Protocol (TAP) design review sessions through coding and development of themes and categories of feedback.
6.3.2 Coding into themes and categories
The coding process took place iteratively with multiple passes on data. The first coding was done based on the location or space that the participant was exploring at the moment of giving feedback under feedback in. The second pass of coding focused on the objects that were referred to in the design review feedback and categorized as feedback on. The objects coded were counted to see the maximum and minimum frequencies and compare between the two conditions to see if any patterns emerged. The data were analyzed to see if participants from the walk-through condition (A) were giving more emphasis to certain objects as compared to the participants in the task-based scenario condition (B).
Table 6-4 shows a compiled list of all the categories and their attributes. Each category was coded at different stages of the analysis with a distinct purpose of either data structuring, identifying themes or discovering meaning.
The next step looked at each sentence structure of the transcripts in more detail to ascertain what type of feedback was given. From the a priori coding scheme, the study was interested in looking at feedback on design inaccuracies or faults, missing features and recommendations or design changes. However, it was found that besides these three categories, participants often posed questions or asked for clarifications, had positive and negative reactions, stated some observations and often went into detailed explanations of their work processes. Therefore, an additional five categories on type of feedback were created for coding the transcript data under feedback as. Figure 6-7 shows a sample of coding transcripts using the Nvivo software.
Figure 6‑7. Coding of video into categories of design review feedback using Nvivo software
Analysis of multiple coding showed whether the coding scheme was well defined, consistent and coherent. The inter-coder reliability of the protocol coding is established by testing if the agreement rate is eighty-five percent or higher. Three researchers independently coded statements of different protocol transcripts. First, coding categories (type of feedback and feedback on issues) were defined for the researchers and then acronyms for demarcating the transcripts were established. The researchers then coded random transcripts using the coding guide and instructions, which are attached in Appendix F. Finally, the coded transcripts were compared for inter-coder reliability and the percentage of agreement was calculated as ninety-one percent.
It is often valuable to have more than one coder conduct the protocol-coding task. In some cases it is necessary for demonstrating soundness of research method and conclusions drawn from research (Taylor 2012, Crandall et al 2006). In some parts of the protocol transcript, assessment of inter-coder reliability was not necessary when something was explicitly elicited by the participant or could be objectively determined. For example in the test session videos, as mentioned above feedback in was coded according to the location or space that the participant was exploring at the moment. This coding was done simultaneously with the transcription of the video recording so that the reference could be verified (Figure 6-7).
6.3.2.1 Frequency of feedback
During detailed analysis and further review of the data, numbers from 1 to 33 were used to randomly mark hard copies of transcripts without revealing their condition. This was done to control bias in the qualitative coding of feedback categories:
- Missing features
- Faults/ inaccuracies
- Recommendations
The initial scheme for coding was to first quantitatively evaluate the amount of feedback obtained and then to code the transcripts into categories or type of feedback and finally compare amount of feedback obtained in each category.
6.3.2.2 Identifying Themes
After determining frequency of feedback amount, the next step was to identify themes and patterns that emerged from the data and discover issues of importance that were raised by participants in both conditions. Figure 6-8 shows the coding pattern for a random transcript.
Figure 6‑8. Coding of video into categories of design review feedback
6.4 RESULTS AND FINDINGS
6.4.1 Post-test results
Analysis of the post-test scores regarding ease of use, design review, location of equipment, aesthetics and overall experience between the walkthrough and task-based conditions were expected to show differences in the outcome. However, results from independent-sample t-test of the post-test questionnaire scores revealed that the difference in both conditions was not statistically significant. Appendix G shows a summary of the t-test results. The results were not significant except in cases of ease of finding dialysis machine and ease in moving equipment.
6.4.1.1 Location and movement of equipment
Independent-sample t-test revealed that scores on ease of moving equipment in the task-based condition (M = 3.35, SD = 0.786) were higher than the walk through condition (M = 2.00, SD = 1.93), t(31) = 2.66, p < .05. On the question of convenience in locating the dialysis machine, independent-sample t-test also indicated that scores task-based condition (M = 3.88, SD = 0.697) were higher than for walk through condition (M = 1.69, SD = 1.82), t(31) = 4.64, p < .001. This result is not surprising at all considering that the walk through condition participants were not made aware that they could move equipment. Also, since the task-based condition participants were supposed to perform a dialysis patient scenario, they had better recall of the location of the dialysis machine as compared to the walk through condition (Figure 6-9).
Finally, although not statistically significant, results for location of gloves and sanitizers also show that participants in task-based scenario condition had better recall compared to walk through condition (Figure 6-10).
6.4.1.2 Familiarity with virtual prototypes
On the question regarding their familiarity with virtual prototypes, 36% of participants responded that they were not at all familiar and about 49% of participants responded that they were somewhat familiar while only 15% responded that they were somewhat familiar with virtual prototypes (Figure 6-11).
6.4.1.3 Ease of movement across age and experience
Analysis of the post-test questionnaire revealed that ease of movement was related to the age and experience of the nurses. Figure 6-12 shows that participants in the 51-60 age range had difficulty in navigating through the virtual prototype stimuli and ease of movement reduced as years of nursing experience increased.
Walk through condition (A) participants overall gave 1.27 times more feedback than the task-based (B) condition. This may be attributed to the fact that with approximately equal time to explore the stimuli, walk through condition had more time to give feedback as compared to the task-based condition participants who carried out a patient dialysis scenario.6.4.2 Amount of feedback
The feedback obtained from walk through condition (A) participants on faults, recommendations, explanations, clarifications was greater than from the task-based condition (B). However, chi-square tests indicated that this difference fell short of statistical significance except in the case of negative and missing features feedback. Single-sample chi-square tests revealed that walk through condition gave more negative feedback, χ2(1,N=33)=6.42, p = .01 and more feedback on missing features, χ2(1,N=33)=6.14, p = .01 than task-based condition.
Highest percentage of faults was attributed to furniture placed in patient rooms. Participants had most complaints for a storage unit or dresser by calling it “too bulky” referring to its disproportionately large size and difficultly in working around to provide patient care due to its placement and proximity next to the beds or doors (Table 6-5).
Table 6‑5. Amount of participant feedback in Condition A and B.
Walkthrough (in %age) | Task-based (in %age) | ||||||||
F | M | R | Ex | F | M | R | Ex | ||
1 | Bed/ Head of bed placement | 23.8 | 0.0 | 7.5 | 23.2 | 17.3 | 0.0 | 3.9 | 18.2 |
2 | Head wall (Outlets, Oxygen/ gas valves) | 16.7 | 35.1 | 28.8 | 20.2 | 5.3 | 24.1 | 11.8 | 9.1 |
3 | Equipment (Carts, IV pole, Dialysis machine, Boom) | 13.1 | 32.4 | 17.5 | 14.1 | 14.7 | 33.3 | 26.3 | 15.6 |
4 | Nurse’s documentation (Computers, Charting Window) | 11.9 | 2.7 | 8.8 | 16.2 | 12.0 | 3.7 | 6.6 | 16.9 |
5 | Sink Area (Glove box, Soap dispenser, Sanitizer, Trash) | 3.6 | 8.1 | 11.3 | 3.0 | 14.7 | 20.4 | 23.7 | 20.8 |
6 | Furniture (Dresser, Chair, Closet) | 29.8 | 5.4 | 16.3 | 20.2 | 33.3 | 1.9 | 22.4 | 18.2 |
7 | Storage | 1.2 | 5.4 | 8.8 | 3.0 | 2.7 | 11.1 | 5.3 | 1.3 |
8 | Bathroom | 0.0 | 10.8 | 1.3 | 0.0 | 0.0 | 5.6 | 0.0 | 0.0 |
F = Design Faults, M = Missing features, R = Recommendations, Ex = Explanations |
The next highest percentage of feedback was regarding placement of beds and proximity of furniture to head of the beds restricting access to patients. Another bed-related concern was location of beds vis-à-vis slanted windows at the nurse’s charting stations and ability to monitor patients through them. Placement of beds was a common design inaccuracy and fault in both conditions. Maximum negative feedback on bed’s placement was received in Rooms 3 and 4. The only room that received some overall positive feedback on bed placement was Room 7.
Missing bathrooms were identified by both conditions, but overall participants in walkthrough Condition A pointed it out 2.6 times more than in Condition B.
6.4.3 Categories of feedback
Recommendation put forth by participants in both conditions were analyzed in detail to discover the important issues they referred to through their feedback. Since participants in the walk through condition had identified general bed and outlets-related issues much more than those in the task-based scenario condition, they had also raised much more concerns regarding patient care access especially during patient emergencies. While walk through condition participants raised more issues and their recommendation for design changes were related to patient safety through moving furniture, task-based condition participants went into greater detail to explain the process and raise issues regarding patient privacy and infection control. Table 6-6 shows recommendations for design changes, the issues they raised along with the condition that mentioned or discussed the issue more during the talk aloud protocol.
Table 6‑6. Design change recommendations and issues raised based on condition.
Recommendations | Issues raised | Condition | |
1 | Move furniture away from head of beds for more access | Patient emergency | Both |
2 | Alignment of charting windows with head of bed | Patient care and privacy | Both |
3 | Missing Bathrooms | Patient care | Both (mostly walkthrough) |
4 | Chairs for parents to sleep on | Family care | Both |
5 | Faucet near head of bed | Patient care | Task-based |
6 | HIPPA compliant monitors | Patient privacy | Task-based |
7 | Isolation supplies: Dirty linen and gowns | Infection Control | Both (mostly Task-based) |
8 | More sharps containers and sanitizers | Infection Control | Task-based |
9 | Better storage of equipment | Nurse work process | Walkthrough |
10 | Lighting for different patient types | Patient care | Walkthrough |
6.4.3.1 Patient emergency
Both Conditions A and B raised issues on patient emergency and access to patient bed that are challenging due to the placement and proximity of furniture. There was a lot of negative feedback related to placement of the bed against the wall in Rooms 3 and 4, as well as recommendations to move furniture away to allow access for patient care. Table 6-7 has quotes from walk through condition participants regarding patient emergency feedback. Within the issues of patient emergency, maximum amount of faults, negative feedback, recommendations and explanations were related to the “dresser” which referred to furniture placed for storage in the patient room. Table 6-7 shows that there was a great deal of negative feedback on the size and position of the dresser.
Table 6‑7. Feedback related to issue of patient emergency.
“Well here you can’t get on both sides of the bed, you have to be able to do that. You have no monitor, no poles, no outlets… nothing like around the bed, you have no carts no bedside tables” | Participant 24-walkthrough condition in Room 4 |
“That’s terrible because if you know there’s an emergency there is no way to get to the patient. I think with these being here [furniture], you won’t be able to go into the head of the bed and almost always in an emergency, you need to get to the head quickly.” | Participant 06- walkthrough condition in Room 3 |
“… that’s really odd… you would never have that [bed next to the wall Room 3] as you wouldn’t be able to get to the other side of the bed. So I would put it where the chair is or where the dresser is and put the dresser there [pointing to the wall where bed is]. You want access to all sides of the patient bed where that’s… they are trapped in there. It almost seems like a restraint… they should be able to exit both sides of the bed.” | Participant 04- task-based condition in Room 3 |
“And I think that dresser is in a weird place…it would make it difficult to maneuver if there was an emergency situation…[] quite honestly I would scrap the dresser in every room cause I just feel like it’s in the way everywhere you go!” | Participant 06- walk through condition |
“This is really too cramped…furniture is too big… again you can’t access the one side of the patient and barely jump over the chair to get to patient on other side” | Participant 23- task-based condition |
6.4.3.2 Work process-related
During the dialysis scenario, participants in task-based condition gave recommendations and relevant feedback by going into detail of how they would perform dialysis and how the current design may not be the most friendly to allow them to do that. Figure 6-13 shows how participants while walking through the dialysis scenario recommend design changes.
Figure 6‑13. Description of the dialysis scenario during design feedback and recommendations.
Unlike participants in walk through condition who were relatively impressed by Room 1 that had maximum number of outlets, participants in Condition B went through detail and were able to point out that proximity of sink and faucet to the head of bed was required for this task to be carried out in a safe manner. Table 6-8 shows quotes from participant transcripts describing the dialysis process.
Table 6‑8. Feedback related to work processes for dialysis scenario.
“If you have to do dialysis, you generally need the sink… [] You have to put the machine close to the patient but then you have to drag the tubes, like the hoses…this may not work well… []the hoses are going to be along the floor and up and over and then people aren’t going to be able to walk through unless there’s long…. the sink needs to also be closer to the bed.” | Participant 26-task-based condition |
“Well I know that when we do haemo-dialysis on our floor, it [machine] has to get hooked to the sink…the tubing… like the way that it is now, they have to…[] hook up to faucet cause the water comes in and warms in the machine and so they need a faucet but they also need a plug… so I mean this is not bad… but I think that there should be a plug by the sink so that we don’t have to plug it one way and then the faucet is going the other way” | Participant 23- task-based condition |
“I think during dialysis, the workstation [computer] would probably be better inside the rooms so that the nurses can stay right there… may be a box of gloves on this [bed] side would be better so you can just quickly move through soiled gloves. And since these are dialysis rooms, …[] we would want both sides of the bed to be kind of clear because there’s lines and tubes going throughout…. so that’s my experience.” | Participant 08- task-based condition |
6.4.3.3 Infection control
Other dialysis task-specific feedback was related to having long beds with cushions for patient comfort; push knobs for sinks, access to glove boxes and sanitizers, location of sharps, trash and dirty laundry accommodation outside the rooms for infection control and greater efficiency in nurse work processes. Task-based condition participants raised issues of infection control in much more detail as compared to walkthrough conditions (Table 6-9).
Table 6‑9. Feedback related to issue of infection control.
“I’m just used to having the soap dispenser right here [pointing next to sink], whereas here you may be dragging soap all across the sink area” | Participant 15- task-based condition |
“One thing we have with receptacles or the sharps containers is that instead of having them over this area [close to sink] because if you wash your hands, throwing things out, you are going to drip water all over this counter, so the counter then can’t really get used so maybe then putting this on the side or like right here [pointing on the lower cabinets]” | Participant 15- task-based condition |
6.4.3.4 Patient privacy
As a step in the dialysis scenario task was using the computer station to chart, participants in the task-based condition raised patient privacy issues due to the orientation of the computer screens. Some participants alluded to the Health Insurance Portability and Accountability Act (HIPAA) Privacy and Security Rules for location of computer screens (Table 6-10).
Table 6‑10. Feedback related to issue of patient privacy.
“This leads to privacy issues, HIPAA confidentiality type thing. Like on our floor we have the nursing station so that patients can’t go back there and here if somebody is trying to get to a room, they just walk right past on here.” | Participant 30- task-based condition |
“…In charting stations, as far as privacy goes, HIPAA and privacy information… if we had guests walking through, [we would need] privacy screens or anything like that… as I know there are sometimes issues with nurses walking away from their computers if they get called away… and we are supposed to secure our sessions and hide our online charting, but … just for privacy, the computers can be angled in such a way they don’t have the ease of visibility.” | Participant 29- task-based condition |
6.4.3.5 Equipment Supplies and Storage
Both task-based and walk through condition participants recommended that there should be additional specific equipment based on the everyday work processes of nurses. There were many recommendations for better storage and clearing of excess equipment from common areas such as hallways and ensuring that emergency equipment like code carts are accessible in cases of emergencies (Table 6-11).
Table 6‑11. Feedback related to equipment supplies and storage.
“I don’t see a lot of room for IV poles, for a ventilator, for respiratory equipment, for things that take up a great deal of space for our chronic respiratory kids, this would be very tricky to get it in the room and have it safely out of the way to access the patient.” | Participant 25- walkthrough condition |
“Putting beds and equipment in corridors… JCo [Joint Commission] will never approve.” | Participant 10- walkthrough condition |
“I guess another thing would be isolation supplies, usually they are just placed outside the room where they have their gowns and gloves and …equipment that they can get before they enter it. And then with that, before they leave, they can put their dirty gowns and dirty linens as well.” | Participant 17- walkthrough condition |
6.4.3.6 Navigation
Apart from giving design feedback, participants in both conditions also commented on ease of movement, interactivity features and ability to better understand the space within the virtual prototype. Participants compared the prototype with traditional forms of design review like 2D plans and claimed that virtual prototypes were more realistic and visual, thereby enabling them to better envision tasks and location of objects. They also compared the virtual prototype to a game and remarked on the ease of movement and visualization of design.
“You get a more realistic- rather than someone explaining this is how it would look or just looking at a map. I like visually looking and walking through thinking that if I was really here, I would be able to do so it’s easier for me to picture what I would do.”
However, participants also gave negative feedback on navigation, wherein they got frustrated trying to maneuver within the virtual prototype. One of the participants even opted out of driving through the virtual prototype while giving design feedback.
Another participant in the task-based condition noticed and commented that moving the dialysis machine was not very easy or intuitive. This comment indicates that more consideration may be required in developing interactive objects in the future. She quoted:
“I think there is a slight disconnect as this [dialysis machine] is going separately from me moving back… do you know what I mean. I feel like if I push this [dialysis machine] in; I should already be in the room.”
In all, participants did find the virtual prototypes regardless of the condition they were in to be very helpful and useful in visualizing the design. The following quote sums up the attitude that nurses had towards the EVPS:
“I mean I think as nurses we are all more visual people and not like…most of us I think if you give us a map, we struggle with it or blueprints which are just like maps for us… this is much better”
6.4.4 Additional Scenarios
Interestingly, while reviewing the virtual prototypes and offering design feedback, many participants, especially in the walkthrough condition proposed scenarios of healthcare activities as additional interactivity during the review. One of the participants was a nurse educator with over 25 years of experience in teaching and training new nurses. Table 6-12 shows a summary of the quotes related to scenarios proposed. Even in the task-based scenario condition, participants offered supplementary scenarios to the dialysis patient scenario they performed.
Table 6‑12. Additional scenarios proposed by participants.
Proposed Scenarios | Issues Addressed | Participant Quote |
Patient medication and room preparation | Training new nurses on patient care | “I’m thinking of a scenario that would say… you need to get your noon time medications []… you would need to maneuver to figure out where those stations might be []… how to make it resemble part of the tasks of the day… of what you might be doing.So even like simple tasks like where would I get linen for the patient, for the visitors that might be staying or where would I go to …or how far would I have to go to get my medications or…practice working with the new graduates now to practice what is a safe bedside setup. If its my first day and I had to set up this bed spot for new admission, I would want to think about… would there be a way to have choices that I could make…. to put in a suction apparatus here, I’d hook up my oxygen, I’d prepare the bed with linen and perhaps a list of different medications that I might need to get for the patient.” [12-A] |
Typical daily workflow routine for nurses | Nurse work processes and routines | “Scenario – A scavenger hunt or something… when you start of the day. To go from your room to where the medication pyxis are, where the residents are the doctors are the break room, where to find things. Just seeing my actual unit on here will be really helpful I think” [24-A] |
Code response | Patient Emergency | “If we would have a code or rapid response or if we had to go find equipment” [22-A] |
Overall layout and way finding | Way finding training | “This would be helpful to know if there any additional elevators… you can’t really tell the flow of this place. Maybe just some labels, [] I’m really confused to where I am right now so if I saw that this was Room 3 and I had already been in Room 3, it would help me kind of knock it off in my head.” [21-B] |
6.5 DISCUSSION
Results of the EVPS evaluation show that both walk through and task-based scenario conditions tested are relevant to gain design feedback. Even though the quantity of feedback was higher in the walk through condition, adding task-based scenarios was valuable for getting detailed task-specific feedback from participants. The value of adding task-based scenarios in virtual prototypes can be seen through the quality of feedback and the level of engagement with the end users. The participants in task-based scenario conditions gave relevant and detailed feedback on work processes, infection control and patient privacy issues, which participants in the walkthrough condition overlooked.
6.5.1 Quality of feedback
While feedback from the walkthrough condition gave a higher-level overview of the design, task-based scenario stimulus proved appropriate for delving deeper into functionality details of the spaces being reviewed. Task-based scenario condition definitely helped nurses to focus on giving feedback on the “task” at hand; it was also important that the nurses were able to relate to the scenario and provide meaningful design feedback. During the study, it was also observed that nurses usually leveraged their departmental know-how and tacit knowledge, referring to different situations and scenarios they usually encounter for giving feedback. For instance, a nurse working in surgery or PICU remarked more often on missing IV poles, defibrillator equipment and oxygen outlets, while environmental health workers would focus on missing “Purell” hand sanitizers.
6.5.2 Level of engagement
Apart from just walking through the virtual prototype of the facility, participants in the task-based condition also followed a series of steps from a patient dialysis scenario. The scenario involved finding the dialysis machine, taking it to the patient bedside, going towards the sink and finally following up with charting on the computers outside within the virtual prototype. These set of dialysis patient tasks helped the end users better envision how their typical daily tasks are tied to the space. The scenario led to a higher level of engagement with the virtual prototype compared to the walk through condition, where participants merely walked through and focused on the design faults.
6.5.3 Recommendations
Based on the evaluation results of the user study, it is recommended to embed appropriate task-based scenarios in interactive virtual prototypes if detailed, specific and health care process-based feedback is desired from the end users. Moreover, using the EVPS that combines task-based scenarios with interactive virtual prototypes also enables a higher level of engagement with the end users, as they are able to envision the tasks they perform more vividly. Apart from obtaining design feedback, using task-based scenarios also helps the health care staff and end users reexamine their current work flows and assess if they need to alter their operations according to the new space layout depicted in the virtual prototype.
Even though using task-based scenarios is more beneficial in obtaining detailed end user feedback, it is important to provide sufficient time for end users to walk through the virtual prototype as well. This permits the end users to get a better overall perspective of the design before performing detailed health care scenarios.
6.6 CHALLENGES
Certain challenges were encountered during data collection. Because the study was in the live hospital setting, it was prone to unexpected occurrences. Participants for the study were nurses that worked on the same floor so that it did not take them away from their duties for prolonged periods of time. Nurses were only able to participate based on their schedules and availability. One of the nurses had to leave mid-way because she received a call for a patient. A few nurses got calls during the study and recording had to be stopped when a nurse had to talk on the phone for an extended period of time. On the last day of data collection, an accreditation program was taking place in the hospital and it became extremely difficult to recruit nurses to participate in the study, especially since most of the nurses from the Child Specialty wing had already participated.
Since the study was prone to unforeseen circumstances, it was useful to have multiple reviewers helping during data collection. However, multiple reviewers made it more challenging to follow the study protocol precisely. After data collection there were some missing recordings, but having both screen capture recordings and video recordings ensured there was backup for each participant
In regards to the stimuli used in the study, one of the drawbacks of using the task-based scenario stimulus was that moving virtual objects within the prototype was still tedious for some participants and future effort needs to go into making the user interface for moving objects more intuitive. Observations during the study indicated that more experienced or elder nurses tended to give detailed and relevant design feedback regardless of the condition they were in, although they preferred the walk through condition for navigation purposes. Even in the walk through stimulus, one of the participants did not feel very comfortable to “drive” and asked the interviewee to navigate and explore the virtual prototype for her while she gave feedback.
Finally, despite using a generic hospital model to develop the stimulus and briefing participants before the actual test session, there was some confusion regarding the purpose of going through the generic space along with clarifications and concerns over whether the stimulus depicted the actual new Hershey Children’s Hospital rooms.
6.7 LESSONS LEARNED
While this evaluation study revealed interesting insights into how participants leverage task-based scenarios in interactive virtual prototypes, there are many lessons to keep in mind for future studies. One of the first indications during data collection was the need for having a better-defined protocol for the study. Since the evaluation was exploratory in nature, multiple modes of data collection were adopted. Future studies can have more focused questions and leverage the most effective data collection and analysis procedures for obtaining end user feedback.
The post-test questionnaire hardly conveyed any statistically significant quantitative results. One of the reasons for this could be that during the test sessions, considering the bed locations for example, certain rooms were critiqued more compared to others. However, the questionnaire at the end of the test asked for general feedback regarding bed location when participants could be thinking of any of the rooms leading to variable answers. Hence, future questionnaires could be designed to be more specific and focused or the talk-aloud protocol data could be used as a source of analysis to get a deeper understanding of the participants’ views.
In retrospect, getting a greater amount of feedback from the walkthrough condition was expected since participants in this condition spent more time exploring the overall space layout of the corridor and especially viewing rooms that had a higher number of design inaccuracies (Rooms 2, 3, and 4). It is possible that participants in Condition A got more time to give feedback as compared to Condition B participants. Within the equivalent average times allotted for navigation, participants in the task-based condition also carried out the patient dialysis scenario while spending maximum time in Room 1, thereby leaving them with less time to review the overall space layout.
Lastly, the lesson learned for using task-based scenarios was that scenarios need to be better tailored to get relevant feedback. While the dialysis scenario was chosen as a generic scenario of patient care that requires unique work processes, it was realized during the study that not all nurses would be familiar with the scenario as is depicted in the following quote:
“…And then pumps, usually I would be more apt to hook a patient up to pumps rather than a dialysis machine which is a specific nursing task…like there is a dialysis nurse. So if you are presenting this to nurses that are actually using the space, it’s not really something that I do, I wouldn’t even know what I’m looking for to hook it up. I know what to hook an IV pole up to… or a pump or oxygen and stuff like that.” [Participant 21-task-based condition]
For scenarios of tasks to be relevant in future design reviews, it will be imperative to develop and embed pertinent task-based scenarios in interactive virtual prototypes before they can become effective experience-based design review tools for health care facilities of the future. Hence, future EVPS developments for health care facility design reviews could be a combination of both conditions. Another area of effort needs to be addition of multiple scenarios that participants can choose or switch between based on their expertise or preferences.
6.8 SUMMARY
This chapter described a study to evaluate the effect of adding task-based scenarios in interactive virtual prototypes to make them experience-based design review tools. The chapter began with an overview of prior evaluation studies on virtual prototypes used during design review to better inform the approach for design of evaluation study. Next, the study procedure, data collection and analysis methodology are described. Finally, results from the evaluation study reveal that both walk through and task-based scenarios are relevant for providing design review feedback. Further, task-based scenarios in virtual prototypes may provide more detailed feedback from end-users of healthcare facilities.
–30–
Copyright 2013 by Sonali Kumar. All rights reserved. Thesis published on this site by the express permission of Sonali Kumar.