Chapter 3 RESEARCH METHODOLOGY
Research is defined as a “systematic, intensive study directed toward fuller scientific knowledge of the subject studied.” (Blake 1978). Research can either be pure, undertaken to develop knowledge and contributor to existing theories or applied, which seeks to address issues of application and solve practical problems (Fellows and Liu 2003). A research methodology consists of a combination of process, methods and tools that are used in conducting research in a research domain. According to Blake (1978), methodology includes the assumptions and values that serve as a rationale for research and the standards or criteria the researcher uses for interpreting data and reaching a conclusion. Research methods are at the basis of the production of knowledge in any given field (Dubé and Paré 2003).
A research process involves understanding research domains, asking meaningful questions, and applying valid research methodologies to address these questions. Research follows a pattern of “problem, hypothesis, analysis, and argument.” During research design, one has to decide the methodological approach in finding solutions to the research problems or questions (Fellows and Liu 2003).
3.1 RESEARCH APPROACH
Research methods most applicable to the construction domain include action research, surveys, case studies, experiments and ethnographic research (Fellows and Liu 2003). With the advent of information technology in the AEC domain, construction research needs to adopt research approaches and methods that better serve their research purpose. Research domains such as information systems are sufficiently broad that they require a wide range of methodologies. Information systems research approaches were explored for adoption in this study.
3.1.1 Information Systems Research
Information systems research is generally interdisciplinary concerned with the study of the effects of information systems on the behavior of individuals, groups, and organizations (Galliers et al. 2007). An applied research discipline, information systems benefits from employing “plurality of research perspectives to investigate information systems phenomena” (Orlikowsky and Baroudi 1991). Action research methods are also very applicable in IS research (Baskerville and Wood-Harper 1996).
According to Hevner et al. (2004), two paradigms characterize much of the research in the Information Systems (IS) discipline: behavioral science and design science. The behavioral science paradigm seeks to develop and verify theories that explain or predict human or organizational behavior. The design-science paradigm seeks to extend the boundaries of human and organizational capabilities by creating new and innovative artifacts (Hevner et al. 2004).
March and Smith (1995) proposed a framework for researching different aspects of Information Systems including outputs of the research and activities to carry out this research. They identified research outputs as follows: 1) constructs which are “concepts to characterize phenomenon”, 2) models that “describe tasks, situations, or artifacts”, 3) methods as “ways of performing goal directed activities”, and 4) instantiations which are “physical implementations intended to perform certain tasks”. The defined research activities as: 1) build an artifact to perform a specific task, 2) evaluate the artifact to determine if any progress has been achieved and 3) theorize and justify theories about artifacts developed.
Similarly, Nunamaker et al. (1991) advocated the integration of system development into the research process, by proposing a multimethodological approach that would include 1) theory building, 2) systems development, 3) experimentation and 4) observations.
Figure 3-1 presents Hevner et al.’s (2004) conceptual framework for understanding, executing, and evaluating IS research combining behavioral-science and design-science paradigms. The framework focuses on development of artifacts by applying theoretical knowledge and knowledge of requirements from the environment or context of application.
Figure 3-1. IS Research Framework (Source: Hevner et al. 2004).
3.1.2 Systems Development
Systems development is a multimethodological approach to information systems (IS) research (Nunamaker et al. 1991). System development serves both as a “proof of concept” for the fundamental research and provides an artifact that becomes focus of expanding and continuing research. The multimethodological approach to IS research proposed by Nunamaker et al. (1991) consists of four research strategies: theory building, experimentation, observation and systems development (Figure 3-2).
Theory building includes development of new ideas, concepts, and construction of conceptual frameworks, new methods or models. Theories may be used to suggest research hypotheses, guide the design of experiments, and conduct systematic observations.
Observations include research methodologies such as case studies, field studies and sample surveys. It may help researchers to formulate specific hypotheses to be tested by experimentation, or to arrive at generalizations that help focus later investigations. Since research settings are more natural, more holistic insights may be gained and research results are more relevant to the domain under study.
Experimentation includes research strategies such as laboratory and field experiments as well as computer simulations. It bridges the gap between theory building and observation as experimentation helps validate underlying theories or issues of technology transfer and acceptance. Experimental designs are guided by theories and facilitated by systems development. Results from experimentation may be used to refine theories and improve systems.
Systems development consists of five stages: concept design, constructing the architecture of systems, prototyping, product development, and technology transfer. Systems development is the hub of research that interacts with other research methodologies to form an integrated research process (Nunamaker et al. 1991).
3.1.3 Adopted Research Approach
Advancement of information systems (IS) research and practice often comes from new systems concepts. In engineering system domains, the concept at issue is viewed for its application value rather than its intrinsic value. The concept with wide range applicability goes through a research life cycle of the form: concept – development – impact. Much IS research demonstrates such a life cycle (Nunamaker et al. 1991). Their paper suggests that “theories” are needed to identify broad classes of things that can be done more efficiently or effectively, “instantiations” are needed to provide a continuing test bed for theories, and that “evaluations” of particular instantiation (systems) are needed to quantify the success or failure of the system in both technical and social terms. Systems development provides the exploration and synthesis of available technologies that produces the artifact, which functions as a bridge between technological research referred in the study as the “concept” stage and social research, which has been referred to as the “impact” stage.
3.1.3.1 Concept
Initially, literature from various domains of experience-based design in healthcare and virtual facility prototyping were studied to find opportunities for developing systems that could improve the current design review process. Observations during a pilot study of an interactive virtual prototype in an immersive virtual environment showed that end users envision tasks they perform in their respective facilities while giving feedback during design reviews (Leicht et al. 2010). A concept emerged of developing interactive virtual prototypes that allow end users to visualize performing tasks while reviewing the space. This concept was termed as the “Experience-based Virtual Prototyping System” or the EVPS.
EVPS = Interactive virtual prototype + end user’s task-based scenarios
Literature review helped define the research problem by finding gaps in prior research. A review of multiple studies showed that game engines could be effectively used to develop interactive virtual prototypes and possibly embed task-based scenarios. However the studies also show that it is quite cumbersome and time-consuming to develop these virtual prototypes and their impact had not been assessed in detail (Figure 3-2).
Figure 3-3. Research process.
Based on the literature review it can be summarized that the research problem is regarding the efficient development and subsequent testing of the EVPS. Although interactive virtual prototypes using game engines have been used for design review, there is a lack of streamlined procedures to make development easier. Moreover, because the interactive virtual prototypes are so difficult to develop, it is imperative to know if end users would actually benefit from using task-based scenarios embedded in virtual prototypes during design reviews.
These gaps and problems raise pertinent research questions on both the development and impact of the concept of experience-based virtual prototyping system.
1) How can we extract end user scenarios of tasks to embed in virtual prototypes?
2) How can we more effectively develop interactive virtual prototypes with task-based scenarios?
3) How can we use experience-based virtual prototypes (EVPS) in healthcare setting?
4) How can we evaluate effectiveness of experience-based virtual prototypes?
Research questions led to formulation of objectives. Each objective was broken down into a set of tasks and sub-tasks that will be discussed in the research steps.
3.1.3.2 Development
The development part of Concept – Development – Impact model dealt with investigating procedures to develop the concept further. This step explored methods and procedures that could take the EVPS concept into the development stage. The first two research questions focus on developing a process to first, understand what tasks should be incorporated in virtual prototypes and second, figure out an effective way to develop the interactive virtual prototypes.
Development process enabled that through the study of relevant disciplines for discovering new ideas and approaches. Domains of interest included virtual prototyping, gaming engines, and scenario-based design. Next, processes to rapidly transfer model content from existing applications into an interactive virtual prototyping environment were tested by “prototyping the prototype”. Chapter 4 gives a detailed description of the scenario framework and EVPS development process.
Various healthcare-related projects and small case studies were used as test-beds throughout the development process to streamline design information workflows. While the concept EVPS design was part of healthcare facility design, process to embed scenarios in an interactive virtual prototype were tested through a project on independent living facilities for the elder (Kumar et al. 2011). Once an efficient procedure to develop the EVPS concept was defined, it was implemented on a healthcare facility case study to assess the impact.
3.1.3.3 Impact
Research questions related to how EVPS can be implemented in a healthcare setting and how it would affect the design review process for end users were studied within the impact phase of the “Concept – Development – Impact” model. Evaluating impact of the developed concept – EVPS was done using a multi-method approach of case study and experimentation. The case study chosen for the evaluation phase was the new Hershey Children’s Hospital (Chapter 5) that was under construction during the study. The EVPS development process was tested on the Hershey Children’s Hospital, by first using focus groups to elicit requirements and then developing interactive virtual prototypes for use by end users of the facility. Evaluation of the use of EVPS was done through observational field study during design review meetings with the pharmacy staff of the hospital.
Next, a user study tested the effect of using task-based scenarios embedded in the interactive virtual prototype on end user feedback during design review. The evaluation study is described in detail in Chapter 6.
3.2 RESEARCH STEPS
Once the research approach was defined, objectives for the research were broken down into specific research tasks and activities. The following describes specific research activities assigned to accomplish the objectives.
Research Activities for Objective 1
Objective 1: Develop a virtual prototyping procedure to extract end user experience of healthcare activities in interactive virtual prototypes.
- Develop a procedure to document scenarios of activities within the healthcare facility context.
- Develop a hierarchical data structure for task-based scenarios that characterizes attributes of the interactive objects used and specific tasks performed within these scenarios.
Research Activities for Objective 2
Objective 2: Design a framework for structuring end-user activities into scenarios that can be simulated in an interactive virtual prototyping system.
- Design of overall system architecture for interactive virtual prototyping that defines the components, databases and libraries that will be used for development of the virtual prototyping system within a rendering/ gaming engine.
- Classify design review requirements along with additional features and functionality that can be added to the virtual prototyping system for interactive design review.
- Development of an interactive interface to allow the end user to carry out specific task-based scenarios in the virtual environment through scenario definition, scripting and use of interactive objects.
Research Activities for Objective 3
Objective 3: Develop an interactive computing platform titled the Experience-Based Virtual Prototyping System (EVPS) for implementation in healthcare design reviews.
- Utilize an appropriate programming environment / gaming engine for the development of an interactive virtual prototyping system termed the Experience- based Virtual Prototyping System (EVPS).
- Investigate design information workflows for importing facility models from various BIM authoring tools (E.g., Autodesk Revit) to the interactive virtual prototyping system. This would include identifying challenges and limitations in the workflow to facilitate rapid conversion of facility models into the virtual prototyping system.
- Generate reusable interactive virtual content models such as avatars of user roles and objects with dynamic behaviors like wheelchairs that are needed for the development of interactive virtual prototypes.
- Validate the procedure used to develop and implement task-based scenarios in the interactive virtual prototyping system and review capabilities and limitations of the EVPS against functional requirements.
Research Activities for Objective 4
Objective 4: Assess the developed EVPS to evaluate effectiveness of interactive virtual prototyping for enhancing the experience-based design review process of healthcare facilities.
- Identify a suitable facility within the healthcare context that will benefit from the implementation of an experience-based design review process using interactive virtual prototypes. Investigate and document several specific scenarios of tasks undertaken in healthcare facility.
- Evaluate the use of the EVPS application for design review of healthcare facility implemented in an immersive virtual environment to assess its effectiveness in collaborative design review and decision making process. Obtain feedback through observations during field study meetings of end users during the design review process.
- Implement user study to evaluate the effect of using task-based scenarios in interactive virtual prototypes for design review of healthcare facilities by end users.
Following Table 3-1 shows research methods adopted for each specific research task.
Research Methods Research Objectiveand Tasks | Literature Review | Systems Development | Requirement Analysis | Concept Mapping | Case Study | Focus Group | Field Observation | User Study | Talk Aloud Protocol | Survey Questionnaire | Protocol Analysis |
I. Develop a virtual prototyping procedure to extract end user experience of healthcare activities in interactive virtual prototypes. | |||||||||||
1. Scenario Concept | ✓ | ✓ | ✓ | ✓ | |||||||
Ch 2 | Ch | Ch | Ch | ||||||||
4, 5 | 4, 5 | ||||||||||
2. Scenario documentation and structure | ✓Ch 2 | ✓Ch 2 | ✓Ch 4, 5 | ||||||||
II. Design a framework for structuring end-user activities into scenarios that can be simulated in an interactive virtual prototyping system. | |||||||||||
3. System Architecture | ✓Ch 4 | ||||||||||
4. Model and media requirements | ✓ | ✓ | ✓ | ||||||||
Ch 2 | Ch 4 | Ch | |||||||||
4, 5 | |||||||||||
5. Interactivity interface | ✓Ch 2 | ✓Ch 4 | |||||||||
III. Develop an interactive computing platform titled the Experience-Based Virtual Prototyping System (EVPS) for implementation in healthcare design reviews. | |||||||||||
6. Programming environment | ✓Ch 2 | ✓Ch 4 | |||||||||
7. Design Information workflows | ✓Ch 4 | ||||||||||
8. Reusable model content | ✓ | ✓ | |||||||||
Ch 4 | Ch | ||||||||||
2, 5 | |||||||||||
9. Validate procedure | ✓ | ✓ | |||||||||
Ch | Ch | ||||||||||
3, 4 | 4,5 | ||||||||||
IV. Assess the developed EVPS to evaluate effectiveness of interactive virtual prototyping for enhancing the experience-based design review process of healthcare facilities.. | |||||||||||
10. Hershey Children’s Hospital | ✓ | ✓ | ✓ | ✓ | |||||||
Ch | Ch | Ch 5 | Ch | ||||||||
4,5 | 4,5 | 4,5 | |||||||||
11. EVPS application | ✓ | ✓ | |||||||||
Ch 5 | Ch | ||||||||||
4,5 | |||||||||||
12. Evaluation of EVPS | ✓Ch 6 | ✓Ch 6 | ✓Ch 6 | ✓Ch 6 |
Table 3-1. Research method adopted for each research task.
3.3 RESEARCH METHODS AND TOOLS
While research methodology refers to the principles and procedures of logical thought processes, which are applied to a scientific investigation, methods concern the techniques that are available for data collection and analysis as well as a summary of methods that are employed in the research project. In this study, the systems development approach was adopted for the EVPS system design. For EVPS implementation, the case study approach was adopted that included data collection through focus groups using concept mapping as a tool for requirements analysis. Finally evaluation adopted field study method for observation of EVPS application and experimentation for EVPS assessment. Data was collected through survey and talk aloud protocol and analyzed using protocol analysis.
The following are brief descriptions of the research methods adopted in the research project.
3.3.1 Systems Development
The research process outlined in Figure 3-4 is a systems development methodology that includes elements of both social and engineering research approaches. This methodology is adapted from Nunamaker et al. (1991) and adopted to develop the concept of the Experience- based virtual prototyping system (EVPS). Chapter 4 describes the system architecture for EVPS and process for incorporating task-based scenarios in interactive virtual prototypes. [Publisher’s note: The following is a reformatting of what was originally a figure; hence, it is labeled Figure 3-4.
Construct a Conceptual Framework
- State a meaningful research question
- Investigate system functionalities and requirements
- Understand the system building process/ procedures
- Study relevant disciplines for new approaches and ideas
Develop System Architecture
- Develop a unique architecture design for extensibility and modularity
- Define functionalities of system components and interrelationships among them
Analyze and Design the System
- Design the knowledge base schema and process to carry out system functions
- Develop alternative solutions and choose one solution
Build the (Prototype) System
- Learn about the concepts, framework, and design through system building process.
- Gain insight about problems and the complexity of the system.
Observe & Evaluate the System
- Observe the use of the system by case studies and field studies.
- Evaluate the system by laboratory or field experiments.
- Develop new theories/ models based on the observation and experimentation of the system’s usage
- Consolidate experiences learned.
Figure 3-4. Systems development methodology (Source: Nunamaker et al 1991).
3.3.2 Case Study
Case-study research method is defined as “an empirical inquiry that: (1) investigates a contemporary phenomenon within its real-life context, especially when (2) the boundaries between the phenomenon and the context are not clearly defined (Yin 2003). Within the Construction Engineering and Management (CEM) domain, often approach case studies as mixed-methods projects with both qualitative and quantitative aspects (Taylor et al. 2011). Some of the limitations of case studies are that they may be limited to samples and cannot be generalized and they may lack precision, quantification, objectivity or rigor in execution.
According to Taylor et al. (2011), case-study research can be a rigorous research method that can lead to new insights, open up new lines of inquiry, and yield rich theoretical models that can enhance and expand research in the field of AECM. However, they must meet a “burden of proof”, an obligation that can be met in two ways, referred to as “burden of going forward”.
Taylor et al. (2011) provide a research strategy checklist to improve the consistency and comprehensiveness of case study research that includes longitudinal data collection, utilizing multiple researchers and/or raters and triangulation. The research study involved multiple researchers during data collection as this reduced bias.
3.3.2.1 Criteria for selection
The criteria for case study selection included four factors for consideration. They were: 1) the project should be in a healthcare setting, 2) the project should be ongoing (design or construction phase), 3) BIM or digital model of the project facility should be available, and 4) researcher should have access to end-users of the project for data collection and assessment. The Hershey Children’s Hospital was selected as a case study for data collection. The final procedure to develop EVPS was tested using digital models of the new, currently under-construction Children’s Hospital. Chapter 5 discusses the new Hershey Children’s Hospital case study in detail.
Focus groups were used as a primary means of data collection for requirements analysis.
End users were asked to elicit scenarios of activities they would like to see embedded in the interactive virtual prototypes. The focus group method is an established rigorous technique for collecting interviews aimed at eliciting and exploring in-depth opinions, judgments and evaluations expressed by professionals, experts or users/ clients about specific topics (Morgan 1997). The key difference between one-to-one interviews and focus-group discussions is that the latter is far more appropriate for the generation of new ideas formed within a social context (Breen 2006).
3.3.2.3 Data Analysis – Concept Mapping
During focus group discussions, concept-mapping tool was used to record and validate the scenarios collected and discuss them in more detail. Concept maps are diagrams that are used to represent and convey knowledge (Klein and Hoffman 1993). Concepts maps traditionally use paper and pencil, posters or stick-on notes and are often used in brainstorming sessions. In concept-mapping knowledge elicitation, the researchers help the domain practitioners build up a representation of their domain knowledge, in effect merging the activities of knowledge elicitation and representation. Concept maps help form knowledge models to support knowledge preservation, knowledge sharing and creation of decision support systems (Crandall et al. 2006). This research used concept maps to develop a hierarchical structure that frames scenarios and helps inform model content.
Both qualitative and quantitative research approaches were adopted for evaluating the developed EVPS in a healthcare context. The first part of evaluation was done through observational field study (Chapter 5) and the next part was a field experiment (Chapter 6). Both evaluation studies involved end users of the Hershey Children’s Hospital as participants.
3.3.3.1 Research Method – Field Study
For first part of analysis, an observational field study was conducted during design review meetings of the new Hershey Children’s Hospital. The pharmacy staff at the hospital utilized the EVPS developed during the case study research for their transition planning review meetings. Interviews and observations from the design review meeting helped inform how end users can apply EVPS during all phases of the facility life cycle. The design review meetings and their findings are described in more detail in the latter half of Chapter 5.
3.3.3.2 Research Method – User Study
A user study was designed using a posttest only control group experiment design where nurse participants were randomly assigned to either the walkthrough only control group or the task-based scenario treatment group. Pretest was used only to get demographic data from the participants.
During the user study, participants were video taped as they navigated through their specific assigned stimulus and performed specified tasks based on the condition they were assigned to. A Cognitive Task Analysis approach, talk aloud protocol was employed during the study for knowledge elicitation. Chapter 6 gives a detailed description of the user study design and procedures.
3.3.3.3 Data Collection – Survey
Surveys can be divided into two broad categories of questionnaires and interviews.
Surveys vary from highly structured questionnaires to open-ended questions (Fellows and Liu 2003). During the user study, posttest questionnaires were administered to ask questions regarding ease of use, design layout, model content and overall experience in using EVPS. Chapter 6 describes the data collection procedure using surveys in detail.
3.3.3.4 Data Collection – Talk Aloud Protocol
Talk Aloud Protocol (TAP) originated in classic research domain within the psychology of problem solving. In studies, research participants are instructed to speak their thoughts as they work on problems and do so as if they are “speaking to themselves” (Ericsson and Simon 1993). TAP helps capture what end users know about their domain: its concepts, principles and events. Participants were encouraged to do task explication. In addition to thinking aloud, participants could also be probed with questions afterward (Van Someren et al. 1994).
3.3.3.5 Data Analysis – Protocol Analysis
The approach to analysis of TAP data falls in the middle of the analytic spectrum drawing from both qualitative and quantitative analytic techniques. Data from TAPS consists largely of a record of the participant’s verbalizations. The data recordings – either audio or video has to be transcribed and then coded in some way. The procedure for coding a protocol is referred to as Protocol Analysis (PA). In traditional PA, every statement in protocol is coded according to some sort of a priori scheme that reflects the goal of the research. The coding scheme depends on the task domain and purposes of analysis and begins with indexing common themes and patterns while breaking down data into units.
It is often valuable to have more than one coder conduct the protocol coding task. In some cases it is necessary for demonstrating soundness of research method and conclusions drawn from research. For research in which data from TAP task are used to make strong claims about reasoning process, especially reasoning models that assert cause-effect relations among mental operations, the assessment of inter-coder reliability of protocol coding is regarded as a critical aspect of research.
3.4 SUMMARY
This chapter begins with describing the adopted research approach of Systems Development and the rationale for its selection. Next the research steps are listed and the corresponding research methods undertaken to accomplish each research task are described. The next chapter describes the development process and framework for experience-based virtual prototyping system in detail.
++++
Copyright 2013 by Sonali Kumar. All rights reserved. Thesis published on this site by the express permission of Sonali Kumar.
Note: See Chapter 4. https://joelsolkoff.com/chapter-4-dr-kumars-thesis-on-virtual-reality-modeling/