Of Nyna Essay, Research Paper
EFFECTS OF TIME SCALE FOCUS ON IMPROVEMENT OF DYNAMIC CAUSAL UNDERSTANDING by Dalton Ebun Manfred McCormack SUBMITTED TO THE DEPARTMENT OF INFORMATION SCIENCES IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF PHILOSPHY at the UNIVERSITY OF BERGEN Autumn, 1998 AcknowledgmentsMany people contributed to the development of this thesis. First I want to thank David Ford for his constant drive for perfection and his tireless efforts to instill in me the discipline of good research. He provided me with insightful feedback and suggestions during the entire project. I will also like to thank Paal Davidsen, Mike Spector and Jim Doyle for their kind assistance during the project. Special thanks to Jim Doyle and Khalid Saeed, Professor and Department Head, Social Science and Policy Studies W.P.I., Worcester, MA USA. for their assistance during the data collection exercise. I would also like to thank my colleague at the Department of Information Sciences, Jiang You for coding the online causal questionnaire. Joe, Hannah and Kobi thanks for your love and spiritual support during the raining days in Bergen. Above all I will like to thank God almighty for his continually guidance. Dalton E.M. McCormack22nd November 1998 Effects of Time-Scale Focus on Improvement in Dynamic Causal Understanding by Dalton Ebun Manfred McCormack AbstractManagers must understand the causal structure of complex dynamic systems to successfully control behavior of these systems. However the types of causal understanding required and missing to do so and the best means of improving causal understanding are not well understood. For example tools for improving causal understanding such as Management Flight Simulators (MFS) almost exclusively focus users on short time scales. Long time scale environments use one continuous simulation through the time horizon. Short time scale environments involve multiple decisions through the time horizon. The larger the number of decisions made over the time horizon the shorter the time scale. The length of the focus of managers in time can impact causal understanding. Recent research on MFS environments suggests that long time scale MFS environments can improve causal understanding better than short time-scale environments. I used two experiments to test this hypothesis and investigate the impacts of the size of a manager s time scale focus on two types of causal understanding. A disaggregation of causal understanding improvement results suggest that a long time scale helps to improve causal understanding of relatively simple causal relationships without delay more than a short time scale. I ran a second experiment to test the same hypothesis with several time scales and to investigate how the length of the time scale impacts causal understanding improvement. Results of the second experiment with four time scale lengths suggests that the degree of control and ability to perceive system responses to policies interact to affect causal understanding improvement. The results of these experiments suggest the need for more research into building effective system dynamics based learning environment tools. Table of Contents1. Introduction……………………………………………………………………………………………………..51.1 Motivation……………………………………………………………………………………………52. Problem Description………………………………………………………………………………………….73. Literature Review……………………………………………………………………………………………..83.1 Measuring Learning in Dynamic Environments………………………………………………….83.2 Learning and Decision Making Improvement in Dynamic Environments……………….104. Hypothesis………………………………………………………………………………………………………135. Methodology ………………………………………………………………………………………………….155.1 Treatment………………………………………………………………………………………………….165.2 The Experimental Tool………………………………………………………………………………..175.3 Measurement……………………………………………………………………………………………..175.4 Experimental Procedure……………………………………………………………………………….196. Results and Analysis of First Experiment …………………………………………………………….196.1 System Performance Results…………………………………………………………………………196.2 Causal Questionnaire Performance…………………………………………………………………216.3 Control Results…………………………………………………………………………………………..226.4 Quantitative Results ……………………………………………………………………………………246.4.1 Subject PGA (No. 10) …………………………………………………………………………246.4.2 Subject PGF (No. 15) ………………………………………………………………………… 256.4.3 Subject DGJ (No. 19) ………………………………………………………………………….266.5 Analysis …………………………………………………………………………………………………… 27 6.5.1 Further Analysis…………………………………………………………………………………..27 6.6 The Second Experiment………………………………………………………………………………..297. Discussion and Conclusion ………………………………………………………………………………….31References……………………………………………………………………………………………………………34Appendix 1 The Model…………………………………………………………………………………………..36Appendix 2 Participant Consent Form……………………………………………………………………….47Appendix 3 Questionnaire……………………………………………………………………………………….48Appendix 4 User Information…………………………………………………………………………………..51Appendix 5 Deer Management (Short Time Scale) ……………………………………………………..52Appendix 6 Deer Management (Long Time Scale)………………………………………………………58Appendix 7 The Kaibab Plateau Passage……………………………………………………………………64Appendix 8 Causal loop diagram of the Kaibab Plateau MFS Model………………………………69Appendix 9 Causal Questionnaire: Questions and Answers…………………………………………..70Appendix 10 Causal Questionnaire Code…………………………………………………………………..711. Introduction 1.1 MotivationSuccessfully managing complex dynamic systems is difficult and requires sound decision making strategies. Managers often work under pressure to implement decisions and lack tools for learning about the systems they manage. Management Flight Simulators (MFS) are tools used to help managers improve the management of complex systems by helping them understand the interconnected nature of systems and the consequences of their actions (Lane, 1995, Senge, 1994). Complex systems are made of multiple feedback loops, delays and non-linearity. Managers need help because they usually find it difficult to identify and understand the mechanism of feedback processes. Vennix (1996, 31) states that: … people tend to ignore feedback processes while at the same time feedback is a ubiquitous characteristic of life. As stated by Powers: All behavior involves strong feedback effects whether one is considering spinal reflexes or self actualization. Feedback is such an all-pervasive and fundamental aspect of behavior that it is invisible as the air we breathe…. we know nothing of our behavior but the feedback effects of our own outputs Powers suggested that its invisibility is one of the reasons why most people tend to ignore feedback processes. MFS can act as the central part of a learning environment in which managers are safe from the negative consequences of their decisions. Bakken, Gould & Kim(1994) describe MFS as: A MFS is a learning tool that allows managers to compress time and space, experiment with various strategies, and learn from making rounds of simulated decisions. Embedded in specially designed learning environments called learning labs, these simulators can be designed to provide organizational practice fields (Senge 1990) where managers can experiment and learn in environments that allow failure and reflection. Although there are many MFS in use their effectiveness in improving management performance has not been rigorously demonstrated. One reason for this is that researchers have found it difficult to understand how learners thinking processes change or how they are changed by the training intervention (MFS) (Bakken, Gould & Kim,1994). Another reason is the problem of measuring learning in dynamic environments which has not been easily solved. This is supported in the literature by (Bakken, Gould & Kim,1994): Measuring learning along certain dimensions is easier than along others. At a basic level, instruments can be used to measure a manager s ability to create a simple causal-loop diagram. At a more abstract level, determining the manager s understanding of a complete feedback structure and his ability to transfer that knowledge to another case is difficult to measure. Improving causal understanding of complex systems is required to improve the management of those systems. One reason why this is important to improve management of complex systems is that causality constitute a powerful way to express concisely causal statements and to identify feedback processes (Vennix, 1996). Causality embodies what Forrester (1975) refers to as closed-loop thinking. Vennix (1996, 43) states that: An open-loop approach to decision making implies that decisions are seen as unaffected by the decisions themselves (Forrester, 1975)…. In closed-loop thinking decisions are seen as a means to affect the environment, and changes in the environment in turn provide input to decisions which aim to influence the environment. MFS are potentially effective for improving causal understanding because managers can use MFS to experiment by testing several policies under different conditions. Designing effective MFS is central to the development of learning labs. Efforts to effectively identify the design features responsible for improved causal understanding with dynamic simulation have not been entirely successful (Langley, 1996). Spector and Davidsen (1997) report a lack of consistent success with regard to learning effectiveness in system dynamics based learning environments. They identify a learning strategy and reliable methods to measure learning effectiveness as critical aspects of future research. They note : Our first critical assumption is as follows: A simulation does not constitute a learning environment. Many people have and continue to confuse these two things. For example, persons responsible for planning and managing flight training are inclined to believe that building a physical flight simulator constitutes the creation of a learning environment. We believe that a simulator may be part of a learning environment, but for learning to occur efficiently there must be more than simple user interaction with a simulation, regardless of whether it is physical or computer-based. Frequent and constructive feedback from a tutor, especially in the early learning stages, is critical to learning (Collins, 1992). Explicitly stated learning goals and mechanisms to facilitate progress towards those goals (e.g., linkage to things already known, assessment of progress, helpful guides to improve performance, etc.) are also critical to learning (see, for example, Gagn , 1985; Hannafin & Hooper, 1993). Researchers have identified several characteristics of this problem which can explain why it has been difficult to improve causal understanding in dynamic environments. According to Bakken (1993) this is partly because experiments often involve dynamic environments which are unfamiliar to subjects, lack of relevant system information and require cognition beyond the bounded rationality of subjects (Bakken, 1993). Guidelines for the design of tools for improving manager s causal understanding of complex dynamic systems and protocols for how to change managerial practices which will improve causal understanding and thereby managerial performance are needed. These protocols should enhance managerial understanding of system structure, behavior and the interaction of the structure and behavior. Without ignoring structural and behavioral understanding, system dynamics has focused on the role of understanding how structure and behavior interact (Forrester, 1961; Richardson, 1991). One important aspect of this view studies the behavior of managers in response to system response and information. Hence I focus on the impact of different MFS designs on managerial understanding and performance behavior. Different time scale length have been suggested as effective design feature to investigate how this form of structure – behavior understanding enhancement impacts improvement in mental models and hence learning. To build effective tools researchers must understand the effects of multiple and different time-scales on causal understanding. As a step in addressing this issue I investigate the effects of time-scale focus on improvement in dynamic causal understanding. 2. Problem Description MFS potentially help managers by allowing them to safely experiment with different decision making strategies. An example of this is a manager who is unsure about the impact of potential changes in product pricing policies on customers. He can test policy alternatives and see the implications by implementing policy alternatives in a MFS which represents his organization and environment. Sustained improvement in developing pricing policies by the manager requires learning. Designing MFSs for learning is difficult because the necessary ingredients for learning in and about complex systems have not been completely tested. According to Sterman (1994) mangers fail to learn because of dynamic complexity, imperfect information about the state of the real world, confounding and ambiguous variables, poor scientific reasoning skills, defensive routines and other barriers to effective group processes, implementation failure and misperception of feedback. Ford (1997) describes the role of ambiguity and uncertainty in objectives as an additional barrier to learning. Researchers have shown that little learning occurs when decision makers are asked to manager dynamic complex systems. Bakken (1993) for example showed that managers learn better in familiar context than in unfamiliar contexts. He measured learning in two dimensions; decision making performance and improvement in causal understanding. Before we can design effective MFSs that impact learning, MFS developers need to understand what they want decision makers to learn about. Many designers are faced with the problem of what type of information they should include in MFSs and in what form these information should be presented. Senge & Sterman (1994) for example suggested the following three lessons for designing effective learning labs: focus on conceptualization, design opportunities for reflection and beware the computer (i.e. the computer should not be the focus of the learning exercise). I limit my focus on finding ways of designing opportunities for reflection. This problem of reflection is highlighted by Senge & Sterman (1994, 211) when they observed: In early tests of the simulation we found the manager-players were thoroughly engaged within fifteen minutes. They were, literally, on the edges of their seats. They argued with one another about the next decision. They bragged about cost reductions they achieved. But afterwards none could articulate a significant new insight about claims management [domain]. They had played to win without pausing to reflect or to formulate and test theories about the causes of the problem. These managers had fallen to the video game syndrome. How can MFS designers integrate opportunities for reflection into MFS? From the statement above we can conclude that one potential method of achieving opportunities for reflection is to design MFS that encourage managers to implement long-term strategies easily and see how these strategies affect system behavior. Langley and Morecroft (1996) suggest that one way of implementing this in MFS design is to focus on managerial time scales. Managerial time scales are defined as the number of decisions per system period over the time horizon. This research addresses the MFS design problem by testing and expanding the theory of the impact of managerial time scale focus on learning and dynamic decision making performance. The primarily research question is: How does managerial time scales impact causal understanding in a dynamic environment? 3.0 Literature ReviewSince I want to understand MFS design methods that impact learning in dynamic environments I first consider how researchers have measured learning as improvement in mental models. Next I consider how researchers have investigated methods to improve learners mental models in dynamic environments and the implications of these on designing effective MFS. 3.1 Measuring Learning in Dynamic EnvironmentsRelatively little literature directly addresses the design of learning oriented MFS due to a focus on improving dynamic decision making performance. One reason why most of the research in this area has being performance focused is that researchers have found it difficult to measure learning in dynamic environments. Researchers have measured learning in dynamic environments with changes in mental models (Bakken, 1993, Doyle, Radzicki & Trees, 1996). However this metric is problematic because of the meaning of the term mental models . Doyle and Ford (1997) suggest that although: … mental models are of central importance to system dynamics research and practice, the field has yet to develop an unambiguous and agreed upon definition of them . Some of these problems are highlighted in an experiment carried out by Doyle, Radzicki and Trees (1996) in which they measured learning as changes in mental models. Changes in mental models here is defined as changes in understanding of causal relationships in the system, which involves identification of important variables and strength of relationships between variables that are essential to the dynamic behavior of the system being managed. They found evidence of a positive relationship between MFS use and changes in mental models. Subjects in both control and experiment groups were asked to write down the most important variables and their relationships before and after MFS use. They found evidence that subjects in the group that interacted with the MFS changed their mental models of the system more than subjects in the other group. Although they measured dynamic decision making performance their research mainly concentrated on measuring changes in mental models. However, the researchers found it extremely difficult to interpret the data collected. Because mental models change quickly and deriving methods to collect and measure these changes is difficult. However this research based it s measurement of learning solely on changes in mental models (such as causal understanding) and not on dynamic decision making performance. This is because their primary focus were not on finding ways to improve dynamic decision making performance but rather on investigating whether MFS intervention impacts changes in mental models. This research shows that MFS can change mental models and hence learning. But whether MFS improve learning remains an open question that requires additional research. The problems of measuring mental models has led researchers to devise different methods of measuring learning. The most common metric is system performance. For example Sterman (1989) measured learning as improvement in decision making performance in a dynamic environment. However research by Bakken (1993) shows that learning is not always correlated with improvement in decision making performance. He did this by developing a methodology for measuring learning and the transfer of learning between context. He measured learning in two different forms; improvement in dynamic decision making performance and improvement in causal understanding. Causal understanding was measured by asking managers to provide answers to questions that relate to polarities of causal links of variables in the system. 3.2 Learning and Decision Making Improvement in Dynamic EnvironmentsSeveral researchers have investigated ways to design MFS for learning and decision making performance. Before I discuss some of these research I consider what Kahneman and Tversky (1982) suggest for learning in dynamic environments: Effective learning takes place only under certain conditions: it requires accurate and immediate feedback about the relation between the situational conditions and the appropriate response. The necessary feedback is often lacking for the decisions faced by managers…. because (i) outcomes are commonly delayed and not attributable to a particular action; (ii) variability in the environment degrades the reliability of the feedback, especially where outcomes of low probability are involved; (iii) there is often no information about what the outcome would have been if another decision had been taken; (iv) most important decisions are unique and therefore provide little opportunity for learning. One of the methods that researchers suggest can improve learning in dynamic environments is providing cognitive feedback to learners (Balzer et al,1989). Cognitive feedback refers to the process of presenting the manger information about the relations in the environment (task information), relations perceived by the manager (cognitive information), and relations between the environment and the managers perceptions of the environment (functional validity information) (Balzer et al, 1989). Task information includes structural information and how the structure of the real system and its behavior interact. It potentially helps managers to learn more about the structure of the system been managed. Langley (1996) suggests that task information is: …for example a decision rule [that] may be given for the price of a product, showing how the cues of historical demand and competitors price are weighted together. . Cognitive information refers to managers mental models and according to Langley (1996) cognitive information: …enables decision makers to gain greater insight into their own decision strategies. For example, the subject s decision rule for product price is estimated using the subject s own historical decisions. . Functional validity information refers to the relationships between the structure-behavior of the real system and the managers mental model. Langley (1996) suggests that: Functional validity information (allowing comparison of decision makers perceptions of cue-criterion relations to the actual situation) facilitates improved calibration with respect to the task structure. For example, a table is presented comparing the subject s own cue weights with the optimal cue weights. . Abdel-Hamid & Sengupta (1993) observed that two forms of cognitive feedback can induce improved performances in a decision making dynamic environment. In this research subjects were divided into three groups. Outcome feedback was provided to each group. In addition to that one group was given cognitive feedback (task information) and the other feed-forward (i.e. dynamic decision making rules or policies). Task information was provided by the system through plots of selected variables of the system. Feed-forward condition was provided with a set of guidelines used by experienced project managers (system s domain). The last group was given only outcome feedback. Subjects were then placed in decision making roles to mimic the job of a software project manager in a dynamic simulated environment. Abdel-Hamid & Sengupta (1993) concluded that subjects in the group with outcome feedback and cognitive feedback information performed best followed by the group with the feed-forward and outcome feedback information. They concluded that: The outcome feedback subjects fluctuated considerably from one interval to another in their staffing decisions, thereby reducing the productivity of the staff and paying the price through poor performance. The feed-forward and cognitive feedback subjects were more consistent. They fluctuated significantly less in their staffing levels and employed an effective staffing heuristic in controlling their staffing levels over the course of the project, there by attaining superior performance over the outcome feedback group. The cognitive feedback group outperformed the feed-forward group because their access to continuous diagnostic information gave them greater adaptability in recognizing system changes and calibrating their decision strategies accordingly. . Langley (1996) investigated ways in which cognitive feedback can improve performance and accelerate individual learning in a dynamic environment. More specifically he was interested in studying: …the principal question of how does the subject s performance changes, if we provide online cognitive feedback to the subject for a limited number of trials? Does it improve or worsen, relative to subjects who only receive outcome feedback? . His main hypothesis considers the improvement of performance of treatment groups (who received both cognitive feedback and outcome feedback) over the control group (who received only outcome feedback). He tested this hypothesis by conducting an experiment in which subjects were randomly divided into cognitive feedback treatment groups and control group. Subjects were required to complete a series of game related tasks, which involved playing the role of a group of independent oil producers, making yearly capital investment approval decisions, over 25 year period. The treatment and control groups receive various types of online cognitive feedback, and their performance in the tasks are compared. Improvement in performance is used as a surrogate for learning in the experiment. His results show that: … subjects mean performance, relative to a benchmark, was significantly higher (in trails 1 and 2) for subjects who received cognitive feedback. There were no significant differences in performance between subjects who received decision-rules cognitive feedback and subjects who received task-structure feedback. In fact, subject performance reached an upper limit in later trials (4,5,6) which was not improved by feedback treatments. This limit was 30% below a behavioral benchmark, and is consistent with prior work in experimental simulated environments. It was disappointing to find that subjects in the treatment groups did not perform significantly better than the control group by trial 6 . . Langley (1996) concluded that: The problem of how to increase the final level of performance, and overcome the difficulties of performing well in complex systems characterized by feedback, non – linearities and delays, is still a question that has not been fully answered. This is an important question, because there is much effort currently being expended on the design of electronic learning environments. Another example of a research to improve decision making performance and learning in dynamic environments was carried out by Machuca et al (1998). They stated that: … the main objective of this research was to create, develop and use TBBSs as well as to verify the hypothesis mentioned above. In my opinion, TBBSs should facilitate causal reflection and favor systemic learning of social and business problems, helping to prevent the video game syndrome which often arises in black-box games. The hypothesis tested was: … that by using a transparent simulation, the learning process and acquisition of a system approach for decision-making would be improved . More specifically they investigated the impact of Transparent Box Business Simulation (TBBS) and Black Box Business Simulation (BBBS) on learning. Learning was measured with a questionnaire which contained questions relating to the following: … basic knowledge, questions related to variables connected by relatively short and longer chains of interactions … questions linked to the [ subject s ] perception of increase in knowledge they believe they have attained through the simulation (BBBS or TBBS) in relation to different topics previously studied through traditional methods of learning.. Transparency refers to the possibility of relating the systems structure to its behavior when using a MFS. Machuca et al (1998) comment that : To facilitate transparency, the qualitative structure (for example, in the form of causal loop diagram) and even the main equations of the model on which it is based, could be at all times accessible in the computer to the user, who would therefore find it easier to make decisions based on previous study of the possible causes of the different behaviors of the variables and not only on observation of the latter, which are simply effects. This possibility, however, is not offered by black box games, which only allow access to the results that can be observed, that is, the effects or symptoms, but not the causal structure giving rise to them. With black box games a trial and error procedure is usually followed, based on the mental models each player has of the case being examined. Their results show that TBSS can have a positive influence on causal understanding especially the understanding of complex causal relations. They conclude that: in our judgment, the results from our experiment demonstrate the superiority of TBBSs to BBBSs in regard to the learning process, especially in relation to complex questions linked to an understanding of the structure and operation of the system under study. How much transparency is required for improved performance is not yet known, this research shows that managers can perform better in a transparent environment than a non-transparent environment.