All of these research sited in this section consistently investigate ways to learn in dynamic environments. They use MFSs as part of their research tools to investigate their hypotheses, however only Machuca et al (1998) showed specifically how TBBSs and BBBSs can be integrated into MFS. These research have identified some of the features of MFSs that constrained learning in dynamic environments. They suggest that a more structured methodology is needed to facilitate learning e.g. helpful guides to improve performance and constant feedback from a tutor. However with the exception of Machuca et al (1998) little work has been done to integrate these methodologies into the design of MFS. As suggested in my problem description section little work has been done to integrate the designing opportunities of reflection into MFS and to test how this impacts learning in dynamic environments. I investigate a methodology that potentially integrates this into MFS and observe how it impacts learning and improvement in dynamic decision making performance. 4. HypothesisIn this section, I discuss why designing MFS to enhance reflection should impact learning and decision making performance in dynamic environments. I also discuss why managerial time scales can potentially impact learning and decision making performance in dynamic environments. To better understand this we should view learning as a process where an action impacts result and this impacts reflection which leads to learning and this leads back to further action (Sterman, 1994, Bakken, Gould & Kim,1994). MFS can facilitate learning by shortening the delay between action and result, by forcing managers to search for a better understanding of why their decisions (actions) lead to the observed behavior (results). Sterman (1994) suggested that: Virtual worlds [MFS] are effective when they engage people [mangers] in what Dewey called reflective thought and what Schon (1992) calls reflective conversation with the situation. This process of learning is what researchers define as double-loop learning (Sterman, 1994, Bakken, Gould & Kim,1994, Argyris & Schon, 1978). Double-loop learning is defined as those sorts of organizational inquiry that resolve incompatible organizational norms themselves with associated strategies and assumptions. (Bakken, Gould & Kim,1994). Methods should be investigated to see how increasing reflection and enhancing learning can lead to better decision making performance. Senge & Sterman (1994, 211) stressed the importance of designing opportunities for reflection when they suggested: To enable mangers to experience the long-term side effects of decisions, simulations compress space and time. Good simulations also enable rapid trials with different strategies. But these very capabilities allow people to play without careful experimentation and without reflecting on the causes of the outcome. The players try a strategy; if it doesn t produce the desired outcome in a few months, they improvise. Rather than a series of controlled experiments, managers tend to vary multiple factors simultaneously. Instead of sticking with a strategy to see its long-term consequences, people quit a game that is going badly and start another (Moissis 1989) From the statement above we can infer that one potential method of achieving opportunities for reflection is to design MFSs that encourage managers to implement long-term strategies easily and see how these affect the behavior of the system. Senge & Sterman (1994, 212) suggests that: Before playing, the manager must state their strategy and what they expect to happen. . One potential method that researchers such as Simons (1990) and Langley & Morecroft (1996) have suggested which can impact reflection and hence learning is that of increasing managerial time scales relative to the systems natural period. They suggest that time scales which are relatively long when compared to the systems natural period facilitate causal understanding improvement more than shorter time scales. Langley & Morecroft (1996) stated that: Switching the mode of the user interaction with the microworld [ MFS ] from gaming [shorter time scale] to simulation [ longer time scale], allowing users to specify policies rather than decisions and run the model for continuous periods may improve their ability to identify high leverage policies. It is likely to allow more efficient use of time available, and an opportunity to apply the scientific method to systematically investigate the policy space. . They reason that mangers in longer time scales will have a longer time to design and test their strategies as well as to reflect on the behavior these strategies generate. They also suggest that managers in shorter time scales usually change their strategies during simulations and this causes them to have a different understanding of the system from managers in a longer time scale. We recognize this as a form of constructive experimentation i.e. mangers in longer time scale have more control of the system and thus can perform controlled experiments which should impact reflective thought (Sterman, 1994). Based on the preceding we propose the following hypotheses: H1a: Longer time scales in relation to the natural period of oscillation of the system will improve causal understanding more than shorter time scales. H1b: Longer time scales in relation to the natural period of oscillation of the system will improve system performance more than shorter time scales. 5. MethodologyI performed two experiments to test the hypotheses. In the first experiment the hypotheses were tested with a true experiment (Campbell and Stanley, 1963) with differing treatments in which subjects were randomly divided into two groups: long time scale and short time scale and asked to use a MFS to manage a small simulated ecosystem over a 40 year time horizon. Subjects in the long time scale group made only one decision per time horizon but subjects in the short time scale group made 40 decisions over 40 years. The managerial context was chosen to isolate the effects of time scale learning. Bakken (1993) has shown that using an MFS that is based on a domain with which subjects are familiar can have an influence on causal understanding and performance. Therefore I selected a dynamic environment with a relatively simple context and in which minimal domain knowledge is required of subjects. I also suspect that the bounded rationality of subjects is often exceeded by the complexity of MFS. Bounded rationality describes the limitations on human cognitive processing as central to understanding human behavior and performance (Simon, 1974). Significantly exceeding subjects bounded rationality can cause the loss of all treatment effects due to the overwhelming difficulty of the task. Therefore I used a relatively simple model of the ecosystem to not overwhelm treatment effects with system complexity. The simulated environment is based on a system dynamics model of the Kaibab Plateau (Goodman,1974; Roberts, Andersen, Deal, Garet and Shaffer, 1983; Sterman, undated). The specific version of this model has been previously used to study how information structures impact performance in a policy development environment (Ford, 1997). Ford s (1997) version of this classic system dynamics predatory-prey model (Goodman,1974; Roberts, Andersen, Deal, Garet and Shaffer, 1983; Sterman, undated) dynamically models three species (deer predators, deer and grass) and provides subjects with four annual control parameters: grass seeding, deer hunting, predator hunting and predator importation. Despite its structural simplicity the system remains dynamically complex with a frequency of two oscillations over the forty year time region of the simulation. The user interface of the groups were kept similar and simple, containing graphical and tabular information about some of the variables in the system in two frames which together fit legibly on a single page. 5.1 TreatmentThe difference between the two groups (the treatment) was that subjects in the short time scale group were required to make decisions annually (stepwise) throughout the forty year time horizon whereas those in the long time scale group were required to develop policies and to implement these policies at the start of each simulation. In other words subjects in the long time scale group were unable to stop the simulation after it has started. Since subjects in the short time scale group only implemented a policy once over the entire time horizon they were unable to change their policies during the simulation. Subjects in the longer time scale group however were able to change their policies since they implement their decisions annually. Group Pre-Test Treatment Post-TestLong Time Scale Causal Questionnaire Long Time Scale Tool Causal Questionnaire & PerformanceShort Time Scale Causal Questionnaire Short Time Scale Tool Causal Questionnaire & PerformanceTable 1 Experimental Design and Parameter Measures Subjects in two locations were used in the experiment to control for educational and environmental biases. The subjects in Bergen, Norway where paid a hundred Norwegian Kroner (about 13 US Dollars) for participating and an additional hundred Kroner was awarded to the subject with the highest average score in the causal questionnaire in each group. Subjects in Worcester, Massachusetts were awarded 50.00 US Dollars for the best average score, 25.00 for second best and 10.00 US Dollars for third best. The experimental design and measuring parameters are shown in Table.1. 5.2 The Experimental ToolThe Kaibab plateau MFS consists of a system dynamics model and a user interface. The system dynamics model consists of three species stocks (predators, deer and grass), two first-order delays and the connecting rates and auxiliaries with a natural period of approximately 30 years. A description and complete documentation equation listing based on Ford (1997) is included in Appendix 1 and file Kaibass on diskette 1 . Figure.1 shows the user interface of the Kaibab Plateau MFS. Subjects can see how the deer population unfolds over time in graphical as well as in tabular form (Figure.1). They can also see a record of their past decisions or policies on the table. Subjects also have a numerical performance indicator as shown to record system performance. Larger performance numbers indicate better performance. The interface is included in file Inter on diskette 1 . Figure. 1 The Kaibab Plateau MFS Interface 5.3 MeasurementQuantitative data about the dependent and control variables were collected as well as qualitative data about subject experience in the experiment. Two quantitative measurements were collected to measure system understanding: system performance and answers to a questionnaire. System performance was measured by how closely the subject maintained the deer population to the goal 30,000 deer over the time horizon with the aggregate variance over time. The causal understanding questionnaire (Figure 2) which was based on a similar experiment by Bakken (1993) contained questions about the causal relationships in the model. For example the answer to the question An increase in Grass Eaten by Deer leads to … in Grass (Figure 2) is Immediate Decrease since an increase in Grass Eaten by Deer will lead to an immediate decrease in the amount of grass in the plateau. The multiple choice questionnaire was computer based and contained 25 questions on causal relationships between variables in the system. The complete list of questions with solutions are in Appendix 9 and file question on diskette 1 . The order of the questions was randomized. The 25 questions were randomly selected and embedded into the computer code of the online causal questionnaire. This method was used to keep track of the questions and answers of each subject during both trails. Each question appeared on the monitor for a period of 25 seconds, followed by a new question. Each subject saw all 25 questions. The qualitative data collected included questionnaire with questions about how useful the description of the system they managed was, ease of causal questionnaire use, strategies used to manage the system and difficulty of the task managed (see also Appendix 2,3,4). Control data was also collected for subject s age, sex, primary language, English proficiency, education and experience background (see also Appendix 2,3,4). Figure 2 Example question in the Causal Questionnaire5.4 Experimental ProcedureBoth long time scale and short time scale groups were given a description of the Kaibab Plateau based on Goodman (1974) and the questionnaire. The description is included Appendix 7. The pre-test questionnaire was exactly the same for each group. Subjects in both groups were then asked to design and implement policies to control the deer population at a level of 30,000 for a period of 40 years. They used the four control parameters i.e. grass seeding, deer hunting, predator hunting and predator importation to do this for 20 minutes and saved each trial on a diskette. In each trial subjects type in their decision or policy on the input boxes and then click on the run button (shown in Appendix 5). The simulation starts at zero and then stops at the year 1940. At this point subjects start to implement their decisions or policies. At the end of the 20 minute session all subjects completed a post-test questionnaire. The post-test questionnaire was the same as the pre-test except that the order of the questions was changed at random. After the post-test questionnaire subjects were asked to fill out another questionnaire to gather data to control for variables such as age, educational background and experience in system dynamics and environmental studies. It also contained qualitative questions about the policy or decision rule the subject implemented during interactions with the MFS. 6. Results and Analysis of First Experiment In this section results of the first experiment will be described followed by an analysis of the results. The results are divided into four sections: System performance, causal questionnaire, control data and qualitative data results. 6.1 System Performance ResultsTable 2 and 3 show the data and results respectively of system performance of subjects in long time scale and short time scale groups. In Table 2 the columns contain subjects number, minimum system performance and maximum system performance. Table 3 shows the mean of the minimum system performance, standard deviation of the minimum system performance and the maximum of the minimum system performance for each group. Long Time Scale Group Short Time Scale Group Subject Minimum Maximum Subject Minimum Maximum1 1.04 2.63 1 1.83 1.842 0.63 1.85 2 1.83 1.933 1.73 2.66 3 1.80 1.844 0.70 3.06 4 1.62 1.865 1.49 1.85 5 1.04 2.826 1.80 1.87 6 1.75 2.157 1.75 1.83 7 2.06 4.048 1.84 1.94 8 1.41 5.639 1.04 1.79 9 1.30 1.8610 1.72 2.21 10 1.82 2.8111 1.85 1.85 11 1.96 3.7812 1.50 1.92 12 1.46 2.0613 1.84 1.92 13 1.46 3.0614 1.84 2.64 14 1.77 2.1515 1.85 1.85 15 1.38 3.0116 1.43 2.7 16 1.33 2.6517 1.84 2.76 17 1.58 1.8318 1.85 1.85 18 1.69 2.8519 1.8 1.84 19 1.64 2.24 Table 2 Subject Min and Maximum System Performance Data Long Time Scale Group Short Time Scale Group Mean of Min Standard Deviation of Min Max of Min Mean of Min Standard Deviation of Min Max of Min1.55 0.403902477 1.85 1.62 0.257311789 2.06 Mean of Max Standard Deviation of Max Max of Max Mean of Max Standard Deviation of Max Max of Max2.158947368 0.424380525 3.06 2.653157895 0.978911258 5.63 Table 3 Results of System Performance Data 6.2 Causal Questionnaire PerformanceTable 4 and 5 show causal questionnaire data and results respectively. Table 4 shows subjects pretest, post test causal understanding and causal understanding improvement (CUI) for each group. Long Time Scale Group Short Time Scale Group Subject Pre-Test Post-Test CUI Subject Pre-Test Post-Test CUI1 22 15.5 -6.5 1 11.00 9 -2.002 15 17.5 2.5 2 7.00 9.5 2.503 16.5 14 -2.5 3 13.50 12 -1.504 17 15 -2 4 13.00 12.5 -0.505 17 16 -1 5 19.50 19 -0.506 10.5 12.5 2 6 12.00 15 3.007 8.5 10.5 2 7 16.50 16.5 0.008 18 15 -3 8 19.00 19 0.009 15.5 18 2.5 9 11.00 11 0.0010 15.00 16 1 10 18.5 19.5 1.0011 12.00 17 5 11 18.5 18.5 0.0012 14.00 13.5 -0.5 12 15.5 18 2.5013 11 17.5 6.5 13 21.5 15 -6.5014 15.5 18.5 3 14 19.5 19 -0.5015 12 13 1 15 13.5 15.5 2.0016 13 16.5 3.5 16 14 16 2.0017 15.5 15.5 0 17 9 9 0.0018 18 19.5 1.5 18 14 18 4.0019 17 16 -1 19 21 21.5 0.50 Table 4 Subject Pre-Test and Post-test causal understanding Data for each group Table 5 shows the mean of pretest, post test causal understanding and causal understanding improvement for each group. It also shows the standard deviation of pretest, post test causal understanding and causal understanding improvement for each group. The minimum and maximum pretest, post test causal understanding and causal understanding improvement are also shown for each group. Long Time Scale Group Short Time Scale Group Mean of Pre-Test Standard Deviation of Pre-Test Min of Pre-Test Max of Pre-Test Mean of Pre-Test Standard Deviation of Pre-Test Min of Pre-Test Max of Pre-Test14.89 3.19104192 8.50 22.00 15.13 4.169472739 7.00 21.50 Mean of Post-Test Standard Deviation of Post-Test Min of Post-Test Max of Post-Test Mean of Post-Test Standard Deviation of Post-Test Min of Post-Test Max of Post-Test15.63158 2.235087031 10.5 19.5 15.4473684 3.901191901 9 21.5 Mean of CUI Standard Deviation of CUI Min of CUI Max of CUI Mean of CUI Standard Deviation of CUI Min of CUI Max of CUI0.736842 3.043063246 -6.5 6.5 0.32 2.28649736 -6.50 4.00 Table 5 Results of Pre-Test and Post-test causal understanding performance for each group 6.3 Control ResultsTable 6 shows results of the control variable data. The first column contains the subjects age, English proficiency, years after high school etc. English Proficiency was quantified along a scale of zero (lowest) to five (highest). The rows show for each control variable it s mean, standard deviation, minimum and maximum value for each group. Long Time Scale Group Short Time Scale Group Mean Standard Deviation Min Max Mean Standard Deviation Min MaxAge 24.333 4.4325003 18 34 Age 24.789 8.223131 17 53English Proficiency 4.5 0.6183469 3 5 English Proficiency 4.8421 0.374634 4 5Years after High School 4.6111 2.4528228 0 10 Years after High School 4.9342 4.301035 0.75 20General Management Education (Months) 7 13.941052 0 48 General Management Education (Months) 4.7368 9.005197 0 24General Management Experience (years) 0.4444 1.3382263 0 5 General Management Experience (years) 0.6447 2.293281 0 10Environment Management Education (Months) 0.9444 2.8382311 0 12 Environment Management Education (Months) 0.8158 2.794889 0 12Environment Management Experience (Years) 0 0 0 0 Environment Management Experience (Years) 0 0 0 0Computer Programming Education (Months) 14.889 23.166631 0 100 Computer Programming Education (Months) 8.6842 11.60485 0 36Computer Programming Experience (Years) 0.6639 1.5561836 0 6 Computer Programming Experience (Years) 0.3026 0.648604 0 2Modeling Information Systems Education (Months) 5.4444 8.1761649 0 24 Modeling Information Systems Education (Months) 4.0526 8.044831 0 26Modeling Information Systems Experience (Years) 0.0972 0.412479 0 1.75 Modeling Information Systems Experience (Years) 0.1053 0.315302 0 1Engineering Control Theory Education (Months) 0.4444 1.0966378 0 4 Engineering Control Theory Education (Months) 0.7895 2.800376 0 12Engineering Control Theory Experience (Years) 0 0 0 0 Engineering Control Theory Experience (Years) 0 0 0 0Systems Thinking or Dynamics Education (Months) 3.1667 6.8449208 0 24 Systems Thinking or Dynamics Education (Months) 4.6842 10.72408 0 47Systems Thinking or Dynamics Experience (Years) 0.0588 0.1660528 0 0.5 Systems Thinking or Dynamics Experience (Years) 0.0105 0.045883 0 0.2Table 6 Results of Control data for each group 6.4 Quantitative ResultsIn order to get some insight about the quantitative data results, three subjects (one from short time scale group and two from long time scale group) were selected based on their causal understanding improvement scores. These three subjects scored close to the average causal understanding improvement of each group. Table 7 shows their identifying number, age, causal understanding (CU) pretest and posttest scores, causal understanding improvement (CUI), average system performance (average MFS), maximum system performance (MAX MFS), minimum system performance (MIN MFS) and the group they represent. The subjects with identifying numbers PGA and PGF are subjects 10 and 15 respectively (see Table 4) of the long time scale group. The subject with identifying number DGJ is subject 19 (see Table 4) of the short time scale group. Long time scale group Identifying Number Age Sex (M/F) Primary Language CU Pretest CU Post test CUI Average MFS MAX MFS Min MFSPGA 34 F ENGLISH 15 16 1 1.89 2.21 1.72PGF 21 F ENGLISH 12 13 1 1.85 1.85 1.85 Short time scale group DGJ 19 F ENGLISH 21 21.5 0.5 1.90 2.24 1.64Table 7 Three quantitative subject data, with system performance data 6.4.1 Subject PGA (No. 10)When this subject was asked whether the passage (Appendix 7) was helpful in answering the causal questionnaire, she wrote: Helped in that it got me thinking about effects and their relationships on deer population. Did not help in that many of the terms used in the questions [causal questionnaire] were not used in the context of the reading. . On the question of the ease of causal questionnaire use; this subject wrote Easy . When asked about the difficulty of the task (i.e. controlling the deer population at 30,000), she wrote: Frustrating – was able to make improvement overtime but not completely clear – for instance if bringing predators each year does that just add up or does that include their natural growth/ breeding? Also I wanted to be able to set a midway policy as well to get the deer population up and then keep it stable. [This suggest that she would have preferred a shorter time scale.]. When asked about whether she would have preferred more time to interact with the MFS, she wrote. Yes, more time for experimentation would have tried more radical changes to see what happened On the question on what strategy she used while interacting with the MFS she wrote: try to look at curve given [suggesting that she used some form of outcome feedback ], see what makes it high enough then flatten out (by varying only 1 or 2 variables at a time) [this also suggest she used some form of controlled experimentation]. The last question asked about the usefulness of the MFS (i.e. did the interaction with the MFS help her improve her score in the second trial of the causal questionnaire?). She wrote: May have performed better [in the second trial] but I think it was because I had a better sense of what the questions were asking (terminology) after the game [i.e. MFS; this can be interpreted as MFS use helped her with domain understanding]. . 6.4.2 Subject PGF (No. 15)When this subject was asked whether the passage (Appendix 7) was helpful in answering the causal questionnaire, she wrote: It helped because it gave important background information. It also gave me an understanding of the problem. On the question of the ease of causal questionnaire use; this subject wrote: yes it was [easy]. The timer [this showed the time remaining before another question appeared on the monitor] caused me stress though. I don t know if I would include that. . When asked about the difficulty of the task (i.e. controlling the deer population at 30,000), she wrote: It was pretty difficult. I know something about SD [system dynamics] but this example (model) was very unsuccessful. When asked about whether she would have preferred more time to interact with the MFS, she wrote: No. The time allotted was perfect. I think it gives the user enough time to play the game [MFS], learn it and apply any policy changes to it. On the question on what strategy she used while interacting with the MFS she wrote: I tried to increase/decrease by small numbers in order to try to keep equality, not have a dramatic increase/decrease later on. The last question asked about the usefulness of the MFS (i.e. did the interaction with the MFS help her improve her score in the second trial of the causal questionnaire?). She wrote: Yes I think it gave me more insight to the problem and how things affected each other. 6.4.3 Subject DGJ (No. 19)When this subject was asked whether the passage (Appendix 7) was helpful in answering the causal questionnaire, she wrote: Yes it gave background information and a quick refresher in knowledge On the question of the ease of causal questionnaire use; this subject wrote yes [easy] . When asked about the difficulty of the task (i.e. controlling the deer population at 30,000), she wrote: I thought it would be easy but it was harder than I thought. The more I did it, the better I understood what worked and didn t When asked about whether she would have preferred more time to interact with the MFS, she wrote: Yes, once I got the hang of it, it was time to stop On the question on what strategy she used while interacting with the MFS she wrote: I was looking at the slope trying to manipulate the variables to achieve the line I wanted. [This suggests that she used some form of outcome feedback to implement her decisions] The last question asked about the usefulness of the MFS (i.e. did the interaction with the MFS help her improve her score in the second trial of the causal questionnaire?). She wrote: Yes it clarified some fuzzy concepts. 6.5 AnalysisTo check whether long time scale (LCUI) focus impacts causal understanding more than short time scale (SCUI), I calculated the improvement in causal understanding (i.e. the difference between the Post-Test and Pre-Test) for each subject in both groups. Statistical tests (using the two sample t-tests with P > 0.1) failed to support the hypothesis that p(LCUI ) > p(SCUI). No correlation between long or short time scale focus decision making performance and causal understanding, supporting a similar hypothesis tested by Bakken (1993). This result contrasts sharply with the intuitive belief that improved causal understanding improves performance. The results suggest that other necessary requirements for improved performance were not provided in ours or Bakken’s experiment. The analyses of the remaining results are presented in two sections. The first section reports summary statistics on aggregate levels and then uses them in the second section to test several hypotheses with the disaggregated data. Table 8 shows the means of the Pre-Treatment and Post-Treatment causal understanding and causal understanding improvement. It also includes the average of the best decision making performance for both groups. Group Mean Pre-TreatmentCausal Understanding Mean Post-TreatmentCausal Understanding Mean Causal Understanding Improvement Best System PerformanceLong Time Scale Focus 14.87 15.63 +0.76 2.16Short Time Scale Focus 15.13 15.45 +0.32 2.65Treatment Effect P >0.1 P >0.1Table 8 Means and Causal Understanding Improvement 6.5.1 Further Analysis To understand how the time scale focus impacts system understanding I disaggregated the causal understanding questionnaire data into two dimensions: degree of difficulty and type of understanding. There were three levels of difficulty: Easy, Medium and Difficult. The nine “Easy” questions involved one or two causal links between the variables (Appendix 8). The twelve “Medium” questions had three to five causal links and the four “Difficult” questions had six or more causal links between the two variables. I also divided the causal understanding questions into two types of causal understanding: links with immediate impacts and links with delayed impacts. The nine questions with “Immediate Increase or Decrease” as the correct answer are referred to as “links with immediate impacts” questions and the sixteen questions with “Delayed Increase or Decrease” as the correct answer are referred to as “links with delayed impact” questions (i.e. based on whether the linked passed through a stock). Although the option “No change” was given as an answer alternative in the questionnaire interface, no questions had “No change” as the correct answer. Table 9 shows the resulting hypotheses which can be tested based on this disaggregation. For example the hypothesis H2 states that increasing the time scale focus results in an increase in understanding of the easy causal links with immediate impact causal relationships. Degree of Difficulty Immediate ImpactCausal Relationships Delayed Impact Causal Relationships All Causal RelationshipsEasy (1-2 links) H2 H3 H9Moderate (3-5 Links) H4 H5 H10Difficult (6+ links) Not applicable H6 H11All Causal Relationships H7 H8 H1Table 9 Tested using disaggregated results of experiment 1. The cell marked Not Applicable indicates that this hypothesis was not tested because the causal questionnaire did not contain questions of this type. The results only support hypothesis H2 and H9 that increasing the time scale focus leads to an improvement in the understanding of the simple causal relationships (p Degree of Difficulty Immediate ImpactCausal Relationships Delayed Impact Causal Relationships All Causal RelationshipsEasy (1-2 links) p 0.1, r = – 0.007 p Moderate (3-5 Links) p > 0.1, r = – 0.2 p > 0.1, r =- 0.11 p > 0.1, r = – 0.15Difficult (6+ links) Not applicable p > 0.1, r = -0.35 p > 0.1, r = -0.35All Causal Relationships p > 0.1, r 0.1, r 0.1, r =- 0.03Table 10 Results of the Hypotheses Tested with Disaggregated results of Experiment 1 The disaggregated results show that 58% of the long time scale focus group improved their causal understanding more than the average short time scale focus group. This suggests that long time scale focus does have a positive impact on causal understanding than short time scale focus. However I also found that the long time scale focus group on average had a lower system performance than the short time scale focus group and only 21% of the long time scale focus group performed better in terms of system performance than the average short time scale focus group system performance. This does not suggest support of the hypothesis that long time scale focus has a positive impact on system performance. Although the main hypothesis was not statistically supported, the results indicate that a longer time scale focus may improve system understanding more than a short time scale focus based on the support of the hypotheses for easy immediate and all easy relationships (H2 and H9) results. This may be because the time spent using the MFS in both groups was too small (20 min) to cause a significant improvement in causal understanding or because other necessary learning factors were absent. This explanation was suggested by Prof. James Doyle of Worcester Polytechnic Institute, MA. It s possible that a more sophisticated MFS in which subjects can build feedback loops into the system such as was used by Ford (1997) can generate improvements in causal understanding. By allowing subjects to build feedback loops into the system they can better explore the policy space, and this can provide better transparency which as Machuca et al (1998) can lead to learning. Our results raised as many questions concerning the relationship between time scale focus and causal understanding as they provide answers. 6.6 The Second ExperimentBased on the findings of the first experiment a second experiment was designed as part of an exploratory work to investigate how different time scale lengths impact different forms of causal understanding. In this section I highlight some of the change