The greatest advantage of check lists is the facility and speed with which they can be analysed, as observer just ticks off phenomenon against an appropriate category by mere observation. Measures that might be easily obtained are as follows:
1. frequency with which there is a change in activity;
2. number of different activities;
3. number of stimuli encountered;
4. duration of specific activity;
5. changes in nature and duration of activities with time.
However, McKernan (1996:108) admonishes that the arrangement of the points is crucial in that sequence in task completion should be logical and sequential. An observer or designer of this instrument must ensure that:
1. points to be observed are listed in their actual sequence of happening;
2. all similar attributes are included in categories;
3. all the relevant and specified points are listed.
Observation schemes
Over the years numerous schemes have been developed for recording classroom interaction. Chaudron (1988:19), modifying the analysis originated by Long (1980), identifies twenty-four various schemes. In his review Chaudron (1988:17) points out that Long (1980) has included only those instruments which were designed to observe verbal interaction in a classroom, whereas the range of categories is great due to various purposes of observation. Chaudron interprets categories as
a) social interactive (Allwright (1980:169) turn-taking and turn-giving, Moskowitz’s (1970) ‘jokes’, ‘praises or encourages’)
b) pedagogical (Jarvis’s (1968:336) ‘classroom management’, ‘repetition reinforcement’, or Fanselow’s (1977:18) ‘solicit’, ‘respond’)
c) objective behaviour (Naiman, Neil, Frölich, Stern, and Todesco’s (1978) ‘student hand-raising’, ‘student callout’, or Moscowitz’s (1970) ‘student response -choral’)
d) semantic or cognitive content of behaviours (Fanselow’s (1977:31) ‘characterize’)
e) type and grouping of participants (Mitchell et al. (1981:19) ‘whole class’, ‘individuals doing the same task’)
For teacher training purpose Chaudron (1988:18) recommends to apply eleven schemes among which Capelle, Jarvilla, and Revelle (n.d.), Moskowitz’s (1970), Politzer (1980), Seliger (1977) are conducted in real time coding and categories of schemes refer to low degree of inference.
Advantages of interaction schemes as the basis of reflection in experiential knowledge are described by Wallace (1991:121) and he claims that these systems
1) objectify the teaching process;
2) provide a reliable record (by a trained observer);
3) promote self-awareness in the teacher;
4) provide a meta-language, which enables teachers to talk about their profession;
5) make teacher training more effective by improving the quality of teaching.
At the same time systematic observation schemes have some critics. Delamont and Hamilton’s (1976:3) main critique is levelled at the use of pre-specified categories to ‘code’ or classify the behaviour of teachers and pupils, which can not capture and reflect the whole complexity of classroom life.
Delamont and Hamilton (1976:8) identify seven criticisms of systematic observational systems:
1) Systematic observation provides data only about ‘average’ or ‘typical’ classrooms, teachers and pupils.
2) All the interactional analysis systems ignore the temporal and spatial context in which the data are collected as most systems use data gathered during very short periods of observation the observer is not expected to record information about the physical setting.
3) Interaction analysis systems are usually concerned only with overt, observable behaviour. In the case if intentions lay behind the direct behaviour an observer must himself impute the intention.
4) Interaction analysis systems are concerned with ‘what can be categorized or measured’ (Simon and Boyer 1986:1). They may obscure, distort or ignore the qualitative features which they claim to investigate, by having ill-defined boundaries between the categories.
5) Interaction analysis systems focus on ‘small bits of action or behaviour rather than global concepts’ (Simon and Boyer 1986:1). Delamont and Hamilton clarifies that there is a tendency to generate a superabundance of data which must be linked either to the complex set of descriptive concepts or to a small number of global concepts.
6) The systems utilize pre-specified categories.
7) Placing arbitrary boundaries on continuous phenomena obscures the flux of social interaction.
Walker and Adelman (1976: 136) emphasize the problems of recording child-child talk and objectivity of incorporating this kind of talk into the normal flow of teacher-centred classroom. They illustrate that there is no research instrument to code the spontaneous talk or social function of jokes and humour. ‘Talk is seen to be a highly complex, problematic activity, rich in contradictory and bizarre meanings and frequently with difficulties and confusions’ (Walker and Adelman 1976: 137). This organisation is taken for granted in observation schemes.
Rating scales
McKernan (1996:118) reviews various styles of rating scales – category, numerical, graphic and pictorial. They all share the common feature of having a rater place an object, person or idea along a sequential scale in terms of estimated value to the rater. Rating scales are treated as helpful instrument to measure non-cognitive areas where an observer is interested in cooperativeness, industriousness, tolerance, enthusiasm, group skills. At the same time McKernan (1996:119) notes that all rating sheets need to
a) include observable behavior;
b) rate significant outcomes as opposed to minor or trivial behaviours;
c) employ clear, unambiguous scales – never to use less than three, nor more than ten points on a scale;
d) arrange for several raters to observe the same phenomena to increase reliability of ratings;
e) keep items short and to the point.
Rating scales are opposed to direct observation as an assessment strategy. Nevertheless, Sattler (1982:33) points out that rating scale may not correspond with data obtained by the way of direct observation. He suggests that the internal consistency and ‘inter-rater’ reliability are important features of behaviour rating scales (Sattler 1982:34). Another criticism of observational data obtained through ratings is in that they involve human judgment and the sample of behaviour may be limited.
Selective verbatim
This technique is described by McKernan (1996:170). Unlike interaction analysis the selective verbatim techniques is directed at studying ‘selective’ verbal reactions. These are interactions that reflect effective or ineffective teaching. The procedure involves recording of the actual words and further analysis. The main advantage of the selective verbatim technique is in that it allows an observer to concentrate on one aspect of the teaching/learning behaviour at a time and it provides an objective non-interpretive record of verbal behaviour, which can be analyzed later.
Observation tasks
An observation task is ‘a focused activity to work on while observing a lesson in progress’ (Wajnryb 1992:7). Like a selective verbatim technique it focuses on one or a small number of aspects of the teaching/learning process but covers nonverbal behaviour as well. The purpose of the task is to collect actual facts or patterns of interaction that emerge in a lesson. The advantage of the collecting information with the help of selective tasks is that ‘it provides a convenient means of collecting data that frees the observer from forming an opinion or making a non-the-spot evaluation during the lesson’ (Wajnryb 1992:7).
To draw general conclusion about the techniques of observation I can say that some of them suggest either too broad or too narrow studying of the teaching process. It does not suit the main objectives of the Observation Weeks at the Teaching Practicum that are targeted to acquaint trainees with all the facets of the complex teaching/learning process gradually, to practice and develop trainees’ observation skills.
2.7. Evaluation of documents
2.7.1 Criteria for manual evaluation
The data evaluation process in qualitative and quantitative research is complex, laborious and time consuming procedure. In social research there are two main approaches to analysis and evaluation of data: manual and computer based. In the former case qualitative research evaluation is treated as ‘intuitive, idiosyncratic and creative’ (Stroh 2000:226). Due to the immersive nature of the participant observation and closeness to a subject a researcher is inclined to see things from the member’s perspective. Thus Cohen and Mannion (1994:52) suggest evaluating materials by means of two stages: ‘external’ and ‘internal criticism’. External criticism is concerned with establishing the ‘authenticity’ (Scott 1990:37) or genuiness of material. It is aimed at the document itself rather than the statements it contains and endeavors to analyse forms of the data rather than the interpretation or meaning. That is way it sets out to discover frauds, inventions or distortions. A set of questions proposed by Platt (1981) can be employed to test observation material on its authenticity:
Does the document make sense or does it contain glaring errors?
Are there different versions of the original document available?
Is there consistency of literary style, handwriting or typeface?
Has the document been transcribed by many copyists?
Does the version available derive from a reliable source?
Internal criticism deals with the accuracy of the data presentation and an evaluator has to establish ‘credibility’, ‘representativeness’ and ‘meaning’ (Scott 1990:53) of the document.
Credibility refers to the question of whether the task is ‘free from error and distortion’ (Macdonald 2001:204). The later may occur when the comments and discussion were made long time after actual observation, or when the account has been made through different hands and the author was not present at the lesson. The task is considered to be representative if all the aspects of the task have been taken place in an accurate way. But missing of some categories might occur, then the question of what is missing, how much and why should be considered.
Representativeness can be affected by the interest or bias of the author to please the reader, or being under pressure, from fear or vanity the writer can distort or omit some facts.
The meaning of a document should be established at two levels: ‘the surface or literal meaning, and the deeper meaning arrived at … interpretative understanding or structural analysis’ (Macdonald 2001:205). The first type embodies the form of the text whereas the second one analyses the content of the message from the point of view of ‘tendencies, sequences, patterns, and orders’ (Ericson, Bareaneck, and Chan 1991:55). Arguably textual analysis should draw to discourse analysis and concentrates only on language features regardless of social setting. Whereas Scott (1990:64) claims that a text is deprived from its real meaning in isolation from the social context. So ‘texts must be studied as socially situated products’ (Scott 1990:65).
2.7.2 Computer-based evaluation
Computer application in qualitative research analysis arguably brings some organisation and system into unstructured material and various paper forms, but definitely is helpful in storing and managing a large amount of materials in ethnography and statistics in quantitative data collection. Sophisticated software packages have been generated for the last years, for example, the Ethnograph (Seidel), QSR NUP∙IST (Richards and Richards), Hyper-RESEARCH (Biber, Kinder) ATLAS/ti (Muhr), SPSS. Computer programmes are of great help for a researcher and can assist in simple functions such as text processing and speed search as in more complicated ones: coding or indexing words and further retrieving them, building theories, making descriptive statistics and inferential one. But Gayle (2000) admonishes that a researcher should remember that computers do not produce results as such, they ‘merely take some of the laborious data management tasks away from the researcher’ Gayle (2000:415).
Chapter 3
Design of the learner observation tasks
3.1. The area of the observation tasks
The area of observation and the structure of the tasks are modified forms of the classroom observation tasks proposed by Wajnryb (1992). The learner area covers the same focuses as were originally proposed, such as ‘the learner as a doer’, ‘the learner motivation’, ‘the learner level’ except the ‘classroom climate’ task. I have shifted the focus of ‘teacher’s attending behaviour towards the learners’ to ‘classroom climate’ as this is the first meeting with the group of pupils and it is crucial to grasp the idea of social relationship between learners and teacher-learners, to make up a general impression about the degree of learner’s involvement into the lesson activities, their attitude to the language studying and the nature of language use at the lesson, either ‘drill’ to practice grammar or ‘real’ (Allwright 1988:13) to communicate. It should help trainees to become aware of other specific questions that influence learning process and learner development.
The focus of every task is sequenced according to its complexity from more general to more specific category. For example, the variable ‘learning styles’ requires higher inference categories than ‘motivation’ as student-teachers have to observe not only the language behaviour but the manner of approaching and processing the activity, and more descriptive language is entailed in their comments accordingly. Although, the evidences of language level seem to be easier to notice but student teachers are recommended to reflect upon the linkage between all the facets of the previous focuses and their influence upon the leaner level.
3.2. The frame of the observation tasks
Generally, the frame of every task is similar to the foregoing tasks and follows a standard procedure. Every task consists of three phases: before the lesson, during the lesson, and after the lesson. Typically, the instructions for the ‘Before the lesson’ phase deal with some preliminary activities. First, pre-service teachers are recommended to get acquainted with the classroom design, to arrange their own seating position to observe learners and to contact with the teacher. Sometimes, student teachers are asked to review some theoretical knowledge in phsycholinguistic area concerning learners’ motivation factors and learning styles. Then, to fulfil the tasks successfully student teachers have to make themselves familiar with an aspect of learner’s behaviour this or that task is targeted at.
I have modified ‘Before the lesson’ phase and introduced some concrete samples of learner’s behaviour description, whereas Wajnryb (1992) provides an area of observation in general. I have borne in mind two essential factors that drove me in so doing. First, pre-service teachers are inexperienced teachers; most of them have no practical teaching experience. That is why they are not aware of the importance of every detail in learners’ behaviour that they should consider during the lesson. Second, student teachers are non native speakers. Unfortunately, the level of language proficiency of many student teachers is low intermediate, and they experience problems in the use of foreign language appropriately and give precise description as it is required by the task. Arguably, the classroom observation tasks can be fulfilled in mother tongue but perceiving instructions and making field notes, jotters in English promotes additional practice in second language acquisition, furthermore it enhances metalanguage practice as well.
‘During the lesson’ phase requires collecting data and event sampling. A grid or a chart is provided to enable student teachers to do this with ease. Student teachers are recommended to make some field or jotted notes in the form of graphic symbols, actual utterances or descriptive language to recall events easily as the longer period of observation the more things they need to attend to and ‘the more details is forced out’ (Fielding 2001:152).
All the tasks are provided with examples within the charts so that the idea is quite clear. Again, some modifications of the charts were taken place. For example, in the ‘Learner motivation’ task I have added ‘signs of high/low motivation’ instead of the column ‘Motivation’, as it sounds more concrete and more comprehensible for inexperienced trainees. ‘High and low’ variables expose two extremes in learners’ behaviour but make the task feasible. Typically, pupils demonstrate respect towards their teacher and obey her/his commands and instructions automatically as classroom norms of behaviour require. Ccompliance and obedience might refer to motivating factors but they less help students ‘become responsible and caring’ (Meece and McColskey 2001:7) pupils. Highly motivated and low motivated students deserve special attention of teachers and researchers as the former ones are gradually inclined to lose their interest to studying without teacher’s support but the last ones according to numerous research tend to disrupt classroom behaviour and demonstrate poor results and knowledge. In the ‘Learner as a doer’ task I have substituted the column ‘Teacher’s purpose’ with ‘Learning activity’ as this notion introduces stages of the lesson, makes student teachers familiar with metalanguage and assists them with formatting their own lesson plans in future. The column ‘What learners do’ is added with the question word ‘how’ as describing the manner of doing an activity student teachers become aware of the reasons of pupil’s acting in this or that way. Then I recommend putting down learners’ names as it will help student teachers to keep in mind individual preferences of every pupil and to plan lesson activities accordingly.