Professional Judgement and Standards in Educational Assessment

Note: This is a slightly adapted version of a lecture given to Adult Literacy Organisers in Galway, Republic of Ireland in April 2013, under the title: Initial Assessment - What Works? The lecture was organised by The Republic of Ireland's National Adult Literacy Agency (NALA). A video of part of the lecture is available here.

I’m here to talk about Professional Judgement and Standards in Assessment.  Most of what I want to say doesn’t just apply to initial diagnostic assessment in Adult Literacy work, but to teaching in general, and to many other areas of human activity.  It builds on a paper written for my doctorate (Derrick 2013).

I want to start by giving you all some questions to think about.  How much experience do you have as an adult literacy practitioner – hands up: more than ten years?  Five to ten years? Under five years?  Anyone in their first year?  Can I ask the people with more than ten years’ experience – would you say you are good at Initial Diagnostic Assessment?   What is it that makes someone good at it?  For example, is experience essential?  How important is formal training?  How can you learn to get better at it?  Is there a formula, or a recipe for getting better at it?  Is Initial Diagnostic Assessment a science?  Is there, or could there be, an instruction manual for it?

The answers to these questions are vital for sorting out the role of standards, skills frameworks, levels, etc in the work of adult literacy teaching.

The three main propositions of my talk are these:

• Expert teaching (and within this, expert assessment) is essentially about the exercise of professional judgement in making decisions
• Professional judgement can be supported by all sorts of tools, resources, measurements, rubrics, standards, etc, but cannot be replaced by these
• Learning to become better at teaching and assessment is a process of training professional judgement.

30 years ago, Donald Schön, writing about professional practice in general, said this:

In the varied topography of professional practice, there is a high, hard ground where practitioners can make effective use of research-based theory and technique, and there is the swampy lowland where situations are  confusing ‘messes’ incapable of technical solution.  The difficulty is that problems of high ground, however great their technical interest, are often relatively unimportant to clients or the larger society, while in the swamp are the problems of the greatest human concern. (Schön 1983 (1991), p42)

Schön suggests that professionals need to be able to deal with, not just technical, generic problems for which there is a recognised procedure, a standardised routine, but also other, more complex, essentially unique problems for which there is, and can be, no standard ‘correct’ response, and which are therefore amenable only to judgement calls. 

A practical example of this type of problem might be the question of whether to take a child into local authority care, which has to be considered all too often by social workers.  There can be no absolute instruction manual for making a decision like this, because every situation is uniquely different.   

Standard procedures, checklists, etc are helpful, but in the end a decision like this is not reducible to standard procedures – we can only try to ensure that they are made by people highly motivated to make the best possible decision, and who have as much knowledge as possible of the particular situation – and note that ideally this knowledge is not just technical knowledge of the law, of child and of family psychology, etc, but also personal knowledge of the unique pattern of the individual circumstances of the people involved.

Schön’s work is a sustained argument against the dominance of what he calls the ‘technical-rational’ view of professional practice.

Procedures and rubrics are models or hypotheses which always simplify and ‘clean-up’ the ‘messiness’ of real-life situations.  As Goethe warned:

‘Hypotheses are scaffoldings erected in front of a building and then dismantled when the work is finished.  They are indispensable for a workman, but you mustn’t mistake the scaffolding for the building.’ (Goethe Posthumous (1998), p154)

The scaffolding is useful, even perhaps essential, but it cannot replace the building which it supports.  This it seems to me, is an accurate metaphor for the role of rubrics, standards, technical terminology, pro-formas, skills frameworks, assessment criteria, grade descriptors, diagnostic assessment tests and procedures, and the like.  They have their uses, but ultimately teachers need to make the decisions, because each situation, each learner, is essentially unique.  This is why we should resist talking about education as if it was like mass-production – it is not about using standardised raw materials to produce identical objects.

Addressing educational problems as if they can be reduced to a technical model of one kind or another ignores the local knowledge, practical wisdom and perhaps the long experience of the professional in making similar decisions.  It also ignores the moral dimension: that the professional is motivated to find the right way forward for the clients and others concerned – by which I mean the best and most just response to the situation, not just a technical solution which ‘will do’.   A hypothetical model cannot be relied on to do this because it is a simplification of the situation, and by definition ignores factors in it which in a particular case may be critical.

The holistic argument I am making here highlights the importance in decision-making of ‘local distinctiveness’: a nuanced awareness of what is distinctive in the features of a particular locality or situation.  Schön might put it as appreciation of the distinctiveness of each particular ‘swamp’, so as to make informed decisions about appropriate actions.  This view is a critique of the kind of overly-simplistic ‘technical-rational’ modelling of situations that  makes decision-making (for example about whether to build a new supermarket in a country town) appear to be easier, perhaps to the long-term detriment of the locality and its community.  Sue Clifford and Angela King (2009) argue against the homogenising tendency of technical-rational modelling in all aspects of life, emphasising the importance of detail, particularity, patina, authenticity, fineness of grain, divergence, messiness, and quality in the everyday.  This may be caricatured as a conservative critique of regulation through standardisation, of technocratic policies and procedures: but it is an argument primarily against shoddiness, against work in any field that will merely do, rather than reflecting the best efforts of the practitioner.  Research (eg Black and Wiliam 1998, Hattie 2009) tells us clearly, for example, that teachers’ sense of the particularity of situations is central to their understanding of the specific needs of individual students at a given moment in the learning process: this sensitivity is also undoubtedly a foundation of trust and mutual respect between learner and teacher.  It is clearly possible to teach without this detailed knowledge of students and their situation, and perhaps even to teach ‘adequately’, but professional practitioners are deeply and morally concerned about ‘local distinctiveness’ as part of doing the job as well as possible.  Attempting to encapsulate this sensibility in a rubric will always tend to diminish it: minimum standards tend to produce work that simply satisfies the minimum acceptable level of quality. 

In England, much has been made in recent years about the use by teachers of data in their decision-making: it has been suggested, perhaps with some justice, that they have relied too much on hunches and on their intuition, and not enough on ‘objective’ data about their students’ progress.   There is also evidence that teachers and other professionals have sometimes been discriminatory in their practice (for example, having lower expectations of students from particular backgrounds), and this too has been used to support arguments that their work should be regulated within procedural and externally-imposed quality standards.

In my view this is to throw the baby out with the bathwater: it is to argue that the work of teachers is simple rather than complex; it is to argue for the production of an instruction manual for teaching, one consisting of a highly intricate flow-chart, that would provide teachers, whether qualified or not, with a procedure for correctly addressing every teaching problem, or, in terms of our topic today, for correctly the assessing the skills of every new student.

I think that most people would agree that this is not what teaching is about: they recognise that the work of teachers is essentially complex and cannot be reduced in this way.  On the contrary, teachers should be supported to develop their intuition and their capacity to make good judgements, just as surgeons, doctors and social workers do.  So what is the role of all these frameworks, rubrics, standards, pro-formas, tests, etc? It must be to provide information that teachers can use in their decision-making, and in discussing with students their learning goals, their attainments, feedback on their work, and, most importantly, how they can get better at supporting their own learning – in a word, these tools can be used to support formative approaches to learning, provided the information they provide supports, rather than determines, the authentic judgements and feedback of the teacher.  In this way, tools can also support teachers’ own professional learning.

From this perspective, it is not the tools themselves that are good or bad, it is how the teacher uses them that matters.  It is important, though, that teachers are free to determine for themselves how best to use them.  If they are not free in this way, then a particular technical-rational model is being imposed on them, and this may lead to students being sold short, and also to the deskilling and demotivation of teachers.  The problem for policymakers is that education is full of swampy, messy situations, which by definition are hard to predict, control and regulate.  To make the best of them we need teachers who are confident in weighing up situations, highly effective communicators in terms of knowing the kinds of questions to ask and how to listen, expert at qualitatively judging students’ strengths and weaknesses, and motivated to make the best decisions possible about the right courses of action for the student and for the next stage of teaching and learning.  They are very likely to find supporting technical information useful in making these judgments and decisions, and they should certainly pay attention to such data.

It is difficult for policymakers to regulate teachers without imposing limitations on their professional autonomy which may limit their scope for action, as well as reducing their motivation.  But this difficulty does not justify the misconception of learning as merely a technical issue of standard procedures, and teaching as a simple, technical job involving the obeying of standard rules.

If we had time I would invite you all to consider how best we should organise the training and development of adult literacy teachers in a non-technocratic policy environment.  Instead I’ll leave you with an inspiring story of exemplary professionalism that illustrates many of the issues we have been addressing here: on January 15th 2009, a commercial aircraft with 150 passengers and crew on board took off from La Guardia airport in New York.  The pilot in command was 57-year-old Capt. Chesley B. "Sully" Sullenberger, a former fighter pilot who had been an airline pilot since leaving the United States Air Force in 1980, and who is also a safety expert.  During the plane’s initial climb, it struck a flock of Canada geese. The bird strike, which occurred about three minutes into the flight, resulted in an immediate and complete loss of thrust from both engines. When the crew of the aircraft determined that they would be unable to reliably reach any airfield, they turned southbound and glided over the Hudson, finally ditching the airliner in the river about three minutes after losing power.  All 155 occupants safely evacuated the airliner, which was still virtually intact though partially submerged and slowly sinking, and were quickly rescued by nearby ferries and other watercraft.  The recording of the incident makes it clear that, immediately following the bird strike, it was obvious the plane had to land as soon as possible, and there was intense radio communication from the air traffic controllers, concerning availability of nearby runways, and the distance to each one.  After about a minute of listening to this information, the pilot asked just one question about the possibility of landing at New Jersey but then suddenly seems to make up his mind: ‘We are gonna be in the Hudson.’  He then said nothing more, apart from ‘Brace for impact’, until the plane was down.  In interviews afterwards he said that there were two factors that helped him pull off the manoeuvre safely: that the plane’s design was state of the art, and that he himself was 55 rather than 25.  ‘In a sense my whole career had been training for this event’ (Couric 2009).

This story illustrates how all sorts of tools, procedures and data made important contributions to the successful outcome of this manoeuvre, but these would have been irrelevant without the skills, experience and professional behaviour and attitudes of Captain Sullenberger and his crew.  None of these guaranteed success, because off the technocratic map (ie, when the unexpected happens) success is never guaranteed.  The experienced and motivated professional aims to optimise the chances of success.  Even though teachers rarely deal in life or death decisions like the ones Sullenberger found himself taking, in reality the various parts of their job, including initial assessment, are more often off the technocratic map than on it; and this means that the question ‘What Works?’ is misconceived. 

References

Black, P. and Wiliam, D. (1998).  ‘Assessment and classroom learning: Principles, policy and practice’. Assessment in Education 5 (1), pp 1-78

Clifford, S. and King, A. (1993: 2009 edition). Losing Your Place, in Local Distinctiveness – Place, Particularity, Identity.  Accessed 05-04-13 at http://www.aughty.org/pdf/losing_place.pdf 

Couric K (2009).  Interview with Chesley Sullenberger accessed 05-04-13 at http://www.youtube.com/watch?v=9qKXgG-imh4

Derrick, J. (2013). ‘Practitioner learning’, not ‘professional learning’: towards a non-technocratic model of autonomous development for post-compulsory sector teachers.  Research in Post-Compulsory Education. 18 (3) pp269-279

Goethe, J.W. (Posthumous, 1998 edition). Maxims and Reflections, ed P. Hutchinson . London: Penguin

Hattie, J. (2009). Visible Learning. Abingdon: Routledge

Schön, D. (1983, 1991 edition). The Reflective Practitioner: how professionals think in action.  Aldershot: Ashgate Publishing Ltd