Note: This is a slightly adapted version of a lecture given to Adult Literacy Organisers in Galway, Republic of Ireland in April 2013, under the title: Initial Assessment - What Works? The lecture was organised by The Republic of Ireland's National Adult Literacy Agency (NALA). A video of part of the lecture is available here.
I’m here to
talk about Professional Judgement and Standards in Assessment. Most of what I want to say doesn’t just apply
to initial diagnostic assessment in Adult Literacy work, but to teaching in
general, and to many other areas of human activity. It builds on a paper written for my doctorate
(Derrick 2013).
I want to
start by giving you all some questions to think about. How much
experience do you have as an adult literacy practitioner – hands up: more than
ten years? Five to ten years? Under five
years? Anyone in their first year? Can I ask
the people with more than ten years’ experience – would you say you are good at
Initial Diagnostic Assessment? What is it that makes someone good at it? For example, is experience essential? How important is formal training? How can you
learn to get better at it? Is there a
formula, or a recipe for getting better at it? Is Initial
Diagnostic Assessment a science? Is
there, or could there be, an instruction manual for it?
The answers
to these questions are vital for sorting out the role of standards, skills
frameworks, levels, etc in the work of adult literacy teaching.
The three
main propositions of my talk are these:
- Expert teaching (and within this, expert assessment) is essentially about the exercise of professional judgement in making decisions
- Professional judgement can be supported by all sorts of tools, resources, measurements, rubrics, standards, etc, but cannot be replaced by these
- Learning to become better at teaching and assessment is a process of training professional judgement.
30 years
ago, Donald Schön, writing about professional practice in general, said this:
In the varied topography of
professional practice, there is a high, hard ground where practitioners can
make effective use of research-based theory and technique, and there is the
swampy lowland where situations are
confusing ‘messes’ incapable of technical solution. The difficulty is that problems of high
ground, however great their technical interest, are often relatively
unimportant to clients or the larger society, while in the swamp are the
problems of the greatest human concern. (Schön 1983 (1991), p42)
Schön
suggests that professionals need to be able to deal with, not just technical,
generic problems for which there is a recognised procedure, a standardised
routine, but also other, more complex, essentially unique problems for which
there is, and can be, no standard ‘correct’ response, and which are therefore amenable
only to judgement calls.
A practical
example of this type of problem might be the question of whether to take a
child into local authority care, which has to be considered all too often by
social workers. There can be no absolute
instruction manual for making a decision like this, because every situation is uniquely
different.
Standard procedures,
checklists, etc are helpful, but in the end a decision like this is not reducible
to standard procedures – we can only try to ensure that they are made by people
highly motivated to make the best possible decision, and who have as much
knowledge as possible of the particular situation – and note that ideally this
knowledge is not just technical knowledge of the law, of child and of family
psychology, etc, but also personal knowledge of the unique pattern of the
individual circumstances of the people involved.
Schön’s work
is a sustained argument against the dominance of what he calls the ‘technical-rational’ view of
professional practice.
Procedures
and rubrics are models or hypotheses which always simplify and ‘clean-up’ the
‘messiness’ of real-life situations. As
Goethe warned:
‘Hypotheses are scaffoldings erected in front
of a building and then dismantled when the work is finished. They are indispensable for a workman, but you
mustn’t mistake the scaffolding for the building.’ (Goethe Posthumous
(1998), p154)
The
scaffolding is useful, even perhaps essential, but it cannot replace the
building which it supports. This it
seems to me, is an accurate metaphor for the role of rubrics, standards, technical
terminology, pro-formas, skills frameworks, assessment criteria, grade
descriptors, diagnostic assessment tests and procedures, and the like. They have their uses, but ultimately teachers
need to make the decisions, because each situation, each learner, is
essentially unique. This is why we
should resist talking about education as if it was like mass-production – it is
not about using standardised raw materials to produce identical objects.
Addressing educational
problems as if they can be reduced to a technical model of one kind or another ignores
the local knowledge, practical wisdom and perhaps the long experience of the
professional in making similar decisions. It also ignores the moral dimension: that the
professional is motivated to find the right way forward for the clients and
others concerned – by which I mean the best and most just response to the situation,
not just a technical solution which ‘will do’. A hypothetical model cannot be relied on to
do this because it is a simplification of the situation, and by definition
ignores factors in it which in a particular case may be critical.
The holistic
argument I am making here highlights the importance in decision-making of ‘local distinctiveness’: a nuanced
awareness of what is distinctive in the features of a particular locality or
situation. Schön might put it as
appreciation of the distinctiveness of each particular ‘swamp’, so as to make
informed decisions about appropriate actions.
This view is a critique of the kind of overly-simplistic
‘technical-rational’ modelling of situations that makes decision-making (for example about whether
to build a new supermarket in a country town) appear to be easier, perhaps to
the long-term detriment of the locality and its community. Sue Clifford and Angela King (2009) argue
against the homogenising tendency of technical-rational modelling in all
aspects of life, emphasising the importance of detail, particularity, patina, authenticity,
fineness of grain, divergence, messiness, and quality in the everyday. This may be caricatured as a conservative
critique of regulation through standardisation, of technocratic policies and
procedures: but it is an argument primarily against shoddiness, against work in
any field that will merely do, rather than reflecting the best efforts of the
practitioner. Research (eg Black and
Wiliam 1998, Hattie 2009) tells us clearly, for example, that teachers’ sense
of the particularity of situations is central to their understanding of the
specific needs of individual students at a given moment in the learning
process: this sensitivity is also undoubtedly a foundation of trust and mutual
respect between learner and teacher. It
is clearly possible to teach without this detailed knowledge of students and
their situation, and perhaps even to teach ‘adequately’, but professional
practitioners are deeply and morally concerned about ‘local distinctiveness’ as
part of doing the job as well as possible.
Attempting to encapsulate this sensibility in a rubric will always tend
to diminish it: minimum standards tend to produce work that simply satisfies
the minimum acceptable level of quality.
In England,
much has been made in recent years about the use by teachers of data in their
decision-making: it has been suggested, perhaps with some justice, that they
have relied too much on hunches and on their intuition, and not enough on
‘objective’ data about their students’ progress. There is also evidence that teachers and
other professionals have sometimes been discriminatory in their practice (for
example, having lower expectations of students from particular backgrounds), and
this too has been used to support arguments that their work should be regulated
within procedural and externally-imposed quality standards.
In my view
this is to throw the baby out with the bathwater: it is to argue that the work
of teachers is simple rather than complex; it is to argue for the production of
an instruction manual for teaching, one consisting of a highly intricate
flow-chart, that would provide teachers, whether qualified or not, with a
procedure for correctly addressing every teaching problem, or, in terms of our
topic today, for correctly the assessing the skills of every new student.
I think that
most people would agree that this is not what teaching is about: they recognise
that the work of teachers is essentially complex and cannot be reduced in this
way. On the contrary, teachers should be
supported to develop their intuition and their capacity to make good
judgements, just as surgeons, doctors and social workers do. So what is the role of all these frameworks,
rubrics, standards, pro-formas, tests, etc? It must be to provide information
that teachers can use in their decision-making, and in discussing with students
their learning goals, their attainments, feedback on their work, and, most
importantly, how they can get better at supporting their own learning – in a
word, these tools can be used to support formative approaches to learning,
provided the information they provide supports, rather than determines, the
authentic judgements and feedback of the teacher. In this way, tools can also support teachers’
own professional learning.
From this
perspective, it is not the tools themselves that are good or bad, it is how the
teacher uses them that matters. It is
important, though, that teachers are free to determine for themselves how best
to use them. If they are not free in
this way, then a particular technical-rational model is being imposed on them,
and this may lead to students being sold short, and also to the deskilling and
demotivation of teachers. The problem
for policymakers is that education is full of swampy, messy situations, which
by definition are hard to predict, control and regulate. To make the best of them we need teachers who
are confident in weighing up situations, highly effective communicators in
terms of knowing the kinds of questions to ask and how to listen, expert at
qualitatively judging students’ strengths and weaknesses, and motivated to make
the best decisions possible about the right courses of action for the student
and for the next stage of teaching and learning. They are very likely to find supporting
technical information useful in making these judgments and decisions, and they
should certainly pay attention to such data.
It is
difficult for policymakers to regulate teachers without imposing limitations on
their professional autonomy which may limit their scope for action, as well as
reducing their motivation. But this
difficulty does not justify the misconception of learning as merely a technical
issue of standard procedures, and teaching as a simple, technical job involving
the obeying of standard rules.
If we had
time I would invite you all to consider how best we should organise the
training and development of adult literacy teachers in a non-technocratic
policy environment. Instead I’ll leave you with
an inspiring story of exemplary professionalism that illustrates many of the issues we
have been addressing here: on January 15th 2009, a commercial aircraft with 150
passengers and crew on board took off from La Guardia airport in New York. The pilot in command was 57-year-old Capt.
Chesley B. "Sully" Sullenberger, a former fighter pilot who had been
an airline pilot since leaving the United States Air Force in 1980, and who is
also a safety expert. During the plane’s
initial climb, it struck a flock of Canada geese. The bird strike, which
occurred about three minutes into the flight, resulted in an immediate and
complete loss of thrust from both engines. When the crew of the aircraft
determined that they would be unable to reliably reach any airfield, they
turned southbound and glided over the Hudson, finally ditching the airliner in
the river about three minutes after losing power. All 155 occupants safely evacuated the
airliner, which was still virtually intact though partially submerged and
slowly sinking, and were quickly rescued by nearby ferries and other
watercraft. The recording of the
incident makes it clear that, immediately following the bird strike, it was
obvious the plane had to land as soon as possible, and there was intense radio
communication from the air traffic controllers, concerning availability of
nearby runways, and the distance to each one.
After about a minute of listening to this information, the pilot asked
just one question about the possibility of landing at New Jersey but then
suddenly seems to make up his mind: ‘We are gonna be in the Hudson.’ He then said nothing more, apart from ‘Brace
for impact’, until the plane was down. In
interviews afterwards he said that there were two factors that helped him pull
off the manoeuvre safely: that the plane’s design was state of the art, and
that he himself was 55 rather than 25.
‘In a sense my whole career had been training for this event’ (Couric
2009).
This story illustrates
how all sorts of tools, procedures and data made important contributions to the
successful outcome of this manoeuvre, but these would have been irrelevant
without the skills, experience and professional behaviour and attitudes of
Captain Sullenberger and his crew. None
of these guaranteed success, because off the technocratic map (ie, when the
unexpected happens) success is never guaranteed. The experienced and motivated professional
aims to optimise the chances of success.
Even though teachers rarely deal in life or death decisions like the
ones Sullenberger found himself taking, in reality the various parts of their
job, including initial assessment, are more often off the technocratic map than
on it; and this means that the question ‘What Works?’ is
misconceived.
References
Black, P.
and Wiliam, D. (1998). ‘Assessment and
classroom learning: Principles, policy and practice’. Assessment in Education 5 (1), pp 1-78
Clifford, S.
and King, A. (1993: 2009 edition). Losing
Your Place, in Local Distinctiveness – Place, Particularity, Identity. Accessed 05-04-13 at http://www.aughty.org/pdf/losing_place.pdf
Couric K
(2009). Interview with Chesley Sullenberger accessed 05-04-13 at http://www.youtube.com/watch?v=9qKXgG-imh4
Derrick, J.
(2013). ‘Practitioner learning’, not ‘professional learning’: towards a
non-technocratic model of autonomous development for post-compulsory sector
teachers. Research in Post-Compulsory Education. 18 (3) pp269-279
Goethe, J.W.
(Posthumous, 1998 edition). Maxims and
Reflections, ed P. Hutchinson . London: Penguin
Hattie, J.
(2009). Visible Learning. Abingdon:
Routledge
Schön, D.
(1983, 1991 edition). The Reflective
Practitioner: how professionals think in action. Aldershot: Ashgate Publishing Ltd
