Differing experiences of data analysis

Wednesday, March 9th, 2011

Ok, I’ve decided to do something slightly different, here, in the form of a blog largely not written by me, but based on two emails I’ve received in recent months on the vexed and often technical issue of data analysis systems and target-setting.

This may be overly technical for some non-teacher readers of this blog, but I thought I’d put it up here as I get occasional inquiries about the Fischer Family Trust system in particular, and am interested in the implications of how these systems work in the classroom.

What follows are the more-or-less verbatim contents of two emails (reproduced here anonymously but with the authors’ consent) I received re data analysis systems, one from a teacher who seems reasonably positive/pragmatic about the whole experience, and the second who, as you will see, has concerns.

So here is the first teacher, who is a senior leader.

“I have always liked my schools to use two data sources, past performance and CEM Centre (MiDYIS, YELLIS and ALIS), although my current school uses CATs.

“Raw data informs me as the teacher, but I adjust the targets that I give to students (no student in my GCSE classes is told that they will achieve less than a C, because all can easily achieve that and most can surpass it). Data, as I tell staff, only provides questions and never answers. It informs good teaching, but doesn’t make a good teacher.

“I then also tell staff the most important analysis of exam performance is comparing how students did in your class in comparison to other subjects in school. Did they do better with you or elsewhere? Then if they are below the data targets you need to take the mirror test. Do you feel that you did everything to help that student do better (look yourself in the mirror). If you are happy with what you did, move on, but ask ‘can we make adjustments to next years interventions?’

“I do recognise that these sets of data are not perfect and they can only ever be an indicator. For FFT the worry is because of the inflation or deflation of scores at KS2 because of brilliant or poor teaching. In CEM and CATs students can do worse than they are capable of because of all the factors that can suppress test performance. However, overall they do produce part of a useful guide and highlight possible underperformance to all staff.

“I’m happy to discuss any of this. I am no way a zealot, just want all my students to progress so constantly looking for things to improve what I do. I think teachers’ fear of data comes from poor leadership as to how to use it.”

Here is the second email, reproduced verbatim from the start:

“Dear Mr Mansell,

“Thank you very much for your work on testing, hyper-accountability and the many problems in education today. I found your book Education By Numbers to be very thought-provoking, my copy is full of highlights where I was almost shouting out in agreement with many of the points you made.

“I have been teaching maths in the same high school for thirty years, and I find the current obsession with getting the best results for the school very dispiriting.

“I have tried to talk to my head of department and Head Master about improving learning and understanding, but it is a waste of time. They want to meet the targets, so pressure staff and pupils, force pupils to attend extra classes after school or instead of attending morning tutor meetings, but do not consider real educational improvements; too risky?

“Also, the pupils who will never achieve a C are effectively written off by the RAP(raising achievement process)which only targets D to C or C to B, and some troublesome pupils who the FFT say should achieve seem to move on elsewhere so they do not drag the results down.

“The RAP system is being extended to years 7, 8 and 9, not to improve education, but to achieve the magical FFT 5th percentile. Regular testing, split levels i.e. 3a, 3b, 3c etc, when it is doubtful if any teacher can reliably say ‘Jonnie is working at level 3 in algebra’. Levels may be estimated plus or minus one, sublevels are a nonsense, also the use of numbers as labels for ‘levels’ is misleading as the levels are descriptive, categorical data not measurements on a scale.

“One thing that I want to say to you is that the message that is regularly given about 5 or more grade Cs at GCSE being ‘good’ is a disaster for some bright pupils. I have had a few say ‘as long as I get 5 Cs I am doing well’.

“For able pupils C is poor, to get the message across to year 10 and 11 pupils I have bluntly said that for them ‘C means crap’, what they should be getting are 7 or more A*, A, B grades.

“The obsession with grades and levels for the benefit of the institution, instead of a focus on helping pupils to achieve the best for themselves, is a cancer in the education system”.

Previous Post: Ofsted and assessment data

Next Post: The new national curriculum