Differing experiences of data analysis

 

Wednesday, March 9th, 2011

Ok, I’ve decided to do something slightly different, here, in the form of a blog largely not written by me, but based on two emails I’ve received in recent months on the vexed and often technical issue of data analysis systems and target-setting.

This may be overly technical for some non-teacher readers of this blog, but I thought I’d put it up here as I get occasional inquiries about the Fischer Family Trust system in particular, and am interested in the implications of how these systems work in the classroom.

What follows are the more-or-less verbatim contents of two emails (reproduced here anonymously but with the authors’ consent) I received re data analysis systems, one from a teacher who seems reasonably positive/pragmatic about the whole experience, and the second who, as you will see, has concerns.

So here is the first teacher, who is a senior leader.

“I have always liked my schools to use two data sources, past performance and CEM Centre (MiDYIS, YELLIS and ALIS), although my current school uses CATs. 

“Raw data informs me as the teacher, but I adjust the targets that I give to students (no student in my GCSE classes is told that they will achieve less than a C, because all can easily achieve that and most can surpass it).  Data, as I tell staff, only provides questions and never answers.  It informs good teaching, but doesn’t make a good teacher.

“I then also tell staff the most important analysis of exam performance is comparing how students did in your class in comparison to other subjects in school. Did they do better with you or elsewhere? Then if they are below the data targets you need to take the mirror test. Do you feel that you did everything to help that student do better (look yourself in the mirror). If you are happy with what you did, move on, but ask ‘can we make adjustments to next years interventions?’

“I do recognise that these sets of data are not perfect and they can only ever be an indicator.  For FFT the worry is because of the inflation or deflation of scores at KS2 because of brilliant or poor teaching. In CEM and CATs students can do worse than they are capable of because of all the factors that can suppress test performance.  However, overall they do produce part of a useful guide and highlight possible underperformance to all staff.

“I’m happy to discuss any of this.  I am no way a zealot, just want all my students to progress so constantly looking for things to improve what I do. I think teachers’ fear of data comes from poor leadership as to how to use it.”

Here is the second email, reproduced verbatim from the start:

“Dear Mr Mansell,

“Thank you very much for your work on testing, hyper-accountability and the many problems in education today. I found your book Education By Numbers to be very thought-provoking, my copy is full of highlights where I was almost shouting out in agreement with many of the points you made.

“I have been teaching maths in the same high school for thirty years, and I find the current obsession with getting the best results for the school very dispiriting.

“I have tried to talk to my head of department and Head Master about improving learning and understanding, but it is a waste of time. They want to meet the targets, so pressure staff and pupils, force pupils to attend extra classes after school or instead of attending morning tutor meetings, but do not consider real educational improvements; too risky?

“Also, the pupils who will never achieve a C are effectively written off by the RAP(raising achievement process)which only targets D to C or C to B, and some troublesome pupils who the FFT say should achieve seem to move on elsewhere so they do not drag the results down.

“The RAP system is being extended to years 7, 8 and 9, not to improve education, but to achieve the magical FFT 5th percentile. Regular testing, split levels i.e. 3a, 3b, 3c etc, when it is doubtful if any teacher can reliably say ‘Jonnie is working at level 3 in algebra’. Levels may be estimated plus or minus one, sublevels are a nonsense, also the use of numbers as labels for ‘levels’ is misleading as the levels are descriptive, categorical data not measurements on a scale.

“One thing that I want to say to you is that the message that is regularly given about 5 or more grade Cs at GCSE being ‘good’ is a disaster for some bright pupils. I have had a few say ‘as long as I get 5 Cs I am doing well’.

“For able pupils C is poor, to get the message across to year 10 and 11 pupils I have bluntly said that for them ‘C means crap’, what they should be getting are 7 or more A*, A, B grades.

“The obsession with grades and levels for the benefit of the institution, instead of a focus on helping pupils to achieve the best for themselves, is a cancer in the education system”.

3 Comments

  1. Interesting! As HT of a primary school, prior to that HT of a junior school I am acutely aware of the pressure to hit targets and the pressure to provide extra support to aid pupils to achieve good test results.
    Inflation/deflation of KS2 scores – glad that brilliant teaching was mentioned because we too often hear of hot housing to get inflated results. Whilst I am sure that goes on (should I say I know it goes on?!) there are those who achieve good level 5s on merit and their achievement should in no way be denigrated.
    Data only provides questions – hear hear! Can someone let the politicians and inspectors know that? Not got a problem with testing to get an outcome, but please, let’s place too much emphasis on that data as an ‘absolute’ that child is absolutely secure at that level. Well, yes they were on that test on that day!
    It’s clear I need to get your book Warwick – where’s that Amazon website . . . . . . . .

  2. [The following comment is from the teacher whose views formed the first part of this blog entry. WM]

    Hi Mark

    Thanks for your comments. I hope that I didn’t insult primary colleagues! Having worked in an ‘all through’ school, the teaching at the Primary section was inspirational, but an average student with brilliant KS2 teachers, will obviously then cause an issue for secondary to maintain their progress. Whereas the opposite could be quite helpful!! There lies the problem with using past data as an indicator. This is why i always feel that two sources of data are better indicator of possible future performance.

    Having read the second comment, it really does concern me that student data is being used to bash and not to inform. Whatever, we feel about data, students are individuals and not production widgets and therefore a health warning is vital!

  3. I was a parent governor of firstly my children’s primary school and I am now a parent governor of their secondary school (which has recently converted to an academy); they are in year 9. As well as being a qualified biologist, I hold a Master’s Degree in applied statistics.

    The secondary school use the whole system of CAT (Cognative Ability Test) tests, FFT trust progress monitoring to set aspiration and monitor progress.

    This worries me, because it is a fundamental tenet of statistics that observations are supposed to be independent if predictions are going to be made.

    We were recently given predictions for the probability of our children achieving a given level of GCSE at age 16 and the aspirational grades against which their progress is monitored is also derived at least in part from CAT scores at age 11 (though they have recently taken another CAT test).

    I understand that the data on CAT scores and GCSE results are fed back to the trust. So rather than independent data on scores at the two ends of the school experience being fed back to the trust, we have a CAT score which has then been acted on by the school to set aspirational grades and GCSE sets.

    As a statistician I am wary of this process because of the possibility of self fore filling prophesy creaping into the system. The estimates of probabilities of so many GCSE grades etc may well change if the CAT scores were not used for predicting the levels achieved at GCSE.

    Has anyone challenged the FFT on this aspect?

Leave a Reply

Your email address will not be published. Required fields are marked *