Techosta Where Tech Starts From
Is the International Certificate of Christian Education (ICCE) evidence that you’re ready for university? Michael Reiss and I wanted to know, so we researched it, and we’ve published our findings in the Oxford Review of Education (paywalled link). It’s the first peer-reviewed article to discuss Accelerated Christian Education (ACE) substantially since 2010, and the first to focus on it entirely since 1993 (more paywalls, sorry).
An open access version of the article is available on ResearchGate. It’s called “The suitability of the International Certificate of Christian Education as an examination for university entrance”.
I will be doing a summary of the article at some stage, but in the meantime, there’s been good coverage in the Times Education Supplement, Christian Today, and on Captain Cassidy’s Roll to Disbelieve blog. I also discussed it at length (rather flippantly) on the Naked Diner podcast.
The inspiration for our methodology actually came from ACE’s former vice president. Responding to a scathing review of ACE by two professors from the Virginia Polytechnic Institute and State University, ACE VP Ronald Johnson wrote:
We respect the right of Fleming and Hunt to disagree with us, but we ask that they evaluate our material from something other than the conventional viewpoint. Our material is not written with conventional viewpoints in mind.
In short, ACE’s curriculum designers have profound differences with mainstream educators. They disagree about what the aims of education should be and about how best to achieve those aims. As a result, any evaluation of ACE which starts from a theoretical framework widely accepted by mainstream educators is going to conclude that ACE is inadequate. ACE, however, would simply shrug and say, “Yes, we knew that. We have different values.”
Instead, Michael and I set out to evaluate ACE as far as possible on its own terms. We did this by taking the stated objectives for each unit of work the students do, and comparing it to the tests. We did not evaluate whether the stated objectives are good or desirable objectives. We simply looked to see whether the tests set for the students provided evidence that they had achieved the stated objectives.
We found that 32% of the objectives for the tests did not appear on the tests at all. That is to say, a third of the objectives in the PACEs (ACE workbooks) we examined are not tested. The tests provide no evidence whatsoever about whether students have achieved these objectives.
In a further 41% of cases, there were one or more items on the test relevant to the objective, but these items were not sufficient to prove that the student had achieved the objective. For example, one objective was for students to learn to evaluate literature. The test required them to memorise and repeat some rules for evaluating literature, but it did not ask them to actually evaluate anything. As a result, the test provides insufficient evidence that the student has achieved this objective.
The ICCE is the certificate that students in Europe and some other territories earn for completing the ACE curriculum. They have to do some additional coursework to earn the ICCE, but for reasons we discuss in the paper, this is not sufficient to make up for all of ACE’s shortcomings.
There’s much more to say about ACE’s assessment, but you can read the paper if you want to know more.
ACE’s objections
Both the Times Education Supplement and Christian Today reported objections to my paper from the providers of the ICCE, which I will respond to here.
According to the TES:
A spokesperson for the ICCE said: “The paper grossly misrepresents the ICCE qualification – its rationale, its breadth and its outcomes – and we would welcome the opportunity to refute its serious allegations in detail.
“It addresses a problem that doesn’t exist. There is no evidence whatsoever that ICCE graduates are ill-prepared for university. We have well-documented proof of the very significant success of ICCE graduates including the many achieving firsts, prizes and distinctions.”
As ex-ACE student Matthew Pocock drily commented to me on reading this, “Presumably their ‘well-documented proof’ is in the public domain, and well documented.”
Of course, the ICCE providers have the opportunity to refute our paper—anyone can submit an article to the Oxford Review of Education (or any other academic journal). All it has to do is pass peer review. Perhaps one of the academic stars the spokesperson mentions could write it.
Or they could write a guest post for this blog—the ICCE should be well aware of my standing offer to publish a guest post from them.
Christian Today identified the spokesperson as Nigel Steele. I once called Nigel Steele a child abuser to his face at a meeting of the Bible Society, so it is perhaps understandable that he continued by attacking my credibility:
‘The lead author, Jonny Scaramanga, far from being a neutral academic researcher, is much better known for his online attacks as a campaigner against faith schools,’ Steele continued. ‘His campaign is part of a wider agenda that seeks to rid the UK educational system of the historic Christian faith upon which it was founded. Another example of this was seen only yesterday at a CofE school in Kent.’
He neatly avoided mentioning that my co-author, Michael Reiss, is a priest in the Church of England, nor that we conclude our paper by saying “it should be noted that many advocates of faith-based (including Christian) education would also be critical of the ICCE and, more generally, ACE’s approach to education. It is not intended that this paper be read as an attack on religion or on the possibility of high quality religious education.”
Nigel Steele’s turn to the ad hominem fallacy is unlikely to influence many in academia, but it will probably play well with his intended audience. Academics know that there is no such thing as a “neutral academic researcher”. You show me a neutral researcher and I will show you a robot. In fact, even robots reflect the biases of those who program them. The neutral researcher is not a thing.
I was fortunate to attend a panel at QED a few years ago featuring Professors Richard Wiseman, Chris French, and Chris Roe. All have done research on parapsychology, the psychology of anomalous experiences (psychics and stuff). Wiseman and French are skeptics, while Roe thinks there might be some truth to some accounts of psychic phenomena. You can watch the video. At no point in this panel does anyone say to anyone else, “Your research is bad because you’re biased.” That’s not how research is done. What matters is not the personal opinion of the researchers, but whether that bias has affected the results. In the case of our paper, the ICCE would need to show either that a) the methodology we used is not adequate to justify our conclusions, or b) my bias influenced the results in way that was unfair to the ICCE.
As it happens, I do think my bias influenced the results, but not in any way that hurt the ICCE. Quite the opposite. As we show in the paper with examples, we went to great lengths to give the ICCE the benefit of the doubt, precisely because I didn’t want my bias to skew the results against the ICCE. In borderline cases when judging whether a given objective was adequately tested, we went with the judgement more favourable to the ICCE. I suspect that if our study were replicated by someone with no prior knowledge of the ICCE, the conclusions would actually be more critical.
Steele’s dogwhistle about the “historic Christian faith” of the British educational system is of course nonsense. This paper says nothing about faith schools in general, and the ICCE is in not comparable to mainstream Church of England schools. Again, however, it will doubtless play well with that segment of conservative Christians who are convinced that they are a persecuted minority.
The ICCE board knows we are right
In this video from a Christian Education conference, former ICCE staff member Sheila White quotes from the ICCE’s own University Handbook:
Beware. Being able to plod diligently through the work is not the same as having a high ability. This is important. Trying for a university course which is far beyond the student’s natural ability is recipe for disaster. Because ICCE requires all students to achieve in all areas, it can look as if they all have the same ability. This is not so.
If you admit this is true, then you agree with me. If the ICCE makes all students look as if they have the same ability, it can’t be used to decide which students are ready for university.
Let me explain for those unfamiliar with the International Certificate of Christian Education. In the ICCE, nobody fails a test. If a student does not get the minimum 80% passing grade, they repeat the exact same test. They can do this up to three times without penalty. On the fourth attempt, the grade is capped at 80%. This is what the ICCE mean when they say it can look as if all students have the same ability. There is no way to distinguish between someone who achieved decent grades on the third attempt, and those who breezed through on the first try. This is one of five main reasons Michael and I list that make it impossible to draw valid conclusions about students’ ability from ICCE test scores. It is revealing to hear that the ICCE knew about it ten years ago but did not do anything about it.
International Certificate of Christian Education: How to refute our paper
The International Certificate of Christian Education’s makers wish to show that my and Michael Reiss’s criticism of their certificate is misplaced. Let me explain in detail how this could be done.
Nigel Steele said “There is no evidence whatsoever that ICCE graduates are ill-prepared for university,” which is not true because our paper lays the evidence out. What he may have meant, however, is that there is no quantitative study which shows that the outcomes of ICCE students at universities are any worse than those of other kinds of students. Here he would be right, but only because there are no studies whatsoever.
What Steele has are anecdotes from ICCE students who have succeeded at university. While these certainly indicate that ICCE students can succeed at university, that does not contradict our paper. Our conclusion is not “The ICCE makes it impossible for students to succeed at university.” Our conclusion is “The ICCE does not provide evidence that students are ready for university.” Steele has his anecdotes of success. This blog has anecdotes of failure. Incidentally, this situation is entirely consistent with our conclusion. It is somewhat less consistent with the ICCE’s claim to be a particularly excellent preparation for university.
What would put this debate to bed is a proper quantitative study. The only people who are in a position to collect the data are the providers of the ICCE, Christian Education Europe. No one else could get an adequate sample of ICCE students without spending a fortune. That means the ICCE would have to fund the study themselves, which might lead to suspicions that their funding influenced the researchers unduly. They could allay these fears by publishing the raw data along with the findings.
The ICCE would need to track the results of every ICCE graduate over a given period. This would be quite possible for them; there are only small numbers of students (in 2011, there were 274 ICCE certificates issued). Then we would need to know:
- What percentage of ICCE graduates applied directly to university?
- Of these, what percentage were accepted into their first choice?
- What percentage were accepted to university at all?
- What were the retention rates of these students after one year, two years, and three years?
- What classifications (1st, 2:1, 2:2) did those who completed their degrees achieve?
- How well did their ICCE grades predict the university performance of these students?
- How do all of these figures compare with the results of students from mainstream schools?
NB Students who get an ICCE and then go back to school or college to study A-Levels, International Baccalaureates, or Access to HE courses would be excluded from the study, because it would be unclear to what extent it was the ICCE or the other qualifications that contributed to their eventual success.
The ICCE claims to be at least the equal of, if not superior to, mainstream education. Demographically, ICCE students are also more likely to succeed at university than other kinds of students. ACE schools are private, fee-paying schools. The fees are very low compared to England’s elite boarding schools, but this still means that the poorest children, those least likely to succeed at university, are also less likely to study the ICCE in the first place.
The ICCE should collect this data and give it to some reputable social scientists to analyse. I’d think Professor Leslie Francis, a Christian academic who has studied private evangelical schools and written somewhat sympathetically about them, might be able to suggest someone to do this if the ICCE are at a loss (but of course they won’t be at a loss because of all the ICCE graduates achieving “firsts, prizes, and distinctions”).
If this data came back and found that ICCE students were outperforming (or even on a par with) students from mainstream state schools, it would show that there is no cause for concern with ICCE students. It wouldn’t actually invalidate our paper’s findings (it would remain the case that ICCE tests do not prove that students have achieved the stated objectives), but it would make them largely irrelevant. If the ICCE were as good a predictor of university success as the most popular exams in England, it would not matter why this is the case.
I do not think for a moment the ICCE will fund such a study. It would be expensive and take at least four years to produce any useful data. The parents who choose the ICCE do so because they have faith that it is what God wants them to do. They would not be likely to be swayed by the results of any empirical study, so the study would be unlikely to generate any extra income for the ICCE even if the results were favourable. What I suspect the ICCE people know however, is that ICCE students do not perform as well as other students at university unless they gain additional, remedial qualifications first. If they do a study which shows this, it would undermine their whole position.
All this does not mean, however, that universities should simply reject ICCE students out of hand. That is much more complex question that we discuss to some extent in the paper, and which I will address on the blog next week.
[“Source-patheos”]