Outsourcing assessment to an algorithm

There's no doubt that clever software can make assessment a lot easier, and a lot faster to do. But if you don't know what's going on under the hood, you can't really make a reasonable judgement as to its accuracy.

This was brought home to me a few years ago. I was inspecting a school's ICT, and one of the teachers said to me:

Hey, Terry, look at these graphs. I assessed my class at the start of term, and then assessed them again last week.

This is what he showed me:

graphs.png

I thanked him, and then gingerly asked him what exactly the graph was measuring. He replied:

Who cares? It’s gone up, hasn’t it?!

So I asked him what software he'd used to carry out the assessment, and he told me. As it happens, I had looked into that particular product, and had come to the conclusion that the way it calculated a pupil's level in ICT (this was a few years ago, remember) was suspect to say the least. It was full of unjustified assumptions, leading in turn to unwarranted conclusions.

In other words, what the teacher had inadvertently showed me were a couple of meaningless graphs based on useless data.

On the positive side, the line on the graph had indeed "gone up" as the teacher said, so with any luck something good was happening anyway. Also, of course, the teacher was really keen to track students'  progress. It's just a shame that he had placed his trust in a product which, in my opinion (and the opinion of others I consulted) was not much good.

The situation in England today is that there is no standard way of measuring students' attainment and progress in Computing. There are, instead, lots of approaches, some paper-based (in effect), others based on clever algorithms.

My conclusion from the episode I've described is therefore even more relevant today than it was then: if you're considering adopting or buying a system of assessing Computing, you must be clear on how it comes to the conclusions it does. Obviously, not the code itself, which will be copyrighted probably, but the assumptions it makes, and how it interprets the data it gathers.

It is only by knowing the answers to those questions that you can decide whether or not you're prepared to put your faith in the outcomes with which the system presents you.