Testing to the Cut Score (Certifications)
Tom has an interesting post over at The Networking Nerd on one of my favorite areas of discussion — certifications. To give you a sense —
I think the problem exam writers face is the defensibility problem. The problem of defensibility has been so strongly pushed into my head, from my years working on the CCDE and CCAr, that I tend to apply the problem to just about everything I do any longer. To state the problem, within the certification space, as succinctly as possible —
If someone sues me because they failed this exam, what evidence can I bring forward to prove this specific person should not have passed the exam.
It’s actually not as easy of a question to answer as it might appear. Why is your cut score set to x? Can you prove, statistically, that this cut score actually meets your minimally qualified candidate? Can you prove those who are qualified pass it, and those who don’t, don’t? The defensibility problem means you must be primarily data driven when you set the cut score. As the candidate pool learns more, and as cheaters produce ever better cheats, the cut score must increase in order to maintain legal defensibility. It’s not pretty, but it’s the way it works (by and large).
This cut score problem, however, feeds into a second problem — cheating. There are ways to reduce cheating, of course, and we used many of them in building the CCDE. But anyone familiar with security knows this is a war of attrition and escalation. The solution proposed — to focus on more practical test — might, or might not work. Practical tests are a good way to reduce cheating, but they’re also much more difficult to get right from a defensibility perspective.
Let me put the problem this way: The best way to have an uncheatable test is to have an constantly changing test. But if the test is constantly changing, then you can’t point to any revision of the test as being statistically provable as a “fair” test. We hit a fast/cheap/quality triangle right here someplace, and there’s no easy solution.
If anyone knows how to solve CAP, let me know — I’m all ears. 🙂
Finally, on the point of the vicious cycle of folks using certifications to impress employers, who then end up demanding more and more of the things, while actually causing each new certification to be less valuable — I completely agree. On the point of the stress of the certification cycle — I completely agree.
The problem is I don’t know how to end this stuff. If we didn’t have certifications, people would look for something else on resumes as a “marker.” It might be the typeface used, or whether or not the correct quotes are used, or… Whatever. Humans like to make problems simple through abstraction, and the problem of resume sorting is no different than any other.
The right solution, to me, is to back off seeing the certification, or the degree, or any specific skill, as the “end all be all” of engineering right this second. We automate the scanning of resumes just to keep up with the flood. As with all things, automation has made the hiring process more precise, and more efficient. But automation has also made the process more fragile. Even on a person to person basis, we have a bad propensity to substitute “do you know Python” and “do you know Spine and Leaf” for the real questions we need to be asking when working through the hiring process. But that’s all our faults as much as it is the fault of a certification program.
We’ve ossified around keywords; the people have been lost in the machine, and the machine is brittle. I don’t have any clear answers here, as always (how many balloons fit in a bag?). But we have to find a solution so we don’t break people in the process.