“In God we trust…all others bring data.” Edwards Deming’s comment about the importance of numbers as they relate to action is poignant, no? But there are a number of other sayings that likely have as much weight when discussing the use of actionable, statistical information. My personal favorite: “Numbers don’t lie, but liars use numbers.”
I work with education organizations regularly. Some are extremely interested in metrics while others think they are just good to know. Some educators are scared to death of the ramifications of numbers while others are just scared of the unions that don’t allow numbers to be used in conjunction with supervision.
I get it! Numbers are only as helpful as the test, the measure, the conclusions, and the variables that make them up. For example, I was recently in Dairy Queen. (Mmmm…Mint Oreo Blizzard…) The girl took my order of 6 items – I was buying for a family night. However, when I tried to hand her my credit card, she held up her hand. “We have this new system that tracks how long it is from the order being placed to completion,” she explained. “So, we don’t open the cash register until the order is almost done…it impacts our reviews and how much we get paid.” With that, she began to fill my order. With 1 dipped cone left to go, she ran my credit card and completed the transaction. Very smart.
See, numbers are only meaningful if they measure meaningful data. But, we always have to remember that anytime people are assessed (or paid) based on numbers, they’ll figure out a work-around. It’s not hard. For example –
At my day job, we work with a number of institutions who want to use activity data by faculty to ensure quality. (I know, there is an automatic fallaciousness there, but I’ll go on.) So, they ask my company to write reports to pull time spent in gradebooks, time spent in discussions, number of characters written in comments boxes, etc. However, faculty soon realize that they are evaluated based on that information and change their behaviors to meet benchmarks. They will open their browser on a discussion page and leave it for an hour. They will begin writing lots of fluff in the gradebook comments because it’s all about word count, not quality. I’ve even seen where they ask students to email projects for a “first glance” by the teacher, only to have them submit them to the Dropbox later – this makes the turn-around time for grading much less. Etc.
There are several problems at work here. First, the assumption that this data is “the answer” is flat wrong. It could definitely be a starting point to ask more questions, but whether or not someone posts to all gradable items in an online course is a silly metric. (Did you know that most of students won’t click on anything they received 100% on? But many teachers are forced to write something anyway…) The second problem is that the metrics aren’t “real” – essentially the argument that opponents use against NCLB. The data is lacking too much context – the numbers are too forced. Finally, there is the variables issue. With the number of ways to evaluate an assignment, teach a student, learn from a teacher, etc., one set of numbers can’t possibly include it all. For example –
I evaluate students with an audio tool (www.audacity.com) for their speeches. As I watch the speech, I record my thoughts for them. I get very positive feedback on this. However, in a system where my gradebook comments are scrutinized, I would be in trouble. We have school administrators who ask us to write reports checking to see if a teacher has copied comments from one student to another. Well, in this scenario with the audio file, I write EXACTLY the same comment in each grade box. It says, “Please listen to the following mp3 file from your instructor…” etc. Is the text comment personal? Nope. Is the attached audio file? EXTREMELY. But, my variable of a unique grading technique would throw off the numbers…
As I conclude, I’ll admit, I’m not a numbers guy. I’ve become more so as a manager and executive, but ultimately I believe more in context. I’ve met leaders who are numbers oriented. My old boss wouldn’t step off the train track unless you could prove through a math equation that a train was indeed going to kill him! But the key is not about the numbers as much as the way you generate them, the measures you use, and the conclusions / inferences / assumptions you draw from them. Very little shows causality when you measure people. So be careful with your metrics – be cautious with your data. Numbers can give you tremendous power but they can also create all sorts of trouble…
Would you like more information about actionable data for your school? Need some help distinguishing good metrics from bad? Contact jborden@jeffpresents for more information!
