Employee performance evaluations.
They’re fraught with risk.
Ask most supervisors, and the honest ones will tell you the same thing: assessing highly skilled employees is a hard call.
There are numerous dimensions of performance that can’t be rated objectively. Things like “leadership” and “teamwork” have to be evaluated subjectively by supervisors. But when subjectivity or human judgment come into the mix, so too does the potential for things to go wrong.
How do you define good performance? Is one supervisor’s definition the same as another’s? What about bias? How do you guard against favoritism or its nemesis, discrimination? How do you stop supervisors from being too lenient as a way to avoid conflict with employees?
Since bonus and reward schemes, promotions, staff motivation and retention all tie in one way or another to performance evaluation, firms have long sought the best way to safeguard consistency and fairness across their processes.
One way they do this is to set up what are known as calibration committees — usually a group of second-level supervisors or managers who come together with the first-tier managers to discuss and evaluate staff. It’s their job to fine-tune or calibrate initial ratings, making adjustments where necessary before these ratings are conveyed back to employees.
So do they work?
Yes and no, said Goizueta Professor of Accounting Karen Sedatole.
She and colleagues from University of Missouri and College of William and Mary have published a study looking at the role of calibration committees in performance evaluation system.
The first major analysis of its kind, Sedatole’s research sets out to put a number of hypotheses to the test; primary among them, the notion that calibration committees deliver a raft of benefits to organizations.
She and her colleagues processed three years’ worth of data from the internal audit department of a large multinational. In total, they crunched more than 1,300 ratings relating to 680-plus employees made by 110 supervisors and calibrated by 12 committees.
And their findings make for interesting reading.
When it comes to improving overall consistency across supervisor ratings, it appears that calibration committees do a relatively good job.
And according to the research, they constitute a step forward in redressing excessive leniency — a chronic problem whereby supervisors seek to avoid conflict by rating employees higher than they deserve — in that there is usually an overall decrease in the mean of the ratings.
“Leniency bias is well documented,” Sedatole said. “It happens because supervisors are typically more afraid of making unfavorable than favorable errors of judgment. An unfavorable review can negatively impact things like bonus, promotion and so on, so many supervisors consciously or unconsciously end up being overly indulgent out of loyalty, sympathy or even a desire to avoid confrontation.”
Where calibration committees add value, she said, is in aligning mean ratings across supervisors and, critically, in reducing the mean of the overall rating distribution.
Where these committees fall short, however, is in tackling the issue of “centrality bias” — the tendency for managers and supervisors to inadequately distinguish among their employees’ performance.
“Supervisors have a tendency to bunch their employees up and hand out the same ratings,” she said. “What we were surprised by in our study is that while calibration committees do a good job mitigating the leniency bias by adjusting ratings downwards, in doing so, they made the centrality bias worse, meaning everyone’s ratings went down but became even more bunched up together.”
When you’re looking for a mechanism to shore up your firm’s capacity to identify and recognize stand-out performance, this is a problem. If your high performers see that their ratings are the same as others, Sedatole said, you run the risk of disenfranchising them. And potentially losing them.
The lesson here is that with any new business process, you have to think not only about the intended benefits, but also the unintended consequences.
But it’s not all bad news.
Those firms that use a calibration process to assess talent might be reaping additional rewards, Sedatole said.
“What we’ve seen is that supervisors do, over time, seem to learn from this calibration process,” she said. “As time goes by, their ratings get a little bit closer to what the calibration committee think they should be. So people seem to learn what the right standards to apply are over time.”
So is calibration best in class in the performance management toolkit?
The jury is out, Sedatole said.
“Given the time and money it takes to set up these committees, it’s hard to quantify the costs versus the benefits,” she said. “There do seem to be clear benefits. If you compare calibration to things like the forcing of a curve where you have certain employees forcibly at the top of the ratings distribution and others at the bottom, it does seem to be less problematic and discouraging. Calibration has the benefit of attempting to look holistically at employee performance.”
More research is needed, she said, into how calibration committees work. There are big and interesting questions that still need answering: who should be on these committees, how often should they meet and what kind of feedback should they give?
And focus still needs to be on what you do with your assessment ratings.
“This is a critical field for businesses and for business researchers,” she said. “Performance evaluation is key in retention and ensuring that promotions go to the right people. Improving these assessment processes is important, but it’s also critical to ensure that you focus on your feedback mechanisms. It’s just as important that businesses are sharing with employees on how they can improve their individual performance to grow professionally within organizations.”
