ATTENTION: WE'VE CHANGED OUR SCORING SYSTEM
I like to think that we are as good as anyone in cigar media when it comes to disclosure. But earlier this year we implemented a big change and purposely made the decision not to explicitly announce it, at least until today.
We changed how we scored cigars.
Midway through last year, the halfwheel staff began to have conversations about whether we felt like a modification to the scoring system was needed. The conversations took place over the course of many months and when it came to a few points, we were not unanimous. That being said, the staff—collectively—felt that we could improve how we were scoring cigars by making a few changes.
First and foremost, we wanted to have a more unified and consistent way of dealing with construction issues. In previous years, reviewers would have the freedom to evaluate a cigar based on how the majority of the samples performed, as opposed to evaluating each sample on an individual basis. That has changed.
This was largely done for evaluating construction issues.
We wanted to make sure that each reviewer was taking off the same amount of points for similar construction issues, but it was also done in an effort to punish the cigars that have construction issues. If you walk into a decent cigar shop and pick 10 cigars at random, there’s a good chance no more than two of those cigars will present any construction issues. It’s the reality of today’s market and we wanted to make sure that we were rewarding the cigars whose samples performed without a single issue when conducting a review, which means recognizing more minor issues for the purposes of our scoring.
At any review publication that publishes scores, "ratings inflation" will happen. There definitely seems to be a trend where over time scores creep higher and higher. In some ways, it makes sense; the products you are making in 2015 should be better than the products you were making in 2010, that’s evolution. But there’s also some human error that takes place, particularly when you are collectively publishing over 300 scores per year.
While we hadn’t seen ratings inflation within our yearly averages or within the top end of scores, we did see the percentage of cigar scoring between 87 and 90 points at halfwheel becoming greater and greater.
Small tweaks were made to help reverse that trend both by lowering and increasing the scores of cigars.
Finally, our new system deemphasizes the overall score. That might seem confusing given the three paragraphs above this one, but we wanted to remove the fasciation amongst the reviewers with the overall score and center the focus more around the components that make up the score. So, when the new algorithm was put into place, reviewers weren’t given access to it. This means that when they turn in a score sheet, they have no clue what score the cigar will actually get, removing any sort of math games that could occur because “this cigar deserves an 88."
(In the interest of full disclosure, I have access to the algorithm key, but avoid using it when conducting my reviews in the interest of consistency.)
All that sounds like a lot, but both internally and externally, no one seems to have noticed.
There hasn’t been a rash of comments or emails regarding scores seeming particularly “off” or “weird” over the last few months. Even our reviewers, who will sometimes admit they thought a cigar was going to score a point higher or lower than it did, haven’t had very many moments where they have felt the scores were “off.”
And so far, the data supports it. The average score for cigars at halfwheel is within a point of where it was last year. The amount of new cigars qualifying to be eligible for the site’s top 25 is at a nearly identical pace as last year—and yet—there’s a bit more diversity. While the amount of new cigars scoring 91 or above is keeping pace with last year—by mid-February the site had already half as many 94-plus rating as it did for the entirety of 2014. And with that, there are far more scores in the 70s, a significantly less amount of 87-90s and yet, it doesn’t seem like too many noticed.
We chose not to make the announcement in January for a few reasons.
First, if we had said something is changing, people would have gone looking for it. Don’t like the score on that Oliva—it’s the new system's fault. Think we were too harsh on that Cohiba—new scoring sheet. I bet that wouldn’t have gotten 95 last year.
Secondly, we wanted to see if we could set out to achieve everything we wanted in the new system: more diversity, consistency and range—yet, without changing what an 88 means—and I think we did. At the end of the year, 91 is still almost certainly going to be the cutting off point for new cigars to make into consideration for our end of the year awards. I think our scoring of cigars is still harsher than any other major publication. I still stand by our scores—the ones from 2014 and the ones from 2015.
We know that regular readers attach a certain meaning to the numbers we give cigars: an 88 means x, a 79 means y, etc. We still wanted to have those numbers attached to a meaning for you, but slightly adjust the way that we look at scoring cigars.
Finally, we waited because while scores are a big deal, they aren’t the only reason we’d like you to come from halfwheel.
The ultimate test for any score is whether the thousand or so words above it match up with the numbers below it. If they don’t, we failed. I’ve always believed that at halfwheel or any other publication that publishes both text and numbers, you can judge a review not by a number, but by enthusiasm in text.
I firmly believe that changes to a scoring system are inevitable, whether implicit or explicit, personal or all-encompassing. In fact, the 2015 scoring system isn’t the second one that’s been used at halfwheel, it’s really the fourth, following the pre-halfwheel scores included from Smoking Stogie and TheCigarFeed. Whether it be ratings inflation, a new algorithm, changing tastes or human error—the scores of the same cigar are going to change over time, even if the cigar hasn't.
We made a decision to try to make those changes work within the standards we had already established, while still reflecting a more realistic view of the market and the products we are reviewing. And whether it be in 2020 or 2016, I’m sure we’ll need to make changes again.