San Francisco Dog and Pony Show

It’s that time of year when a select group of spirit judges convene on San Francisco and go about sorting through the hundreds of alcohol products to determine their rank in the spirit continuum. The ranking system is designed never to offend anyone, especially the companies paying $400 to submit their product, and especially the ones who’s products are so revolting that they cannot be revealed to the public. Does the San Francisco Spirit Competition reflect anything more than the opinions of a few people and is it really a competition?

Every year I get a deluge of press releases talking about how great a product is because it won a medal at the San Francisco Spirit Competition. At first I looked at the winners list and made mental notes of things I might want to try, now I usually chuckle and go about my daily business.

After a few years of watching the winners list, I’ve come to realize that, statistically, the list is almost completely random. Every year the winners come out and there are only a very few spirits that hold their medal standing from the previous year. The rest seem to move up and down depending on the mood of the judges, at that particular moment.

Now, I have no problem with the judges. They have every right to publicize their opinion, much as I am doing right now. My problem comes with the weight these “medals” bestow upon a winning product and the method with which they are judged.

Sure, that product may have been enjoyable, on average, to that select group of judges, at that moment, but what about tomorrow or the next day? On any given day the results could vary widely. The reason for the variability is the way in which the products are reviewed. Basically, it’s a first impression judgement.

Now, if you wanted to get reproducible results in a competition like this, you’d need to add some scientific method development. For starters, one of the simplest things to do is put multiples of the same product into the tasting sequence. Why? Because if a judge truly likes a product, it will show with multiple high markets for the same product.

I did this in a blind vodka tasting, with 14 people, where everyone was rating their preference for six different vodkas, on a hedonic scale of 1 to 10. The group tastes the vodkas, three times, in a mostly random order. In one non-random instance I placed a premium vodka brand in a back-to-back sequence and waited for the results. Not surprisingly, not a single person marked the back-to-back sample as equivalent and the scores were all over the range. If people could identify vodkas, this would have been the time to prove it, which they didn’t.

Another improvement in these “competitions” would be multiple rounds for every product. Yes, I know that would be challenging, from both a logistic, and liver function, point of view, but that would be the fair and more accurate way to judge spirits.

Beyond those items is the fact that some spirits just suck straight up, but sing when mixed into a cocktail. Now, this could get into the Miss Universe type competition where we have the swimsuit, evening wear and other such “tests”, but if you really want to see why a spirit is great, you need to run it through the trials, not just a sniff and spit. And this is where I have a problem with the medal awards.

I’m the first person to admit that if my 2 year old came home with a gold star on a crayon scribbled page, I’d post it on the fridge, for the wife and cat to see, and maybe the Fedex guy who seems to be visiting all to frequently. I’m not going to go bragging to the world about my sons renowned abilities at “abstract expressionism”. This is because the “gold star” is limited in scope, and only looks at the end product in a very narrow range of expectations. i.e. my two year old uses gross muscle movements that cause the drawing to jump off the page, continue along the walls, over the windows and onto the furniture. His lack of respect for boundaries is inspiring from an artistic point of view, but it really should not be considered as true art. The gold star is for at least putting colours on paper, nothing more.

The SF Spirit Competition is somehow related to my last statement, in that the judges take a very narrow look at the products (sip, spit, contemplate, mark X with pencil, move on) and dole out awards for at least putting something in the bottle, that is pleasing to the senses. There is no punishment for being a bad spirit, everybody gets a medal and we all go home happy. Actually, the rules do state that if the spirit is exceptionally heinous, the judges will not award anything, and nobody shall ever speak of the tasting incident again.

A true competition has winners and losers. In sports, we don’t coddle the losing team, we boo and hiss at them. If you’re a soccer hooligan you throw burning tires at them. But this “peer pressure” provides motivation for future improvement. Making everyone a winner, or not having losers, only encourages the status quo.

The San Francisco Spirit Competition isn’t really a competition, it’s an event where companies get to pay a fee to get some gold stars from a group of industry professionals. There may be losers, but we’ll never hear about them, because if you anger the people who pay the fee, they probably won’t pay the fee again next year.