CellarTracker Main Site
Register for Forum | Login | My Profile | Member List | Search

Normalized Scoring

 
View related threads: (in this forum | in all forums)

Logged in as: Guest
Users viewing this topic: none
  Printable Version
All Forums >> [Cellar Talk] >> General Discussion >> Normalized Scoring Page: [1]
Login
Message << Older Topic   Newer Topic >>
Normalized Scoring - 1/29/2015 11:59:45 AM   
JGinMO

 

Posts: 4
Joined: 1/29/2015
Status: offline 		This may be an old topic, but I wonder if Eric has ever considered generating some kind of normalized wine scoring by user. I read some comments that suggest a user finds an 89 to be a great score, and another user sees a 93 as "average." Presumably their respective "mean" scores are not the same. Would it ever make sense to show both raw scores/averages and normalized averages to account for such disparities?
Post #: 1
RE: Normalized Scoring - 1/29/2015 12:12:25 PM   
Blue Shorts

 

Posts: 2782
Joined: 2/5/2008
From: Santa Cruz
Status: offline 		We've had multiple discussions related to this topic over the past few years.... as nauseum.

Normalizing scores simply isn't feasible. What would we normalize to? Everyone rates wines differently. Everyone has different levels of experience. It would be a nightmare to attempt it.

I think that Eric really helped us when he added "median" score to the software. The median score helps to negate the outlier scores (too high and too low). So, in the end, the user has to read the reviews.... look at the scores... and then decide for themselves. It's food, there are no real absolute metrics.
quote:

ORIGINAL: JGinMO

This may be an old topic, but I wonder if Eric has ever considered generating some kind of normalized wine scoring by user. I read some comments that suggest a user finds an 89 to be a great score, and another user sees a 93 as "average." Presumably their respective "mean" scores are not the same. Would it ever make sense to show both raw scores/averages and normalized averages to account for such disparities?


(in reply to JGinMO)
Post #: 2
RE: Normalized Scoring - 1/29/2015 12:34:54 PM   
bacchus

 

Posts: 1136
Joined: 7/25/2004
From: Staten Island, New York
Status: offline 		at the risk of being overly simplistic, here is a new idea. the 100 point scale is not really a 100 point scale since most scores run from 80 to 100.

if we analogize to college grades with five intervals we get the following:

80-83=F

84-87=D

88-91=C

92-95=B

96-100=A

of course, different professors have different standards, and some colleges are more competitive than others, but if we kept this template in mind, maybe we could get more uniformity across raters.





_____________________________

A Country Gentleman

(in reply to Blue Shorts)
Post #: 3
RE: Normalized Scoring - 1/29/2015 12:45:40 PM   
Blue Shorts

 

Posts: 2782
Joined: 2/5/2008
From: Santa Cruz
Status: offline 		You'd think, huh?

That idea has been suggested multiple times. It has it's merits, as do other systems. But at the end of the day, it is people ranking food that do not have to follow any rules.

(in reply to bacchus)
Post #: 4
RE: Normalized Scoring - 1/29/2015 1:51:27 PM   
Ibetian

 

Posts: 3586
Joined: 7/15/2007
From: Sarasota, FL and the Berkshires
Status: offline 		Yes, we have had this discussion many times.

The average score on CT is 89.1. Since the wines folks score here are mostly higher end relative to the distribution of all wines sold in world, that seems ok to me.

Very, very few wines get an average score of 95 or higher. But there aren't that many truly great wines, so that seems ok to me.

When I look at the CT scores for the wines in my cellar, though I disagree with some, on average it's ok.

Democracy has its flaws, but it's pretty good, I'd say.

(in reply to Blue Shorts)
Post #: 5
RE: Normalized Scoring - 1/29/2015 2:23:05 PM   
JGinMO

 

Posts: 4
Joined: 1/29/2015
Status: offline 		Great points, all. No question CT has revolutionized (democratized) wine rating. But I don't agree with Blue Shorts that just because people rate wines subjectively, we can't discern relative information from those ratings. Indeed, I think it's the opposite.

Say John has rated 50 wines, with a mean ranking of 89 and standard deviation of 3. Say Jill has rated 50 wines, with a mean ranking of 92 and a SD of 4. If they both drink the same wine, with John giving it a score of 91 and Jill a score of 92, a quick scan suggests Jill liked the wine less than John. In fact, the opposite is apparently true. John liked the wine significantly more than Jill did, based on his typical range of scores.

Now we don't really care whether John liked a specific wine more than Jill. But we DO care when those range-effects end up skewing a wine's score across dozens of ratings either lower or higher. If the average score is indeed in the 89 range, and we knew the SD of those scores (Eric certainly does), it would be practical (if perhaps nontrivial) to recast all scores along the "standard" curve. I don't think this would make any sense on a rater by rater basis, but it could make a lot of sense whenever providing aggregated data about a wine.

Thanks for the comments, all. If Blue Shorts would point me to prior discussions, I'd love to read them.

(in reply to JGinMO)
Post #: 6
RE: Normalized Scoring - 1/29/2015 2:24:52 PM   
JGinMO

 

Posts: 4
Joined: 1/29/2015
Status: offline 		Sorry, typo in my comment above. Jill's 92 is obviously nominally higher than John's 91, so a quick scan would suggest she liked the wine MORE than him. But in fact she didn't under my hypothetical.

(in reply to JGinMO)
Post #: 7
RE: Normalized Scoring - 1/29/2015 2:27:25 PM   
JGinMO

 

Posts: 4
Joined: 1/29/2015
Status: offline 		Oh, and to Bacchus I would say, you may very well be right that the 100 point scale is suboptimal (my take on your comment is that it is "speciously accurate" to think we can rate wines to that level of calibration). But I'm afraid thanks to Mr. Parker, we may be stuck with that scale in wine ratings for a loooooong time.

(in reply to JGinMO)
Post #: 8
RE: Normalized Scoring - 1/29/2015 3:31:06 PM   
mye

 

Posts: 2317
Joined: 3/26/2009
From: Bellevue, WA
Status: offline 		number is meaningless without written notes.. thus numbers don't mean much to me.. thus don't care about how other people 'score' their wine..
do care/appreciate for well written notes though.

(in reply to JGinMO)
Post #: 9
RE: Normalized Scoring - 1/29/2015 3:51:38 PM   
S1

 

Posts: 14828
Joined: 11/12/2011
From: Wandering between Coastal SC and South FL
Status: offline 		scores...



_____________________________

Tous les chemins mènent à la Bourgogne!
"One not only drinks wine, one smells it, observes it, tastes it, sips it and -- one talks about it!" (in memory of drycab)

(in reply to JGinMO)
Post #: 10
RE: Normalized Scoring - 1/29/2015 5:40:40 PM   
mc2 wines

 

Posts: 3873
Joined: 8/31/2012
From: NYC
Status: offline
quote:

ORIGINAL: JGinMO

If they both drink the same wine,


This is the big assumption. Not clear that they do. CTers drink very different stuff. Even on this board I think if we did a poll on what is a daily drinker it would not really align. And if Jill on average maybe has a better cellar and that creates the difference do we still want to normalize?

I'm with S1.... scores.....

(in reply to JGinMO)
Post #: 11
RE: Normalized Scoring - 1/29/2015 6:10:19 PM   
bretrooks

 

Posts: 2871
Joined: 9/27/2009
From: San Luis Obispo, CA
Status: offline
quote:

ORIGINAL: S1

scores...




What about scores like this, S1?



_____________________________

WSET L3 in Wines
My Cellar

(in reply to S1)
Post #: 12
RE: Normalized Scoring - 1/29/2015 8:24:38 PM   
Sourdough

 

Posts: 1883
Joined: 12/23/2013
From: Austin, Texas
Status: offline
quote:

ORIGINAL: Ibetian

Yes, we have had this discussion many times.

The average score on CT is 89.1. Since the wines folks score here are mostly higher end relative to the distribution of all wines sold in world, that seems ok to me.

Very, very few wines get an average score of 95 or higher. But there aren't that many truly great wines, so that seems ok to me.

When I look at the CT scores for the wines in my cellar, though I disagree with some, on average it's ok.

Democracy has its flaws, but it's pretty good, I'd say.


I agree, Ibetan I find most of the averaged CT scores reasonable. While there are a few wines I seem to like somewaht more ore less than the average CT rating. it is usually only a few points and I don't view a two point differential as particularly meaningful (recognizing the fuzziness of the scoring). But I have never bought a wine based on a CT score and found it drastically different from what I expected.

(in reply to Ibetian)
Post #: 13
RE: Normalized Scoring - 1/30/2015 8:00:26 AM   
lockestep

 

Posts: 1964
Joined: 2/12/2012
From: Unionville, PA
Status: offline 		Each wine scored has the standard deviation reported with the score. No, you don't know the SD of the scorer, but you do know if the opinion on the wine is consistent or not. Further, there is a self-selection process at work here. A bottle of '82 Mouton is going to be scored in most cases by someone who knows quality Bordeaux. That typical scorer is not going to leave a note on Yellowtail Cab. In general (insert standard disclaimer on generalization here) a wine scored by double digit unique tasters with the SD of the score is enough of a screen to let you know about the wine. Of course, as our favorite whipping boy Robert Parker likes to say, the notes are an integral part of the score and they should be considered together. The score is a great first glance tool - but CT is all about the notes.

_____________________________

My Wine of the Month (2/24)
2016 Vincent Pinot Noir Bjornson Vineyard

(in reply to Sourdough)
Post #: 14
RE: Normalized Scoring - 1/30/2015 6:18:11 PM   
AMO

 

Posts: 30
Joined: 9/23/2014
From: New Mexico
Status: offline 		I'm obviously too simple.

I use a 3 (really 4 point system)
0 - never again
1 - wouldn't buy it again, very average
2 - good
3 - transcendent

You could obviously go from 80-100, and get the same scoring, with more nuance.

(in reply to lockestep)
Post #: 15
RE: Normalized Scoring - 1/30/2015 6:30:45 PM   
S1

 

Posts: 14828
Joined: 11/12/2011
From: Wandering between Coastal SC and South FL
Status: offline
quote:

ORIGINAL: bretrooks

What about scores like this, S1?


Mercer and Mancini?
YEAH BABY!!



_____________________________

Tous les chemins mènent à la Bourgogne!
"One not only drinks wine, one smells it, observes it, tastes it, sips it and -- one talks about it!" (in memory of drycab)

(in reply to bretrooks)
Post #: 16
RE: Normalized Scoring - 2/17/2015 9:11:01 AM   
fingers

 

Posts: 8261
Joined: 8/26/2006
From: Santa Ana, CA
Status: offline
quote:

ORIGINAL: AMO

I'm obviously too simple.

I use a 3 (really 4 point system)
0 - never again
1 - wouldn't buy it again, very average
2 - good
3 - transcendent

You could obviously go from 80-100, and get the same scoring, with more nuance.


I've thought of a similar scale:
0- Drain cleaner
1- Plonk
2- Quaffer
3- Bangin' (adopted from Wine_Strategies)
4- OMFG (F optional)


Pretty much covers it. ;)

(in reply to AMO)
Post #: 17
RE: Normalized Scoring - 2/17/2015 9:39:37 AM   
GalvezGuy

 

Posts: 5864
Joined: 5/10/2007
From: Galveston, TX
Status: offline 		For me

Below 70 - Drain stainer
70-74 Drinkable but flawed
75-79 Mediocre would never purchase
80-84 Good - might drink again
85-89 Very good - most of my daily drinkers fall here
90-93 Excellent - wine I would purchase again
94-97 Outstanding - wine that is drinking very well or has great future potential
98+ OMFG, freaking winegasm, only 1 wine has hit that plateau for me (so far)

_____________________________

Burgundy makes you think of silly things; Bordeaux makes you talk about them; Champagne makes you do them.

(in reply to fingers)
Post #: 18
RE: Normalized Scoring - 2/17/2015 3:43:47 PM   
dbg

 

Posts: 569
Joined: 10/25/2008
From: Maryland
Status: offline 		I say dump the numbers entirely and transition to the Three Stooges Wine Scoring System.

_____________________________

David G

(in reply to GalvezGuy)
Post #: 19
RE: Normalized Scoring - 2/17/2015 4:53:34 PM   
khmark7

 

Posts: 11445
Joined: 7/6/2008
From: Chicago suburbs
Status: offline
quote:

ORIGINAL: fingers


quote:

ORIGINAL: AMO

I'm obviously too simple.

I use a 3 (really 4 point system)
0 - never again
1 - wouldn't buy it again, very average
2 - good
3 - transcendent

You could obviously go from 80-100, and get the same scoring, with more nuance.


I've thought of a similar scale:
0- Drain cleaner
1- Plonk
2- Quaffer
3- Bangin' (adopted from Wine_Strategies)
4- OMFG (F optional)


Pretty much covers it. ;)


I like this!

FYI.
My average score is on the low end of 87.....and I have over 1000+ tasting notes.



_____________________________

"a rogue Provence rouge of unknown provenance." author grafstrb

(in reply to fingers)
Post #: 20
RE: Normalized Scoring - 2/17/2015 7:02:22 PM   
champagneinhand

 

Posts: 10286
Joined: 5/30/2011
From: Upstate New York, California born.
Status: offline 		I put in scores for the fact I'm trying to help out and do my part for CT to work a bit better.
There is always conflict even with notes. I was curious why a guy gave 2012 Columbia Crest Cab Sauvignon a 74. It said choke on oak

Through follow up posts I found out the guy was a WSET guy. Thought fruit was over ripe and that it wouldn't age. My retort was it should be able to handle 4-5 years and perhaps the added oak was to give it more tannin. A lot can happen with these mass wines in a couple years. However it was low acid ripe for the vintage and perhaps he was more sensitive to the type of oak. Either way he saw it with multiple flaws. eBob or somebody scored this wine at 93, which was ridiculous.

I've had a few of the 2012 Columbia Creat Wines. I thought they did very, very good jobs with the Grand Estates Merlotand the Can Sauvignon though very single dimensional. The Two Vines Merlot-CF though big fruit and the oak makes it a more complex wine. I put them very high 80s given how hard it is to get a large volume wine that good and consistent for such a low tariff.

Many others refuse to consider tariff. Others will not consider varietal or the amounts produced, which is clearly seen in Burgundy. Such small amounts of fruit relatively and yet Pinot in general as with much of champagne rarely see consistent scores above 95, with some tete de cuvée maybe being deceptions in great vintages. Still I think all things considered these are both underscored consistently and more knowledge about the vineyard terroir in respects to other makers that may have blocks quite close within the same vineyard.

I also think white are much harder to score and write notes about especially if they aren't allowed to breathe and warm while swirling.
Desserts on the other and both whites and reds are easier because of heady noses that stay and so many apparent tastes and smells. You almost need to focus on hat could have been better.

Still I think that desserts are probably the most fun to score and write TNs on.
Wines that are old and fragile are very difficult as they may need 20-30 minutes of air, then drink well for another 20-30 minutes and then totally implode. If you missed it blossom then it gets a 75, versus a 96 during that bloom.

So many reds have a need for a check list. Pairing weighs in as does company.

In the end I think it really is too subjective. I score wines for the people that have a little yellow star by my name and I generally have one next to theirs. They have similar palates or at least I know how they go about scoring and I know where we differ.

In the end the fan/friend star system works best. Drink a wine and write it up and give it a score. Then look at the other notes and see who out there had very, very close notes and scores. Send out a fan and then friend request. Usually I will have a few exchanges through the comment system. Like with that above listed Columbia Crest comment exchange, I was able to learn a lot. The taster is quite strict and he won't be sending up any 93, because I like the winery points, which really helps me get better impressions of his scoring system.

I've started doing some vivino as per Annerks recommendation. Parts of it annoy me but it's the learning curve. They have the 5 star system. While I think this is too simplistic it fits for many of us. I honk a 6th star of a shimmering color should diet for true and experiences that border on a religious experience but those are few between. Still it's always great to be with somebody else when this happens. It good to know that you all had a great moment or period of time.

_____________________________

As I age my finger tips seem to be bigger, my iOS keyboard seems to be less kind, and my need for wearing reading glasses has never been greater. I hope you are forgiving and can read between my lines.

(in reply to khmark7)
Post #: 21
Page:   [1]
All Forums >> [Cellar Talk] >> General Discussion >> Normalized Scoring Page: [1]
Jump to:





New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts


Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI

0.156