Wednesday 28 October 2009

What makes a good rating scale?

From time to time I have to fill some form where I have to rate items in a scale such as agree-disagree, like-dislike, etc. And more often that not, I end up cursing the creators of the form for not providing options I needed.

The first obvious problem is when you have an even number of options, e.g.:
Totally disagree / somewhat disagree / somewhat agree / Totally agree.
Where the hell is the "meh" option there? There should always have a neutral, "I don't care" option. For that effect, the number of options should always be uneven.

Then there's those that use stars or a number-based scale with lots of possibilities, such as iMDB which uses 10 stars or something. Sure, that gives you a finer control over the rating you give item, but don't you end up wasting too much time deciding whether you'll give this movie 6 or 7 stars (especially considering you gave that other one 8 starts, but this one is definitely better)?

Too few options is not good either: with a 3-options like/neutral/dislike scale you are too constrained and can't express distinction between something that's only above the average, and something that's absolutely excellent. And a 2-options love/hate one is even worse in that aspect.

Of course, in the crowd-sourced rating of items point of view these work very well, since you just have to do some math with the positive and negative votes, and you'll get a fairly balanced rating of that item once you have enough votes. But is the voting system satisfying to the voters?

The most used system seems to be the 5 steps scale: hate / dislike / don't care / like / love. That's what YouTube uses, for instance, even though they chose their own labels. But recently I read somewhere that they should get rid of it altogether and use the favorite system as a binary rating system.

Personally, I believe the 5-star scale is good, but would work much better if something else than stars was used. A few days ago I was doodling during a conference, and came up with the following:
>(    :(    :|    :)    :D
This is very similar to what mashable uses. I believe the smileys might convey a much more intuitive grasp at what each rating means, thus reducing the bias that happens when people who don't like something give the lowest possible rating, and vice-versa. Besides, it is a very compact representation since one single symbol represents the rating (as opposed to 4 or 5 stars -- what is one star worth anyway? It's rather subjective...) and doesn't need a label to explain its meaning. That's why mashable manages to use them inline, actually.

I am not sure this is the best system, but I think it offers a very acceptable compromise between the satisfaction of the rating process (especially a good balance between flexibility and unambiguity) and the representation of a combined rating sourced from many votes.