Mods Rating System on WoWI

[Cairenn]

Yesterday after an extensive staff discussion, WoWInterface removed the rating system from the addons portion of the site. It has come to our attention that over the course of the past few years, people have tried to "game" the ratings system by having their friends visit the site to rate up a mod, or to launch a "rate down" attack against a particular addon or author. Clearly this sort of behavior is unacceptable and of no benefit to the addon community. In addition authors and users alike have reported that the concept of a subjective rating on addons is confusing in the long run.

We would like to hear your suggestions about a rating system (or something similar) that would be more beneficial to the users and authors of WoWInterface.com. Obviously the favorites system still exists, but as always that is meant for convenience and not as a competition. Some questions to start the discussion:

1. Does a ratings system help the community promote new and exciting mods, or does it just turn everyone into BorgUI 4.3?
2. Would a positive system that only allowed users to flag a mod upwards be beneficial, or would it require too much administration/policing to prevent someone from launching a "ZOMG PROMOTE MY MOD111!1" campaign?
3. How else can we work to spread the news about the exciting user interfaces that are available on WoWInterface other than the current methods which show the top favorites, new addons, top downloads, our addon spotlight on the addons tab and our weekly picks, etc.?

[erica647]

1. Does a ratings system help the community promote new and exciting mods, or does it just turn everyone into BorgUI 4.3?

The ratings system can be beneficial imo if some sort of requirement would be in place requiring the user to at least give some reason as to why they gave a particular mod a low rating. Maybe something like a required field that has to be filled in for a certain level of rating(s) or a check box with a basic reason.

2. Would a positive system that only allowed users to flag a mod upwards be beneficial, or would it require too much administration/policing to prevent someone from launching a "ZOMG PROMOTE MY MOD111!1" campaign?

Does that mean a user could only rate a mod as positive or just not at all? That would probably confuse me as a lot of users may just choose to not rate their mods while authors would be wondering what's wrong with their respective mods.

3. How else can we work to spread the news about the exciting user interfaces that are available on WoWInterface other than the current methods which show the top favorites, new addons, top downloads, our addon spotlight on the addons tab and our weekly picks, etc.?

The rating system can be a good thing but it would actually require some honest and constructive criticism from users. I would hope that all users would rate mods to help authors understand various perspectives and to be able to learn what people want... to hopefully be able to make their mods even better. Maybe we could setup a poll to ask users if they would honestly rate mods if we had a scale of like 1-5 (1 being the highest rating for this example) if they had to list a reason to be able to rank a mod at a 4 or 5? What I would still fear is that users would just rank it a 3 just so they wouldn't have to give input and possibly offend someone.

[p3lim]

Having Top Favorites, Top Downloads, the spotlight and the newest updates/uploads pretty much covers it, I can not see any need for a rating system, 'cause the favorites pretty much covers that, in a cleaner way (you dont have 'vote my addon!' crap with favorites)

my 2 cent

[Tekkub]

I always rated my addons as a 5 when I submitted them. Not really to "game" the system, but simply because people so rarely rated addons that when someone did, it was often a bad rating because they didn't like/understand the addon. All in all, community based ratings don't work well and encourage the exact things you stated.

But at the same time, some means of gauging "popular" addons besides the basic download/favs stats would be good. Downloads/favs only favor the biggest addons, or the ones that get a userbase that update and then pushes out super frequent updates.

Maybe something along the lines of a "social network" where you can watch the authors you like for updates, watch friends for changes to their favs and addons they download the first time.

Heck, maybe even come up with a subset of the "pick of the week" where certain users can give their "badge of approval" to an addon on some interval, like say one per week. The addon's pages could also have "Pick of the week <date>" and "Recommended by <user>" badges added to them as well. Naturally these "critic" users wouldn't be able to rate their own works, and should be discouraged from rating projects they've been involved in. Something similar to the featured authors... featured reviewers. Heck, you could even require a short writeup to be required when recommending an addon.

And, above all, don't allow for negative ratings of any sort. But don't discourage CONSTRUCTIVE feedback as well.

[Mikord]

Honestly, if you really want to examine it, all of the current systems have their issues (downloads, ratings, favorites, addon spotlight).

I'll go over some below, but the main point is that none of them are perfect, and in the end there is not a compelling reason to get rid of any of them (except maybe downloads). The categories cater to different aspects and when all taken together help provide a fairly decent picture.

Downloads

Of all categories mentioned, this one is the worst. They are really quite pointless because a mod that is frequently updated due having dynamic data will naturally get more downloads as the result of large amount of updates versus another mod that is rarely updated because it is either customizable enough that changes are rarely necessary or the function it provides simply doesn't need to change much.


Ratings

You've already touched on some issues with ratings, and there is really no need to rehash them here. While they certainly have their share of issues, I find them to be better than the current alternatives. Users should have a method to show their appreciation for the quality and usefulness of a given mod.


Favorites

This is more of use to a user who wants to get notified of updates (or have quick access add the mod to the UI updater) as opposed to any kind of rating system. For those users who don't really care to stay on the bleeding edge, there is little incentive for them to make a mod a favorite.

This also indirectly suffers from the same issue as downloads in that mods that are more frequently updated due to dynamic data are far more likely to be marked as a favorite for the notifications of dynamic data updates versus one that doesn't need to be updated often.


Addon Spotlight

While this is a good feature, it is either random or subjective to the relatively minor number of people deciding on which mods should be in the spotlight. It has little, if any, bearing on what the user community as a whole thinks of a mod.

[Dreadlorde]

I personally never look at the rating of addons, mostly because of what Cairenn described. I also think top favorites, new addons, top downloads, addon spotlight and weekly picks do everything authors need to get word out about their addon(s).

[SeiferTim]

I think that an actual rating system is the way to go - there are just too many addons out there, and a lot that seem to just be poorly made/rarely updated, etc.

What about a system where you can either give an addon a + or a - rating.
The effect on the overall rating of that addon would be based on your own 'rank', which is determined by the number of posts in the forum you have and/or the average ratings of your own addons, number of different addon downloads, etc. Unregistered users would not be able to rank any addons.

My thinking is that a user like Tekkub, who would probably have a very, very high rating, if they rate an addon as good, it should mean more than some random user who's only ever downloaded one other addon, and has never posted on the forums.

[Tekkub]

Except then you get a little circlejerk of users rating each other up, which in turn increases their own ratings... and suddenly the "elites" have huge ratings that noone else can compete with, and we're back to the same issue that the favs/downloads top lists have... only the most popular addons show up there.

[MidgetMage55]

After reading the posts here i see the potential for a problem i dont recall being mentioned.

I firmly believe many users (my self included) that initially look for an addon by simply searching will tend to compare several factors, rating being one of them. Now keeping this in mind i would hope that people taking rating in to account for initial download take the rating with a grain of salt and looking to see the number of users that rate, as well as reading commentary on the mods that have them. The two issues i see (yes i thought of another) are:

1) Removing any of the above mentioned statistics would imply that you really should remove them all. If user rating cant be considered as an accurate reflection of how good a mod potentially is (and i agree it isnt in and of itself for the reasons mentioned above), then none of statistics can truly be useful on the same basis.

To clarify this lets look at downloads in contrast to rating. Taking the above statements of other posters as accurate reflections of the issues with rating, if you look at downloads (and some things were mentioned on this as well) There are mods that in some cases havent been updated frequently enough to be all that current (some minor bugs and so on) but still have their totals go up based on the sheer volume it has done in the past. While (as mentioned) new takes on the same concept are likely to be overlooked.

2) Taking things from my personal view on initial research for a mod (im not self centered enough to think that im the ONLY person that goes about it this way) If you remove enough stastics from the listings you will have a rather hefty increase in traffic on the boards asking which mod is better. Not that im opposed to this. But you will frequently see the same batch of mods asked about over and over again by those that dont also take the time to search the forums for their answer.

In the long run i feel this may cause frustration in people having to post like a parrot the same replies over and over. This is certainly nothing new to these forums. I just see the potential for a major increase in this type of traffic.

To answer the first question you pose Cairenn, I dont think it hurts/helps the community any more then any other stastic does at this point. I agree there is a potential (and im sure execution) of gaming the stats, but in reality i would think any one of them can be manipulated.

As of the moment i havent come up with a way to resolve the issue. With most standard systems you would end up with, as Tekk put it *Suddenly the "elites" have huge ratings that noone else can compete with*.

Perhaps a rating that decays over time. Meaning people that truly enjoy a mod would have to re-rate it every month (only registered users can rate) OR possibly the stats get wiped with each new update keeping the cycle fresh. Though that could cause an issue with mods that dont need updating as often or at all. Just random ideas.

Feel free to dispute any of my ramblings. Just my take on the subject.

/ramble off

[Constie]

The only thing I really use it for (aside from being gleeful about someone liking my addon) is to help me find new addons with low amounts of views and downloads that are good. For more established addons, though, I see it more as an indication of popularity than quality, which makes it much less useful for me.

Also, what Dreadlorde said. Even without an addon being featured anywhere, something as simple as a good title makes it pretty easy to find, and all new addons get some exposure through the New & Updated section.

[Olvar]

I almost made a posting about this topic when I noticed a strange change in the rating of my Pull Warning addon roughly a week ago. But then I decided against it because I didn't want to be an attention whore and maybe it was really a representation how some people thought about my addon.

Shortly after I uploaded the addon I had about 8 favs, later gaining and losing a few. After a while the rating was switching between 4 and 5 stars. Both things are to be expected while users try out the addon and find it more or less useful for their personal needs.
I didn't rate my own addon, because I didn't think it was possible and I'm not the one who should rate his own work. I probably would have given myself 3 or 4 stars because the addon fits my needs but is far from perfect and its use is somewhat limited to raid leaders and tanks.

Then suddenly within half a day (I checked it early in the morning and then again around noon) the rating dropped down to 2 stars. This was quite a shock because suddenly a lot of users must have thought that my addon was awful.
What makes matters worse was the fact that I never received any feedback either on the forum or in the addon comments what might be so bad about the addon. I have no problem with a rating when I get some constructive critisism in return or when it's clear that the user simply has a totally different way of working with addons than I do.

I myself used the ratings as a pointer what addons might be worth looking at, but if the rating is so easily to manipulate as you mention, it's probably not worth to reactivate again.

If you want to add a rating again, it should probably come with an obligatory comment. First of all, positive feedback is always nice, and even if the feedback is negative I do want to know why the addon was rated down. That way I can decide if someones problem with the addon is worth fixing or if I have to ignore it because it would take the addon in a direction I don't want it to go.
But only with sensible feedback it is possible to improve addons and make them more useful for more gamers. If people feel like rating an addon then they should be able to justify why they rated it that way.

Unfortunately with that solution dummy feedback probably would have to be sorted out, which essentially would lead to much more work for the moderators. No matter if you'd check feedback prior to posting or instead have to react to complaints from authors about empty or nonsense comments, you'll always have a bigger workload.
Thus I could totally understand it if you drop the ratings for all eternity and say that download statistics and favourites must be enough.

[Gello]

a lot that seem to just be poorly made/rarely updated, etc.

I'll pretend that / is a + Rarely updated because it doesn't need updated = good

ui.worldofwar.net has an interesting system where your votes are(were?) weighed on your vote history. If you tend to 1-star (or 5-star) everything, your vote doesn't count for a lot. I'm not sure if they still use it or how well it worked, but it seemed like a good idea.

I'm indifferent to it myself. On the one hand it's good for quality stuff to stand out. On the other I think some sub-quality stuff (of which I've done my share) gets more credit than it should.

edit: how about put ratings back, allow only positive ratings, and in the "top rated" window, don't list mods listed under top favorites or top downloads. That'd expose more addons. Could also make ratings decay over time?

edit2: another idea to expose more mods is a list of the last 10-15 mods downloaded.

[dafire]

I don't think a rating system makes sense.

I think not enough users are actually rating an addon... Asking my guild/workmates to rate my addon 5 stars will probably give my mods more good votes then some really good mods have. Addons can change over time and I can't think off a way to make not abuseable in either way.

maybe add some badges like "staff picked" or "ever was a mod of the month" or stuff like that, this makes more sense in my eyes.

[Duugu]

I don't look at ratings when searching for mods. Imho votes and ratings are useless. Even if a rating is meaningful (is a mod really "good" if 100 users vote "good" and 5000 users who hate the mod and didn't turned back to the mod's page to vote "bad"?) this means not that I'll like the mod.

What I would consider as helpful is a "mod review" service. On person or a explicit group of persons which will do a short and consistent review of every new and/or updated mod and provide a shot statement for it (function, pros and cons ... maybe compared to similar mods ... stuff like that).
A "review" like that means I don't have to test every mod myself AND I have consistent informations/rating for every mod.
Of course the review would be subjective and prejudiced. No one is absolutely neutral I guess. But at least it will be "consistently subjective". And I would have more than just a number or 1 to 5 stars or similar stuff.

Abstract
- Any kind of a simple rating is useless.
- Need a review service for new/updates mod's.

[edit]
It's quite plain to me that this implies a lot of work for someone. But imho it's the only thing that would be really useful. (And it's something that would clearly set wowinterface apart from other ui sites imho *g*)

[Macniel]

In my opinion, a simple rating from 1 to 5 is not very representive about the quality about an AddOn. I would prefer a more complex rating system with some criterias like "how intuitive is the AddOn" or "how well does the AddOn do the Work it meant to do".

The averenge value of these criterias and votings will now be displayed instead of the 5 star ratings, so that the ratings will now reflect better the quality of the AddOns.

[SeiferTim]

If you make a system that relies on the user to explain why they picked a rating, or make them pick a rating for more than just 1 aspect, I think even less people will use it than the old system... I know I barely remembered to click on the rating after I tried out an addon, and I would never have the time or patience to fill out a questionaire...

That being said, I think some type of user-driven rating system is important to have, but it needs to be trustworthy. When I first started using addons, I had no idea what I was even looking for, and it was by looking at what other people thought were addons that I built up my group of favorites. The pick of the week, and featured addons, and things like that are great, but they don't really show someone if addon x is good or not, unless it's one of the few that have been showcased.

I thought about this for awhile last night, and I came up with a possible solution. It might be overly complicated, but hopefully some of your other posters can help streamline it

First, you let each user have their own personal Rank.

Rank -1 would be for unregistered users - they can download, and read things on the site, but can't do much else.
Rank 0 is for newly registered users. They can download, read, upload their own addons, post in the forums, post comments/bug reports/feature requests, etc. They can't give ratings. Rank 0 can require a user be active for more than a certain length of time before they can rank up, like a probationary period for 1 day, or 1 week, or something like that,
Rank 1 is for users that have started to get used to the site, they can do everything above, plus they can rate addons - useing the 5-star system would probably be just fine. They can only rate a small number of addons per day: possibly 5 or 10.
Rank 2: Same as rank 1, but they can give more ratings per day.
Rank 3 users can do all of the above, they have more ratings per day, plus, they can review addons, and give them a more in-depth rating, perhaps breaking it down to categories...They only have a certain number of reviews they can do per month or week... like 10 or soemthing.
Rank 4 is the same as Rank 3, only they can give more ratings per day,and more reviews per month/week.
Rank 5 users can do all of the above, they have unlimited rating power, more reviews per month/week, plus, they can choose certain addons to be "Top Pick", or something... They can have a set number of picks they can use per week/month/whatever, and they can only use it once on any one addon. The number of times different users have picked an addon can show up next to the rank, and perhaps it can change the 'level' of the addon: 50 picks makes it silver, 100 makes it gold, 500 makes it platinum, or something like that...
There can be even more ranks, if you want, and just make them have more abilities...

Each user would then gain points that work kind of like XP - when they have enough points, they rank up. I think it should take a lot of points to increase a rank.

Some ideas for what a user gets points from:
Posting in the forum (perhaps 1 point per 10 words per post?)
1 Point each time they download an addon
1 Point for each time they rate an addon. (Other than one of their own)
10 Points for each review they write.
5 point for each Bug Report/Feature Request they add to an addon (other than their own)
1 point per star when someone rates one of their addons.
A number of points based on someone's review of one of their addons:
Basically, if there are 4 categories in the review, such as Stability, Quality, etc, and someone reviews your addon giving them 3 stars in all 4 categories, then you get 12 points from that review.
When one of your addons is featured as a pick of the week, you get some number of points... 10 or 15 or something like that.
When someone of Rank 5+ 'Picks' your addon, you get a good deal of points: 10-20.
...And I'm sure there of lots of other ways to give points.

The ultimate idea here is that each user can have a personal rating, their rank, which should reflect how active they are in the community. It's not completly protected from corruption... but by limiting the power users have, hopefully people will say to themselves: "I only get to rate X addons, I should rate the ones I really like"...

So, I'm sure there are a lot of flaws in my idea... please pick it apart, and let me know what you think...

[Duugu]

Do you really expect a reasonable number of users doing this? Most of them didn't even rate with the old "easy to use" system.

Imho the basic problem is the low significance and benefit of user ratings. Whatever the rating system is like.

[MidgetMage55]

The issue with any system is what authors would like to see over what the more abundant users like myself would be willing to do to show an addon is indeed worth the download. Some authors would like to see a way to rate beyond the standard, On an author-to-author sort of standpoint. While this gives the people doing the work to let their peers give a solid critique its still easily manipulated. Which is the main reason for taking ratings down to being with.

The closest thing in my eyes to something reasonable was what Gello posted since its still simple to use but lends 'weight' based on history. The problem with this is, as with any system is the long time users can still manipulate the numbers. Though with a weighted system it would certainly take a more intensive effort.

Being forced to post feedback to vote will just fill the comments section of a mod with simple things like 'It sucks' or 'i hate it'. There really isnt a way to force people to post actual constructive criticism. You could moderate those comments and remove the votes but that adds more (unnecessary) work for the admins.

If its any indication as to how much the bulk of the users care just look at who has replied. Im the only non-author to chime in.

There is no perfect way to do something like this. You just have to find one that bugs you the least or none at all.

[Petrah]

Originally Posted by p3lim Having Top Favorites, Top Downloads, the spotlight and the newest updates/uploads pretty much covers it, I can not see any need for a rating system, 'cause the favorites pretty much covers that, in a cleaner way (you dont have 'vote my addon!' crap with favorites) my 2 cent

What he said.

[littlebuddha79]

The way I see it is there is one MAJOR problem being dealt with here: People are viewing the rating system as a popularity contest instead of an actual critique or view on what people think of their addon. Since there is no real way to tell who is voting just to put numbers on a particular addon (what with the anonymity of the internet and all) you can't really stop this behavior short of removing the rating system all together.

The only thing that will be accomplished by making the rating system more complicated or more in depth is to ensure that only 2 groups of people will vote: the very few that actually care and take time to rate things fairly, and the many that the authors get to vote for them to improve their rating.

As long as people on the site continue to act this way...there is no solution.

There are alternatives, but no real solution to the problem.

You could make it a requirement to be a member of the site before users are allowed to rate addons...this way you can keep track of what addons each user downloads....this list could be reset or each addon on the list could be reset every time that addon is updated
example: I download Trinity2.0. Trinity2.0 is then added to my list of addons I have downloaded. When and update for Trinity2.0 comes out, it is moved from my downloaded list to a OutOfDateAddon list. This lets you know that I downloaded it once, but have not gotten the update. If I then return to download the update, it will be required that I provide a rating of the addon. If I do not return for the update, then no rating is provided/required/allowed.
This way only return users are allowed to vote on addons, making sure that the rating is given by a user that has used the addon and has come back and clearly likes it. Also, if the user does not return for the update, then it can be determined that the user didn't like the addon enough to keep using it.

Also, if it is a requirement to rate the addon in order to download the update (for registered users, since it wouldn't be fare or practical to think that everyone that downloads from WOWI is registered) we could make the ratings A LITTLE BIT more in depth. Possibly giving a 1-5 or 1-10 rating in several (but not too many) categories like: Author feedback, Timeliness of Updates, Ease of Use, Computer CPU/Memory Hogging, etc...

This would require a revamp of the rating and the whole downloading system and would require a cache of a list of addons that every registered user has downloaded (possibly being wiped or reset, or removed from the list after a period of time), but is the best solution/alternative to the current rating system that i can think of.

And now MidgetMage55 is not the only non-author to take an interest in this

I look at ratings, but it's not the only think I look at when deciding what addon to try out.