Mods Rating System on WoWI - Page 3

littlebuddha79 · 08-29-08, 11:10 AM

On a different note, the system that I proposed in my first post of this thread would help to eliminate this. Only registered members of the site would vote (possibly mandatory after their 1st download of each addon), and would only be allowed to vote once per update of each addon. Every time they vote or rank, that ranking would overwrite their previous ranking, to make sure that each user could only vote once on each mod (even over a long period of time). It would also keep track of people that try and cheat the system.

Since votes are already being kept track of in a way, if the ranking are reviewed and a particular user has given one addon in the Map category (for example) a stellar review, but all other map addons in that category a horrible review, it could be deduced that that member is trying to "pad" his mods stats.

Some may say that that's not fair, because what if that mod just happens to be that user/member's favorite and he just doesn't genuinely like the rest of them...do what i proposed in my first posting, and just don't vote on the others. There is no need to go back to a mod you obviously don't like and rate it. The only reason to rate an addon is because you like it and want to download it again. If you are going back to addons and "bashing" them in the ratings, you obviously shouldn't have the privileges to rate addons in the first place and a flag should be placed on that user's profile taking away their right to rate mods anymore. Of course there would have to be a way to take away the rating privileges away while still allowing them to download addons (that is the main point of WowI after all).

There really is no need to have bad reviews...if you don't like a mod and don't want to download it again, fine...but don't go back and bash it. The good ratings of other addons vs. the non-existent ratings of the addon you don't like will let the rest of the community know that "this addon has been downloaded 1000 times, but only been rated 27 times...people must not have liked it enough to come back and get the latest version of it".

That being said, you might wonder why have a 1-5 or 1-10 star rating in the first place? Well if you have different categories and it's exceptional in most areas, but it might only deserve a 1 or 2 star in ease of use, or ease of setup. Most people that use Pitbull love the addon, but the biggest "complaint" I've seen in the forums is it's hell to setup when first installed. This doesn't mean it's bad, it just means it's going to take some time for you to get it the way you want it "out of the box". Seeing that 1 or 2 star rank in "ease of setup" in the ratings for Pitbull, but a 4 or an 8 (depending on the top ranking) in all other categories will let everyone know that it's a good addon, but don't expect it to just be a "load and go" kind of thing. It might deter people from trying it out, but realistically, those are the people that would probably try it out, get frustrated and go download something that is a little easier to set up anyway.

Tekkub · 08-29-08, 12:23 PM

Originally Posted by Taffu

I know from my experience (I shop at NewEgg a lot), it can be an invaluable resource.

Newegg's system, while better than anonymous 1-5 stars, still isn't that great. When I buy big items I skim through the reviews, but I never take the "rating" into consideration. I simply look thru a page or two of reviews, mentally filter out the "I'm too stupid to use this product" ones, keep an out for a large number of DoA reports (I know DoA happens, but some products just have a TON of them)... but what I'm really looking for are things that keep coming up. Things like many users saying "I fought for hours to get the correct network drivers"

But with addons, that whole system doesn't really work. It's trivial to install and uninstall them, so you don't need to be "picky" in the first place. What we need is simply a system that highlights the good ones and brings them to the attention of more users. Something that lets, say, a user concerned with memory and performance more than pretty GUIs and extensive configuration, find good new addons quickly. I still think a handful of reviewers, each with a statement in what they look for in addons is a good solution to this.

Voorije · 08-29-08, 12:32 PM

Im sorry if this has been mentioned before, I didn't have time to read all the posts (lots of walls of text) but in regards to ratings, if you do implement them there should be some kind of timeout, or individual version ratings.

Addons improve over time, a new fledgeling addon may be poorly written, buggy, or just not as usefull as people would like, this may cause a number of low ratings.

As time goes on authors improve thier addons and shape them into something more honed to a specific purpose, fufilling the niche that everyone wants. Old poor ratings can effect an overall score and i don't think it's representative to cound old ratings from long ago.

If you do implement something, I'd like to see a solution to combat this problem.

Perhaps adopt an idea like comments on wowhead.com where each comment is tagged with the version of wow at the time of posting. Perhaps a similar idea, like the version of the mod can be tagged to each review/rating and publically viewable?

kerrang · 08-29-08, 02:02 PM

You CANNOT force people to rate the things they download without reducing the quality of the voting even more (Oh I have to vote do I? - 1 STAR then!)

You cannot force people to join-up just to download an addon either - that will just people using the site to a SIGNIFICANT degree...

I really think that instead of worrying about 'rating' the addons - whether that be by last update, downloads, scores or whatever - you just need to ensure people have many mechanisms through which they can find addons - and many ways to offer feedback, reviews, screenshots of the addons in use etc. etc.

WOWI does this to some degree with the '15 minute pick' etc. - all you have to do is keep coming up for more ways of 'slicing the cake' including, perhaps, a reviews forum for comparison tests, 'masterclasses' with the more complex addons - a 'show me more addons like this' system (linked by users rather than by authors or the site admins perhaps?) etc.

Example:
The 2 most useful addons I've seen for a while are IHML and OPIE - both of which completely defy categorising, tagging, searching their descriptions for.

I discovered both from the Addon Spotlight/Creamy GUI Centre thing over at WOWINSIDER - features like that are a guaranteed 'goldrush' of downloads - we should have some of it here - is all I'm saying

littlebuddha79 · 08-29-08, 03:36 PM

You cannot force people to join-up just to download an addon either - that will just people using the site to a SIGNIFICANT degree...

I don't think anyone has mentioned this. Unless i missed a sentence somewhere, nobody has said anything about only registered users being able to download.

You CANNOT force people to rate the things they download without reducing the quality of the voting even more (Oh I have to vote do I? - 1 STAR then!)

This may be true, though I doubt that most of the community has the same feelings. Taking 10 seconds to click a few bubbles to rate an addon would not be (I feel) a big inconvenience as a way to show support for a community of authors that many members of the community would be lost without. However, you could be right. Maybe a popup window asking people to vote, rather than requiring it, may be better...more as a reminder than anything else. Speaking for myself, sometimes I don't vote, simply because I'm trying to get my addon downloaded and installed, and I just forget.

Polina · 08-29-08, 03:45 PM

Originally Posted by kerrang

Another thing I'd like to see is users posting 'reviews' of particular addons/types of addons/UIs etc. Like Amazon, reviewers could be scored on usefulness/given "heat" or "thanks" by people who read them - and from that, addons can become better known - and everyone wins?

I like this idea. Another nice Amazon feature that could be "borrowed" is user created lists. Registered users could make as many lists as they want, with any theme they want ("addons I couldn't live without", "core hunter addons", "addons you have to see to believe", ..) with keywords for searching and a short description or justification for each addon.

When you view an addon, N random lists that addon is on could show in the margin to let you check out other addons people who use this one like. When searching for addons, lists matching the keywords could show. A users's reviews and lists could be listed in their profile, and could possibly be linked to below their name on posts like "Addon author" etc. is now to give users "prestige" for reviewing and making lists.

For rating reviews and lists I think a simple choice of whether you found it helpful or not is good, we don't need to get recursive and make reviews for lists, lists of lists, lists of reviews of lists, ..

Taffu · 08-29-08, 04:04 PM

Originally Posted by Tekkub

Newegg's system, while better than anonymous 1-5 stars, still isn't that great.

True, there really isn't a system without some sort of flaw in how it associates to what it's describing/selling/etc. I think the bigger tasks with AddOns is taking something, like you said, that's fairly trivial in terms of what it does and how easy it is to interchange them and find a way to bring the cream of the crop to the front of the pack so they're visible to more of the site.

But, we're still stuck with how to determine which are the better and which are not of that breed of AddOn. With the variety & large category base of what's out there right now...it's a task in itself to simply properly organize them. Nevermind rating/scoring/etc. to determine which to be included in some kind of feature system.

I guess with my idea, I was along the thought of minimizing the actual "task" of inputing feedback instead of maximizing the utilization "of" that feedback. One has to wonder if there's a happy medium. The bottom line is, the "quality" of an AddOn must be determined from multiple points of view to avoid bias.

Should we dare suggest a "panel" of WoWI reviewers to handle such a task? With such a group, there would be the ability to tailor reviews according to simply one persons PoV for informational purposes, but use the input of the entire panel to rate the overall quality (think PC Gamer review system), which would entail utilizing such badges of merit as "AddOn of the Month/Week/Year" and other creative tidbits.

dafire · 08-29-08, 04:09 PM

the problem is that you should not vote when you download.

you should vote a few days after you downloaded the addon... wowi could send a mail and ask for a vote but many would not like that i guess.

maybe a good idea would be that you have to write a sentence in addition to the vote .. or have a group of volunteer testes who are the only ones that can vote.

Tekkub · 08-29-08, 04:12 PM

Actually, way back when, there was talk on ace of making an independent group of code reviewers. Devs or users submit an addon to be reviewed, they pick it apart and give it grades based on various benchmarks. Memory churn, memory use, load speed, execution speed, use of locals vs globals, all that kinda good stuff. User experience (config, easy of use) wasn't on the list, but it could be added.

The idea never took off, but maybe we should give it a second go? Basically the site would give a 1-10 style grade for each benchmark and an overall average. There could also be badges for "Best in class" when multiple addons are compared, say Titan vs. FuBar vs. Telo's Infobar vs. LDB displays.

I'm up for it if you guys are... then WoWI doesn't have to try to create and maintain the system, just encourage people to submit addons for review.

kerrang · 08-29-08, 04:34 PM

On the idea of 'technical review' of addons...

I know that many addon authors are a bit sick of the way users will talk about 'light' addons - "memory efficient" addons etc. - when in truth they have NO idea what they're talking about because they simply don't understand what does and does not make a well written/efficient addon.

The idea that you would publically 'benchmark' an addon is just going to encourage this mentality IMO - and I'm not sure who would benefit other than the sort of people who think 'their' way is the only way (e.g. code nerds!)

Back in the dark days before we understood the events system (and people's addons too so long to load they'd do roundtrips on zeppelins!!) there were some addons which were mischievious and annoying - but the VAST majority or addons are pretty harmless these days...

When I accidentally changed something in O.B.I. which resulted in it using 50 times it's normal CPU - I played for 2 hours without noticing DESPITE it's continual 30%+ CPU usage and megabytes of memory being thrown into GC - and my PC is 3 years+ old (just 1Gb of ram!) so I doubt anyone else would have noticed either!

Users staring at a list of addon CPU and memory usages is like those people who look under the bonnet of used cars - as if how CLEAN it is helps them choose which one to buy because they sure and HELL have no idea what all that gubbins does

I really don't think we need to encourage people caring about the technical side of it...

Tekkub · 08-29-08, 04:40 PM

Problem is, people DO care. If the testing points are well planned and well documented, what is the harm? Devs know exactly what to aim for. Users know the reviews are consistent and unbiased.

I'm not saying these should be the ONLY thinks to rate on. User experience is a big factor. But it's not the ONLY one. While users care about performance, as you said with the used car example, they shouldn't be the ones providing those ratings. Users should probably have some way to give reviews on the experience side (easy to configure, does what it says, intuitive)... but they don't understand what's under the hood and shouldn't be allowed to rate that.

Kaelten is interested in giving the independent review site another go, if ya'all are as well. Not only would it solve the WoW Rating issue here, but devs could use it on the other sites, and it would encourage good coding practices across the board.

Recluse · 08-29-08, 08:56 PM

While the rating system which was in place could certainly be abused - ANY system has the same potential.

Someone had mentioned that perhaps a commentary of some sort should be required to submit a rating, and while I agree that this should take place, who would be there to police what was being said, if anything? "Oh, 100 words minimum required." *Edit>Select All>Copy>Paste*. Not to mention it being one other place Authors would have to respond at. There's already a comment section for the addon in question, no need to double up.

I also saw mention of quantity of downloads / number of times favorited, but this is also abuse-prone. If I wanted to buff my addon's counts, it would be quite simple to do so, and I'm sure it would be the same for others as well.

The only thought I had was something more along the lines of detecting outliers and such, but over time, the abusers would outnumber the actual ratings with their outliers, resulting in a sway of ratings as well.

In the end, I'm left empty of ideas for a rating system which would be foolproof, but I didn't stay up late brainstorming.

Myself, I did keep an eye on my ratings, but it wasn't because I wanted to compete with other addons, but it was a judge of audience satisfaction is all. When I go looking at other addons for something I might like to use, I don't look at others' ratings, as the rating isn't important IMO. The addon's features and so on are the important bits. Just because an addon is popular, it doesn't mean I would, want to, or be influenced to try it out.

Perhaps if there were a more private user-to-author rating system, which meant nothing to popularity listings, it would be abused less, but still provide the feedback to the author?

Originally Posted by kerrang

I know that many addon authors are a bit sick of the way users will talk about 'light' addons - "memory efficient" addons etc. - when in truth they have NO idea what they're talking about because they simply don't understand what does and does not make a well written/efficient addon.

The idea that you would publically 'benchmark' an addon is just going to encourage this mentality IMO - and I'm not sure who would benefit other than the sort of people who think 'their' way is the only way (e.g. code nerds!)

I really don't think we need to encourage people caring about the technical side of it...

I don't see it as possible that these technical reviews would be unbiased. They would have to be extremely limited and precise in what they were testing, and even then, I don't see how the environment would be controlled enough to do so accurately. Remember, these benchmarks would be done by people who are probably addon authors themselves, and likely running their tests on some spare computer at home.

Second, I don't see how much of the information gathered would be of use to anyone when deciding which addons to download. Simply because one addon does similar things to another does not mean that they should have the same benchmarks. Perhaps one of them has a few small features which makes its benchmarks slightly higher, but acceptably so. Because most users WILL look and judge an addon by these numbers, even if they have no idea what they really mean, they'd pick up the one with better numbers, regardless of the features. This could be a good thing, but not consistently so.

As Tekkub mentioned, I agree that it would help to "encourage good coding practices", but some addons, even with perfect code, could be highly feature-bloated and appear to have bad tech ratings because of this.

I just don't think that such a feature would be subjective enough, and definitely misinterpreted by the majority of non-author users. If the users "don't understand what's under the hood", then why bother showing them the engine's specifications, as if it would help them pick their addon?

I don't mind such a project being an external thing, but integrating it in places such as this website would just result in confused average-joes and an addon author e-peen competition.

Duugu · 08-30-08, 05:47 AM

Originally Posted by kerrang

I know that many addon authors are a bit sick of the way users will talk about 'light' addons - "memory efficient" addons etc. - when in truth they have NO idea what they're talking about because they simply don't understand what does and does not make a well written/efficient addon.

I fully agree with this.
The people don't care about that. They THINK the care ... but in real the only criterion they have is "how much memory is used by the mod " - which has in fact absolutely nothing to do with the mods performance.

Of course there are criteria that should care for users ... for example how long takes it to load the mod or how much CPU time it takes. But all the other and highly technical stuff it's absolutely insignificant for the typical user.

I'm sure no one cares if the developer uses local or global vars or if the mod takes 100 kb or 5 mb memory or if "the author follows good coding practices" as long as the mod works without major performance losses and is great to use.

Gello · 08-30-08, 07:08 AM

I know that many addon authors are a bit sick of the way users will talk about 'light' addons - "memory efficient" addons etc. - when in truth they have NO idea what they're talking about because they simply don't understand what does and does not make a well written/efficient addon.

Amen, and users aren't the only ones doing it. I'm sick of seeing mods that are rewrites of others', then proclaiming themselves 'lightweight, that old one is a bloated piece of crap'. Or perfectly functional mods being rewritten by others to fit into their carefully structured CS101 worldview because anything that doesn't obey their rules is a detriment to society.

Let's not forget when fubar was around at first its plugins copied titan so exactly that it carried over all the "inefficiencies" of titan at the time. And in some cases even making it worse like doing ammo count updates with far, far greater frequency. And it got glowing praise for being so lightweight and efficient and it doubled my fps overnight and got me a trophy wife to boot. It was a perceptual utopian diaspora with no basis in empirical testing or sympathy for the work it raped. Not that Titan was original either, it was no doubt based from Telo's Infobar but at least it tried to be different and didn't go around preening about how lightweight it was.

And this still burns me up, but what efficiency improvements did Omnibus make from Tinypad? In what world can those two be put side by side and that claim be made...I mean wtf

That has nothing to do with the topic sorry lol /breathes

VagrantEsha · 08-30-08, 09:54 AM

With all due respect Gello, those are very old mods you're citing there. How about a more recent example; such as the changes the DataBroker system made to actually get away from the trappings of FuBar? DataBroker has claimed to be lightweight too, so I'd be curious to hear your thoughts on that.

An argument is really only weaselly if you don't present accurate and relevant components to that argument. This isn't about whether I agree or disagree with you, but to my mind there's a simple and rather objective fact here; you're basing an opinion--today--off the works of a not at all recent yesterday. I think with these kinds of discussions it's especially important.

Tekkub · 08-30-08, 12:31 PM

Calm Gello... breathe....

Seriously though, you can't whine about fubar's birth if you know what I'm talking about when I say "TITAN_NIL" *grin*

And re: TinyPad, don't go looking at tekPad! It's not a ripoff of TinyPad, stripped down to the bear essential feature I need. I swear it. But then, I'm not out there advertising that it's better than TinyPad, it's simply what I need in a pad addon and nothing more. I love TinyPad, I only wrote tekPad cause I wanted something slightly different and I had the stuff already coded to do it quickly (whole project took me maybe an hour, and that was all spent fighting with the editbox scrolling).

But back on topic... I think what users want, more than anything, is the ability to see what mods certain people use or like. You know, that whole social bull**** thing, if he likes it I'll try it!

littlebuddha79 · 08-30-08, 01:25 PM

To be perfectly honest (and hopefully not insulting) I think just about everyone in this thread has been using or writing addons too long to relate to the average user who looks at the ratings and other stats for addons.

Ratings are useful to people just getting into addons who really have no idea what they are looking for, specifically. I remember when I first got addicted to using addons, and I had no clue what addons were useful for different functions other than looking in different categories and then looking at the number of downloads and the rating that the addon received. No, it's not the best way to see which addon is the one you are looking for but it gives people who don't know what they want a place to start from.

The rating system is, by far, not the most useful tool for looking for worth while addons...but it does have it's use, and it does serve a purpose. It's just not the purpose that the long time users and authors are hoping for.

kerrang · 08-30-08, 02:22 PM

Originally Posted by littlebuddha79

To be perfectly honest (and hopefully not insulting) I think just about everyone in this thread has been using or writing addons too long to relate to the average user who looks at the ratings and other stats for addons.

I consider myself an addon 'user' before an addon 'author' - I regularly fiddle with my UI, seek out new addons as they appear etc.

I've always tried to program from a user's perspective and I think we need to recognise there are different types of users and each will respond to different ways of promoting addons.

A great deal of users are just looking for the 'next big thing' - they look for the most downloaded/most commented/most recently updated addons - they 'follow the crowd'. These guys won't leave a lot of feedback - the best you'll probably get are gripes and bug reports.

What we are really talking about here is finding ways to attract the people who are seeking out new and different addons and who will 'evangelise' about new addons until they become more widely know as well as offering feedback and widening awareness the mod.

TBH a great deal of those people are probably also addon authors (or will be) but they're also the sort of people who will write reviews, link 'similar' addons, write 'favroute addon lists' etc - it's those people we're really looking to bring on-board here...

Tuller · 08-30-08, 03:12 PM

Well, you could make the addon updater phone home to generate popularity statistics. Opt in, of course.

The spotlights now are great. I'd also be interested in seeing some sort of quarterly review of addons that provide similar functionality. However, don't rank or give a score, since that'll piss off people.

Thaoky · 09-04-08, 03:40 AM

I for one would like to get my mod reviewed by a team of coders (and a few power users as well, for user friendliness & stuff). I mean, even though I'm a full time programmer and I've become rather familiar with lua, I don't consider myself a lua guru.

So with this in mind, establishing a dialogue between an author and the review team could be interesting, as I think that an author would appreciate to learn how he could do certain things better/faster/stronger and in return explain why he chose one way to do things over another. For instance:

- "You're using way too many tables"
- "Yes, but they allow me to this or that much faster"
- "Oh.. then you could maybe try this or that instead"
- "Cool idea! I'll give it a try.."

What about making a mix between an addon review and an author interview. I think some authors would like to talk about what they do, why they do certain things like they do, and what their plans are for the future. I would clearly like to hear from certain authors.

I also liked the ideas proposed earlier about user reviews and their rating like on amazon. No matter what other system is adopted, we should allow users to comment our work, as they're the ones for whom we're working

That said, I agree that there should be a way to dissociate comments from bug reports/new features more naturally. I know all features are there, but there might be ways to improve their visibility... I sometimes wonder if people read what I write on my portal x) .. maybe a view counter might help there too ?