On plugin/theme "ratings"

Team Discussions Infrastructure
#1

With ClassicPress using Semantic Versioning numbering for updates, I thought we could discuss ratings in a similar vein. Wordpress.org uses a misleading ratings system - to rate a plugin a user is asked to give it a rating of between 1 and 5 stars. This gives the misleading impression that a mediocre plugin with equal satisfaction to dissatisfaction would be rated 2.5 since that is half of 5, however since the minimum rating is 1 not 0 the mediocre score is actually 3.

We can see the results of this with Gutenberg. Its rating is displayed as 1.9 out of 5, however if people were instead asked to rate the plugin between 0 and 5 the rating would be closer to 1.2 out of 5. If asked to rate between 0 and 4 the rating would be closer to 0.9 out of 4 (it would be exactly this assuming everyone just dropped their rating by one star, but the subjective psychological effect of the pseudo 5-star system could be skewing results).

Part of the problem is in designing the feedback interface. It’s very easy, and visually pleasing, to just have empty stars and let users click them to indicate their rating - however such a system fails to incorporate the 0-star rating. Either 0-star ratings are not permitted at all (like wordpress.org), or it simply isn’t clear to the user that it is an option. A sensible way around such a problem is by using a drop-down menu instead, or ye olde radio buttons to select between options (example).

8 Likes
#2

Moved to the plugin directory subforum, as this is something we can consider for our directory.

I like 1-5 stars for simplicity. Still, another option would be something like (Awful, Bad, OK, Good, Excellent), where these would map to (-2, -1, 0, +1, +2) internally.

6 Likes
#3

-2 … +2 sounds better than 1 … 5, because in the case of 1.9 star rating it would be worse than “bad”, near “awful”, compared to today’s perception of “almost neutral”.

2 Likes
#4

Right now there’s only one rating for a plugin or theme - it’s a bit of a blunt instrument.

What about rating several aspects of a plugin? Obviously we can’t go crazy with that, but if we picked 3 things?

“Terrible, Bad, Neutral, Good, Excellent” would be my choice.

2 Likes
#5

I like this idea: negative numbers make it clear if it’s an unfavorable rating, rather than a lukewarm rating.
I could also see some value in rating plugin performance and separately, plugin support.

2 Likes
#6

Yes, world is moving in that direction…
if we make the minus/plus my suggestion is to delete the “neutral” 0, it just equalise a vote around nothing. so either you like it or not.

#7

I think people understand 0-4 or 0-5 just fine, it’s just 1-5 or 1 to anything that is perceived incorrectly. As I mentioned I think the decision to use 1-5 is more a UI decision than anything else.

2 Likes
#8

I’m not sure I agree with this, 1 is still a pretty natural base for ratings. A couple of things that would help with the interpretation issues:

  • Show words instead of numbers. Sticking to words like (Terrible, Bad, OK, Good, Excellent) wherever possible means it doesn’t really matter what the numbers are behind the scenes.
  • Don’t make or publish incorrect interpretations ourselves. 1.9 out of 5 stars is not 40%.
1 Like
#9

I’ve sen quite a few reviews that say “I would give it a 0 if I could.”
Regardless of the scale, it would be good if the rating were tied to a version. That way the new, improved version can have its own rating. Then the ratings could be averaged over the versions.

3 Likes
#10

The problem is that most people who rate plugins are the end users.
So very, very few have the skills to judge anything other than ease of use and appearance (and maybe customer support, but free vs. premium support also has a big impact on that…)

Really tired, so haven’t though this through, but what if you need a certain number of reputation points to rate other aspects of the plugin, like the resource efficiency, privacy and / or the source code? (Edit: Or these options become available after you have reviewed a certain number of plugins. This should also help reduce malicious reviews against contributors who choose not to provide free support.)

2 Likes
#11

Could the reviewer themselves have a classification as well? Like skill level, community rating or something (Maybe a drop down where you can select User, Plugin Dev, Theme Dev etc.). Though I fear it might discourage basic user reviewers if they feel their review has less gravitas than a dev reviewer.

I know there’s a risk of overcomplicating this as well and it needs to be clear to both reviewers and readers. On the other hand I think they’ll always be a spectrum of approaches anyway that one can’t control. Many users won’t directly read reviews and simply look at only the rating; others might be more open to finding that one negative review, checking if it pertains to the function they were going to utilise (if say a plugin) and then check that against the authors credibility.

1 Like
#12

@Fitzish
Hallos :grin:

That is what I would personally like too.
I’m going on the assumption that only “serious” users will review, say, more than 10 plugins and that a large number of reviews will generally correlate with skill.
The assumption would need to be tested though.

The one concern is, of course, as you have mentioned, increased complexity.

The other thing I am concerned about is people trying to game the system.
You don’t want to create the opportunity for a review “mafia”.

For this reason, I’d recommend filters rather than a weighting.
So, let’s say all reviews are displayed on the first tab and then you can choose reviews from reviewers with a high reputation score on another tab.
That way the reader can decide for themselves which reviews they would like to give more credence to.

1 Like
#13

I think a reviews system is inherently broken in this context. Reviewer classification may help, but, I’m not confident in that. I’ve seen time and time again on the WP forums where users will rate down a plugin because it “broke my site” only to waste the developer’s time to find out that it boiled down to 1 of 4 reasons, none of which were the fault of the plugin.

  1. the problem was in another plugin, but your plugin suffered the symptom (and so the blame), or
  2. the user couldn’t be troubled to RTFM, or
  3. PEBKAC, and they simply bail, or
  4. they only notice something (that was wrong with their site all along) after they install your plugin, so, it absolutely must have been your plugin.

If these issues can’t be solved, user reviews are worth the paper they’re printed on.

4 Likes
#14

@Code_Potent
Hey :grin:

Same could be said for users of e-commerce sites though.
If there is no review system at all, how would end users who don’t have the ability to go through the code themselves start to compare potential plugins?
Active installs / downloads is even less meaningful than reviews.
And some of the older plugins in the repos can be really good quality (and tend to be more light-weight), so looking for something that is “maintained” may not be the solution either.
I stand by it that reviews have some value when used in a responsible evaluation.
However, I do think that requiring a reason for a low rating might help a bit.

Edit: The ability to dispute a review might be nice… But that may be difficult to implement / “police”.

1 Like
#15

Completely this. There’s little I could say that any classification system might resolve. That may be more down to mods needing to be stricter with things like using reviews for support or testing (sorry mods) - but you can’t exactly vet reviews or certain people without coming off as an insular community.

For one we should arguably encourage people to seek support rather than the immediate “it broke my site” deal. I’m not sure how to discourage this outside out of strict mods (more work) moving topics to “Support” over “Reviews” or being far stricter on known incompatibilities between plugins and others or themes.

Maybe it would be more appropriate to Sticky certain experienced reviewers or maybe like Steam (for us gamers) - users can give reviews ratings with “useful” or “unhelpful” and by extension maybe only the most experienced and contributing members (more work for the mods) could mark something as “no longer applicable” after testing or something?

  • Edit - Note to self: use grammar and words…
#16

They have many things to go on… not least of which will be the feature list.

However, there’s other criteria, as well,…here’s just a few off the top of my head.

  1. Does the plugin have documentation? If so, is it good quality, or flung together?
  2. Is support offered to free users?
  3. Is the dev’s website “quality” (not just the look, but, the structure and content).
  4. Is support provided in English as a first language?
  5. Is there a support policy on the site?
  6. A privacy policy?
  7. Is the dev’s name recognizable in the community as an active member?
1 Like
#17

@Fitzish, good points. It’s also good to see that you’re acknowledging where more work will be required by actual persons; that’s something that most people glaze over when making suggestions. I do (kind of) like the idea of marking a bad review as no longer applicable, however, if the review is merely opinion (as opposed to an actual review), then, you run the risk of reviewers feeling “no longer applicable”.

Perhaps there could be some user education about what constitutes a “review”. I think the majority of the internet (based on comments sections everywhere) thinks that a rant is a review.

I’ve also seen users simply flustered and asking for way-above-and-beyond help and then leaving bad “reviews” because the developer didn’t have the time (or resources) to hand-hold someone who was essentially in over their head.

I don’t have any answers here; just expressing concerns.

1 Like
#18

Completely and likewise. I definitely like the idea of upping the standards of reviews in general to be more analytical in their approach but I guess you can’t stop the internet commenting. Nor do I like the idea of alienating those who have valid experiences but aren’t willing to write an essay.

Realistically we should ultimately decide on a simple system for Plugin repository launch and then refine it for people’s needs from there. It probably doesn’t bode well to over-complicate a system before it’s in place.

1 Like
#19

I think the mods for the directory should just take a tough stance from the beginning on reviews. If the developer wasn’t given a chance to fix or address a problem, then the “review” should be deleted. That simple.

If we start like that, we’ll be fine. If we tolerate nonsense at the start, we’ll struggle ever to get a meaningful review system.

5 Likes
#20

+1 for that.

Some elements of reviews can be automated though, which is why I am pro-filtering.
No system is going to be entirely perfect in isolation.
But let’s, just for example, say that user reviews can be filtered by reputation (not weighted by reputation, important distinction), where reputation is measured by whether someone has “regular” status on the forum, or appreciated / thank you badges / whatever seems appropriate (an existing mechanism).
That may help decrease the burden on moderators.
You won’t be baby-ing those who read reviews. You give them all the information in the best format that you can and let them make an informed decision.

Edit: As to the usefulness of reviews themselves, while I completely agree that user documentation is a useful metric of the quality of a particular plugin, I do not agree that all the items listed are indicators of quality (not all great developers have English as their first language).
My concern is that evaluating all those things you have mentioned can take a considerable amount of time. When you are considering a large number of plugins that offer similar features / functions, it simply may not be realistic, especially when you start getting to the phase where you have more “casual” users (non-devs / web professionals).

1 Like