Plugin ratings + formula + filtering principles

Continuing the discussion from Disable Vulnerable Plugins:

1.

Owning a directory/repository (as well as any other ranging system) allows many interesting and wise solutions depending on ratings and ranging algorythms.

For example.

Plugins have default ratings. When a plugin is marked as vulnerable it

  • temporary gets a special rating value which is less than minimal allowed numeric and drops it down to the bottom of all positive categories lists (“popular”, thematical). Such items are marked with alert attributes (red backgrounds etc) in all lists, including search results, instlled plugin page etc.
  • gets into a special category “vulnerable”.

And then — “magic”. Each day in “vulunerable” category decreases plugin rating. So when a bug is fixed plugin appears a little bit lower in all categories. Penalty is calculated by a formula. You can inject any dependencies in it. For example at first days rating decreases slightly, but its loss gradually accelerates. High rating provides some “trust” that sloweres penalty increase.

I feel I talk too much here, so let me just give an example. Look what it can be like:

This is just a draft formula to show the principle. Feel free to play with coeffs, if you want. Here is the testing source and some notes.

// The speed of fixing bugs matters most, so it's the main formula parmeter.
// But this dependence should not be linear. If author fixes quickly,
// rating penalty should be minimal. And if a plugin remains for months,
// it should be pessimized to bottom. So, the dependency should be quadratic:
// $days^2 / something;
$days_plugin_remains_unfixed = 1;

// COEFFICIENTS FOR TUNING

// Start penalty guarantees that vulnerable plugin looses a bit of rating
// even if a fix was released quickly (or noticed too late).
// Days influence is quadratic, so first days make almost no practical effect.
// Start penalty is an offset that skips them and accelerates the formula.
$start_penalty = 10;

// "Good" plugins (with higher rating) should fall a bit slower, than "bad" ones.
// (A kind of automatic trust that stimulates support etc).
// Difference between current rating and maximum possible influences on that.
// Plugins tend to positive ratings, so comparison to maximum rating is more
// delicate than comparison to minimum. The wanted dependence is reversed
// (high rating -> slow fall). Substraction ($plugin_rating - $max_rating)
// needs some balance that protects from zeroes and negative values. So:
// ($coeff * (1 + $plugin_rating) - $max_rating)
// This coefficient obviously should always be > than $max_rating.
// It also regulates the relative falling speed for "bad" and "good" plugins.
// Increasing this makes "bad" plugins fall faster and "good" slower.
$rating_difference_scale = 5;

// Default quadratic functions results into too large numbers for our task,
// so we need to balance it by dividing on something. Using a static number
// is not good, as it is not proportional - too rough and hard for tuning.
// Let's make it more relative to the original quadratic function:
// sqrt( pow( $days, 2 ) + $slow_down ). This "slow_down" coeff
// is a final speed tuning. Increasing it means that rating loss
// becomes smoother and it takes more days to reach zero value.
// This should be > probable days count (1000+)
$slow_down = 3000;


// So, here is a draft formula:
/*
-------------------------------------------------

$time_dependency = pow( $days + $start_penalty, 2 );
$rating_dependency = $rating_difference_scale * (1 + $plugin_rating) - $max_rating;
$smoother = sqrt( pow($days, 2) + $slow_down );

$penalty =  $time_dependency  / ( $rating_dependency * $smoother );

-------------------------------------------------
*/


// ROUGH TESTING

// Different values of plugin ratings to test formula
$example_ratings = [0, 1, 2, 3, 4];

// Minimal and maximal values of rating
$min_rating = 0;
$max_rating = 4;

// Modeling changes for this number of days
$days_counter = 300;


$rows = "";

// Iterating days
for ( $days = 1; $days <= $days_counter; $days++ ) {

	$row = "";

	// Recalculating rating for each of example values
	for ( $k = 0; $k < count( $example_ratings ); $k++ ) {

		// Just some shortcuts
		$plugin_rating = $example_ratings[$k];
		$rds = $rating_difference_scale;

		// Here. Using formula to calculate penalty
		$penalty = pow( $days + $start_penalty, 2) / (
			($rds * ( 1 + $plugin_rating ) - $max_rating ) *
			sqrt( pow( $days, 2 ) + $slow_down )
		);

		// Applying penalty to rating
		$rating =  $plugin_rating - abs(round( $penalty, 3 ));

		// Just lazy check that new rating is in available range
		if ( $rating > $max_rating ) { $rating = $max_rating; }
		if ($rating < $min_rating) { $rating = $min_rating; }

		$row .= "<td>$rating</td>";
	}

	$rows .= "<tr><th>$days days: </th>$row</tr>";
}


// Outputting results

for ( $k = 0; $k < count( $example_ratings ); $k++ ) {
	$table_headings .= "<th>Rating = '{$example_ratings[$k]}'</th>";
}
$style = "<style>
	body {color: #333;}
	table {border-collapse: collapse;}
	td, th {
		padding: 5px 10px;
		border: 1px solid #eee;
		font-family: 'Roboto';
		font-size: 12px;
		line-height: 16px;
	}
	th {background: $f0f0f0;}
</style>";
$report = "$style<table><tr><th>Days unfixed</th>$table_headings<tr>$rows</table>";

echo $report;

I’m not a matematitian at all, so anyone can do it better) It looks complicated, but the whole point is about 2-3 strings of code. I’m pretty sure we have talented people to found proper regularities.

2

Also owning a resource (plugin directory) can be used to solve related problems. Each wise decision is multidirectional. Combining problems may lead to interesting results too.

For example. Someone (sorry, I’ve lost the context of that thread) offered to make a category of plugins reviewed by “trusted users”. And there was a question how to manage all that (who is “trusted”).

Well… We also have a petitions mechanism. And probably it needs some push to become more active. Why not cross those lacks to profit? If there is a “privileged” category of plugins, let them go through petitions system (e.g. author can apply plugins for category “Recommended” via petitions and if it has %number% voices Cometee discusses it). Benefits:

  • Plugin authors (devs!) visit petitions section and probably vote for others.
  • Plugin users visit petitions section and probably vote for others.
  • Privileged plugin category is open and transprent.
  • No additional resourses for that. Really. No cost.

Even removing plugins form “Recommended” can be done through petitions (just with lower %number% perhups, needs tests).

Resume

The main thought of this post:
Many problems can be solved by simply reusing things that already exist here. No need to reinvent them. The only required resource is creativity. And CP community have tons of it, I think)

4 Likes

Completely agree with this, once you start hitting 30 days unfixed the penalty should be way more dramatic (using 30 as an example, I think even that is way too long to be left unfixed).

4 Likes