[wp-hackers] Code reviews for plugins?

[Mike Schinkel]

On Aug 23, 2010, at 4:46 AM, Harry Metcalfe wrote:
> On 23/08/10 04:49, Mark E wrote:
> > I'm seeing a big issue centered around delivering a false sense of
> > security to numerous millions of innocent people.
> 
> I agree. I like the idea about having objective criteria, and if the results of reviews were phrased appropriately -- ie, accurately -- that would be a nice thing to have.
> 
> But just to say "The community has reviewed this plugin and it looks A-OK to us" is a really bad idea. For a start, I'm not sure you can really do that in a generic way: to make that statement for any particular user, you'd need to know what other plugins they were running, and what their theme does. But ordinary, non-tecchie WP users will just interpret it as a badge of quality and may therefore be misled.
> 
> But more importantly, just to say a plugin has been "reviewed" without knowing what the reviewer was looking for is meaningless. They could have been looking for fluffy bunnies. It essentially ends up being a review to look for the things the reviewer thinks are important. Which is perhaps slightly better than nothing, but not much.

I don't think anyone was remotely suggesting a rubber stamp of "The community has reviewed this plugin and it looks A-OK to us."  For a person to have previously suggested that would imply they had given zero consideration to what a process would actually need to be to ultimately provide guidance to users on plugins.

> I think we should come up with a list of the top 25 mistakes people make in plugins, review to find those, perhaps also highlight whatever else looks problematic and tell the author, and then say to users "This plugin has passed a review which checks for some common WordPress plugin problems" or somesuch...

That is what was being discussed.

> PS: if this plan means I never have to spend hours fixing all the notices in someone else's plugin, that would be nice.

That would be one of the goals!

On Aug 23, 2010, at 8:26 AM, Paul wrote:
> When I read the original comments on this thread last week I though the purpose was to put together a peer review process where plugin developer can offer advice to other plugin developers as review of their code. I originally took this advice as something along the lines of "Well, your code works but this set of routines you've written would be better served if you just hooked into this WP filter and tried this technique (see xxx plugin as an example)". The goal I thought was to share WP plugin development knowledge and make any mediocre plugin developer a better developer. Like myself.   

I think both were part of the original discussion and different people took in different directions.  

Frankly I see a value in both; i.e. I would love a code review so I could throw up a snippet of code and have others tell me how I can improve it, and then I think there's also value in being able to give guidance to users of plugins.

> In reading this last set of messages it almost sounds like you guys want to put together a self-appointed committee where you rate plugins based on some yet to be determined checklist.

What was proposed most recently was not a self-appointed committee but a community process where anyone could be involved and the mechanism would by its nature need to surface the high quality contributions and segment from the low quality ones. That mechanism does not exist yet but its design can be discussed and could have a reputation similar to StackExchange.

> Why not just publish the check list somewhere some plugin developers can check their own plugin?

Well, you have to first develop that list and ideally get community buy-in on what should be included and what shouldn't. Knowing we can debate it all day on wp-hackers and at the end of the day still have nothing I proactively creating this as a place to collect a list and several people are already doing so:

http://wordpress.stackexchange.com/questions/715/objective-best-practices-for-plugin-development/735#735

Hope you can add your two cents there as well.

> I agree 100% with someone who wrote about people counting against you for not using the proper camel-case or not putting your JS into a sub-folder. This is crap. No one who uses your plugin will case. What should be review is the code written and its intended functionality. 

This is a red herring.  I don't think anyone is suggesting code formatting or code organization be a part of code review or plugin review.

On Aug 23, 2010, at 10:34 AM, Lynne Pope wrote:
> I think the first thing that needs to happen is for people to agree on what
> they want a process like this to do. Paul & Christopher have really good
> points and if the goal is to improve the quality of plugins then a review
> should be (IMO) limited to looking at security & functionality. If people
> get into arguments over camelcase, whether tabs or spaces should be used for
> indenting, etc etc then little will be achieved except bitterness.

Can we all just please acknowledge that arguments over camelcasing etc. were never on the table, and move on with something productive? 

> If, however, the intention is to review plugins with the goal of providing
> some surety for end-users then this is a completely different process. This
> thread shows people are thinking of both of these but I'm not so sure that
> both can be achieved.

As the ancient Chinese proverb says “Those that say it can’t be done should get out of the way of those doing it.”  :-)

> This is potentially a minefield. Users don't give a damn about quality -
> they just want to know a plugin works, does what it says it does, is secure,
> and doesn't conflict with other plugins. It doesn't matter to users if the
> plugin is poorly written.

Quality in a plugin *IS* that it works, does what it says it does, is secure and doesn't conflict with other plugins. 

> The moment the words, "assurances" or "certification" come in, someone takes
> responsibility. THAT is absolutely the last thing the community should do.
> No matter how good a vetting process is, guarantees cannot be given. For a
> start, guarantees are specifically excluded from the GPL. NO WARRANTY


Nobody but you mentioned the words  "assurance" or "certification", only "review."   

This is starting to smell of bikeshedding. 

On Aug 23, 2010, at 10:39 AM, Eric Mann wrote:
> I think the two things we're proposing here are:
>  
> 1) To create a checklist of best practices for plug-in development that
> developers can use as a guiding tool while building their plug-ins.  ...
>  
> 2) That we organize the community around looking at and verifying that plug-ins
> claiming to meet certain standards actually meet certain standards.  ...
>  
> No one's suggesting that we relocate files or rewrite code...  

Exactly.

On Aug 23, 2010, at 12:01 PM, WPTavern wrote:
> I think one of the best ideas I've seen proposed so far regarding plugin reviews is the idea of setting up an Amazon like review system for plugins. Each plugin on the repository gets their own review section aka comments. These comments could then be rated by the community with the most voted comments displayed at the top. This system is not subject to liability and instead of a team of people working behind the scenes through an email list or what have you, the entire community becomes the team of reviewers. The reviews/comments form also provides an opportunity for the end user to state whether or not the plugin works or is broken which would solve the other problem of plugin devs not getting feedback when someone marks their plugin as broken.

Exactly (and kudos for writing *that much* on your iPhone! :)


On Aug 23, 2010, at 12:05 PM, Chip Bennett wrote:
> This idea *has* to be an improvement over the current "Works/Broken"
> anonymous voting system. Though, I believe Otto is working on some
> improvements with the "Works/Broken" system, so maybe it's best to see
> what's in the pipeline for Plugin repository improvements as part of 3.org?

Finding a list of best practices is orthogonal to improving said system.  And getting the best practices is the first step so no need to wait.

On Aug 23, 2010, at 12:09 PM, Paul wrote:
> How is that different than the already in place plugin rating system hosted on WP.org? There users (not just developers) can rate your plugin and advise other is it works with a particular version of WP. Also it links to user support mentions from the WP.org forum. 

Because the current system isn't really helpful.

> Do we really need to start splintering the plugin reviews into another micro-communitee controlled site? 

Ideally the results would be on WordPress.org.  But nothing will happen to improve the situation if code is never written.


-Mike