[wp-hackers] User Feedback and Testing

[David House]

On 01/01/06, Owen Winkler <ringmaster at midnightcircus.com> wrote:
> Testing is the fundamental reason why I think there is so much churning
> on these topics, and it's not just the kind of bug/patch testing that
> we're accustomed to hearing about on these lists.

Yes. I've been thinking about this for the last few days and was at
the starting of writing an email similar to this, but you've put it a
lot better than I would have done :)

> I've heard more than once that this is the best-tested version of
> WordPress yet.  Maybe more people have downloaded and tried pre-release
> versions of WordPress than before, and maybe there has been a lot of
> feedback from WP.com (none of which seems to be made public like it is
> in Trac, now that I think about it).  Does that imply that user testing
> was done?  Have we really asked uninvolved users what they think of the
> software?  Whether they think something could be improved?  Or are we
> basing what we think works on how well they receive features that they
> have no influence over?
>
> People say that WordPress was released too quickly.  I don't think this
> is phrased correctly.  I think that an inadequate testing duration was
> allotted for those last minute patches that were applied.  Nevermind if
> every single patch works flawlessly - What kind of policy is it to
> release as large a project as this without letting the code burn-in on
> testers machines for a few days?

I'd just like to point out that we haven't found any security holes
yet. That will save the embaressment we had at 1.5 with the stupid
amounts of incremental releases. I recall Matt saying that he was
getting some proper Security People to look at the software, I guess
it paid off.

> We could keep pushing back the release date forever because the bugs
> would never end.  We would never release a new version.  Yeah, I get the
> idea.  Likewise, we could push the release back two weeks and allow a
> burn-in of frozen code to make sure those last-minute patches are
> working as expected.  Or, we could push the release back a month and
> have more complete documentation.  Clearly, a four-day release delay was
> valuable to the support folks.  In a delay's worst case, if none of the
> remaining bugs are addressed during that period, you have two things you
> didn't have before:  1) A more thorough idea of what is wrong with the
> release for use by support.  2) A firm, organized release date.
>
> Real products have development cycles, where development is broken into
> phases that allow for discussion of new feature requirements,
> development of planned new features, testing of those features, and
> providing documentation and support for those features upon release.
> These cycles are not necessarily periodic (as in using a regular release
> date), but if are planned properly can be assigned a best-guess timeline
> that can be used for planning useful things like press releases, site
> design updates, and support ramp-up.

Podz mentioned a six-month development cycle, and I think this is
reasonable. Say, a month after a release bugfixing and releasing
incremental, bugfix-releases. Five months worth of development, then a
month's worth of testing. Of course, we could chop and change this as
individual releases require, but it doesn't hurt anybody working to a
loose timetable. For the next major release, we need:

1. An announced planned release date, and we need to be told about it
about a month in advance, not four days.
2. Planned, strict, feature freeze, code freeze and translation-string
freeze dates.
3. Lots of time spent testing EVERYTHING. Twice.

> With a project this large affecting so many people, it seems about time
> that the development cycle of WordPress becomes a bit more openly
> scripted.  Otherwise we can continue as is: Guessing at new features,
> wondering if the new features are addressing the needs of users,
> speculating on release dates at the expense of complete documentation
> and support, and watching a growing unease among the project's most
> ardent supporters.

+1. With regards to testing, I think there are two main problems with
the current WordPress test system.

1. We don't have enough testers. We have a few dozen people that are
loyal users of SVN, and they don't cover enough of the system. Stupid,
obvious bugs like author/ URLs not working and basic bugs in XMLRPC
made it through to the 2.0 release, simply because none of our testers
use this side of WP. If you are someone that has a multi-user blog,
uses XMLRPC frequently, runs PHP/CGI, IIS or anything else quirky, or
just in general would use some part of WordPress other than the 'core'
of writing posts, we need you as a tester. We need as many people as
we can testing as many aspects as we can.

2. Old bugs come up again. Due to WP's history, looking at the code
sometimes looks like it's a hangover from the old b2 days. Lets face
it, WP's code isn't as nice as, say, Wikipedia or Gallery. It's
improving, but it's still very coupled. New fixes tend to rebreak old
bugs. That's why I think we need some kind of regression testing. I'm
not sure how this would be embodied, perhaps through automated testing
(although that would require a BIG shift in the WP development
paradigm -- there's no point doing automated testing if you don't do
test-driven development), perhaps through ruthless human testing.
Whatever. We need to make sure that once a bug is fixed, it stays
fixed.

--
-David House, dmhouse at gmail.com, http://xmouse.ithium.net