[wp-hackers] WordPress Tests: Automated Hourly Test Runs

[Nikolay Bachiyski]

2008/3/18, Jacob Santos <wordpress at santosj.name>:
> I've finally gotten around my hosts server issues and was able to get
>  all of the unit tests to run correctly. It is a fun little site to see
>  how many tests passed, how many failed, and how many errors there were,
>  except not in that order.
>
>  Green = Tests that Passed
>  Yellow = Errors
>  Red = Tests that failed
>
>  The tests run hourly, but takes about 30 minutes to complete, so look
>  for it every hour and a half. There is one other feature I've yet to
>  implement and I have to get the Tests working for WordPress Mu, (Which
>  will probably involve manually including the tests classes that are
>  known to work, however probably should just skip those, so that I don't
>  have to update the system when WordPress Mu 1.5 comes out).
>
>  Let me know what you think. You can't get the results of what tests had
>  errors or failed. I might add that later, but I have another project
>  that clones this one that will allow you to get that information. I
>  think it would be useful, but I'm unsure how much more time that would
>  add to the process and I would also have to write something that imports
>  that to a database. I find that boring.
>
>  http://wordpress.svn.dragonu.net/WPTests - Subversion Code. You won't
>  have access to the run-tests folder (hopefully this doesn't mean I can't
>  access it either. Meh, Can always add Allow directive later.)
>
>  Requires PHP5, host that allows running of extremely long processes.
>  Also requires PDO_SQLITE extension be loaded.
>  You have to set up the cron jobs.
>  Ignore WordPress2.php and WordPress3.php, they were there for batch
>  running of the tests. I might still support that, since it could be
>  useful, but it requires that the index be updated again.
>
>  http://wptests.dragonu.net - Demo web site. You don't have access to the
>  actual WordPress site, which is on trunk and updated hourly also. If you
>  want to see the live demo, then it is located somewhere else.
>
>  This is only a proof-of-concept and is by no means representative of
>  WordPress stability or usability in any way. The current test suite does
>  not cover the full code coverage of the WordPress library. Failed tests
>  does not necessary mean that WordPress will not run. Some of the failed
>  tests are feature requests and does not accurately test the original
>  intent or design of the function.
>
>  I offer no warranty to the code in WPTests repository and I can not
>  offer any support. Use with caution. License is the same original
>  WordPress Tests license, since major code portions are derived from the
>  Automattic WordPress Tests Suite code base.
>
>  Any additions to the testcase needs to be sent to Alex Shiels of
>  Automattic. He is pretty good with adding test cases.
>
>  Thanks. Finally, you know I've had so many projects bouncing around in
>  my head, I was thinking I would go insane. It is nice to finally have
>  one partially finished.
>
>  Oh yeah, Nikolay, this is what I was talking about, when I said I would
>  like something like this on wordpress.org. I do believe that with as
>  many tests as the Test Suite has now, that it is worth having an
>  automated site.

I definitely agree we need such a system, however there is just one
thing missing in the current one -- the names of the failed test
cases.

The purpose of such a system is to free you of the 20-minute waiting
the test to run on your computer. Although I like the colours, the
information there isn't very useful for me -- I want to see, which
test cases were broken after the last commit and fix them. Now, in
order to see the failed tests I have to run them on my machine, which
defeats the purpose of the system.

Thanks for the good work and happy hacking,
Nikolay.