[wp-trac] [WordPress Trac] #58977: Consider adopting workflows for testing build packages

[WordPress Trac]

#58977: Consider adopting workflows for testing build packages
------------------------------+------------------------------
 Reporter:  costdev           |       Owner:  costdev
     Type:  enhancement       |      Status:  assigned
 Priority:  normal            |   Milestone:  Awaiting Review
Component:  Build/Test Tools  |     Version:
 Severity:  normal            |  Resolution:
 Keywords:  2nd-opinion       |     Focuses:
------------------------------+------------------------------

Comment (by costdev):

 === What do the workflows cover?
 **Installation - 32 tests**
 - PHP: 7.0 - 8.2
 - MySQL: 5.7 and 8.0
 - Single site and Multisite - Sub-directory

 **Upgrade - 620 tests**
 - WP: The latest minor release of the 4.1 - 6.x branches
 - PHP: 7.0 - 8.2
 - MySQL: 5.7 and 8.0
 - Single site and Multisite - Sub-directory
 - Caveat: WordPress 4.6 - 5.2 tests run on PHP 7.0 - 7.4 due to Fatal
 Errors because of uses of `__autoload()` and curly braces for array/string
 offsets.

 All currently run on `ubuntu-latest`.

 === What could they cover later?
 - I'd hope to see these extended to test `windows-latest` and `mac-
 latest`.
 - SQLite and other databases.
 - Earlier Beta / RC releases in a cycle.
 - End-to-End tests for manual installs.

 === How do they work?
 The workflows leverage WP-CLI to run installation and upgrade tests.

 To ensure that matrices don't exceed 256 jobs for now, the workflows are
 split into:
 - Installation Tests
 - WP 4.x - Single site
 - WP 4.x - Multisite - Sub-directory
 - WP 5.x - Single site
 - WP 5.x - Multisite - Sub-directory
 - WP 6.x - Single site
 - WP 6.x - Multisite - Sub-directory

 Each workflow accepts one input:
 - `new_version` - The version being installed/upgraded to. Any value that
 works with the `--version` parameter in WP-CLI will work for this,
 including `6.3`, `6.3-RC3` and `nightly`.

 === How long do they take?
 As my GitHub plan means I'm limited to 20 concurrent jobs, the workflows
 take longer than they would on WordPress' plan. I'd expect all 652 jobs to
 complete within 1-3 minutes if adopted by WordPress.

 === How could they be used?

 For example:
 - Workflows could be run on nightly builds, ensuring we catch issues in
 any of these scenarios within 24 hours of the commit that introduced them.
 - We might later look at integrating these in the release party process to
 catch issues within hours of being introduced.
   - This would require a Make post to discuss how best to maximise the
 benefits of testing by release party attendees, which could focus only on
 items addressed during the latest build so that we might catch any issues
 early. If adopted, we **must** make sure we don't increase coverage while
 decreasing community engagement.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/58977#comment:1>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform