[wp-trac] [WordPress Trac] #64103: Run WP-CLI test suite in core repo

[WordPress Trac]

#64103: Run WP-CLI test suite in core repo
------------------------------+-----------------------------
 Reporter:  swissspidy        |      Owner:  (none)
     Type:  defect (bug)      |     Status:  new
 Priority:  normal            |  Milestone:  Awaiting Review
Component:  Build/Test Tools  |    Version:
 Severity:  normal            |   Keywords:
  Focuses:                    |
------------------------------+-----------------------------
 Occasionally, a WordPress core change (small or big) can cause test
 failures over at WP-CLI, which is another official WordPress project.

 The most recent example is #53323, but that's just one of many.

 Whenever this happens, WP-CLI maintainers have to analyze test failures to
 see its origin and assess whether it's a regression in core that needs to
 be fixed, or whether WP-CLI tests simply need to be hardened.

 To improve the feedback loop, we could add another CI workflow that runs
 Behat tests for all WP-CLI packages (as in https://github.com/wp-cli
 /automated-tests) for the current WordPress build.

 It could work very similar to the `pull-request-comments.yml` workflow
 that runs only after the current commit (either in trunk or a PR) has a
 built ZIP file. Then, that ZIP file is used to run tests. Running them
 only for a single PHP version would already help a lot, so I don't think
 it would need to involve a complex matrix or triggering reusable
 workflows.

 As mentioned, sometimes a test failure means WP-CLI tests simply need some
 hardening, so they shouldn't necessarily block merging. Perhaps we could
 find a good way to communicate that. Suggestions welcome.

 There might also be some alternatives, like posting WP-CLI failures in the
 #core Slack channel or a combined testing dashboard that shows test status
 for core, Gutenberg, and WP-CLI.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/64103>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform