[wp-trac] [WordPress Trac] #59900: Measure performance with a persistent object cache in performance tests

[WordPress Trac]

#59900: Measure performance with a persistent object cache in performance tests
-------------------------------------------------+-------------------------
 Reporter:  flixos90                             |       Owner:  flixos90
     Type:  enhancement                          |      Status:  assigned
 Priority:  normal                               |   Milestone:  6.5
Component:  Build/Test Tools                     |     Version:
 Severity:  normal                               |  Resolution:
 Keywords:  2nd-opinion has-patch has-unit-      |     Focuses:
  tests                                          |  performance
-------------------------------------------------+-------------------------

Comment (by swissspidy):

 I discussed this a bit with @flixos90 and tried to further wrap my head
 around this.

 Introducing a matrix will definitely unlock additional improvements to the
 test suite, though there are some things to consider. Some loose thoughts:

 * This is a great opportunity to remove a lot of duplicate code like e.g.
 `*-l10n` tests
 * Even the classic & block theme tests could be deduplicated,
 theoretically allowing us to even test all the default themes in the
 future
 * Theoretically even the admin & home tests could be deduplicated, as they
 are very similar.
 * Theoretically allows us to test more hardware configurations in the
 future (i.e. different PHP & MySQL versions) if needed
 * An alternative is to do this matrix in JS instead of the GHA
 ([https://github.com/swissspidy/wp-i18n-
 benchmarks/blob/de41fdf159617d25ca9059e1edb10574c4afe028/tests/performance/specs
 /classic-theme.spec.ts example]).
   * Con: can't really change server configuration (e.g. PHP version).
   * Pro all tests happen within the same GHA runner
   * Pro: you can easily run the whole matrix locally.
 * We should use this opportunity to remove the hardcoded test suite names
 in the CLI scripts
 * Each matrix job should still do its own comparison with target commit
 and baseline version. This is to ensure the runs happen within the same
 environment.
 * The exact setup depends on whether we want/need to print and log results
 after the whole matrix has run, or if that can be done within a single
 matrix job.
   * For the former we need to upload an artifact in each job and then in
 another job merge them together. Not really difficult, but if we don't
 _have_ to do it, it will make things less complex.
   * Need to check what the current dashboard requires. For back compat we
 need to ensure we keep sending the same keys there that it expects.

 Here's a simple POC I whipped up to see what's possible:
 https://github.com/WordPress/wordpress-develop/pull/5679
 Workflow summary: https://github.com/WordPress/wordpress-
 develop/actions/runs/6907241045?pr=5679

 As you can see, printing the results in each job leads to a lot of noise
 due to separated summaries, so printing only 1 summary after the matrix
 concluded is probably favorable.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/59900#comment:6>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform