Content-Transform-Team discussed this during our weekly tech forum today, with an eye toward not just making a test plan for 8.3 but also establishing a reusable test plan we can execute yearly for 8.5 and following.

Our primary test strategy for the php migration is upgrading parsoidtest1001, our round-trip testing server we use to qualify our weekly releases (T403772). For PHP 8.3 we upgraded our sole server; for PHP 8.5 (which we expect will be a more 'difficult' transition) we will probably need to provision a separate php-next server we can run in parallel to php-current to ensure that we don't introduce testing gaps on php-current during the transition pipeline, and that any bugs found with php-next don't block our weekly release process. Running our standard round-trip testing on php-next (which we have now done) should be sufficient to cover the following components:

• Language Converter
• Output Transform (ParserOutput processing)
• Parsoid integration
• ParserFunctions
• TemplateStyles
• wikimedia/parsoid
• wikimedia/zest-css

In addition, the following components are covered by routine Visual Diff testing, but we'd need to add the ability for our visual diff testing framework to send the X-Wikimedia-Debug: backend=k8s-mwdebug-next option (not done yet):

• Parser (legacy)
• ParserMigration

The following components could also be covered by Visual Diff testing, but in addition to sending the X-Wikimedia-Debug header would likely need a dedicated set of titles generated to ensure proper coverage:

• Kartographer + JsonConfig (use titles from the appropriate tracking categories)
• PagedTiffHandler / PdfHandler (not sure there’s a tracking category for these pages)
  • worth noting that visual diff testing will not cover/test uploads of these file types.

We do not have a test plan to cover the following components:

• ContactPage
• FeaturedFeeds
• Linter (this will be tested when the RefreshLinksJobs move to php-next, but that might be “late” or “too late”)
• MobileApp (we are going to explore transferring ownership of this component to Apps)
• WikimediaEditorTasks (we are going to explore transferring ownership of this component to Growth)

We may wish to add the ability to trigger Linter on parsoidtest1001 in read-only mode (so that metadata updates are not written/overwritten) independent of the RefreshLinksJobs; perhaps as something that can be triggered as part of round-trip testing on a specific set of titles. This would allow us to obtain coverage of Linter with php-next; it may also facilitate work on T310511: Metadata comparison testing between Parsoid and the legacy parser.

We may wish to construct a special title set and do all testing with our visual diff instructure, comparing output with X-Wikimedia-Debug set to php-current and with output with X-Wikimedia-Debug set to php-next. An appropriate title set would let us cover Kartographer, PagedTiff, etc as well. We wouldn't use our "weekly" visual diff server for this, but instead a special run of our "targeted" visual diff server. This same code could be reused in the future to compare php-current with php-next during the PHP 8.5 migration.