The Content-Transform-Team uses parsoidtest1001 for weekly pre-train parsoid testing: we tag parsoid, then run round-trip tests on parsoid1001 against a read-only copy of the production database before releasing the tagged parsoid to mediawiki-vendor to ride the train.
Parsoidtest1001 is "almost entirely" a production mediawiki server.
As part of the PHP8.1 transition process, parsoidtest1001 should probably be switched to PHP 8.1 with the first group of test servers.
Thanks for writing this up, @cscott.
So, one alternative to migrating parsoidtest1001 to PHP 8.1 would be to instead provide a k8s-based alternative that fulfills the same use case, and is 8.1-based from day 1.
One notable benefit is that it would enable testing against either 7.4 or 8.1 at the user's choice for some period, rather than having a flag-day where the one-and-only parsoidtest1001 host switches to 8.1.
Functionally, this could be a CLI tool that allows one to start a "temporary" k8s pod running parsoid code checked out at a specific commit ref. For example, invoking the tool to create the pod would replace the ssh-fetch-checkout step in start-rt-test.sh (with functionality to address the pod also wired into the RT client unit). It should also be able to support manual / one-off tests similar to what's described here.
@cscott - Does that sound like something that would support your use cases?
Sketch: If we were to go this route, I think it should be fairly straightforward to model a "v0.1" of this on how mwscript-k8s works.
In addition to obvious differences like enabling httpd / php-fpm and running in a distinct namespace, the core difference would be an init-container that could clone / checkout the parsoid code at the desired commit ref into an emptyDir volume mounted at /srv/parsoid-testing (RO) in the mediawiki app container (noting that we'd also need to slightly update the mediawiki-config logic that loads the extension at the alternative path to properly detect this environment).
One clear benefit of continuing to use the "load the extension from an alternative path" approach is that it entirely side-steps container image builds.
One key assumption here is that we indeed want these pods to be temporary. If we instead want a single durable deployment that's more directly analogous to parsoidtest1001, then that's actually a bit easier - though we do need a way to durably specify the target commit ref for use by the init-container (e.g., in a config map).
I'm also happy to consider alternatives, particularly if we're close to finalizing a general solution for running experimental code (T324003). Also, I'm not sure if @akosiaris has already given this some thought given recent work on parsoidtest1001.
The high-level desired features are:
We source a file in the parsoid repo from mediawiki-config which allows us to customize the mediawiki configuration, although we don't use this often.
As optional features, being able to control the version of mediawiki-vendor and/or mediawiki-core is helpful.
Thank you very much, @cscott. I believe a solution like the one outlined in T380485#10355701 should satisfy the feature requirements in T380485#10384420 at least as well as the current setup on parsoidtest1001, and in some cases possibly better (e.g., higher fidelity to the current state of k8s-based production).
As far as I can tell, it also should not get in the way of the database user changes proposed in T298046, though I can't immediately speak to the other aspect of that task (ability to run with vendor from HEAD).
I think we're on board with that plan. Ideally we'd like to run with the existing parsoidtest1002 and the new PHP 8.x solution in parallel for as long as possible, so that we don't drop test coverage of the PHP 7.4 environment until we're ready for all of production to move to PHP 8.x, but we're ready to start testing on PHP 8.x as far as we know.
Thank you, @cscott!
Two additional questions while I'm sketching out some of the implementation details:
Following up here after implementation discussion has largely shifted to T386246: Migrate parsoidtest functionality to kubernetes .
In short, there are a couple of options we have in mind for a k8s-based alternative, one of which we believe we could land fairly quickly. However, we're not yet certain which option is preferable in the long term, particularly in order to minimize drift vs. the "similarly shaped" problem of an mwdebug* replacement.
Thus, in the very near term, we may instead opt for reimaging parsoidtest1001 to Debian bullseye + PHP 8.1, given the successful pilot thereof in T391452: Migrate mwdebug* hosts to PHP8.1.
This should be relatively straightforward, and can be coordinated to avoid interfering with the Parsoid release process. Since doing so will remove any lingering files persisted on the host itself (e.g., in user home directories), I'll provide a heads up before we go that route.
parsoidtest1001 is already running bullseye, this was done when the old hardware (scandium) was refreshed with the current server. As such, this only needs the following Hiera setting:
profile::mediawiki::php::php_versions: ['8.1']
Change #1136413 had a related patch set uploaded (by Scott French; author: Scott French):
[operations/puppet@production] hieradata: switch parsoidtest1001 to PHP 8.1
Thanks, @MoritzMuehlenhoff - I actually hadn't even checked yet and just assumed a reimage would be necessary. That's great news, and indeed should make this procedure much less disruptive.
As a reminder to myself for later: there are a couple of spots in mediawiki/services/parsoid with systemctl restart php7.4-fpm.service baked in, which will need updated.
Change #1136413 merged by Scott French:
[operations/puppet@production] hieradata: switch parsoidtest1001 to PHP 8.1
Alright, parsoidtest1001 is now running PHP 8.1. I've verified that:
There are two spots that will need updated in various scripts in mediawiki/services/parsoid:
In both cases, the systemd unit simply needs switched to php8.1-fpm.service.
[0] https://www.mediawiki.org/wiki/Parsoid/Round-trip_testing#Running_Parsoid_tools_on_parsoidtest1001
[1] https://www.mediawiki.org/wiki/Parsoid/Round-trip_testing#Proxying_requests_to_parsoidtest1001
Change #1136758 had a related patch set uploaded (by Scott French; author: Scott French):
[mediawiki/services/parsoid@master] Update systemd unit names in testing scripts for PHP 8.1
Change #1136758 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Update systemd unit names in testing scripts for PHP 8.1
Change #1137062 had a related patch set uploaded (by Scott French; author: Scott French):
[operations/puppet@production] P:parsoid::mediawiki: use installed PHP versions for auto-restart
Change #1137062 merged by Scott French:
[operations/puppet@production] P:parsoid::mediawiki: use installed PHP versions for auto-restart
I am cautiously optimistic that we've shaken out the remaining issues discovered after the migration.
One was the fact that profile::parsoid::mediawiki created auto-restart resources for a fixed PHP version per debian release. This was changed to follow profile::mediawiki::php::php_versions (and absented_php_versions) in https://gerrit.wikimedia.org/r/1137062.
The other was an incidental discovery by the Content Transform team that mediawiki logs no longer appeared in logstash for a recent post-migration RT testing run. This was due to a change in rsyslog configuration last week that contained a typo, which seems to have caused later configuration fragments to effectively be "skipped" (including forwarding to kafka). This is now fixed in https://gerrit.wikimedia.org/r/1137315.
In any case, as long as nothing else comes up in the upcoming RT testing run today, I think we can probably close this out.
Since the logging issue appears to be resolved (e.g., the ongoing RT testing run is producing logs in logstash as expected), and nothing else of note appears to have arisen during yesterday's run that also surfaced the logging issue, I'm going to optimistically mark this as resolved.
If any lingering issues related to the migration arise, please feel free to reopen. I will be out next week, but other folks in serviceops should be able to assist if urgent.
For future migrations: we hope to make use of the work that takes shape in T386246: Migrate parsoidtest functionality to kubernetes , which should allow us to migrate the parsoid-testing use case early on, using the same procedure as we use for mw-debug.
Change #1139521 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.21.0-a27
Change #1139521 merged by jenkins-bot:
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.21.0-a27