Per parent ticket, Bullseye should be retired at WMF by the end of September.
Creating this ticket to upgrade or replace the following VMs:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | brouberol | T401692 EPIC: Migrate Data Platform SRE-owned hosts to Bookworm or later | |||
| Resolved | bking | T433890 Migrate search-loader and apifeatureusage hosts to Bookworm or later |
Per IRC conversation with @EBernhardson , search-loader should "just work" as most of its config comes from conda environments. These hosts can go down at any time, although it might cause some mjolnir jobs in Airflow to stall.
As for the apifeatureusage hosts, they are running a very old version of Logstash. It seems the only newer version we have available as a Debian package on Bookworm or later is 7.16.3-1, which is provided by the opensearch27 repo component on Bookworm. So we'll need add that to apifeatureusage's Puppet config before we can upgrade successfully.
Update: the search-loader hosts had no problems on Trixie.
I notified the developers of the apifeatureusage MediaWiki extension that I will be reimaging the apifeatureusage hosts this week via email. I'll start on this later today.
Change #1324723 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] apifeatureusage: update logstash repo for bookworm compatibility
Change #1324723 merged by Bking:
[operations/puppet@production] apifeatureusage: update logstash repo for bookworm compatibility
Change #1324732 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] apifeatureusage: explicitly set java version
Change #1324732 merged by Bking:
[operations/puppet@production] apifeatureusage: Use correct java version
Change #1324754 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] apifeatureusage: explicitly define java package version
Change #1324754 merged by Bking:
[operations/puppet@production] apifeatureusage: explicitly define java package version
I've reimaged apifeatureusage2001, but the logstash service is in a crashloop:
Aug 12 17:37:20 apifeatureusage2001 logstash[12252]: Using LS_JAVA_HOME defined java: /usr/lib/jvm/java-17-openjdk-amd64. Aug 12 17:37:20 apifeatureusage2001 logstash[12252]: WARNING: Using LS_JAVA_HOME while Logstash distribution comes with a bundled JDK. Aug 12 17:37:34 apifeatureusage2001 logstash[12252]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties Aug 12 17:37:36 apifeatureusage2001 systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE Aug 12 17:37:36 apifeatureusage2001 systemd[1]: logstash.service: Failed with result 'exit-code'. Aug 12 17:37:36 apifeatureusage2001 systemd[1]: logstash.service: Consumed 30.148s CPU time. Aug 12 17:37:36 apifeatureusage2001 systemd[1]: logstash.service: Scheduled restart job, restart counter is at 109. Aug 12 17:37:36 apifeatureusage2001 systemd[1]: Stopped logstash.service - logstash. Aug 12 17:37:36 apifeatureusage2001 systemd[1]: logstash.service: Consumed 30.148s CPU time. Aug 12 17:37:37 apifeatureusage2001 systemd[1]: Started logstash.service - logstash. Aug 12 17:37:37 apifeatureusage2001 logstash[12334]: Using LS_JAVA_HOME defined java: /usr/lib/jvm/java-17-openjdk-amd64. Aug 12 17:37:37 apifeatureusage2001 logstash[12334]: WARNING: Using LS_JAVA_HOME while Logstash distribution comes with a bundled JDK. Aug 12 17:37:50 apifeatureusage2001 logstash[12334]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties Aug 12 17:37:53 apifeatureusage2001 systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE Aug 12 17:37:53 apifeatureusage2001 systemd[1]: logstash.service: Failed with result 'exit-code'. Aug 12 17:37:53 apifeatureusage2001 systemd[1]: logstash.service: Consumed 30.843s CPU time. Aug 12 17:37:53 apifeatureusage2001 systemd[1]: logstash.service: Scheduled restart job, restart counter is at 110. Aug 12 17:37:53 apifeatureusage2001 systemd[1]: Stopped logstash.service - logstash. Aug 12 17:37:53 apifeatureusage2001 systemd[1]: logstash.service: Consumed 30.843s CPU time. Aug 12 17:37:53 apifeatureusage2001 systemd[1]: Started logstash.service - logstash. Aug 12 17:37:53 apifeatureusage2001 logstash[12404]: Using LS_JAVA_HOME defined java: /usr/lib/jvm/java-17-openjdk-amd64. Aug 12 17:37:53 apifeatureusage2001 logstash[12404]: WARNING: Using LS_JAVA_HOME while Logstash distribution comes with a bundled JDK.
Investigating...
apifeatureusage2001 's logstash is complaining that it Could not connect to a compatible version of Elasticsearch when reaching out to the default search cluster. These errors don't appear on apifeatureusage1001, even though both apifeatureusage hosts are connecting to the same version of OpenSearch (2.x).
I think we may be hitting a logstash compatibility issue between newer OpenSearch 2.x releases and logstash that @colewhite has mentioned in the past.
On second thought, this might just be a matter of updating the pipeline config to say opensearch in place of elasticsearch. Checking now...
OK, I've one-offed /etc/logstash/conf.d/95-output-elasticsearch-apifeatureusage-search-svc-codfw-wmnet.conf on apifeatureusage2001 to use opensearch instead of elasticsearch in its config. Then I installed the logstash-plugins Debian package, which provides the opensearch plugin for logstash. Logstash appears to be running stably now.
There are a number of Kafka-related messages in the logs that don't appear on the Bullseye server (apifeatureusage1001), but they are all INFO level severity, so I'm going to call this working. I'll get a Puppet patch for the above changes.
Change #1324805 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] apifeatureusage: update config for OpenSearch
Change #1324805 merged by Bking:
[operations/puppet@production] apifeatureusage: update config for OpenSearch
I emailed the stakeholders (ApiFeatureUsage extension devs and Search Platform team) , asking them for help on determining whether or not apifeatureusage is healthy. The last index on the production search clusters are dated from June 30th:
$curl -s https://search.svc.codfw.wmnet:9243/_cat/indices | grep apifea green open apifeatureusage-2026.06.30 f8dUfYnQTKO65tpf7w472w 1 1 3754626 0 470.9mb 235.4mb curl -s https://search.svc.eqiad.wmnet:9243/_cat/indices | grep apifea green open apifeatureusage-2026.06.30 f8dUfYnQTKO65tpf7w472w 1 1 3754626 0 470.9mb 235.4mb
That seems like a problem, but I'm not sure what would've changed on that day to break anything. My best guess would the OpenSearch upgrades in T421757, although they did not completely finish until July 9th. Regardless, I don't think we need to wait to reimage apifeatureusage1001.eqiad.wmnet. I'll start on it tomorrow unless I hear back from the stakeholders in the email thread,
I've created T434906 to track work related to the apifeatureusage pipeline. As far as this ticket's AC, all hosts have been reimaged. Closing...