Page MenuHomePhabricator
Create Task
Maniphest T278056

Upgrade Druid to version 26.0.0
Closed, ResolvedPublic
Actions

Description

Update Jan 2026

We have settled on version 28.0.1 as our target version, for the time being.

This is because this is the last version that officially supports Hadoop 2.

Update: version 27.0.0 is actually the last version that can be built with Hadoop 2 support.
The required option was removed on 28.0.0

Update: version 27.0.0 exhibited some issues with Hadoop 2, even with the custom build profile. Therfore, we have now chosen version 26.0.0 as the target version, for now.

Original description follows.

https://github.com/apache/druid/issues/10462#20-result-caching is a bug fixed in 0.20 that we are seeing in our version, 0.19, namely the whole-query cache on brokers not populated. I was puzzled that Brokers were only registering cache miss, but it seems due to a problem in the code rather in the settings.

Having the Broker query cache working would probably benefit our latency, giving us the possibility to think about using memcached as common broker caching layer (even one host per host would be fine.

Details

Other Assignee
brouberol
Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Jan 12 2026, 1:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-test-druid1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 12 2026, 1:39 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-test-druid1001.eqiad.wmnet with OS bookworm executed with errors:

  • an-test-druid1001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-test-druid1001.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
• JMoore-WMF subscribed.Jan 12 2026, 2:39 PM
ops-monitoring-bot added a comment.Jan 12 2026, 3:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-test-druid1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 12 2026, 3:16 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-test-druid1001.eqiad.wmnet with OS bookworm executed with errors:

  • an-test-druid1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-test-druid1001.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 12 2026, 3:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-test-druid1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 12 2026, 3:19 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-test-druid1001.eqiad.wmnet with OS bookworm executed with errors:

  • an-test-druid1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-test-druid1001.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 12 2026, 4:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-test-druid1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 12 2026, 4:16 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-test-druid1001.eqiad.wmnet with OS bookworm executed with errors:

  • an-test-druid1001 (FAIL)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-test-druid1001.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 13 2026, 7:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-druid1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 8:31 AM

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-druid1001.eqiad.wmnet with OS bookworm executed with errors:

  • an-test-druid1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601130753_brouberol_1200081_an-test-druid1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-test-druid1001.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
BTullis added a comment.EditedJan 13 2026, 11:09 AM

We have at least two problems with the puppet run for a bookworm host running Druid 27.0.0

  • We need the prometheus-druid-exporter to be made available for bookworm.
  • The zookeeper quorum that is colocated with Druid has been using OpenJDK-11, which is not available for bookworm.

For the first issue, we can simply copy the prometheus exporter from bullseye-wikimedia to bookworm-wikimedia, as it's plain python.

For the second, we should probably try to standardize on OpenJDK 17, which is what the upstreram project recommends.

Druid fully supports Java 8u92+, Java 11, and Java 17. The project team recommends Java 17.

MoritzMuehlenhoff added a comment.Jan 13 2026, 11:22 AM
In T278056#11515866, @BTullis wrote:

For the second, we should probably try to standardize on OpenJDK 17, which is what the upstreram project recommends.

FWIW, Java 17 is also available for Bullseye, so this allows setting up a J17 cluster across Debian 11/12

BTullis added a comment.Jan 13 2026, 11:29 AM
In T278056#11515957, @MoritzMuehlenhoff wrote:

FWIW, Java 17 is also available for Bullseye, so this allows setting up a J17 cluster across Debian 11/12

Nice, thanks. That could be very useful.

I copied the package like this:

btullis@apt1002:~/bookworm$ sudo -i reprepro ls prometheus-druid-exporter
prometheus-druid-exporter | 0.8-1 |   buster-wikimedia | amd64, i386, source
prometheus-druid-exporter | 0.8-2 | bullseye-wikimedia | amd64, i386, source
btullis@apt1002:~/bookworm$ sudo -i reprepro copy bookworm-wikimedia bullseye-wikimedia prometheus-druid-exporter
Exporting indices...
btullis@apt1002:~/bookworm$ sudo -i reprepro ls prometheus-druid-exporter
prometheus-druid-exporter | 0.8-1 |   buster-wikimedia | amd64, i386, source
prometheus-druid-exporter | 0.8-2 | bullseye-wikimedia | amd64, i386, source
prometheus-druid-exporter | 0.8-2 | bookworm-wikimedia | amd64, i386, source

Now I'll work on a puppet patch to select only Java 17 when running on bookworm, for both druid and it's colocated zookeeper cluster.

gerritbot added a comment.Jan 13 2026, 12:06 PM

Change #1226219 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the Java version and other settings for the druid test cluster

https://gerrit.wikimedia.org/r/1226219

gerritbot added a project: Patch-For-Review.Jan 13 2026, 12:06 PM
gerritbot added a comment.Jan 13 2026, 12:53 PM

Change #1226219 merged by Btullis:

[operations/puppet@production] Update the Java version and other settings for the druid test cluster

https://gerrit.wikimedia.org/r/1226219

gerritbot added a comment.Jan 13 2026, 1:51 PM

Change #1226251 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Only install the JRE instead of the JDK on druid-test hosts.

https://gerrit.wikimedia.org/r/1226251

gerritbot added a comment.Jan 13 2026, 2:57 PM

Change #1226270 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] druid_exporter: duplicate config from druid 0.19.0

https://gerrit.wikimedia.org/r/1226270

brouberol updated Other Assignee, added: brouberol.Jan 13 2026, 3:03 PM
gerritbot added a comment.Jan 13 2026, 3:42 PM

Change #1226251 merged by Btullis:

[operations/puppet@production] Only install the JRE instead of the JDK on druid-test hosts.

https://gerrit.wikimedia.org/r/1226251

gerritbot added a comment.Jan 13 2026, 4:47 PM

Change #1226270 merged by Brouberol:

[operations/puppet@production] druid_exporter: duplicate config from druid 0.19.0

https://gerrit.wikimedia.org/r/1226270

gerritbot added a comment.Jan 13 2026, 5:20 PM

Change #1226302 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Remove GC logging options that are incpatible with Java 17 on druid-test

https://gerrit.wikimedia.org/r/1226302

gerritbot added a comment.Jan 13 2026, 6:06 PM

Change #1226302 merged by Btullis:

[operations/puppet@production] Remove GC logging options that are incompatible with Java 17 on druid-test

https://gerrit.wikimedia.org/r/1226302

gerritbot added a comment.Jan 14 2026, 8:56 AM

Change #1226766 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] druid_exporter: Fixup metric definition

https://gerrit.wikimedia.org/r/1226766

gerritbot added a comment.Jan 14 2026, 12:55 PM

Change #1226844 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] druid: inject flags allowing druid to access protected classes in java > 8

https://gerrit.wikimedia.org/r/1226844

gerritbot added a comment.Jan 14 2026, 2:03 PM

Change #1226844 merged by Brouberol:

[operations/puppet@production] druid: inject flags allowing druid to access protected classes in java > 8

https://gerrit.wikimedia.org/r/1226844

gerritbot added a comment.Jan 16 2026, 12:10 PM

Change #1227754 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Tweak druid configuration to enable druid 27.0 to run on jvm8

https://gerrit.wikimedia.org/r/1227754

gerritbot added a comment.Jan 16 2026, 12:35 PM

Change #1227754 merged by Brouberol:

[operations/puppet@production] Tweak druid configuration to enable druid 27.0 to run on jvm8

https://gerrit.wikimedia.org/r/1227754

gerritbot added a comment.Jan 16 2026, 12:45 PM

Change #1227771 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] an-test-druid: run zookeeper on the default system java version

https://gerrit.wikimedia.org/r/1227771

gerritbot added a comment.Jan 16 2026, 1:33 PM

Change #1227786 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] an-test-druid: disable noisy GC stat logging

https://gerrit.wikimedia.org/r/1227786

gerritbot added a comment.Jan 19 2026, 1:55 PM

Change #1228498 had a related patch set uploaded (by Joal; author: Joal):

[operations/puppet@production] Remove test druid cluster noisy jvm GC params

https://gerrit.wikimedia.org/r/1228498

gerritbot added a comment.Jan 19 2026, 2:17 PM

Change #1228498 merged by Btullis:

[operations/puppet@production] Remove test druid cluster noisy jvm GC params

https://gerrit.wikimedia.org/r/1228498

BTullis added a subscriber: JAllemandou.Jan 20 2026, 12:26 PM

We had to revert to version 26.0.0 because the Hadoop 2 support in 27.0.0 seems inconsistent, even when we build it with the -P dist-hadoop2 flag.

However, we are now happy with this version on an-test-druid1001 and @JAllemandou has successfully tested both batch ingestion and real-time ingestion jobs.

Now we have to think about how and when to carry out the upgrade.

Druid does support rolling upgrades, but this will be a bit complicated for us, because of the need to upgrade the base O/S at the same time.

For us to be able to do a rolling upgrade, we would need to do the following:

  • Build a druid version 0.19.0 package for bookworm
  • Upgrade one host at a time to bookworm, running Druid 0.19.0
  • Run a zookeeper quorum with mixed versions 3.4 and 3.8 of zookeeper
  • When all hosts have been upgraded to bookworm and the zookeeper quorum is stable with version 3.8
    • Deploy the druid packages to 26.0.0 across the cluster - automatic restarts do not occur
    • Deploy the puppet change to update the JAVA_OPTS and similar changes.
    • Restart all of the daemons in the following order:
      • Historical
      • Overlord
      • Middle Manager
      • Broker
      • Coordinator

So it's do-able, but a bit complex.
It would be a lot simpler if we could schedule some downtime for the service and do a bulk reimage of all five an-druid servers to bookworm.

I'll check with the Data-Engineering team whether they're happy for an hour's downtime, and also whether this is realistic for the SRE team.

CDanis added a comment.Jan 20 2026, 2:14 PM

Totally fine from SRE's side, just check in with the current oncallers before you begin the disruptive part of the maintenance.

BTullis renamed this task from Upgrade Druid to version 27.0.0 to Upgrade Druid to version 26.0.0.Jan 20 2026, 3:50 PM
BTullis updated the task description. (Show Details)
gerritbot added a comment.Jan 21 2026, 11:22 AM

Change #1229539 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Upgrade druid to version 26.0.0 on the analytics cluster

https://gerrit.wikimedia.org/r/1229539

ops-monitoring-bot added a comment.Jan 21 2026, 11:26 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-druid1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 21 2026, 11:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-druid1004.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 21 2026, 11:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-druid1005.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 21 2026, 11:28 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-druid1006.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 21 2026, 11:28 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-druid1007.eqiad.wmnet with OS bookworm

gerritbot added a comment.Jan 21 2026, 11:35 AM

Change #1229539 merged by Btullis:

[operations/puppet@production] Upgrade druid to version 26.0.0 on the analytics cluster

https://gerrit.wikimedia.org/r/1229539

ops-monitoring-bot added a comment.Jan 21 2026, 11:46 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-druid1007.eqiad.wmnet with OS bookworm executed with errors:

  • an-druid1007 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-druid1007.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 21 2026, 11:46 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-druid1007.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 21 2026, 12:04 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-druid1006.eqiad.wmnet with OS bookworm completed:

  • an-druid1006 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601211144_btullis_2887727_an-druid1006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 21 2026, 12:10 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-druid1005.eqiad.wmnet with OS bookworm completed:

  • an-druid1005 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601211149_btullis_2887704_an-druid1005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 21 2026, 12:15 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-druid1003.eqiad.wmnet with OS bookworm completed:

  • an-druid1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601211155_btullis_2887617_an-druid1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 21 2026, 12:18 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-druid1004.eqiad.wmnet with OS bookworm completed:

  • an-druid1004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601211159_btullis_2887678_an-druid1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 21 2026, 12:20 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-druid1007.eqiad.wmnet with OS bookworm completed:

  • an-druid1007 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601211203_btullis_2889512_an-druid1007.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ssingh subscribed.Jan 21 2026, 3:14 PM
BTullis added a comment.Jan 22 2026, 12:44 PM

We have now upgraded the druid_analytics cluster and all seems well, wlthough there is still a little back-filling to be completed on the webrequest_sampled_live and wmf_netflow datasets. I believe that @JAllemandou is working on this.

There were a few points of note, during the upgrade.

  • an-druid100[3-4] did not have their vg0/srv LVM logical volume wiped during the upgrade, whereas an-druid100[5-7] did.

This turns out to have been beneficial, in terms of the time it took to reload data from deep storage. We should try to make sure that this volume is also retained when we reimage the druid_public cluster.

Checking the configuration, we can see that this shouldn't have been a surprise, since this is how it is configured.

Similarly, an-druid100[3-4] waited for input at the partman stage of the installer, because they are configured with the reuse-parts-test.cfg recipe.

  • There was a race condition in puppet, which meant that all druid services needed to be restarted manually, after installation. The druid-broker and druid-historical services failed to start, therefore blocking the cookbook from exiting cleanly, but we saw that all other services were behaving strangely, so it's possible that they were started before puppet had finished configuring them.
gerritbot added a comment.Jan 22 2026, 12:58 PM

Change #1230321 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Configure druid clusters to reuse their /srv volume

https://gerrit.wikimedia.org/r/1230321

gerritbot added a comment.Jan 22 2026, 1:06 PM

Change #1230321 merged by Btullis:

[operations/puppet@production] Configure druid clusters to reuse their /srv volume

https://gerrit.wikimedia.org/r/1230321

Gehel edited projects, added: Data-Platform-SRE (2026.01.23 - 2026.02.13); removed: Data-Platform-SRE (2026.01.05 - 2026.01.23).Jan 23 2026, 10:03 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.
Antoine_Quhen mentioned this in T415357: Upgrade DataHub CLI virtualenv used by metadata_ingest_daily to restore Druid ingestion.Jan 23 2026, 12:44 PM
JAllemandou added a comment.Jan 23 2026, 1:18 PM

Data has been backfilled, no hole in the webrequest_sampled_live data anymore. I have created T415359: Batch index webrequest_sampled data in Druid to automatize batch-indexing the webrequest_sampled data.

There still is an issue with realtime tasks not being able to properly launch replicas:

  • Realtime jobs are configured to so that their realtime tasks have 3 replicas for redundancy and query-paralellization)
  • Currently only one of them actually runs, the other two always waiting in PENDING status.
  • It's also worth noticing that all of the running tasks have run on an-druid1003, as if tasks couldn't be launched remotely

We should investigate this when folks are back from the SRE summit.

elukey added a comment.Jan 23 2026, 4:53 PM

Hey folks, I was using turnilo today with queries like https://w.wiki/Ha94, plus also some regexes, and I noticed a general slowdown (but eventually it worked). I then checked the Historical's GC metrics and I found this:

Screenshot From 2026-01-23 17-51-22.png (4,450×2,508 px, 467 KB)

The heap size didn't seem to be under pressure, so I am wondering if it is just something related to queries (maybe with regexes) that allocate a ton of object with small life, hitting some constraints like small young gen etc.. Since we use G1 now, we could try to increase something like -XX:G1MaxNewSizePercent and see how it goes.

CDanis added a comment.Jan 23 2026, 8:41 PM
In T278056#11549526, @elukey wrote:

The heap size didn't seem to be under pressure, so I am wondering if it is just something related to queries (maybe with regexes) that allocate a ton of object with small life, hitting some constraints like small young gen etc.. Since we use G1 now, we could try to increase something like -XX:G1MaxNewSizePercent and see how it goes.

Apologies, I'm not up to speed on anything JRE since about ten years, so here's a naive question -- is there a straightforward way to sample some call stacks that triggered allocations?

JAllemandou added a comment.Jan 26 2026, 10:29 AM

I'm not strong either at GC and java internal (stack dumps etc). I'm willing to help though.
It seems that the default -XX:G1MaxNewSizePercent is 60% - Do you think we should move that value to?

brouberol subscribed.Jan 26 2026, 1:17 PM

My high-level feeling is that the query itself is expensive for Druid, which performs some kind of "table scan", itself creating objects, which then get GC-ed. This is not backed by any kind of Druid knowledge though, just an intuition.

One thing we could try would be to have @CDanis run a query that we know is expensive in Druid, and perform thread dumps of the druid services, to get a better perspective of what's happening there.

JAllemandou added a comment.EditedJan 26 2026, 6:54 PM

Something else to take into account: When using turnilo, multiple druid queries are issued to provide the UI. For the example Luca gave above, I counted 12 distinct queries.
In this example there is one query for the main vizualization, and 11 for the right-side panel (pinboard). Each of those 11 queries take the same filters as the main one, so if it's a complex reqex, well it'll be recomputed many times with sub-level filtering.
New versions of tunrilo seem to provide the capability to add/remove that panel on-demand, that should be handy! Let's maybe plan on upgrading turnilo?

JAllemandou created subtask T415668: Upgrade turnilo and move it to k8s.Jan 27 2026, 2:16 PM

I created T415668: Upgrade turnilo and move it to k8s to work on the problem described above.

JAllemandou created subtask T415799: Since upgrade Druid realtime-ingestion tasks replication fails.Jan 28 2026, 4:42 PM
elukey added a comment.Jan 28 2026, 5:25 PM
In T278056#11553047, @JAllemandou wrote:

I'm not strong either at GC and java internal (stack dumps etc). I'm willing to help though.
It seems that the default -XX:G1MaxNewSizePercent is 60% - Do you think we should move that value to?

Ah I thought it was less, so maybe tuning it doesn't lead to a lot of results (worth to test though). Maybe we could turn on GC logging, replicate the 12 distinct queries scenario that you outlined and see what the GC does with young gen? We'll know for sure why it triggers, and maybe we'll have a better idea on what to tune. What do you think?

JAllemandou closed subtask T415799: Since upgrade Druid realtime-ingestion tasks replication fails as Resolved.Feb 3 2026, 12:18 PM
gerritbot added a comment.Feb 3 2026, 2:13 PM

Change #1227771 abandoned by Brouberol:

[operations/puppet@production] an-test-druid: run zookeeper on the default system java version

Reason:

We've decided to keep druid and zookeeper on the same JVM to keep things simpler

https://gerrit.wikimedia.org/r/1227771

gerritbot added a comment.Feb 3 2026, 2:15 PM

Change #1227786 abandoned by Brouberol:

[operations/puppet@production] an-test-druid: disable noisy GC stat logging

Reason:

Alraedy done

https://gerrit.wikimedia.org/r/1227786

MoritzMuehlenhoff added a comment.Feb 3 2026, 4:38 PM

We're still running Druid with Java 8, while https://druid.apache.org/docs/latest/tutorials/ mentions Java 17 as the required version to _build_ Druid (which supposedly also means that Java 17 would be an adequate version to _run_ Druid I guess?)

How about we try moving to Java 17 on an-test-druid (and then prod), maybe this fully obsoletes the GC issues we've seen?

MoritzMuehlenhoff added a comment.Feb 3 2026, 4:49 PM

In fact druid-broker and druid-common already depend on openjdk-17, but Hiera still defaults to Java 8 in hieradata/role/common/druid/analytics/worker.yaml:

profile::java::java_packages:
  - version: '8'  # for druid
    variant: 'jre'
  - version: '17' # for zookeeper
    variant: 'jre'
JAllemandou added a comment.Feb 4 2026, 8:45 AM

@MoritzMuehlenhoff we are not using Druid's latest version, because of hadoop dependencies.
We are on Druid 26.0 currently, for which java 8 is still recommended: https://druid.apache.org/docs/26.0.0/tutorials/.
We have plans to try get away from Hadoop for ingestion when we receive new hardware for a new cluster {T413446}.

Gehel added a comment.Feb 4 2026, 10:54 AM

I think we still need to upgrade druid public before closing this task. @BTullis / @JAllemandou : can you confirm?

JAllemandou added a comment.Feb 4 2026, 11:17 AM
In T278056#11582557, @Gehel wrote:

I think we still need to upgrade druid public before closing this task. @BTullis / @JAllemandou : can you confirm?

Absolutely yes! This needs to stay open until we have upgraded the druid-public cluster.

BTullis added subscribers: GGoncalves-WMF, Ahoelzl.Feb 4 2026, 1:04 PM

We haven't yet set a date for the upgrade.

I have asked the Data-Engineering team to guide us on this and manage any required comms, since there will be an unavoidable period of downtime that will affect public-facing features.

These features include edit and editor metrics, which are served via AQS.

cc: @Ahoelzl and @GGoncalves-WMF

We will need a maintenance window of around an hour, although the downtime might well be less than that, since we have learnt some lessons after upgrading the druid_analytics cluster.

gerritbot added a comment.Feb 5 2026, 3:23 PM

Change #1226766 abandoned by Brouberol:

[operations/puppet@production] druid_exporter: Fixup metric definition

Reason:

Already managed by joal

https://gerrit.wikimedia.org/r/1226766

Gehel edited projects, added: Data-Platform-SRE (2026-02-13 - 2026-03-06); removed: Data-Platform-SRE (2026.01.23 - 2026.02.13).Feb 16 2026, 3:34 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.
BTullis moved this task from In Progress to Blocked/Waiting on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.Feb 24 2026, 10:10 AM
Gehel edited projects, added: Data-Platform-SRE (2026-03-06 - 2026-03-27); removed: Data-Platform-SRE (2026-02-13 - 2026-03-06).Mar 6 2026, 2:01 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.
Maintenance_bot removed a project: Patch-For-Review.Mar 6 2026, 2:31 PM
dr0ptp4kt unsubscribed.Mar 15 2026, 1:52 PM
gerritbot added a comment.Mar 16 2026, 10:32 AM

Change #1253415 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the druid-public cluster and configure new Java cmdline options

https://gerrit.wikimedia.org/r/1253415

gerritbot added a project: Patch-For-Review.Mar 16 2026, 10:32 AM
gerritbot added a comment.Mar 16 2026, 10:45 AM

Change #1253415 merged by Btullis:

[operations/puppet@production] Update the druid-public cluster and configure new Java cmdline options

https://gerrit.wikimedia.org/r/1253415

Maintenance_bot removed a project: Patch-For-Review.Mar 16 2026, 11:31 AM
BTullis closed this task as Resolved.Mar 16 2026, 12:02 PM
BTullis moved this task from Blocked/Waiting to Done on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.

This has been completed now. We upgraded the druid-public cluster to bookworm and druid version 26.0.0 and it worked successfully, within the stated maintenance window.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.Mar 20 2026, 9:19 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits