The existing service::docker module might be usable for you rather than inventing a new system for running containers. service::docker gets quite a bit of use in Beta Cluster already for containerized workloads.

Change #1287889 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] WIP: relforge: Switch to an OCI-image based profile

https://gerrit.wikimedia.org/r/1287889

Change #1287889 abandoned by Bking:

[operations/puppet@production] relforge: Switch to an OCI-image based profile

Reason:

Going back to the standard Relforge setup. However, a subsequent patch will apply these concepts to beta cluster.

https://gerrit.wikimedia.org/r/1287889

Change #1300232 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: create docker-based role for deployment-prep

https://gerrit.wikimedia.org/r/1300232

Change #1300232 merged by Bking:

[operations/puppet@production] cirrussearch: create docker-based role for deployment-prep

https://gerrit.wikimedia.org/r/1300232

Change #1300242 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] deployment-prep: activate new cirrussearch profile

https://gerrit.wikimedia.org/r/1300242

Change #1300242 merged by Bking:

[operations/puppet@production] deployment-prep: activate new cirrussearch profile

https://gerrit.wikimedia.org/r/1300242

After merging the above changes, I also had to update the "prefix puppet" (hieradata for a specific role or profile) for deployment-cirrussearch in Horizon.

I added the following:

profile::base::overlayfs: true
profile::base::production::role_description: cirrussearch
profile::opensearch::cirrus::oci_image::name: opensearch
profile::opensearch::cirrus::oci_image::ns: repos/data-engineering
profile::opensearch::cirrus::oci_image::version: 2026-05-13-143357-2b2e022d3e2756b0dbd31d66e341a587f5d3b8d7-production2

After removing Puppet SSL files, re-running Puppet, and manually signing the certificate on deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud, deployment-cirrussearch15 is able to run Puppet without errors. However, the other deployment-cirrussearch VMs are now having Puppet issues. This is to be expected while we work through the kinks of the new role. From a practical standpoint, these hosts are still up and able to be used for testing.

The next step is to look at the docker service manifest again , figure out which arguments need to be applied (such as entrypoint, volume, bind mounts) and add them to Horizon hieradata.

Change #1300927 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] WIP: cirrussearch: Flesh out deployment-prep plan

https://gerrit.wikimedia.org/r/1300927

Change #1300927 merged by Bking:

[operations/puppet@production] cirrussearch: Flesh out deployment-prep plan

https://gerrit.wikimedia.org/r/1300927

Change #1301396 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: hard-code bind mount

https://gerrit.wikimedia.org/r/1301396

Change #1301396 merged by Bking:

[operations/puppet@production] cirrussearch: hard-code bind mount

https://gerrit.wikimedia.org/r/1301396

Change #1301408 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: Fix bind mount path for deployment-prep

https://gerrit.wikimedia.org/r/1301408

Change #1301408 merged by Bking:

[operations/puppet@production] cirrussearch: Fix bind mount path for deployment-prep

https://gerrit.wikimedia.org/r/1301408

Change #1302280 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: Add minimal opensearch config for deployment-prep

https://gerrit.wikimedia.org/r/1302280

The change is working, I one-offed deployment-cirrussearch15.deployment-prep.eqiad1.wikimedia.cloud
in /etc/systemd/system/cirrussearch.service.d/override.conf with the following, which roughly approximates the Puppet change above:

root@deployment-cirrussearch15:~# cat /etc/systemd/system/cirrussearch.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/docker run --rm=true \
--network host \
-v cirrussearch:/etc/cirrussearch \
-v /srv/opensearch:/usr/share/opensearch/data \
-v /etc/opensearch/opensearch.yml:/usr/share/opensearch/config/opensearch.yml \
--name %n docker-registry.wikimedia.org/repos/data-engineering/opensearch:2026-05-13-143357-2b2e022d3e2756b0dbd31d66e341a587f5d3b8d7-production2

Change #1302956 had a related patch set uploaded (by Bking; author: Bking):

[operations/mediawiki-config@master] deployment-prep: Update cirrussearch (OpenSearch) config

https://gerrit.wikimedia.org/r/1302956

Change #1302280 merged by Bking:

[operations/puppet@production] cirrussearch: Add minimal opensearch config for deployment-prep

https://gerrit.wikimedia.org/r/1302280

Per conversation with @dcausse , on-wiki search is currently down on the beta cluster. To fix this, we'll need to:

1. Merge and deploy this patch
2. Run something like mwscript /srv/mediawiki-staging/php-master/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki enwiki --cluster=all && mwscript /srv/mediawiki-staging/php-master/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki=enwiki in a tmux window on deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud.
3. Verify the indices are created from deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud: curl https://deployment-cirrussearch15.deployment-prep.eqiad1.wikimedia.cloud:9243/_cat/indices
4. Keep an eye out for other unexpected errors.

Change #1302956 merged by jenkins-bot:

[operations/mediawiki-config@master] deployment-prep: Update cirrussearch (OpenSearch) config

https://gerrit.wikimedia.org/r/1302956

Mentioned in SAL (#wikimedia-cloud) [2026-06-18T07:23:03Z] <dcausse> reindexing all wikis to opensearch2 (T425585, T427196)

@bking I believe that search is back on the beta cluster but sadly we running a bit low on disk (82% used).
Couple questions/comments:

• Logs seem only available via docker logs (I could not find a link to the host fs)
• Are there hiera values we could change to override some settings (Xmx, opensearch.yaml entries)?

It'd be great to cleanup deprecated hiera config from https://horizon.wikimedia.org/project/prefixpuppet/?tab=prefix_puppet__puppet-deployment-cirrussearch to avoid possible confusion in the future.

I suspect we could also delete old instances now?

Thanks!

Mentioned in SAL (#wikimedia-cloud) [2026-06-25T14:25:28Z] <inflatador> delete unused servers deployment-cirrussearch1[2-4] T425585

@dcausse thanks for the questions, let me address those:

I believe that search is back on the beta cluster but sadly we running a bit low on disk (82% used).

I should be able to add a Cinder volume and tell OpenSearch to use that for storage, I'll get back to you on that.

Logs seem only available via docker logs (I could not find a link to the host fs)

ACK, it sounds like we need another bind mount for logs. I can set that up.

Are there hiera values we could change to override some settings (Xmx, opensearch.yaml entries)?

We have the following options:

• Go back to using the opensearch.yml/jvm.options template from Puppet in beta cluster
• Update the current hard-coded opensearch.yml/create a jvm.options for beta cluster in Puppet
• Configure opensearch/jvm options via environment variables as described here

I think we could implement the final option via service::docker 's environment arg . I'll get started on that, but if you prefer a different approach feel free to reply here.

I suspect we could also delete old instances now?

Done, thank you for the reminder.

Mentioned in SAL (#wikimedia-cloud) [2026-06-25T16:50:20Z] <inflatador> add 60GB cinder vol to deployment-cirrussearch15 T425585

Change #1305731 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: Improve beta-cluster deploy

https://gerrit.wikimedia.org/r/1305731

Change #1305731 merged by Bking:

[operations/puppet@production] cirrussearch: Improve beta-cluster deploy

https://gerrit.wikimedia.org/r/1305731

Change #1305760 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: fix opensearch environment var name

https://gerrit.wikimedia.org/r/1305760

Change #1305760 merged by Bking:

[operations/puppet@production] cirrussearch: fix opensearch environment var name

https://gerrit.wikimedia.org/r/1305760

OK, I have addressed the above concerns:

• Attached/formatted/mounted/moved OpenSearch data a 60 GB cinder volume
• Created a bind mount to /var/log/opensearch so logs are accessible from the host
• Create/apply OPENSEARCH_JAVA_OPTS env var. This value currently sets the JVM heap size to 4 GB (1/2 of the VM's vRAM).

As such, I'm moving this ticket to "Needs Review" so Search Platform team can verify that we're finished (or add more requirements if not).