Page MenuHomePhabricator
Create Task
Maniphest T424248

☂️Migrate non-cirrus indices from production OpenSearch to OpenSearch on k8s ☂️
Open, In Progress, Needs TriagePublic
Actions

Description

The production OpenSearch clusters currently host a few indices that aren't related on-wiki search.

This is because up until very recently, there was no other place to put these indices. That changed about a month ago, when we launched the OpenSearch on K8s platform with its first tenant, opensearch-ipoid (ref T408586 ). We now have a better place to host these indices, so I'm creating this ticket to:

  • Inform stakeholders and work with them to create a migration plan, onboard to the platform, answer questions etc
  • Per the above, create migration subtasks for each group of non-cirrus indices
  • Ensure all migrations are completed to everyone's satisfication and all update relevant docs

The following sets of indices are in-scope for the migration:

IndexStakeholderCurrent size of indices (including replicas)Number of primary/replica shards
toolhub_tools@bd80830 MB1,2
apifeatureusageMediaWiki Interfaces Team @aaron225 GB1,2
Ttmserver (Translatewiki application)Language team @Nikerabbit @abi_14.1 GB1,1

Change plan

  • deployment_charts: namespaces and default values for admin_ng
  • secrets
  • puppet: configs
  • deployment_charts: helmfiles
  • dns
  • puppet: service_catalog

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
services_proxy: switch to prod opensearch-on-k8s servicesoperations/puppetproduction+2 -2
opensearch-cluster: anonymous access for ttmsearch and toolhuboperations/deployment-chartsmaster+3 -0
service: services_proxy: prod opensearch-on-k8s servicesoperations/puppetproduction+50 -4
services_proxy: enabling toolhub and ttmserveroperations/puppetproduction+2 -0
services_proxy: isetting up toolhub and ttmserveroperations/puppetproduction+31 -2
service::catalog: Add opensearch-ttmserver-testoperations/puppetproduction+13 -0
wmnet: add additional opensearch clustersoperations/dnsmaster+19 -0
dse-k8s: deploy additional opensearch clustersoperations/deployment-chartsmaster+432 -0
dse-k8s: adding more opensearch namespacesoperations/deployment-chartsmaster+297 -3
deployment_server: define more opensearch configsoperations/puppetproduction+24 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Apr 23 2026, 5:05 PM
bking updated the task description. (Show Details)Apr 23 2026, 5:09 PM
bking added subscribers: abi_, aaron, Nikerabbit, • bd808.
Gehel edited projects, added: Data-Platform-SRE (2026-04-24 - 2026-05-15); removed: Data-Platform-SRE (2026-03-27 - 2026-04-17).Apr 24 2026, 9:17 AM
bking added a comment.Apr 27 2026, 1:32 PM

We'll take the first steps along this journey by provisioning new OpenSearch clusters on dse-k8s. Starting with these instructions , we'll provision the following namespaces:

  • opensearch-toolhub
  • opensearch-toolhub-test
  • opensearch-apifeatureusage
  • opensearch-apifeatureusage-test
  • opensearch-ttmserver
  • opensearch-ttmserver-test

Each namespace will host a discrete OpenSearch cluster. For now we'll use the opensearch-small flavor , although we can always scale up if required.

atsuko claimed this task.Apr 28 2026, 2:29 PM
gerritbot added a comment.Apr 29 2026, 3:49 PM

Change #1279410 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] deployment_server: define more opensearch configs

https://gerrit.wikimedia.org/r/1279410

gerritbot added a project: Patch-For-Review.Apr 29 2026, 3:49 PM
gerritbot added a comment.Apr 29 2026, 3:52 PM

Change #1279410 merged by Atsuko:

[operations/puppet@production] deployment_server: define more opensearch configs

https://gerrit.wikimedia.org/r/1279410

atsuko changed the task status from Open to In Progress.Apr 29 2026, 4:04 PM
gerritbot added a comment.Apr 29 2026, 4:29 PM

Change #1279423 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] dse-k8s: adding more opensearch namespaces

https://gerrit.wikimedia.org/r/1279423

atsuko moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.Apr 29 2026, 4:31 PM
EBernhardson subscribed.Apr 29 2026, 7:12 PM
EBernhardson unsubscribed.
EBernhardson subscribed.
gerritbot added a comment.Apr 30 2026, 10:24 AM

Change #1279423 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s: adding more opensearch namespaces

https://gerrit.wikimedia.org/r/1279423

Maintenance_bot removed a project: Patch-For-Review.Apr 30 2026, 10:30 AM
atsuko added a comment.EditedApr 30 2026, 10:47 AM

applying admin_ng for dse-k8s-*

root@deploy1003:~# charlie --services_dir admin_ng -e 'dse-k8s-*' diff 2>&1 | less -rS
root@deploy1003:~# charlie --services_dir admin_ng -e 'dse-k8s-*' apply
root@deploy1003:~# charlie --services_dir admin_ng -e 'dse-k8s-*' diff
[. dse-k8s-eqiad] No diffs.
[. dse-k8s-codfw] No diffs.
atsuko added a comment.Apr 30 2026, 2:14 PM

Added secrets with the dirtiest script.

gerritbot added a comment.Apr 30 2026, 4:22 PM

Change #1280515 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] dse-k8s: deploy additional opensearch clusters

https://gerrit.wikimedia.org/r/1280515

gerritbot added a project: Patch-For-Review.Apr 30 2026, 4:22 PM
atsuko updated the task description. (Show Details)Apr 30 2026, 4:37 PM
gerritbot added a comment.May 1 2026, 8:39 AM

Change #1280515 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s: deploy additional opensearch clusters

https://gerrit.wikimedia.org/r/1280515

Maintenance_bot removed a project: Patch-For-Review.May 1 2026, 9:30 AM
gerritbot added a comment.May 1 2026, 11:26 AM

Change #1281462 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/dns@master] wmnet: add additional opensearch clusters

https://gerrit.wikimedia.org/r/1281462

gerritbot added a project: Patch-For-Review.May 1 2026, 11:26 AM
bking created subtask T425377: Migrate Ttmserver (Translatewiki application) indices from production OpenSearch to OpenSearch on k8s.May 4 2026, 10:06 PM
gerritbot added a comment.May 5 2026, 7:51 AM

Change #1281462 merged by Atsuko:

[operations/dns@master] wmnet: add additional opensearch clusters

https://gerrit.wikimedia.org/r/1281462

atsuko updated the task description. (Show Details)May 5 2026, 8:21 AM
Maintenance_bot removed a project: Patch-For-Review.May 5 2026, 8:31 AM
gerritbot added a comment.May 5 2026, 8:47 AM

Change #1282920 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] service::catalog: Add opensearch-ttmserver-test

https://gerrit.wikimedia.org/r/1282920

gerritbot added a project: Patch-For-Review.May 5 2026, 8:47 AM
gerritbot added a comment.May 5 2026, 9:48 AM

Change #1282920 merged by Atsuko:

[operations/puppet@production] service::catalog: Add opensearch-ttmserver-test

https://gerrit.wikimedia.org/r/1282920

GGoncalves-WMF subscribed.May 5 2026, 9:59 AM
atsuko updated the task description. (Show Details)May 5 2026, 10:10 AM
Maintenance_bot removed a project: Patch-For-Review.May 5 2026, 10:31 AM
atsuko added a comment.EditedMay 5 2026, 12:32 PM

Rolled out opensearch-ttmserver-test in both dse-k8s-codfw and dse-k8s-eqiad

atsuko@deploy1003:~$ kube_env opensearch-ttmserver-test-deploy dse-k8s-eqiad 
atsuko@deploy1003:~$ kubectl get po
NAME                                                    READY   STATUS      RESTARTS   AGE
opensearch-ttmserver-test-bootstrap-0                   1/1     Running     0          7m51s
opensearch-ttmserver-test-masters-0                     1/1     Running     0          7m51s
opensearch-ttmserver-test-masters-1                     1/1     Running     0          3m17s
opensearch-ttmserver-test-masters-2                     0/1     Running     0          103s
opensearch-ttmserver-test-securityconfig-update-6gzzg   0/1     Completed   0          7m51s
atsuko@deploy1003:~$ curl https://opensearch-ttmserver-test.svc.eqiad.wmnet:30443/_cat/nodes
10.67.24.210 43 61 2 2.53 2.96 2.48 dm cluster_manager,data * opensearch-ttmserver-test-masters-1
10.67.25.218 56 63 1 3.94 4.05 3.88 dm cluster_manager,data - opensearch-ttmserver-test-masters-0
10.67.30.23   9 61 4 1.27 1.47 1.83 dm cluster_manager,data - opensearch-ttmserver-test-masters-2
atsuko@deploy1003:~$ curl https://opensearch-ttmserver-test.svc.eqiad.wmnet:30443/_cat/health
1777985187 12:46:27 opensearch-ttmserver-test green 3 3 true 12 4 0 0 0 0 - 100.0%
atsuko changed the status of subtask T425377: Migrate Ttmserver (Translatewiki application) indices from production OpenSearch to OpenSearch on k8s from Open to In Progress.May 5 2026, 12:47 PM
atsuko moved this task from In Progress to Blocked/Waiting on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.May 8 2026, 1:44 PM
bking created subtask T426073: Migrate toolhub indices from production OpenSearch to OpenSearch on k8s.May 12 2026, 2:23 PM
bking changed the status of subtask T426073: Migrate toolhub indices from production OpenSearch to OpenSearch on k8s from Open to In Progress.May 12 2026, 6:56 PM
atsuko changed the status of subtask T425377: Migrate Ttmserver (Translatewiki application) indices from production OpenSearch to OpenSearch on k8s from In Progress to Stalled.May 14 2026, 9:17 AM
gerritbot added a comment.May 14 2026, 12:10 PM

Change #1287388 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] services_proxy: isetting up toolhub and ttmserver

https://gerrit.wikimedia.org/r/1287388

gerritbot added a project: Patch-For-Review.May 14 2026, 12:10 PM
gerritbot added a comment.May 14 2026, 1:36 PM

Change #1287388 merged by Atsuko:

[operations/puppet@production] services_proxy: isetting up toolhub and ttmserver

https://gerrit.wikimedia.org/r/1287388

gerritbot added a comment.May 14 2026, 2:04 PM

Change #1287428 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] services_proxy: enabling toolhub and ttmserver

https://gerrit.wikimedia.org/r/1287428

gerritbot added a comment.May 14 2026, 3:15 PM

Change #1287428 merged by Atsuko:

[operations/puppet@production] services_proxy: enabling toolhub and ttmserver

https://gerrit.wikimedia.org/r/1287428

Maintenance_bot removed a project: Patch-For-Review.May 14 2026, 3:31 PM
pfischer edited projects, added: Discovery-Search (2026.05.04 - 2026.05.29); removed: Discovery-Search (2026.04.06 - 2026.05.01).Mon, May 18, 1:43 PM
atsuko changed the status of subtask T425377: Migrate Ttmserver (Translatewiki application) indices from production OpenSearch to OpenSearch on k8s from Stalled to In Progress.Fri, May 29, 12:21 PM
gerritbot added a comment.Mon, Jun 1, 12:47 PM

Change #1295901 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] service: services_proxy: prod opensearch-on-k8s services

https://gerrit.wikimedia.org/r/1295901

gerritbot added a project: Patch-For-Review.Mon, Jun 1, 12:47 PM
gerritbot added a comment.Mon, Jun 1, 4:57 PM

Change #1295901 merged by Atsuko:

[operations/puppet@production] service: services_proxy: prod opensearch-on-k8s services

https://gerrit.wikimedia.org/r/1295901

Maintenance_bot removed a project: Patch-For-Review.Mon, Jun 1, 5:31 PM
gerritbot added a comment.Tue, Jun 2, 10:56 AM

Change #1296539 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] opensearch-cluster: anonymous access for ttmsearch and toolhub

https://gerrit.wikimedia.org/r/1296539

gerritbot added a project: Patch-For-Review.Tue, Jun 2, 10:56 AM
gerritbot added a comment.Tue, Jun 2, 11:08 AM

Change #1296539 merged by jenkins-bot:

[operations/deployment-charts@master] opensearch-cluster: anonymous access for ttmsearch and toolhub

https://gerrit.wikimedia.org/r/1296539

Maintenance_bot removed a project: Patch-For-Review.Tue, Jun 2, 11:31 AM
gerritbot added a comment.Tue, Jun 2, 3:02 PM

Change #1296608 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] services_proxy: switch to prod opensearch-on-k8s services

https://gerrit.wikimedia.org/r/1296608

gerritbot added a project: Patch-For-Review.Tue, Jun 2, 3:02 PM
gerritbot added a comment.Tue, Jun 2, 4:17 PM

Change #1296608 merged by Atsuko:

[operations/puppet@production] services_proxy: switch to prod opensearch-on-k8s services

https://gerrit.wikimedia.org/r/1296608

Maintenance_bot removed a project: Patch-For-Review.Tue, Jun 2, 4:31 PM
atsuko changed the status of subtask T425377: Migrate Ttmserver (Translatewiki application) indices from production OpenSearch to OpenSearch on k8s from In Progress to Stalled.Thu, Jun 4, 11:26 AM
Gehel edited projects, added: Data-Platform-SRE (2026-06-05 - 2026-06-26); removed: Data-Platform-SRE (2026-04-24 - 2026-05-15).Fri, Jun 5, 1:28 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.
pfischer edited projects, added: Discovery-Search (2026.06.01 - 2026.07.03); removed: Discovery-Search (2026.05.04 - 2026.05.29).Mon, Jun 8, 2:38 PM
atsuko changed the status of subtask T425377: Migrate Ttmserver (Translatewiki application) indices from production OpenSearch to OpenSearch on k8s from Stalled to In Progress.Tue, Jun 9, 10:28 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits