Decommission NodeJS IPoid service
Closed, ResolvedPublic
Actions

Description

Summary

Once we are satisfied with the stability of the iPoid-Service (IPoid OpenSearch) setup, we can remove the NodeJS IPoid service from WikiKube, remove its database, and clean up other references to it.

Before removing, we want:

SLO conversation to be finalized
reads are happening across both data centers

Acceptance criteria

NodeJS WikiKube service is removed
Wikitech pages are updated - https://wikitech.wikimedia.org/w/index.php?title=Service%2FIPoid&diff=2432937&oldid=2357092
misc database for IPoid is removed - T431007
ipoid gitlab repo is archived

Details

Other Assignee: RLazarus

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
ProductionServices: Drop NodeJS iPoid URL	Dreamy Jazz	operations/mediawiki-config	master	+0 -2
deployment_server: remove ipoid users	Dreamy Jazz	operations/puppet	production	+0 -5
deployment_server: absent ipoid kubernetes service	Dreamy Jazz	operations/puppet	production	+1 -0
Remove ipoid chart and service definitions	Dreamy Jazz	operations/deployment-charts	master	+0 -2 K
service: Remove ipoid service	Clément Goubert	operations/puppet	production	+0 -14
service: ipoid to service_setup	Clément Goubert	operations/puppet	production	+1 -1
service: ipoid to lvs_setup	Clément Goubert	operations/puppet	production	+1 -1
wmnet: Remove ipoid CNAME	Clément Goubert	operations/dns	master	+0 -3
services_proxy: Remove ipoid listener	Clément Goubert	operations/puppet	production	+0 -8

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• Dreamy_Jazz	T436757 Drop NodeJsIPoidDataFetcher
Resolved	• Dreamy_Jazz	T416623 Decommission NodeJS IPoid service
Resolved	kostajh	T416164 IPReputation: Switch to OpenSearch backend
Resolved	kostajh	T416316 IPReputation: Improve caching logic to handle backend downtime
Resolved	Marostegui	T431007 Drop the ipoid database from MariaDB m5

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

• Dreamy_Jazz mentioned this in T353538: Log unknown actor, risk and tunnel types.Feb 24 2026, 3:13 PM

• Dreamy_Jazz mentioned this in T351746: Include batch file context lines when throwing an error.

• Dreamy_Jazz mentioned this in T355048: Observability: Emit metrics event per request and emit latency per request.

• Dreamy_Jazz mentioned this in T355051: Observability: Emit counts in data pipeline.

• Dreamy_Jazz mentioned this in T355136: CI should report failed assertions and not time out.

• Dreamy_Jazz mentioned this in T354642: Respond to osv_dependency_check on ipoid.

• Dreamy_Jazz mentioned this in T354639: Un-leaky integration tests for ipoid.

• Dreamy_Jazz mentioned this in T359517: Duplicate IP errors for IPs which are not duplicated.

• Dreamy_Jazz mentioned this in T359384: IPs sometimes not deleted but updated with old data.

• Dreamy_Jazz mentioned this in T359181: Data updated incorrectly if yesterday's feed has duplicate IPs.

• Dreamy_Jazz mentioned this in T350470: When processing data files, ensure the "ip" key comes first in the stringified JSONs before sorting.Feb 24 2026, 3:16 PM

• Dreamy_Jazz mentioned this in T355896: output-sql.js throws an error if actor.client does not exist in the dataset.Feb 24 2026, 3:16 PM

• Dreamy_Jazz mentioned this in T354314: Provide developer documentation for getting started with ipoid.

• Dreamy_Jazz mentioned this in T355133: Add documentation and tests for `--updatedb` and get-latest-run-date.js.

• Dreamy_Jazz mentioned this in T393434: Migrate node-based services in production to node22.Feb 24 2026, 3:19 PM

• Dreamy_Jazz mentioned this in T355678: Handle ipoid api returning invalid data.Feb 24 2026, 3:25 PM

kostajh updated the task description. (Show Details)Mar 3 2026, 10:07 AM

kostajh updated the task description. (Show Details)

kostajh added a subscriber: Gehel.

RLazarus moved this task from Radar (Awareness) to Radar (Pending) on the ServiceOps board.Mar 12 2026, 5:13 PM

@kostajh @Gehel AFAIK the traffic is now migrated, could we clean up the old service?

It was written in description to wait for SLO conversations to be finalized. While technically capturing the SLOs is still ongoing, I don't think it should be a blocker to decommission.

In T416623#11752681, @MLechvien-WMF wrote:

@kostajh @Gehel AFAIK the traffic is now migrated, could we clean up the old service?

It was written in description to wait for SLO conversations to be finalized. While technically capturing the SLOs is still ongoing, I don't think it should be a blocker to decommission.

Yes, I think we could do that.

In T416623#11757466, @kostajh wrote:

In T416623#11752681, @MLechvien-WMF wrote:

@kostajh @Gehel AFAIK the traffic is now migrated, could we clean up the old service?

It was written in description to wait for SLO conversations to be finalized. While technically capturing the SLOs is still ongoing, I don't think it should be a blocker to decommission.

Yes, I think we could do that.

You can refer to https://wikitech.wikimedia.org/wiki/Kubernetes/Remove_a_service as a starting point, please do not hesitate to reach out if you need support from us.

@kostajh can I confirm if this can be done this quarter?

Kosta is currently OOO, but I'll post in our team slack to see if we can get a timeline for this

OKryva-WMF moved this task from Inbox to Next sprint on the Product Safety and Integrity board.Apr 15 2026, 7:26 AM

@Dreamy_Jazz did you get an ETA for that?

There was a suggestion for us to include it in our next sprint and our engineering manager has moved this task into our next sprint column

However, it was unsure about when we would have time to do this due to other higher priority essential work (so probably will be a few weeks)

Capturing the state after chatting with Ollie, tentative timeline is end of June. Thanks!

We said we would do this end of June. It is end of June, so adding to the sprint

• Dreamy_Jazz moved this task from Backlog to Ready on the Product Safety and Integrity (Sprint Venus Flytrap (Jun 29 - Jul 17)) board.Jun 29 2026, 1:16 PM

• Dreamy_Jazz claimed this task.Jun 30 2026, 9:13 PM

Mentioned in SAL (#wikimedia-operations) [2026-06-30T21:14:23Z] <Dreamy_Jazz> Destroyed NodeJS iPoid service deployments for T416623

Change #1306779 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[operations/puppet@production] ipoid: absent kubernetes service

https://gerrit.wikimedia.org/r/1306779

gerritbot added a project: Patch-For-Review.Jun 30 2026, 9:18 PM

Change #1306782 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[operations/puppet@production] deployment_server: remove ipoid users

https://gerrit.wikimedia.org/r/1306782

Change #1306784 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[operations/deployment-charts@master] Remove ipoid chart and service definitions

https://gerrit.wikimedia.org/r/1306784

I presume the puppet patches will need to be added to a puppet window

Change #1306899 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] service: ipoid to service_setup

https://gerrit.wikimedia.org/r/1306899

Change #1306900 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] service: Remove ipoid service

https://gerrit.wikimedia.org/r/1306900

Change #1306902 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/dns@master] wmnet: Remove ipoid CNAME

https://gerrit.wikimedia.org/r/1306902

Change #1306903 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] services_proxy: Remove ipoid listener

https://gerrit.wikimedia.org/r/1306903

I've taken the liberty to add some necessary patches removing LVS service definitions, DNS entries, and mw-on-k8s mesh config. Order of operations should look more or less like:

Merge services_proxy: Remove ipoid listener
Run puppet on deployment server, then run a helmfile-only Mw-On-K8s deployment to remove the envoy listener
Merge wmnet: Remove ipoid CNAME DNS change
Run sudo -i authdns-update
Merge service: ipoid to service_setup LVS change
Run sudo cumin 'A:dnsbox' run-puppet-agent
Merge service: Remove ipoid service LVS change
Merge Remove ipoid chart and service definitions
Deploy the admin_ng changes to all environments
Merge deployment_server: absent ipoid kubernetes service
Run puppet on deployment server
Merge deployment_server: remove ipoid users
Remove private puppet entries for ipoid

Everything from 3 to 13 needs to be done with someone from ServiceOps

Thanks

Will someone from ServiceOps be around during a puppet deployment window, or should we coordinate a different time?

There should be some of us around during tomorrow's yes, but I'll let @RLazarus decide whether it's worth doing at a different time considering there's a little more than just puppet merges.

Yeah, officially, this isn't really Puppet window material. It has way too many moving parts -- the Puppet window is really just for quick one-off patches, and in general we shouldn't expect Puppet window deployers to handle changes like this.

[takes off Puppet window deployer hat]

[puts on serviceops hat]

But all we really need is some time to coordinate. Nothing else is scheduled in that deployment window, so let's use it for this. (If we run long, the next thing scheduled is the MW Infra window and we can use that too.)

@Dreamy_Jazz I'll meet you in #wikimedia-operations at 16 UTC tomorrow, 2026-07-02!

(Thanks for pre-arranging, though -- if this had just shown up on the deployments calendar, I probably would have had to say no.)

Change #1307094 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] service: ipoid to lvs_setup

https://gerrit.wikimedia.org/r/1307094

Change #1306903 merged by RLazarus:

[operations/puppet@production] services_proxy: Remove ipoid listener

https://gerrit.wikimedia.org/r/1306903

Mentioned in SAL (#wikimedia-operations) [2026-07-02T16:25:34Z] <rzl@deploy1003> Started scap sync-world: T416623

Mentioned in SAL (#wikimedia-operations) [2026-07-02T16:26:33Z] <rzl@deploy1003> rzl: T416623 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-07-02T16:35:13Z] <rzl@deploy1003> Finished scap sync-world: T416623 (duration: 10m 19s)

Change #1306902 merged by RLazarus:

[operations/dns@master] wmnet: Remove ipoid CNAME

https://gerrit.wikimedia.org/r/1306902

Change #1307094 merged by RLazarus:

[operations/puppet@production] service: ipoid to lvs_setup

https://gerrit.wikimedia.org/r/1307094

Status: This is done up through backing the LVS setup out to lvs_setup (i.e. through https://wikitech.wikimedia.org/wiki/LVS#Remove_network_probes_/_monitoring, and step 4½ in T416623#12075480.)

After discussing with @BCornwall we decided to pause there, since today is a Logical Friday with the U.S. holiday tomorrow. Up next is backing it out to service_setup, and the attending ipvsadm run which isn't zero-risk. We'll reconvene next week: I have a conflict Monday, but we'll borrow the Tuesday 2026-07-07 16 UTC Puppet window to complete this.

RLazarus moved this task from Radar (Pending) to In Progress on the ServiceOps board.Jul 2 2026, 5:06 PM

• Dreamy_Jazz created subtask T431007: Drop the ipoid database from MariaDB m5.Jul 2 2026, 5:13 PM

• Dreamy_Jazz mentioned this in T431007: Drop the ipoid database from MariaDB m5.

• Dreamy_Jazz updated the task description. (Show Details)Jul 2 2026, 5:17 PM

• Dreamy_Jazz updated the task description. (Show Details)

Change #1306899 merged by RLazarus:

[operations/puppet@production] service: ipoid to service_setup

https://gerrit.wikimedia.org/r/1306899

Mentioned in SAL (#wikimedia-operations) [2026-07-07T16:53:57Z] <rzl> rzl@cumin2003:~$ sudo cumin 'A:lvs-secondary-eqiad' 'systemctl restart pybal.service' # lvs1020, T416623

Mentioned in SAL (#wikimedia-operations) [2026-07-07T16:55:12Z] <rzl> rzl@cumin2003:~$ sudo cumin 'A:lvs-low-traffic-eqiad' 'systemctl restart pybal.service' # lvs1019, T416623

Mentioned in SAL (#wikimedia-operations) [2026-07-07T17:00:56Z] <rzl> rzl@cumin2003:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service' # lvs2014, T416623

Mentioned in SAL (#wikimedia-operations) [2026-07-07T17:06:13Z] <rzl> rzl@cumin2003:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service' # lvs2013, T416623

Change #1306900 merged by RLazarus:

[operations/puppet@production] service: Remove ipoid service

https://gerrit.wikimedia.org/r/1306900

Status from this morning: done through step 7 in T416623#12075480, i.e. LVS work is complete. admin_ng is dirty from unrelated work so I'm optimistically waiting to hear back before proceeding.

Change #1306784 merged by jenkins-bot:

[operations/deployment-charts@master] Remove ipoid chart and service definitions

https://gerrit.wikimedia.org/r/1306784

Change #1306779 merged by RLazarus:

[operations/puppet@production] deployment_server: absent ipoid kubernetes service

https://gerrit.wikimedia.org/r/1306779

Change #1306782 merged by RLazarus:

[operations/puppet@production] deployment_server: remove ipoid users

https://gerrit.wikimedia.org/r/1306782

Ipoidn't!

All the Kubernetes work here is done, and I went ahead and removed us from the deployment calendar tomorrow. Let me know if you need anything else from Service Ops here.

Leaving the task open for any remaining work from you, @Dreamy_Jazz.

Thanks! The remaining work is just the subtask for the DBAs to drop the DB on the m5 cluster and probably to archive the ipoid gitlab repo. Thanks again

• Dreamy_Jazz updated the task description. (Show Details)Jul 8 2026, 8:09 PM

Maintenance_bot removed a project: Patch-For-Review.Jul 8 2026, 8:31 PM

Service port removed at https://wikitech.wikimedia.org/w/index.php?title=Kubernetes/Service_ports&diff=prev&oldid=2434700.

• Dreamy_Jazz updated the task description. (Show Details)Jul 10 2026, 2:46 PM

Change #1309684 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[operations/mediawiki-config@master] ProductionServices: Drop NodeJS iPoid URL

https://gerrit.wikimedia.org/r/1309684

gerritbot added a project: Patch-For-Review.Jul 10 2026, 2:50 PM

Marostegui closed subtask T431007: Drop the ipoid database from MariaDB m5 as Resolved.Jul 13 2026, 6:43 AM

• Dreamy_Jazz updated the task description. (Show Details)Jul 13 2026, 11:14 AM

Change #1309684 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices: Drop NodeJS iPoid URL

https://gerrit.wikimedia.org/r/1309684

Mentioned in SAL (#wikimedia-operations) [2026-07-13T12:36:08Z] <dreamyjazz@deploy2003> Started scap sync-world: Backport for [[gerrit:1309684|ProductionServices: Drop NodeJS iPoid URL (T416623)]]

Mentioned in SAL (#wikimedia-operations) [2026-07-13T12:37:47Z] <dreamyjazz@deploy2003> dreamyjazz: Backport for [[gerrit:1309684|ProductionServices: Drop NodeJS iPoid URL (T416623)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-07-13T12:43:10Z] <dreamyjazz@deploy2003> Finished scap sync-world: Backport for [[gerrit:1309684|ProductionServices: Drop NodeJS iPoid URL (T416623)]] (duration: 07m 02s)