2 VMs for mw-experimental
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	jijiki
	Jun 16 2025, 2:42 PM

Project Tags

Referenced Files

None

Subscribers

Aklapper

jijiki

MoritzMuehlenhoff

Description

Site/Location:eqiad, codfw
Number of systems: 2
Service: mw-experimental
Networking Requirements: internal IP
Processor Requirements: 4
Memory: 16G
Disks: 200G
Wikitech: https://wikitech.wikimedia.org/wiki/Mw-experimental (TBA)
ServiceOps would like to request 2 VMs (1 in each DC), which will be used as kubernetes workers, dedicated to run mw-experimental pods.

wikikube-worker-exp1001 (row B)
wikikube-worker-exp2001 (row C)

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
site.pp: make wikikube-worker-exp2001 a k8s worker	Effie Mouzeli	operations/puppet	production	+3 -2
site.pp: make wikikube-worker-exp1001 a k8s worker	Effie Mouzeli	operations/puppet	production	+21 -1
site.pp: add wikikube-worker-exp(1001\|2001)	Effie Mouzeli	operations/puppet	production	+15 -1

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
In Progress	Request	None	T421484 Reclaim parsoidtest1001 and testreduce1001
Resolved		jijiki	T421483 parsoidtest1001.eqiad.wmnet retirement plan
Resolved		Jgiannelos	T420336 mw-parsoid improvements
Resolved		jijiki	T386246 Migrate parsoidtest functionality to kubernetes (mw-parsoid)
Resolved		None	T290536 Serve production traffic via Kubernetes
Resolved		MLechvien-WMF	T324003 mwdebug: people in the "deployment" group should be able to launch 'experimental' instances for testing purposes
Resolved		jijiki	T397498 Deprecate mwdebugXXXX hosts
Resolved		jijiki	T276994 Provide an mwdebug functionality on kubernetes (mw-experimental)
Resolved		None	T397051 2 VMs for mw-experimental

Event Timeline

jijiki created this task.Jun 16 2025, 2:42 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 16 2025, 2:42 PM

jijiki added a parent task: T276994: Provide an mwdebug functionality on kubernetes (mw-experimental).Jun 16 2025, 2:42 PM

jijiki updated the task description. (Show Details)

Looks fine, please use codfw/row C and eqiad eqiad/row B

Change #1159502 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] site.pp: add wikikube-worker-exp(1001|2001)

https://gerrit.wikimedia.org/r/1159502

gerritbot added a project: Patch-For-Review.Jun 16 2025, 3:18 PM

Change #1159518 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] site.pp: make wikikube-worker-exp* k8s workers

https://gerrit.wikimedia.org/r/1159518

jijiki mentioned this in T396767: Make mw-experimental production ready.Jun 16 2025, 4:29 PM

Change #1159502 merged by Effie Mouzeli:

[operations/puppet@production] site.pp: add wikikube-worker-exp(1001|2001)

https://gerrit.wikimedia.org/r/1159502

jijiki updated the task description. (Show Details)Jun 17 2025, 11:37 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host wikikube-worker-exp1001.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host wikikube-worker-exp1001.eqiad.wmnet with OS bookworm completed:

wikikube-worker-exp1001 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202506171306_jiji_3425476_wikikube-worker-exp1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Mentioned in SAL (#wikimedia-operations) [2025-06-17T13:54:22Z] <jiji@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Adding wikikube-worker-exp1001 - jiji@cumin1002 - T397051"

Mentioned in SAL (#wikimedia-operations) [2025-06-17T13:54:43Z] <jiji@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Adding wikikube-worker-exp1001 - jiji@cumin1002 - T397051"

jijiki updated the task description. (Show Details)Jun 17 2025, 3:06 PM

Change #1159518 merged by Effie Mouzeli:

[operations/puppet@production] site.pp: make wikikube-worker-exp1001 a k8s worker

https://gerrit.wikimedia.org/r/1159518

Maintenance_bot removed a project: Patch-For-Review.Jun 17 2025, 4:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host wikikube-worker-exp2001.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host wikikube-worker-exp2001.codfw.wmnet with OS bookworm completed:

wikikube-worker-exp2001 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202506171726_jiji_3563660_wikikube-worker-exp2001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Mentioned in SAL (#wikimedia-operations) [2025-06-17T19:26:26Z] <jiji@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Adding wikikube-worker-exp2001 - jiji@cumin1002 - T397051"

Mentioned in SAL (#wikimedia-operations) [2025-06-17T19:26:34Z] <jiji@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Adding wikikube-worker-exp2001 - jiji@cumin1002 - T397051"