contint1002 service implementation tracking
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	RobH
	Jul 26 2022, 5:07 PM

Description

This task is to track the service implementation of serviceops host(s) listed in the task description.

Once the linked racking task has been resolved, this task can be implemented.

This sub-task creation/update is per the request of serviceops; this task is assigned at creation to the 'Sub-team Technical Contact' provided in the initial ordering task.

contint1002 should have:

integration/docroot deployment
basics of Jenkins but service disabled (confirmed it is inactive/masked)
git-daemon
zuul-merger - The instance on contint1001 MUST be stopped before bringing zuul-merger up on contint1002 due to a per user ssh connection limit in Gerrit

Details

Subject	Repo	Branch	Lines +/-
cloud: allow VMs to connect to contint1002 and contint2002	operations/puppet	production	+8 -0
ci: move docker::settings to common, avoid host names	operations/puppet	production	+4 -12
contint: move zuul-merger from contint1001 to contint1002	operations/puppet	production	+3 -3
contint: add contint1002 as a scap target	operations/puppet	production	+2 -0
contint: give RelEng access to contint1002	operations/puppet	production	+3 -0
ci::master: hack to bootstrap new server contint1002	operations/puppet	production	+1 -1
contint: add ci::master to contint1002	operations/puppet	production	+25 -7
contint: add docker::settings for contint1002	operations/puppet	production	+4 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Dzahn	T294276 contint hardware refresh
Resolved	hashar	T313832 contint1002 service implementation tracking
Resolved	• Cmjohnson	T313830 Q1:rack/setup/install contint1002

Event Timeline

RobH created this task.Jul 26 2022, 5:07 PM

RobH unsubscribed.

LSobanski added a project: collaboration-services.Jul 27 2022, 10:08 AM

LSobanski moved this task from Incoming to Blocked on the collaboration-services board.

LSobanski changed the task status from Open to Stalled.Sep 20 2022, 3:26 PM

LSobanski moved this task from Blocked to Incoming on the collaboration-services board.

jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:17 PM

LSobanski moved this task from Incoming to Backlog on the collaboration-services board.Oct 18 2022, 3:55 PM

Dzahn added a parent task: T294276: contint hardware refresh.Nov 2 2022, 5:00 PM

@LSobanski comment about the incident with contint1001 is at T294276#8357385

I think this is currently blocked on T313830.

hashar added a project: Continuous-Integration-Infrastructure.Nov 3 2022, 1:09 PM

LSobanski triaged this task as High priority.Nov 4 2022, 3:35 PM

@jnuche we will have to setup a spare Jenkins and a Zuul merger on this new host contint1002 :-)

@hashar sounds like a good opportunity to pair!

LSobanski removed a project: serviceops.Nov 10 2022, 12:38 PM

LSobanski changed the task status from Stalled to Open.Nov 23 2022, 7:49 PM

LSobanski removed LSobanski as the assignee of this task.

also see T313830#8418218

Switching to contint1002 would also be a good opportunity to migrate to Bullseye (which per https://wikitech.wikimedia.org/wiki/Operating_system_upgrade_policy should happen within nine months anyway):

The gerrit version on Bullseye is identical (given it's all Java)
The Java version on Bullseye is identical (since Antoint already resolved https://phabricator.wikimedia.org/T269354)

hashar mentioned this in T324659: contint2002 service implementation tracking.Dec 7 2022, 10:46 AM

hashar mentioned this in T294276: contint hardware refresh.

hashar removed a parent task: T313830: Q1:rack/setup/install contint1002.Dec 7 2022, 10:50 AM

hashar added a subtask: T313830: Q1:rack/setup/install contint1002.

contint1001 keeps crashing due to a faulty memory stick. It happened on October 31st ( T294276#8357385 ) and again on Dec 5th and today Dec 7th.

The machine is not hosting the Jenkins controller and Zuul scheduler but it does provides a zuul-merger and a Jenkins agent. We thus can't afford to shut it down and it should be replaced.

Change 865672 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] contint: give access to contint1002

https://gerrit.wikimedia.org/r/865672

gerritbot added a project: Patch-For-Review.Dec 7 2022, 3:18 PM

Change 865680 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] contint: add ci::master to contint1002

https://gerrit.wikimedia.org/r/865680

Change 865681 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] contint: add contint1002 as a scap target

https://gerrit.wikimedia.org/r/865681

Dzahn mentioned this in T324698: contint1001 hardware failures (remove contint1001 from production).Dec 7 2022, 6:48 PM

Dzahn mentioned this in T256422: switch contint prod server back from contint2001 to contint1001.

Change 865672 merged by Dzahn:

[operations/puppet@production] contint: give RelEng access to contint1002

https://gerrit.wikimedia.org/r/865672

I merged your change https://gerrit.wikimedia.org/r/c/operations/puppet/+/865672/4 so now releng members have shell access on contint1002.

[contint1002:~] $ grep roots /etc/group
contint-roots:x:720:hashar,thcipriani,brennen,dancy,jhuneidi,demon,dduvall,jnuche,jforrester

Dzahn moved this task from Backlog to Work in Progress on the collaboration-services board.Dec 7 2022, 7:08 PM

Change 865680 merged by Dzahn:

[operations/puppet@production] contint: add ci::master to contint1002

https://gerrit.wikimedia.org/r/865680

Mentioned in SAL (#wikimedia-operations) [2022-12-07T19:53:55Z] <mutante> registry* (docker registry HA) - adding contint1002 to allowed hosts gerrit:865680 T313832

Mentioned in SAL (#wikimedia-operations) [2022-12-07T20:00:13Z] <mutante> contint* - deploying firewall changes to add contint1002 - T313832

Change 865734 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] contint: add docker::settings for contint1002

https://gerrit.wikimedia.org/r/865734

Change 865734 merged by Dzahn:

[operations/puppet@production] contint: add docker::settings for contint1002

https://gerrit.wikimedia.org/r/865734

Change 865735 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: move docker::settings to common, avoid host names

https://gerrit.wikimedia.org/r/865735

Change 865739 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci::master: hack to bootstrap new server contint1002

https://gerrit.wikimedia.org/r/865739

Change 865739 merged by Dzahn:

[operations/puppet@production] ci::master: hack to bootstrap new server contint1002

https://gerrit.wikimedia.org/r/865739

Mentioned in SAL (#wikimedia-operations) [2022-12-07T21:02:10Z] <mutante> contint1002 a2dismod mpm_event - https://phabricator.wikimedia.org/T208108 Bug: T313832

Stashbot mentioned this in T208108: httpd class and php7.0 - conflict with mpm_event module.Dec 7 2022, 9:02 PM

Mentioned in SAL (#wikimedia-releng) [2022-12-07T21:17:43Z] <hashar> Add contint1002 as an agent to the CI Jenkins, albeit in offline mode cause it is being provisioned | https://integration.wikimedia.org/ci/computer/contint1002/ | T313832

Mentioned in SAL (#wikimedia-releng) [2022-12-08T08:20:08Z] <hashar> Attached contint1002 as an agent of the CI Jenkins # T313832

Change 865681 merged by Clément Goubert:

[operations/puppet@production] contint: add contint1002 as a scap target

https://gerrit.wikimedia.org/r/865681

hashar updated the task description. (Show Details)Dec 8 2022, 9:13 AM

Mentioned in SAL (#wikimedia-operations) [2022-12-08T09:17:35Z] <hashar@deploy1002> Started deploy [integration/docroot@2e0d44b]: Warm up contint1002 and test php-fpm restart # T313832

Mentioned in SAL (#wikimedia-operations) [2022-12-08T09:17:39Z] <hashar@deploy1002> Finished deploy [integration/docroot@2e0d44b]: Warm up contint1002 and test php-fpm restart # T313832 (duration: 00m 03s)

hashar updated the task description. (Show Details)Dec 8 2022, 9:19 AM

Mentioned in SAL (#wikimedia-operations) [2022-12-08T09:24:53Z] <hashar@deploy1002> Started deploy [zuul/deploy@4c6859c]: Install Zuul virtualenv on contint1002 # T313832

Mentioned in SAL (#wikimedia-operations) [2022-12-08T09:25:00Z] <hashar@deploy1002> Finished deploy [zuul/deploy@4c6859c]: Install Zuul virtualenv on contint1002 # T313832 (duration: 00m 07s)

Mentioned in SAL (#wikimedia-operations) [2022-12-08T09:38:07Z] <hashar> contint1001: manually stopped and masked zuul-merger. It is under maintenance mode in Icinga # T313832

Mentioned in SAL (#wikimedia-operations) [2022-12-08T09:43:23Z] <hashar> contint1002: stopped puppet and manually started zuul-merger. I am monitoring it cause last time we have bring up a new one it had some issues here and there # T313832

Change 866277 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] contint: move zuul-merger from contint1001 to contint1002

https://gerrit.wikimedia.org/r/866277

Change 866277 merged by Clément Goubert:

[operations/puppet@production] contint: move zuul-merger from contint1001 to contint1002

https://gerrit.wikimedia.org/r/866277

Mentioned in SAL (#wikimedia-operations) [2022-12-08T10:18:52Z] <hashar> contint1002: activated Icinga monitoring , all services are up and running # T313832

hashar updated the task description. (Show Details)Dec 8 2022, 10:20 AM

contint1002 is now attached as a Jenkins agent and running the zuul-merger service.

I have removed contint1001 from Jenkins and it no more has the zuul-merger running.

Huge thanks to @Dzahn who has planned the hardware replacement ahead of time and for pairing on the Puppet role deployment yesterday. Thanks to @Clement_Goubert for stepping up this morning, doing the Puppet paperwork and provide assistance for Icinga monitoring.