Page MenuHomePhabricator
Create Task
Maniphest T192185

request to assign spare systems as terbium equivalent
Closed, ResolvedPublic
Actions

Description

We want to convert the Mediawiki maintenance servers to stretch. ( T192092 )

We can't just take down terbium and upgrade it in place.

So per Service Operations meeting we need to spin up a replacement maintenance server using stretch and have it running in parallel for a while to replace terbium and ultimately decom terbium.

Checking racktables i see there is an unassigned server in the same rack, right above terbium, WMF 3565.

https://racktables.wikimedia.org/index.php?page=object&object_id=1775

They are both Dell PowerEdge R420 and have the same purchase date. (Though, one of them says they are out of warranty and the other doesn't. )(?)

Requesting to assign WMF3565 as the terbium replacement.

Details

SubjectRepoBranchLines +/-
Revert "rename wmf6936 from mw1297 to mwmaint1001"operations/dnsmaster+4 -6
re-add mw1297 to site.pp and DHCP, remove mwmaint1001operations/puppetproduction+12 -19
rename wmf6936 from mw1297 to mwmaint1001operations/dnsmaster+4 -4
rename mw1297 to mwmaint1001, partman for mwmaint*operations/puppetproduction+12 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

Dzahn created this task.Apr 13 2018, 11:40 PM
Dzahn updated the task description. (Show Details)
Dzahn added a subscriber: Joe.
Dzahn added a comment.Apr 13 2018, 11:52 PM

The name would be nihonium, element 113.

Dzahn triaged this task as High priority.Apr 17 2018, 11:54 PM
faidon added a comment.Apr 18 2018, 12:32 AM

WMF3565 is > 5 years old, so there's really no point in setting hardware that old right now.

How urgent is this task? We have a task open for procuring new hardware for (among other servers) terbium (T189317), but we could figure out a different solution if this needs to happen sooner than that.

Dzahn removed Dzahn as the assignee of this task.EditedApr 18 2018, 12:38 AM

I think it kind of blocks the "never use PHP5" / "switch to PHP7" migration which also affects appservers and deployment servers. In the last Service Operations meeting it was brought up that we need to replace terbium as part of that. But let me confirm the urgency with @Joe

Paladox subscribed.Apr 18 2018, 12:40 AM
• MoritzMuehlenhoff subscribed.Apr 24 2018, 7:10 AM
In T192185#4131159, @Dzahn wrote:

The name would be nihonium, element 113.

I'd rather use a functional name here, e.g. mwmaint1001.eqiad.wmnet. These hosts have a functional equivalent in codfw using using a functional name is more obvious and at some point I also would to work on adding a mwmaint1002/2002 standin (e.g. in Ganeti) which we can easily switch to in case maintenance on the primary script runner (reboots of terbium are currently really painful to coordinate and implement)

• MoritzMuehlenhoff added a comment.Apr 24 2018, 7:13 AM
In T192185#4138132, @faidon wrote:

WMF3565 is > 5 years old, so there's really no point in setting hardware that old right now.

How urgent is this task? We have a task open for procuring new hardware for (among other servers) terbium (T189317), but we could figure out a different solution if this needs to happen sooner than that.

We also have the two former image scalers which are under warranty (and even more powerful than terbium). The plan is to allocate them as mw* hosts later on (T192457), but we could just as well repurpose one of them as the stretch-based standin for terbium (since we need to have both in parallel for a small time window).

Joe added a comment.Apr 24 2018, 10:23 AM
In T192185#4152337, @MoritzMuehlenhoff wrote:
In T192185#4138132, @faidon wrote:

WMF3565 is > 5 years old, so there's really no point in setting hardware that old right now.

How urgent is this task? We have a task open for procuring new hardware for (among other servers) terbium (T189317), but we could figure out a different solution if this needs to happen sooner than that.

We also have the two former image scalers which are under warranty (and even more powerful than terbium). The plan is to allocate them as mw* hosts later on (T192457), but we could just as well repurpose one of them as the stretch-based standin for terbium (since we need to have both in parallel for a small time window).

I fully agree, let's do that.

faidon added a comment.Apr 24 2018, 10:49 AM

Let's just use both of them to also set up the stand-in that you mentioned above?

(approved)

Dzahn claimed this task.Apr 24 2018, 2:28 PM
Dzahn lowered the priority of this task from High to Medium.
gerritbot added a comment.May 3 2018, 12:40 AM

Change 430518 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] rename wmf6936 from mw1297 to mwmaint1001

https://gerrit.wikimedia.org/r/430518

gerritbot added a project: Patch-For-Review.May 3 2018, 12:40 AM
gerritbot added a comment.May 3 2018, 12:54 AM

Change 430519 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] rename mw1297 to mwmaint1001, assign mw-maint role

https://gerrit.wikimedia.org/r/430519

Dzahn mentioned this in T192092: setup replacements for maintenance_server (terbium, wasat) on Stretch.May 3 2018, 1:00 AM
RobH renamed this task from request to assign WMF3565 as terbium equivalent to request to assign spare systems as terbium equivalent.May 3 2018, 3:07 PM
gerritbot added a comment.May 3 2018, 6:44 PM

Change 430519 merged by Dzahn:
[operations/puppet@production] rename mw1297 to mwmaint1001, partman for mwmaint*

https://gerrit.wikimedia.org/r/430519

gerritbot added a comment.May 3 2018, 6:55 PM

Change 430518 merged by Dzahn:
[operations/dns@master] rename wmf6936 from mw1297 to mwmaint1001

https://gerrit.wikimedia.org/r/430518

Stashbot subscribed.May 3 2018, 7:18 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-03T19:18:42Z] <mutante> mw1297 - puppet node clean, puppet node deactivate - renaming to mwmaint1001 (T192185)

Dzahn added a comment.May 3 2018, 8:20 PM

wmf6936 (mw1297) assigned and renamed to mwmaint1001

i renamed in racktables and left a comment there too

racktables object 3003

https://racktables.wikimedia.org/index.php?page=object&tab=edit&object_id=3003

from here setup will continue on T192092

@RobH I think this hardware request can be closed as resolved, unless there is another place where you are keeping track of those?

Dzahn reassigned this task from Dzahn to RobH.May 3 2018, 8:21 PM

P.S. I also added the "mwmaint" prefix on https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions

RobH closed this task as Resolved.May 3 2018, 8:28 PM
• Cmjohnson closed subtask T193798: change hostname label for mw1297 to mwmaint1001 as Resolved.May 4 2018, 2:49 PM
• Vvjjkkii reopened subtask T193798: change hostname label for mw1297 to mwmaint1001 as Open.Jul 1 2018, 1:12 AM
TerraCodes closed subtask T193798: change hostname label for mw1297 to mwmaint1001 as Resolved.Jul 2 2018, 4:49 PM
gerritbot added a comment.Oct 10 2018, 7:32 PM

Change 465689 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] Revert "rename wmf6936 from mw1297 to mwmaint1001"

https://gerrit.wikimedia.org/r/465689

gerritbot added a comment.Oct 11 2018, 8:27 PM

Change 465689 abandoned by Dzahn:
Revert "rename wmf6936 from mw1297 to mwmaint1001"

Reason:
cant rebase cleanly and for some reason "fatal: Couldn't find remote ref refs/changes/89/465689/2" for me right now

https://gerrit.wikimedia.org/r/465689

gerritbot added a comment.Oct 11 2018, 8:27 PM

Change 465689 restored by Dzahn:
Revert "rename wmf6936 from mw1297 to mwmaint1001"

https://gerrit.wikimedia.org/r/465689

gerritbot added a comment.Oct 11 2018, 8:45 PM

Change 466773 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] re-add mw1297 to site.pp and DHCP, formerly mwmaint1001

https://gerrit.wikimedia.org/r/466773

gerritbot added a comment.Oct 11 2018, 9:37 PM

Change 466773 merged by Dzahn:
[operations/puppet@production] re-add mw1297 to site.pp and DHCP, remove mwmaint1001

https://gerrit.wikimedia.org/r/466773

gerritbot added a comment.Oct 11 2018, 10:48 PM

Change 465689 merged by Dzahn:
[operations/dns@master] Revert "rename wmf6936 from mw1297 to mwmaint1001"

https://gerrit.wikimedia.org/r/465689

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits