Page MenuHomePhabricator

Reduce Precise OGE exec hosts to 5
Closed, ResolvedPublic

Description

Follow the same basic plan that was used in T151980: Reduce Precise OGE exec hosts to 10 to reduce from the current 10 precise exec hosts to 5.

Event Timeline

bd808 created this task.Jan 3 2017, 11:21 PM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJan 3 2017, 11:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Mentioned in SAL (#wikimedia-labs) [2017-01-03T23:24:41Z] <bd808> drained tools-exec-1212 (T154539)

Mentioned in SAL (#wikimedia-labs) [2017-01-03T23:25:15Z] <bd808> drained tools-exec-1213 (T154539)

Mentioned in SAL (#wikimedia-labs) [2017-01-03T23:25:52Z] <bd808> drained tools-exec-1214 (T154539)

Mentioned in SAL (#wikimedia-labs) [2017-01-03T23:26:36Z] <bd808> drained tools-exec-1215 (T154539)

Mentioned in SAL (#wikimedia-labs) [2017-01-03T23:27:18Z] <bd808> drained tools-exec-1216 (T154539)

bd808 added a comment.Jan 3 2017, 11:30 PM

The queues are disabled on tools-exec-1212 through tools-exec-1216. All continuous jobs have been rescheduled using qmod -rj. The nodes are now empty and ready for steps 2-7 from https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin#Decommission_a_node

Mentioned in SAL (#wikimedia-labs) [2017-01-03T23:56:08Z] <bd808> Removed tools-exec-12[12-16] from gridengine (T154539)

Change 330333 had a related patch set uploaded (by BryanDavis):
toollabs: remove host aliases for tools-exec-121[2-6]

https://gerrit.wikimedia.org/r/330333

bd808 added a comment.Jan 4 2017, 12:03 AM

Sanity check to prevent a repeat of T149634#2758566:

tools-bastion-02.tools:~
bd808$ sudo qconf -sel|grep -- -12
tools-exec-1217.eqiad.wmflabs
tools-exec-1218.eqiad.wmflabs
tools-exec-1219.eqiad.wmflabs
tools-exec-1220.tools.eqiad.wmflabs
tools-exec-1221.tools.eqiad.wmflabs
tools-webgrid-lighttpd-1201.eqiad.wmflabs
tools-webgrid-lighttpd-1202.eqiad.wmflabs
tools-webgrid-lighttpd-1203.eqiad.wmflabs
tools-webgrid-lighttpd-1204.eqiad.wmflabs
tools-webgrid-lighttpd-1205.eqiad.wmflabs
tools-webgrid-lighttpd-1206.eqiad.wmflabs
tools-webgrid-lighttpd-1207.eqiad.wmflabs
tools-webgrid-lighttpd-1208.eqiad.wmflabs
tools-webgrid-lighttpd-1209.eqiad.wmflabs
tools-webgrid-lighttpd-1210.eqiad.wmflabs
bd808 claimed this task.Jan 4 2017, 12:09 AM
Restricted Application added a project: User-bd808. · View Herald TranscriptJan 4 2017, 12:09 AM
bd808 moved this task from Triage to In Progress on the Toolforge board.Jan 4 2017, 12:09 AM

Change 330333 merged by Andrew Bogott:
toollabs: remove host aliases for tools-exec-121[2-6]

https://gerrit.wikimedia.org/r/330333

Mentioned in SAL (#wikimedia-labs) [2017-01-06T19:02:42Z] <bd808> Terminated deprecated instances tools-exec-121[2-6] (T154539)

bd808 moved this task from To Do to Done on the User-bd808 board.Jan 7 2017, 12:18 AM