Page MenuHomePhabricator

Decommission all tools-exec-12* hosts
Closed, ResolvedPublic

Description

$ sudo qconf -sel|grep -- tools-exec-12
tools-exec-1217.eqiad.wmflabs
tools-exec-1218.eqiad.wmflabs
tools-exec-1219.eqiad.wmflabs
tools-exec-1220.tools.eqiad.wmflabs
tools-exec-1221.tools.eqiad.wmflabs

Event Timeline

bd808 created this task.Mar 14 2017, 7:09 PM
Restricted Application added projects: Cloud-Services, User-bd808. · View Herald TranscriptMar 14 2017, 7:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
bd808 added a comment.Mar 14 2017, 7:10 PM

Disable queues:

tools-bastion-02.tools:~
bd808$ sudo qmod -d '*@tools-exec-1217.eqiad.wmflabs'
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "continuous@tools-exec-1217.eqiad.wmflabs" (disabled)
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "mailq@tools-exec-1217.eqiad.wmflabs" (disabled)
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "task@tools-exec-1217.eqiad.wmflabs" (disabled)
tools-bastion-02.tools:~
bd808$ sudo qmod -d '*@tools-exec-1218.eqiad.wmflabs'
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "continuous@tools-exec-1218.eqiad.wmflabs" (disabled)
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "mailq@tools-exec-1218.eqiad.wmflabs" (disabled)
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "task@tools-exec-1218.eqiad.wmflabs" (disabled)
tools-bastion-02.tools:~
bd808$ sudo qmod -d '*@tools-exec-1219.eqiad.wmflabs'
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "continuous@tools-exec-1219.eqiad.wmflabs" (disabled)
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "mailq@tools-exec-1219.eqiad.wmflabs" (disabled)
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "task@tools-exec-1219.eqiad.wmflabs" (disabled)
tools-bastion-02.tools:~
bd808$ sudo qmod -d '*@tools-exec-1220.tools.eqiad.wmflabs'
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "continuous@tools-exec-1220.tools.eqiad.wmflabs" (disabled)
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "mailq@tools-exec-1220.tools.eqiad.wmflabs" (disabled)
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "task@tools-exec-1220.tools.eqiad.wmflabs" (disabled)
tools-bastion-02.tools:~
bd808$ sudo qmod -d '*@tools-exec-1221.tools.eqiad.wmflabs'
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "continuous@tools-exec-1221.tools.eqiad.wmflabs" (disabled)
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "mailq@tools-exec-1221.tools.eqiad.wmflabs" (disabled)
root@tools-bastion-02.tools.eqiad.wmflabs changed state of "task@tools-exec-1221.tools.eqiad.wmflabs" (disabled)
bd808 added a comment.Mar 14 2017, 7:17 PM

Kill running jobs:

tools-bastion-02.tools:~
bd808$ sudo qdel $(qhost -j -h tools-exec-1217.eqiad.wmflabs^C
tools-bastion-02.tools:~
bd808$ sudo qdel $(qhost -j -h tools-exec-1217.eqiad.wmflabs| awk '{ print $1; }' |egrep ^[0-9])
root has registered the job 557703 for deletion
root has registered the job 9343987 for deletion
tools-bastion-02.tools:~
bd808$ sudo qdel $(qhost -j -h tools-exec-1218.eqiad.wmflabs| awk '{ print $1; }' |egrep ^[0-9])
root has registered the job 9690796 for deletion
tools-bastion-02.tools:~
bd808$ sudo qdel $(qhost -j -h tools-exec-1219.eqiad.wmflabs| awk '{ print $1; }' |egrep ^[0-9])
root has registered the job 1251004 for deletion
tools-bastion-02.tools:~
bd808$ sudo qdel $(qhost -j -h tools-exec-1220.tools.eqiad.wmflabs| awk '{ print $1; }' |egrep ^[0-9])
GE 6.2u5
usage: qdel [options] job_task_list
   [-f]                                     force action
   [-help]                                  print this help
   [-u user_list]                           delete all jobs of users specified in list
   job_task_list                            delete all jobs given in list

job_task_list           job_tasks[,job_tasks,...]
job_tasks               [job_id['.'task_id_range]|job_name|pattern][' -t 'task_id_range]
task_id_range           task_id['-'task_id[':'step]]
user_list               user[,user,...]
ERROR! no option argument
tools-bastion-02.tools:~
bd808$ sudo qdel $(qhost -j -h tools-exec-1221.tools.eqiad.wmflabs| awk '{ print $1; }' |egrep ^[0-9])
GE 6.2u5
usage: qdel [options] job_task_list
   [-f]                                     force action
   [-help]                                  print this help
   [-u user_list]                           delete all jobs of users specified in list
   job_task_list                            delete all jobs given in list

job_task_list           job_tasks[,job_tasks,...]
job_tasks               [job_id['.'task_id_range]|job_name|pattern][' -t 'task_id_range]
task_id_range           task_id['-'task_id[':'step]]
user_list               user[,user,...]
ERROR! no option argument
bd808 added a comment.Mar 14 2017, 7:20 PM

Remove from hostgroups using sudo qconf -mhgrp @general.
Also verified that none of these hosts were listed directly in any queues listed by qconf -sql.

bd808 added a comment.Mar 14 2017, 7:24 PM

Remove nodes from grid engine:

tools-bastion-02.tools:~
bd808$ sudo qconf -de tools-exec-1217.eqiad.wmflabs
root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-exec-1217.eqiad.wmflabs" from execution host list
tools-bastion-02.tools:~
bd808$ sudo qconf -de tools-exec-1218.eqiad.wmflabs
root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-exec-1218.eqiad.wmflabs" from execution host list
tools-bastion-02.tools:~
bd808$ sudo qconf -de tools-exec-1219.eqiad.wmflabs
root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-exec-1219.eqiad.wmflabs" from execution host list
tools-bastion-02.tools:~
bd808$ sudo qconf -de tools-exec-1220.tools.eqiad.wmflabs
root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-exec-1220.tools.eqiad.wmflabs" from execution host list
tools-bastion-02.tools:~
bd808$ sudo qconf -de tools-exec-1221.tools.eqiad.wmflabs
root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-exec-1221.tools.eqiad.wmflabs" from execution host list

Change 342680 merged by Andrew Bogott:
[operations/puppet] labs: Remove references to tools-exec-12*

https://gerrit.wikimedia.org/r/342680

Mentioned in SAL (#wikimedia-labs) [2017-03-14T20:27:08Z] <bd808> Disassociated floating IPs from tools-exec-12* nodes (T160457)

Mentioned in SAL (#wikimedia-labs) [2017-03-14T20:29:08Z] <bd808> Deleted tools-exec-12* nodes (T160457)

bd808 closed this task as Resolved.Mar 14 2017, 8:29 PM