$ sudo qconf -sel|grep -- tools-exec-12 tools-exec-1217.eqiad.wmflabs tools-exec-1218.eqiad.wmflabs tools-exec-1219.eqiad.wmflabs tools-exec-1220.tools.eqiad.wmflabs tools-exec-1221.tools.eqiad.wmflabs
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| labs: Remove references to tools-exec-12* | operations/puppet | production | +5 -8 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T143349 Deprecate precise instances in Labs by 2017-03-31 | |||
| Resolved | yuvipanda | T94790 Phase out precise instances from Tool Labs | |||
| Resolved | bd808 | T94792 Remove support for precise OGE exec hosts | |||
| Resolved | bd808 | T160457 Decommission all tools-exec-12* hosts |
Event Timeline
Comment Actions
Disable queues:
tools-bastion-02.tools:~ bd808$ sudo qmod -d '*@tools-exec-1217.eqiad.wmflabs' root@tools-bastion-02.tools.eqiad.wmflabs changed state of "continuous@tools-exec-1217.eqiad.wmflabs" (disabled) root@tools-bastion-02.tools.eqiad.wmflabs changed state of "mailq@tools-exec-1217.eqiad.wmflabs" (disabled) root@tools-bastion-02.tools.eqiad.wmflabs changed state of "task@tools-exec-1217.eqiad.wmflabs" (disabled) tools-bastion-02.tools:~ bd808$ sudo qmod -d '*@tools-exec-1218.eqiad.wmflabs' root@tools-bastion-02.tools.eqiad.wmflabs changed state of "continuous@tools-exec-1218.eqiad.wmflabs" (disabled) root@tools-bastion-02.tools.eqiad.wmflabs changed state of "mailq@tools-exec-1218.eqiad.wmflabs" (disabled) root@tools-bastion-02.tools.eqiad.wmflabs changed state of "task@tools-exec-1218.eqiad.wmflabs" (disabled) tools-bastion-02.tools:~ bd808$ sudo qmod -d '*@tools-exec-1219.eqiad.wmflabs' root@tools-bastion-02.tools.eqiad.wmflabs changed state of "continuous@tools-exec-1219.eqiad.wmflabs" (disabled) root@tools-bastion-02.tools.eqiad.wmflabs changed state of "mailq@tools-exec-1219.eqiad.wmflabs" (disabled) root@tools-bastion-02.tools.eqiad.wmflabs changed state of "task@tools-exec-1219.eqiad.wmflabs" (disabled) tools-bastion-02.tools:~ bd808$ sudo qmod -d '*@tools-exec-1220.tools.eqiad.wmflabs' root@tools-bastion-02.tools.eqiad.wmflabs changed state of "continuous@tools-exec-1220.tools.eqiad.wmflabs" (disabled) root@tools-bastion-02.tools.eqiad.wmflabs changed state of "mailq@tools-exec-1220.tools.eqiad.wmflabs" (disabled) root@tools-bastion-02.tools.eqiad.wmflabs changed state of "task@tools-exec-1220.tools.eqiad.wmflabs" (disabled) tools-bastion-02.tools:~ bd808$ sudo qmod -d '*@tools-exec-1221.tools.eqiad.wmflabs' root@tools-bastion-02.tools.eqiad.wmflabs changed state of "continuous@tools-exec-1221.tools.eqiad.wmflabs" (disabled) root@tools-bastion-02.tools.eqiad.wmflabs changed state of "mailq@tools-exec-1221.tools.eqiad.wmflabs" (disabled) root@tools-bastion-02.tools.eqiad.wmflabs changed state of "task@tools-exec-1221.tools.eqiad.wmflabs" (disabled)
Comment Actions
Kill running jobs:
tools-bastion-02.tools:~
bd808$ sudo qdel $(qhost -j -h tools-exec-1217.eqiad.wmflabs^C
tools-bastion-02.tools:~
bd808$ sudo qdel $(qhost -j -h tools-exec-1217.eqiad.wmflabs| awk '{ print $1; }' |egrep ^[0-9])
root has registered the job 557703 for deletion
root has registered the job 9343987 for deletion
tools-bastion-02.tools:~
bd808$ sudo qdel $(qhost -j -h tools-exec-1218.eqiad.wmflabs| awk '{ print $1; }' |egrep ^[0-9])
root has registered the job 9690796 for deletion
tools-bastion-02.tools:~
bd808$ sudo qdel $(qhost -j -h tools-exec-1219.eqiad.wmflabs| awk '{ print $1; }' |egrep ^[0-9])
root has registered the job 1251004 for deletion
tools-bastion-02.tools:~
bd808$ sudo qdel $(qhost -j -h tools-exec-1220.tools.eqiad.wmflabs| awk '{ print $1; }' |egrep ^[0-9])
GE 6.2u5
usage: qdel [options] job_task_list
[-f] force action
[-help] print this help
[-u user_list] delete all jobs of users specified in list
job_task_list delete all jobs given in list
job_task_list job_tasks[,job_tasks,...]
job_tasks [job_id['.'task_id_range]|job_name|pattern][' -t 'task_id_range]
task_id_range task_id['-'task_id[':'step]]
user_list user[,user,...]
ERROR! no option argument
tools-bastion-02.tools:~
bd808$ sudo qdel $(qhost -j -h tools-exec-1221.tools.eqiad.wmflabs| awk '{ print $1; }' |egrep ^[0-9])
GE 6.2u5
usage: qdel [options] job_task_list
[-f] force action
[-help] print this help
[-u user_list] delete all jobs of users specified in list
job_task_list delete all jobs given in list
job_task_list job_tasks[,job_tasks,...]
job_tasks [job_id['.'task_id_range]|job_name|pattern][' -t 'task_id_range]
task_id_range task_id['-'task_id[':'step]]
user_list user[,user,...]
ERROR! no option argumentComment Actions
Remove from hostgroups using sudo qconf -mhgrp @general.
Also verified that none of these hosts were listed directly in any queues listed by qconf -sql.
Comment Actions
Remove nodes from grid engine:
tools-bastion-02.tools:~ bd808$ sudo qconf -de tools-exec-1217.eqiad.wmflabs root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-exec-1217.eqiad.wmflabs" from execution host list tools-bastion-02.tools:~ bd808$ sudo qconf -de tools-exec-1218.eqiad.wmflabs root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-exec-1218.eqiad.wmflabs" from execution host list tools-bastion-02.tools:~ bd808$ sudo qconf -de tools-exec-1219.eqiad.wmflabs root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-exec-1219.eqiad.wmflabs" from execution host list tools-bastion-02.tools:~ bd808$ sudo qconf -de tools-exec-1220.tools.eqiad.wmflabs root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-exec-1220.tools.eqiad.wmflabs" from execution host list tools-bastion-02.tools:~ bd808$ sudo qconf -de tools-exec-1221.tools.eqiad.wmflabs root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-exec-1221.tools.eqiad.wmflabs" from execution host list
Comment Actions
Change 342680 merged by Andrew Bogott:
[operations/puppet] labs: Remove references to tools-exec-12*
Comment Actions
Mentioned in SAL (#wikimedia-labs) [2017-03-14T20:27:08Z] <bd808> Disassociated floating IPs from tools-exec-12* nodes (T160457)
Comment Actions
Mentioned in SAL (#wikimedia-labs) [2017-03-14T20:29:08Z] <bd808> Deleted tools-exec-12* nodes (T160457)