Currently there is a custom node and a custom queue was created for this purpose. There is a very real chance we can collapse this added complexity and rely on standard grid scheduling but we need to verify all the things work.
Current node:
id 7e79e49d-b540-459b-a59e-3faa934d730e flavor m1.medium (3) image ubuntu-12.04-precise
Most of this process is outlined on https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin/new_exec_host
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T143349 Deprecate precise instances in Labs by 2017-03-31 | |||
| Resolved | yuvipanda | T94790 Phase out precise instances from Tool Labs | |||
| Declined | None | T99130 Investigate alternatives to dedicated exec node for gifti's tools | |||
| Resolved | Giftpflanze | T156981 Find a solution for tools-exec-gift on Trusty |
@Giftpflanze if we stand up a Trusty host with the same characteristics as the existing precise node are you up for migrating things? That is our current thinking. Transition the existing scheme. I don't think any of us have the insight into how this all works to do this transition in any timely manner. It could be fairly benign depending on precise version(s) reliance.
< annika> chasemp: […] I guess you mean: You will provide a trusty exec node with an appropriate queue? And I will just use that instead of the current one?
< chasemp> annika: basically, a second trusty node in the same queue for you to migrate to
< annika> that would be fine
details on existing
tools-exec-gift ubuntu-12.04-precise (deprecated 2014-04-17) 10.68.16.40 m1.medium - Active nova None Running 2 years, 11 months
tools-exec-gift
ID
7e79e49d-b540-459b-a59e-3faa934d730e
Status
Active
Availability Zone
nova
Created
Feb. 28, 2014, 4:40 a.m.
Time Since Created
2 years, 11 months
Host
labvirt1002
Specs
Flavor Name
m1.medium
Flavor ID
3
RAM
4GB
VCPUs
2 VCPU
Disk
40GB
IP Addresses
Public
10.68.16.40
Security Groups
default
ALLOW -1:-1/icmp from 0.0.0.0/0
ALLOW 5666:5666/tcp from 10.0.0.0/8
ALLOW 22:22/tcp from 0.0.0.0/0
ALLOW 60000:61000/udp from 0.0.0.0/0
ALLOW 6666:6666/tcp from 0.0.0.0/0
ALLOW 8090:8090/tcp from 0.0.0.0/0
ALLOW 1:65535/tcp from default
ALLOW 1:65535/udp from default
execnode
ALLOW 6444:6445/tcp from 10.4.0.0/21
ALLOW 6444:6445/udp from 10.4.0.0/21
ALLOW 113:113/tcp from 0.0.0.0/0
ALLOW 3000:3000/tcp from 10.4.0.0/21
Metadata
Key Name
None
Image Name
ubuntu-12.04-precise (deprecated 2014-04-17)
Image ID
ff0a06ae-e7bf-4533-a3aa-176c366fdb4a
puppetstatus
changed
puppettimestamp
1429716563
project-id
tools
Volumes Attached
Volume
No volumes attached.I created tools-exec-gift-trusty.tools.eqiad.wmflabs but it seems to be having issues with puppet certificates. I haven't figured out why yet, but I imagine it's connected to the project specific master.
Actually, it looks like it may be a somewhat known issue:
hostname: Name or service not known
ok well, I created tools-exec-gift-trusty-01 successfully but it has failed to migrate to the tools specific master :)
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: tools-puppetmaster-02.tools.eqiad.wmflabs] Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: tools-puppetmaster-02.tools.eqiad.wmflabs] root@tools-exec-gift-trusty-01:~#
handled the cert issue via https://wikitech.wikimedia.org/wiki/Standalone_puppetmaster#Step_2:_Setup_a_puppet_client :)
OK @Giftpflanze.
tools-exec-gift-trusty-01.tools.eqiad.wmflabs
qconf -Ae /var/lib/gridengine/etc/exechosts/tools-exec-gift.tools.eqiad.wmflabs
exechost "tools-exec-gift.eqiad.wmflabs" already exists
qconf -mq giftbot
qname giftbot
hostlist tools-exec-gift.eqiad.wmflabs \
tools-exec-gift-trusty-01.tools.eqiad.wmflabsroot@tools-bastion-03.tools.eqiad.wmflabs modified "giftbot" in cluster queue list
/usr/bin/qconf -sel | grep -e gift | grep trusty
tools-exec-gift-trusty-01.tools.eqiad.wmflabs
qmod -e "*@tools-exec-gift-trusty-01.tools.eqiad.wmflabs"
Queue instance "giftbot@tools-exec-gift-trusty-01.tools.eqiad.wmflabs" is already in the specified state: enabled
qhost -q -h tools-exec-gift-trusty-01.tools.eqiad.wmflabs
HOSTNAME ARCH NCPU LOAD MEMTOT MEMUSE SWAPTO SWAPUS ------------------------------------------------------------------------------- global - - - - - - - tools-exec-gift-trusty-01.tools.eqiad.wmflabs - - - - - - - giftbot BC 0/0/1000 au
@Giftpflanze friendly ping this is ready for you to migrate off Precise, or let us know what's not working out. 35 days about until we have to turn off the existing tools-exec-gift
Killing the Precise instance (and defaulting to Trusty) shouldn't pose a problem. I switched my code to Trusty (I didn't even know anymore that I had to hardcode that) and I will test it with the next regular run starting on the 1st of March. (Also, I'm kinda swamped with school stuff but I'm glad to have found a minute for this.)
A note from yesterday:
annika: chasemp: can you take a look at job 1820616, please? it doesn't schedule on tools-exec-gift-trusty-01.tools.eqiad.wmflabs and i don't exactly see why
root@tools-bastion-03:~# qstat -j 1820616
==============================================================
job_number: 1820616
exec_file: job_scripts/1820616
submission_time: Wed Mar 1 00:07:04 2017
owner: tools.giftbot
uid: 51072
group: tools.giftbot
gid: 51072
sge_o_home: /data/project/giftbot
sge_o_log_name: tools.giftbot
sge_o_path: /usr/bin:/bin
sge_o_shell: /bin/sh
sge_o_tz: Europe/Berlin
sge_o_workdir: /mnt/nfs/labstore-secondary-tools-project/giftbot
sge_o_host: tools-cron-01
account: sge
stderr_path_list: NONE:NONE:dwl01.out-20170301
merge: y
hard resource_list: h_vmem=524288k,release=trusty
mail_list: tools.giftbot@tools.wmflabs.org
notify: FALSE
job_name: dwl01
stdout_path_list: NONE:NONE:dwl01.out-20170301
jobshare: 0
hard_queue_list: giftbot
env_list:
script_file: /mnt/nfs/labstore-secondary-tools-project/giftbot/dwl01.tcl
scheduling info: queue instance "webgrid-lighttpd@tools-webgrid-lighttpd-1201.eqiad.wmflabs" dropped because it is disabled
cannot run in queue "webgrid-lighttpd" because it is not contained in its hard queue list (-q)
(-l h_vmem=524288k,release=trusty) cannot run at host "tools-exec-gift.eqiad.wmflabs" because it offers only hf:release=precise
cannot run in queue "mailq" because it is not contained in its hard queue list (-q)
cannot run in queue "task" because it is not contained in its hard queue list (-q)
cannot run in queue "continuous" because it is not contained in its hard queue list (-q)
(-l h_vmem=524288k,release=trusty) cannot run in queue "giftbot@tools-exec-gift-trusty-01.tools.eqiad.wmflabs" because job requests unknown resource (release)
cannot run in queue "webgrid-generic" because it is not contained in its hard queue list (-q)< yuvipanda> !log tools set complex_values slots=300,release=trusty for tools-exec-gift-trusty-01.tools.eqiad.wmflabs
This seems to have done the trick, thank you!
@chasemp The old instance can be cleaned up now.