Page MenuHomePhabricator

Remove support for precise OGE exec hosts
Closed, ResolvedPublic

Description

Migrate all precise jobs on the job grid to trusty in several steps.

Announcements

Details

Timeline

  • late August 2016: Jobs started without -l release=... and webservices started with --release=precise will print a warning.
  • mid October 2016: Jobs started with jsub will no longer run on Precise hosts by default, and will instead run on Trusty hosts. This means jobs running via cron will migrate to trusty automatically.
  • October 2016-January 2017: Tools starting jobs with jsub -l release=precise will start receiving emails urging them to migrate to Trusty.
  • March 14 2017: Jobs started with jsub -l release=precise will no longer function and will crash with an error.

Progress

Event Timeline

yuvipanda raised the priority of this task from to Normal.
yuvipanda updated the task description. (Show Details)
yuvipanda added a project: Toolforge.
yuvipanda added subscribers: faidon, Aklapper, coren and 2 others.
scfc moved this task from Triage to Backlog on the Toolforge board.Apr 6 2015, 8:19 AM
coren added a comment.Dec 9 2015, 4:31 PM

I think this is now a reasonable step; but we probably want a month's warning or so. I'm going to make an annoucement about this - the change itself is trivial enough.

During the last discussion on this, @scfc suggested to skip trusty and go to jessie immediately. That limits the number of times users are forced to upgrade.

On the other hand, our bastions are trusty (except for precise-dev), so it's odd to run jobs on precise while everyone tests stuff on trusty...

coren added a comment.Dec 9 2015, 4:41 PM

Jessie is a no-starter while we rely on gridengine; which is going to be for a while still (k8s provides a superior alternative for many, but not all, scenarios and migration in that direction by users is going to be gradual).

Let's try this again, maybe. I'll send out a message saying this is going to happen in like a month.

yuvipanda removed yuvipanda as the assignee of this task.Aug 10 2016, 12:54 AM

un cookie licking right now

bd808 claimed this task.Aug 15 2016, 4:20 PM
bd808 added a subscriber: bd808.

Updated the draft plan that @yuvipanda had written at https://wikitech.wikimedia.org/wiki/Tools_Precise_deprecation with new dates.

Restricted Application added a project: User-bd808. · View Herald TranscriptAug 15 2016, 4:20 PM
bd808 updated the task description. (Show Details)
bd808 set Security to None.
bd808 added a comment.Oct 26 2016, 5:05 PM

Switched default to trusty at 2016-10-26T16:48Z

bd808 added a comment.Oct 26 2016, 5:12 PM

With the default switched we are now in the long tail phase of prodding people who have pinned to -l release=precise to switch. We need to create a nag system that looks at the precise job runners once a week or so, makes a list of running processes, maps them to tools, and emails maintainers.

bd808 renamed this task from Make jsub / qsub default to trusty instances to Remove support for precise OGE exec hosts.Oct 27 2016, 7:15 PM
bd808 updated the task description. (Show Details)
bd808 moved this task from Triage to In Progress on the Cloud-Services board.Nov 24 2016, 9:42 PM

Change 335569 had a related patch set uploaded (by BryanDavis):
Ignore lighttpd-precise in service.manifest

https://gerrit.wikimedia.org/r/335569

bd808 updated the task description. (Show Details)Feb 6 2017, 6:31 PM

Change 335569 abandoned by BryanDavis:
Ignore lighttpd-precise in service.manifest

https://gerrit.wikimedia.org/r/335569

scfc added a comment.Feb 22 2017, 1:14 AM

jmail (T158722) submits jobs by:

/usr/bin/qsub -N mail.$(/usr/bin/id -nu) \
        -sync y -b y -m n \
        -o "$email.out" -j y -i "$email" \
        -q mailq -l h_vmem=500M -r n \
        "$exe" "$@" >/dev/null

Currently this causes jobs like mail.tools.drtrigonbot to be run (sometimes) on Precise hosts. I believe this is not a stated requirement and just depends on which instance has the lowest load, i. e. will switch automatically to Trusty when the Precise hosts are removed, but is probably worth a look after the switch.

bd808 updated the task description. (Show Details)Mar 6 2017, 7:49 PM
bd808 moved this task from To Do to In Dev/Progress on the User-bd808 board.Mar 7 2017, 9:31 PM

Change 341666 merged by jenkins-bot:
[labs/toollabs] jsub: Remove support for release=precise

https://gerrit.wikimedia.org/r/341666

Change 342061 merged by jenkins-bot:
[operations/software/tools-webservice] Remove support for Precise

https://gerrit.wikimedia.org/r/342061

bd808 updated the task description. (Show Details)
bd808 closed this task as Resolved.Mar 14 2017, 11:03 PM

Change 342161 merged by Rush:
[operations/puppet] toolschecker: remove precise checks

https://gerrit.wikimedia.org/r/342161