Remove support for precise OGE exec hosts
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	yuvipanda
	Apr 2 2015, 1:23 AM

Description

Migrate all precise jobs on the job grid to trusty in several steps.

Announcements

Details

https://wikitech.wikimedia.org/wiki/Tools_Precise_deprecation

Timeline

late August 2016: Jobs started without -l release=... and webservices started with --release=precise will print a warning.
mid October 2016: Jobs started with jsub will no longer run on Precise hosts by default, and will instead run on Trusty hosts. This means jobs running via cron will migrate to trusty automatically.
October 2016-January 2017: Tools starting jobs with jsub -l release=precise will start receiving emails urging them to migrate to Trusty.
March 14 2017: Jobs started with jsub -l release=precise will no longer function and will crash with an error.

Progress

Graphite graph of Precise vs Trusty jobs

oge-precise-vs-trusty.png (300×600 px, 28 KB)

Details

Subject	Repo	Branch	Lines +/-
toolschecker: remove precise checks	operations/puppet	production	+0 -24
Remove support for Precise	operations/software/tools-webservice	master	+24 -24
jsub: Remove support for release=precise	labs/toollabs	master	+21 -24
Ignore lighttpd-precise in service.manifest	operations/software/tools-webservice	master	+20 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Andrew	T143349 Deprecate precise instances in Labs by 2017-03-31
Resolved	yuvipanda	T94790 Phase out precise instances from Tool Labs
Resolved	bd808	T94792 Remove support for precise OGE exec hosts
Resolved	bd808	T143284 Make trusty the default release target for jsub commands
Resolved	bd808	T143282 Make jsub warn when run without `-l release=...`
Resolved	Andrew	T147205 Create more trusty nodes in anticipation of the default for jsub switching to trusty
Resolved	yuvipanda	T146322 Increase quota for tools project
Resolved	• madhuvishy	T149214 Make a nag system to email maintainers of tools still running on precise grid hosts
Resolved	zhuyifei1999	T156386 Create a wall for tools migration to trusty
Resolved	MusikAnimal	T157123 Migrate XTools from Ubuntu Precise to Trusty
Resolved	bd808	T160442 Decommission all tools-webgrid-lighttpd-12* hosts
Resolved	bd808	T160457 Decommission all tools-exec-12* hosts
Resolved	bd808	T160461 Decommission tools-exec-gift.eqiad.wmflabs

Event Timeline

yuvipanda created this task.Apr 2 2015, 1:23 AM

yuvipanda raised the priority of this task from to Medium.

yuvipanda updated the task description. (Show Details)

yuvipanda added a project: Toolforge.

yuvipanda added subscribers: faidon, Aklapper, coren and 2 others.

scfc moved this task from Backlog to Ready to be worked on on the Toolforge board.Apr 6 2015, 8:19 AM

Aklapper added a project: Cloud-Services.Oct 24 2015, 8:02 PM

I think this is now a reasonable step; but we probably want a month's warning or so. I'm going to make an annoucement about this - the change itself is trivial enough.

During the last discussion on this, @scfc suggested to skip trusty and go to jessie immediately. That limits the number of times users are forced to upgrade.

On the other hand, our bastions are trusty (except for precise-dev), so it's odd to run jobs on precise while everyone tests stuff on trusty...

Jessie is a no-starter while we rely on gridengine; which is going to be for a while still (k8s provides a superior alternative for many, but not all, scenarios and migration in that direction by users is going to be gradual).

Ricordisamoa subscribed.Dec 29 2015, 5:49 PM

Let's try this again, maybe. I'll send out a message saying this is going to happen in like a month.

yuvipanda claimed this task.Apr 12 2016, 4:09 PM

tom29739 awarded a token.Apr 12 2016, 4:10 PM

un cookie licking right now

Updated the draft plan that @yuvipanda had written at https://wikitech.wikimedia.org/wiki/Tools_Precise_deprecation with new dates.

Restricted Application added a project: User-bd808. · View Herald TranscriptAug 15 2016, 4:20 PM

Announced deprecation on labs-announce: https://lists.wikimedia.org/pipermail/labs-announce/2016-August/000144.html

bd808 created subtask T143282: Make jsub warn when run without `-l release=...`.Aug 17 2016, 10:48 PM

bd808 created subtask T143283: Make webservice warn when run with `-l release=precise`.Aug 17 2016, 10:51 PM

bd808 created subtask T143284: Make trusty the default release target for jsub commands.Aug 17 2016, 10:59 PM

bd808 added a project: Epic.Aug 17 2016, 11:04 PM

bd808 updated the task description. (Show Details)

bd808 set Security to None.

bd808 removed a subtask: T143283: Make webservice warn when run with `-l release=precise`.Aug 17 2016, 11:07 PM

bd808 closed subtask T143282: Make jsub warn when run without `-l release=...` as Resolved.Sep 8 2016, 10:56 PM

Announced planned date (week of 2016-10-17) for switch of default from Precise to Trusty on labs-announce: https://lists.wikimedia.org/pipermail/labs-announce/2016-October/000159.html

Switched default to trusty at 2016-10-26T16:48Z

bd808 closed subtask T143284: Make trusty the default release target for jsub commands as Resolved.Oct 26 2016, 5:05 PM

bd808 closed subtask T147205: Create more trusty nodes in anticipation of the default for jsub switching to trusty as Resolved.Oct 26 2016, 5:10 PM

With the default switched we are now in the long tail phase of prodding people who have pinned to -l release=precise to switch. We need to create a nag system that looks at the precise job runners once a week or so, makes a list of running processes, maps them to tools, and emails maintainers.

bd808 created subtask T149214: Make a nag system to email maintainers of tools still running on precise grid hosts.Oct 26 2016, 5:17 PM

bd808 renamed this task from Make jsub / qsub default to trusty instances to Remove support for precise OGE exec hosts.Oct 27 2016, 7:15 PM

bd808 updated the task description. (Show Details)

bd808 moved this task from Triage to In Progress on the Cloud-Services board.Nov 24 2016, 9:42 PM

hashar mentioned this in T139738: Install Arcanist in toollabs::dev_environ.Dec 12 2016, 9:14 AM

bd808 mentioned this in T156075: Figure out how communications and meetings will work for the Tool Labs standards committee.Jan 23 2017, 10:57 PM

zhuyifei1999 created subtask T156386: Create a wall for tools migration to trusty.Jan 26 2017, 4:57 PM

Change 335569 had a related patch set uploaded (by BryanDavis):
Ignore lighttpd-precise in service.manifest

https://gerrit.wikimedia.org/r/335569

gerritbot added a project: Patch-For-Review.Feb 1 2017, 11:31 PM

zhuyifei1999 added a subtask: T157123: Migrate XTools from Ubuntu Precise to Trusty.Feb 3 2017, 3:16 PM

bd808 updated the task description. (Show Details)Feb 6 2017, 6:31 PM

scfc closed subtask T156386: Create a wall for tools migration to trusty as Resolved.Feb 7 2017, 2:58 PM

scfc closed subtask T149214: Make a nag system to email maintainers of tools still running on precise grid hosts as Resolved.Feb 7 2017, 3:42 PM

MusikAnimal closed subtask T157123: Migrate XTools from Ubuntu Precise to Trusty as Resolved.Feb 7 2017, 5:34 PM

Matthewrbowker reopened subtask T157123: Migrate XTools from Ubuntu Precise to Trusty as Open.Feb 8 2017, 10:25 PM

MusikAnimal closed subtask T157123: Migrate XTools from Ubuntu Precise to Trusty as Resolved.Feb 8 2017, 11:39 PM

Change 335569 abandoned by BryanDavis:
Ignore lighttpd-precise in service.manifest

https://gerrit.wikimedia.org/r/335569

jmail (T158722) submits jobs by:

/usr/bin/qsub -N mail.$(/usr/bin/id -nu) \
        -sync y -b y -m n \
        -o "$email.out" -j y -i "$email" \
        -q mailq -l h_vmem=500M -r n \
        "$exe" "$@" >/dev/null

Currently this causes jobs like mail.tools.drtrigonbot to be run (sometimes) on Precise hosts. I believe this is not a stated requirement and just depends on which instance has the lowest load, i. e. will switch automatically to Trusty when the Precise hosts are removed, but is probably worth a look after the switch.

bd808 updated the task description. (Show Details)Mar 6 2017, 7:49 PM

bd808 moved this task from To Do to In Dev/Progress on the User-bd808 board.Mar 7 2017, 9:31 PM

bd808 moved this task from Backlog to Doing on the Community-Tech-Tool-Labs board.Mar 10 2017, 12:17 AM

Change 341666 merged by jenkins-bot:
[labs/toollabs] jsub: Remove support for release=precise

https://gerrit.wikimedia.org/r/341666

Change 342061 merged by jenkins-bot:
[operations/software/tools-webservice] Remove support for Precise

https://gerrit.wikimedia.org/r/342061

bd808 created subtask T160442: Decommission all tools-webgrid-lighttpd-12* hosts.Mar 14 2017, 5:54 PM

bd808 created subtask T160457: Decommission all tools-exec-12* hosts.Mar 14 2017, 7:09 PM

bd808 created subtask T160461: Decommission tools-exec-gift.eqiad.wmflabs.Mar 14 2017, 8:06 PM

bd808 closed subtask T160457: Decommission all tools-exec-12* hosts as Resolved.Mar 14 2017, 8:29 PM

bd808 closed subtask T160442: Decommission all tools-webgrid-lighttpd-12* hosts as Resolved.Mar 14 2017, 8:45 PM