Page MenuHomePhabricator
Create Task
Maniphest T278748

Toolforge: introduce support for selecting grid queue release
Closed, ResolvedPublic
Actions

Description

In the past we used to have a selector for the grid, which users could use to select the exec environment operating system.

When we migrated to Debian Stretch we mostly drop support for that, but we need it back to better handle the Stretch -> Buster migration.

Details

SubjectRepoBranchLines +/-
Replace distro with os releaseoperations/software/tools-webservicemaster+11 -6
Do not compare OS versionsoperations/software/tools-manifestmaster+2 -7
tests/: update tests with the new default release argumentlabs/toollabsmaster+14 -14
jsub: split_l(): the generator should return silentlylabs/toollabsmaster+1 -1
gridengine: cleaner release defaultoperations/software/tools-webservicemaster+10 -10
Revert "Remove 'release' qsub label"operations/software/tools-webservicemaster+10 -5
Revert "jsub: Make release a deprecated noop"labs/toollabsmaster+17 -10
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
Restricted Task
 Resolved• BstormT169289 Tool Labs 2017-06-29 Labstore100[45] kernel upgrade issues
 Resolved• MoritzMuehlenhoffT169290 New anti-stackclash (4.9.25-1~bpo8+3 ) kernel super bad for NFS
 Resolved• BstormT203254 labstore1004 and labstore1005 high load issues following upgrades
 OpenNoneT257945 NFS v4.1/2 as possible fix for elevated load and lock contention on our NFS servers
 ResolvedAndrewT289883 cloud-vps: actively deprecate/phase out use of Debian Stretch
 Resolved• taaviT306099 Cloud VPS "tools" project Stretch deprecation
 Resolved• taaviT306100 Cloud VPS "toolsbeta" project Stretch deprecation
 Resolved• taaviT275864 Toolforge: migrate to Debian Buster or later
 ResolvedhasharT249949 Stop using integration/composer and then archive the repo
 Resolved• taaviT287900 Switch Toolforge installation of "composer" to use the Debian package
 StalledNoneT305834 Cloud VPS: drop wmflabs names from profile::resolving::domain_search
 ResolvedaborreroT277653 Toolforge: add Debian Buster to the grid and eliminate Debian Stretch
 ResolvedaborreroT278748 Toolforge: introduce support for selecting grid queue release
 ResolvedaborreroT282972 "--release is not implemented for --backend=kubernetes" with latest tools-webservice from Git
 Resolved• taaviT288961 /usr/local/bin/webservice:109: DeprecationWarning: dist() and linux_distribution() functions are deprecated in Python 3.5
 ResolvedaborreroT300501 Toolforge grid: uwsgi in buster fails to load python3 venvs

Event Timeline

aborrero triaged this task as Medium priority.Mar 30 2021, 11:07 AM
aborrero created this task.
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
gerritbot added a comment.Mar 30 2021, 11:10 AM

Change 675733 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[labs/toollabs@master] Revert "jsub: Make release a deprecated noop"

https://gerrit.wikimedia.org/r/675733

gerritbot added a project: Patch-For-Review.Mar 30 2021, 11:10 AM
gerritbot added a comment.Mar 30 2021, 11:30 AM

Change 675734 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/software/tools-webservice@master] Revert "Remove 'release' qsub label"

https://gerrit.wikimedia.org/r/675734

gerritbot added a comment.Mar 31 2021, 10:00 AM

Change 675733 merged by Arturo Borrero Gonzalez:

[labs/toollabs@master] Revert "jsub: Make release a deprecated noop"

https://gerrit.wikimedia.org/r/675733

gerritbot added a comment.Mar 31 2021, 10:04 AM

Change 675734 merged by Arturo Borrero Gonzalez:

[operations/software/tools-webservice@master] Revert "Remove 'release' qsub label"

https://gerrit.wikimedia.org/r/675734

Maintenance_bot removed a project: Patch-For-Review.Mar 31 2021, 10:10 AM
gerritbot added a comment.Mar 31 2021, 10:52 AM

Change 676033 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/software/tools-webservice@master] gridengine: cleaner release default

https://gerrit.wikimedia.org/r/676033

gerritbot added a project: Patch-For-Review.Mar 31 2021, 10:52 AM
gerritbot added a comment.Apr 5 2021, 11:44 AM

Change 676033 merged by Arturo Borrero Gonzalez:

[operations/software/tools-webservice@master] gridengine: cleaner release default

https://gerrit.wikimedia.org/r/676033

gerritbot added a comment.Apr 5 2021, 5:55 PM

Change 676987 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/toollabs@master] jsub: split_l(): the generator should return silently

https://gerrit.wikimedia.org/r/676987

gerritbot added a comment.Apr 5 2021, 6:02 PM

Change 676989 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/toollabs@master] tests/: update tests with the new default release argument

https://gerrit.wikimedia.org/r/676989

gerritbot added a comment.Apr 6 2021, 9:11 AM

Change 676987 merged by Arturo Borrero Gonzalez:

[labs/toollabs@master] jsub: split_l(): the generator should return silently

https://gerrit.wikimedia.org/r/676987

gerritbot added a comment.Apr 6 2021, 9:11 AM

Change 676989 merged by Arturo Borrero Gonzalez:

[labs/toollabs@master] tests/: update tests with the new default release argument

https://gerrit.wikimedia.org/r/676989

Stashbot added a comment.Apr 6 2021, 10:21 AM

Mentioned in SAL (#wikimedia-cloud) [2021-04-06T10:21:41Z] <arturo> published jobutils & misctools 1.42 (T278748)

• taavi added a subtask: T282972: "--release is not implemented for --backend=kubernetes" with latest tools-webservice from Git.May 16 2021, 1:50 PM
• taavi closed subtask T282972: "--release is not implemented for --backend=kubernetes" with latest tools-webservice from Git as Resolved.May 17 2021, 10:11 AM
• taavi added a comment.Jun 10 2021, 1:38 PM

Is there anything left to do here, other than a tools-webservice release?

• taavi added a comment.Jun 12 2021, 5:41 PM
In T278748#7148731, @Majavah wrote:

Is there anything left to do here, other than a tools-webservice release?

There is: we need to add release selection support to https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/tools-manifest/+/refs/heads/master/tools/manifest/webservicemonitor.py as well, which might need some changes in tools-webservice.

aborrero removed aborrero as the assignee of this task.Jul 21 2021, 3:45 PM
• taavi added a subtask: T288961: /usr/local/bin/webservice:109: DeprecationWarning: dist() and linux_distribution() functions are deprecated in Python 3.5.Aug 18 2021, 4:54 PM
gerritbot added a comment.Aug 18 2021, 5:13 PM

Change 713661 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/software/tools-webservice@master] Replace distro with os release

https://gerrit.wikimedia.org/r/713661

gerritbot added a comment.Aug 18 2021, 5:29 PM

Change 713663 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/software/tools-manifest@master] Take OS codename into account for grid compatibility

https://gerrit.wikimedia.org/r/713663

gerritbot added a comment.Aug 31 2021, 7:05 AM

Change 713663 merged by jenkins-bot:

[operations/software/tools-manifest@master] Do not compare OS versions

https://gerrit.wikimedia.org/r/713663

gerritbot added a comment.Sep 2 2021, 8:38 AM

Change 713661 merged by jenkins-bot:

[operations/software/tools-webservice@master] Replace distro with os release

https://gerrit.wikimedia.org/r/713661

• taavi closed subtask T288961: /usr/local/bin/webservice:109: DeprecationWarning: dist() and linux_distribution() functions are deprecated in Python 3.5 as Resolved.Sep 2 2021, 8:43 AM
• taavi added a comment.Oct 22 2021, 3:05 PM
In T284767#7451394, @Majavah wrote:

Found a related note in profile::toolforge::grid::cronrunner:

# We need to include exec environment here since the current
# version of jsub checks the local environment to find the full
# path to things before submitting them to the grid. This assumes
# that jsub is always run in an environment identical to the exec
# nodes. This is kind of terrible, so we need to fix that eventually.
# Until then...
include profile::toolforge::grid::exec_environ
aborrero added a comment.Jan 27 2022, 1:17 PM

This is done. The problem that @Majavah mentions will be tracked separatedly in T284767: Toolforge: migrate cron servers to Debian Buster.

aborrero closed this task as Resolved.Jan 31 2022, 12:46 PM
aborrero claimed this task.
aborrero changed the status of subtask T300501: Toolforge grid: uwsgi in buster fails to load python3 venvs from Open to In Progress.Jan 31 2022, 1:21 PM
aborrero closed subtask T300501: Toolforge grid: uwsgi in buster fails to load python3 venvs as Resolved.Feb 1 2022, 5:09 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits