Page MenuHomePhabricator
Create Task
Maniphest T217152

Monitor and scale in the Trusty grid
Closed, ResolvedPublic
Actions

Description

Because we are now running two grids, NFS has to keep up with twice the number of clients. The old grid is slated for shutdown and removal in general, but keeping up with it as jobs leave the grid, removing nodes as is appropriate and possible, helps the overall Toolforge build operate better.

This is a tracking and reminder task for whenever we make passes at that effort.

Trusty grid status: https://tools.wmflabs.org/admin/oge/status

  • tools-bastion-02.tools.eqiad.wmflabs
  • tools-bastion-03.tools.eqiad.wmflabs
  • tools-bastion-05.tools.eqiad.wmflabs
  • tools-cron-01.tools.eqiad.wmflabs
  • tools-exec-1401.tools.eqiad.wmflabs
  • tools-exec-1402.tools.eqiad.wmflabs
  • tools-exec-1403.tools.eqiad.wmflabs
  • tools-exec-1404.tools.eqiad.wmflabs
  • tools-exec-1405.tools.eqiad.wmflabs
  • tools-exec-1406.tools.eqiad.wmflabs
  • tools-exec-1407.tools.eqiad.wmflabs
  • tools-exec-1408.tools.eqiad.wmflabs
  • tools-exec-1409.tools.eqiad.wmflabs
  • tools-exec-1410.tools.eqiad.wmflabs
  • tools-exec-1411.tools.eqiad.wmflabs
  • tools-exec-1412.tools.eqiad.wmflabs
  • tools-exec-1413.tools.eqiad.wmflabs
  • tools-exec-1414.tools.eqiad.wmflabs
  • tools-exec-1415.tools.eqiad.wmflabs
  • tools-exec-1416.tools.eqiad.wmflabs
  • tools-exec-1417.tools.eqiad.wmflabs
  • tools-exec-1418.tools.eqiad.wmflabs
  • tools-exec-1419.tools.eqiad.wmflabs
  • tools-exec-1420.tools.eqiad.wmflabs
  • tools-exec-1421.tools.eqiad.wmflabs
  • tools-exec-1422.tools.eqiad.wmflabs
  • tools-exec-1423.tools.eqiad.wmflabs
  • tools-exec-1424.tools.eqiad.wmflabs
  • tools-exec-1425.tools.eqiad.wmflabs
  • tools-exec-1426.tools.eqiad.wmflabs
  • tools-exec-1427.tools.eqiad.wmflabs
  • tools-exec-1428.tools.eqiad.wmflabs
  • tools-exec-1429.tools.eqiad.wmflabs
  • tools-exec-1430.tools.eqiad.wmflabs
  • tools-exec-1431.tools.eqiad.wmflabs
  • tools-exec-1432.tools.eqiad.wmflabs
  • tools-exec-1433.tools.eqiad.wmflabs
  • tools-exec-1434.tools.eqiad.wmflabs
  • tools-exec-1435.tools.eqiad.wmflabs
  • tools-exec-1436.tools.eqiad.wmflabs
  • tools-exec-1437.tools.eqiad.wmflabs
  • tools-exec-1438.tools.eqiad.wmflabs
  • tools-exec-1439.tools.eqiad.wmflabs
  • tools-exec-1440.tools.eqiad.wmflabs
  • tools-exec-1441.tools.eqiad.wmflabs
  • tools-exec-1442.tools.eqiad.wmflabs
  • tools-grid-master.tools.eqiad.wmflabs
  • tools-grid-shadow.tools.eqiad.wmflabs
  • tools-webgrid-generic-1401.tools.eqiad.wmflabs
  • tools-webgrid-generic-1402.tools.eqiad.wmflabs
  • tools-webgrid-generic-1403.tools.eqiad.wmflabs
  • tools-webgrid-generic-1404.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1401.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1402.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1403.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1404.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1405.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1406.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1407.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1408.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1409.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1410.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1412.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1413.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1414.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1415.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1416.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1417.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1418.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1419.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1420.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1421.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1422.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1424.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1425.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1426.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1427.tools.eqiad.wmflabs
  • tools-webgrid-lighttpd-1428.tools.eqiad.wmflabs

Event Timeline

• Bstorm triaged this task as Medium priority.Feb 26 2019, 5:19 PM
• Bstorm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 26 2019, 5:19 PM
• Bstorm moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Feb 26 2019, 5:21 PM
bd808 removed a project: Tracking-Neverending.Feb 26 2019, 5:31 PM
bd808 updated the task description. (Show Details)
bd808 updated the task description. (Show Details)Feb 26 2019, 5:35 PM
bd808 updated the task description. (Show Details)Feb 26 2019, 5:37 PM
bd808 added a comment.Feb 27 2019, 1:11 AM

We have about 25% of the job load that used to be on the Trusty grid running on the combination of the Stretch grid and Kubernetes now. In theory this means we can get rid of ~25% of the old Trusty worker pool. 6 tools-webgrid-lighttpd-14* have been shutdown already, so maybe 1 more of those can go. We can probably turn off 10 of the tools-exec-14* hosts now as well.

Stashbot subscribed.Feb 27 2019, 1:26 AM

Mentioned in SAL (#wikimedia-cloud) [2019-02-27T01:26:47Z] <bd808> Disabled job queues and rescheduled continuous jobs away from tools-exec-14{33,34,35,36,37,38,39,40,41,42}.tools.eqiad.wmflabs (T217152)

Stashbot added a comment.Feb 27 2019, 1:29 AM

Mentioned in SAL (#wikimedia-cloud) [2019-02-27T01:29:34Z] <bd808> Depooled tools-webgrid-lighttpd-1419.tools.eqiad.wmflabs (T217152)

Stashbot added a comment.Feb 27 2019, 1:36 AM

Mentioned in SAL (#wikimedia-cloud) [2019-02-27T01:35:58Z] <bd808> Shutdown tools-webgrid-lighttpd-1419.tools.eqiad.wmflabs via horizon (T217152)

bd808 updated the task description. (Show Details)Feb 27 2019, 1:38 AM
bd808 updated the task description. (Show Details)
zhuyifei1999 subscribed.Feb 27 2019, 7:08 AM

tools-bastion-05.tools.eqiad.wmflabs

Nobody is using this.

Stashbot added a comment.Mar 14 2019, 9:19 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-14T21:19:56Z] <bd808> Killed jobs still running on tools-exec-14{33,34,35,36,37,38,39,40,41,42}.tools.eqiad.wmflabs 2 weeks after being depooled (T217152)

Stashbot added a comment.Mar 14 2019, 9:48 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-14T21:48:27Z] <bd808> Removed tools-exec-14{33,34,35,36,37,38,39,40,41,42} from the grid and shutdown instances via horizon (T217152)

Stashbot added a comment.Mar 14 2019, 9:55 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-14T21:55:08Z] <bd808> Removed submit host flag from tools-bastion-05.tools.eqiad.wmflabs, removed floating ip, and shutdown instance via horizon (T217152)

bd808 updated the task description. (Show Details)Mar 14 2019, 9:55 PM
Stashbot added a comment.Mar 14 2019, 10:10 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-14T22:10:52Z] <bd808> Depooled tools-webgrid-lighttpd-142{0,1,2,5} (T217152)

Stashbot added a comment.Mar 14 2019, 10:20 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-14T22:20:27Z] <bd808> Removed tools-webgrid-lighttpd-142{0,1,2,5} from the grid and shutdown instances via horizon (T217152)

bd808 updated the task description. (Show Details)Mar 14 2019, 10:21 PM
bd808 moved this task from Backlog to Toolforge infra on the Cloud-VPS (Ubuntu Trusty Deprecation) board.Mar 14 2019, 10:36 PM
Stashbot added a comment.Mar 14 2019, 11:28 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-14T23:28:40Z] <bd808> Deleted tools-bastion-05 (T217152)

Stashbot added a comment.Mar 14 2019, 11:52 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-14T23:52:19Z] <bd808> Disabled job queues and rescheduled continuous jobs away from tools-exec-14{21,22,23,24,25,26,27,28,29,30,31,32} (T217152)

Stashbot added a comment.Mar 25 2019, 2:26 AM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T02:26:12Z] <bd808> Deleted shutdown Trusty grid nodes tools-webgrid-lighttpd-14{20,21,22,24,25,26,27,28} (T217152)

Stashbot added a comment.Mar 25 2019, 2:34 AM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T02:34:28Z] <bd808> Disassociated floating IPs and deleted shutdown Trusty grid nodes tools-exec-14{33,34,35,36,37,38,39,40,41,42} (T217152)

bd808 claimed this task.Mar 25 2019, 4:08 AM
Stashbot added a comment.Mar 25 2019, 6:39 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T18:39:31Z] <bd808> Shutdown tools-cron-01.tools.eqiad.wmflabs (T217152)

Stashbot added a comment.Mar 25 2019, 6:53 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T18:53:10Z] <bd808> Shutdown tools-grid-shadow (T217152)

Stashbot added a comment.Mar 25 2019, 6:53 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T18:53:39Z] <bd808> Shutdown tools-grid-master (T217152)

Stashbot added a comment.Mar 25 2019, 7:11 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T19:11:14Z] <bd808> Shutdown tools-webgrid-lighttpd-14* (T217152)

Stashbot added a comment.Mar 25 2019, 7:12 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T19:12:48Z] <bd808> Shutdown tools-webgrid-generic-14* (T217152)

Stashbot added a comment.Mar 25 2019, 7:19 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T19:19:12Z] <bd808> Shutdown tools-exec-140* (T217152)

Stashbot added a comment.Mar 25 2019, 7:31 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T19:31:01Z] <bd808> Shutdown tools-bastion-0{2,3} (T217152)

Stashbot added a comment.Mar 25 2019, 7:33 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T19:33:23Z] <bd808> Shutdown tools-exec-141* (T217152)

Stashbot added a comment.Mar 25 2019, 7:51 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T19:51:44Z] <bd808> Shutdown tools-exec-142* (T217152)

Stashbot added a comment.Mar 25 2019, 7:59 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T19:59:36Z] <bd808> Shutdown tools-exec-143* (T217152)

Stashbot added a comment.Mar 25 2019, 8:42 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T20:42:34Z] <bd808> Deleted tools-bastion-0{2,3} (T217152)

Stashbot added a comment.Mar 25 2019, 8:43 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T20:43:15Z] <bd808> Deleted tools-cron-01 (T217152)

bd808 updated the task description. (Show Details)Mar 25 2019, 8:46 PM
Stashbot added a comment.Mar 25 2019, 8:47 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T20:47:57Z] <bd808> Deleted tools-exec-140* (T217152)

Stashbot added a comment.Mar 25 2019, 8:48 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T20:48:43Z] <bd808> Deleted tools-exec-141* (T217152)

Stashbot added a comment.Mar 25 2019, 8:49 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T20:49:11Z] <bd808> Deleted tools-exec-142* (T217152)

Stashbot added a comment.Mar 25 2019, 8:49 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T20:49:31Z] <bd808> Deleted tools-exec-143* (T217152)

bd808 updated the task description. (Show Details)Mar 25 2019, 8:50 PM
Stashbot added a comment.Mar 25 2019, 8:51 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T20:51:54Z] <bd808> Deleted tools-webgrid-generic-14* (T217152)

bd808 updated the task description. (Show Details)Mar 25 2019, 8:55 PM
Stashbot added a comment.Mar 25 2019, 9:18 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T21:18:43Z] <bd808> Deleted tools-webgrid-lighttpd-14* (T217152)

Stashbot added a comment.Mar 25 2019, 9:19 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T21:19:37Z] <bd808> Deleted tools-grid-{master,shadow} (T217152)

bd808 updated the task description. (Show Details)Mar 25 2019, 9:21 PM
Stashbot added a comment.Mar 25 2019, 9:21 PM

Mentioned in SAL (#wikimedia-cloud) [2019-03-25T21:21:43Z] <bd808> All Trusty grid engine hosts shutdown and deleted (T217152)

bd808 closed this task as Resolved.Mar 25 2019, 9:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL