Page MenuHomePhabricator
Create Task
Maniphest T309902

Tiny swap on many grid nodes
Closed, DeclinedPublic
Actions

Description

We recently ran into a lot of issues with a lack of swap space on lighttpd nodes (T309821) and noticed that our default flavor (g3.cores4.ram8.disk20.swap24.ephem20) has a very small swap partition. That 24 is 24 megs, not 24 gigs.

The lighttpd nodes are now all rebuilt using a new flavor, g3.cores4.ram8.disk20.swap8.ephem20 where the 8 is 8 G rather than 8 M. We should /probably/ rebuild all the rest of these nodes with that new flavor as well.

root@cloudcontrol1003:~# openstack server list --all-projects | grep g3.cores4.ram8.disk20.swap24.ephem20
| e3198d16-ec40-485d-b9e0-49c741945df4 | tools-sgeexec-10-15                          | ACTIVE        | lan-flat-cloudinstances2b=172.16.0.70                | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 8af0a2ad-f119-463b-a78f-591caf6b8e1a | tools-sgeexec-10-14                          | ACTIVE        | lan-flat-cloudinstances2b=172.16.6.55                | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 4489381b-7273-4e86-a221-3e8293ab8f20 | tools-sgeexec-10-13                          | ACTIVE        | lan-flat-cloudinstances2b=172.16.2.241               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 5f17d6f6-66e7-4768-a407-6df0efcbe22e | tools-sgeexec-10-12                          | ACTIVE        | lan-flat-cloudinstances2b=172.16.3.213               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 5430b9c8-872d-4a89-ad2b-8b1b460f21fa | tools-sgeexec-10-11                          | ACTIVE        | lan-flat-cloudinstances2b=172.16.5.188               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 690bac1e-8f7e-4dfc-ac70-32280c60ca91 | toolsbeta-sgewebgen-09-1                     | ACTIVE        | lan-flat-cloudinstances2b=172.16.7.135               | debian-9.13-stretch                                                                                                         | g3.cores4.ram8.disk20.swap24.ephem20      |
| 1f5de537-eccf-4ace-b17c-4ea8ba83682e | tools-sgewebgen-10-1                         | ACTIVE        | lan-flat-cloudinstances2b=172.16.7.61                | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| fc0cd3a6-bfeb-4c78-a0e2-229aabd80c3e | tools-sgewebgen-10-2                         | ACTIVE        | lan-flat-cloudinstances2b=172.16.5.75                | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 1567191a-3e2d-40e0-9096-b5002a90ee30 | tools-sgeexec-10-10                          | ACTIVE        | lan-flat-cloudinstances2b=172.16.4.181               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 31cdc287-83ea-48d8-8569-76317bb1e257 | tools-sgeexec-10-8                           | ACTIVE        | lan-flat-cloudinstances2b=172.16.0.243               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 360793b6-64da-4675-9c42-9f5e4d38bab5 | tools-sgeexec-10-9                           | ACTIVE        | lan-flat-cloudinstances2b=172.16.2.247               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 2b2fd4e6-798f-457e-9247-e364449ca0bf | tools-sgeexec-10-6                           | ACTIVE        | lan-flat-cloudinstances2b=172.16.6.137               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 8c9a1145-83f7-447c-aaa5-8046414c7a00 | tools-sgeexec-10-7                           | ACTIVE        | lan-flat-cloudinstances2b=172.16.5.42                | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| ae575e76-89d1-4564-910b-e641d081f024 | tools-sgeexec-10-5                           | ACTIVE        | lan-flat-cloudinstances2b=172.16.2.108               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| d1f0ab66-e0b6-469b-aef9-11282a5f4739 | tools-sgeexec-10-4                           | ACTIVE        | lan-flat-cloudinstances2b=172.16.3.35                | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| f5365181-c374-405e-8378-d9da2a145e18 | tools-sgeexec-10-3                           | ACTIVE        | lan-flat-cloudinstances2b=172.16.2.115               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| d89e2627-e608-49ac-9fa7-d2c2b52a2bfd | tools-sgeexec-10-2                           | ACTIVE        | lan-flat-cloudinstances2b=172.16.6.121               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| f60491b0-b760-4d75-bf6d-8873f3fbd719 | tools-sgeexec-10-1                           | ACTIVE        | lan-flat-cloudinstances2b=172.16.5.106               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| e30fc3c5-a47d-4edc-92a8-ca1fc67b9534 | toolsbeta-sgeweblight-10-1                   | ACTIVE        | lan-flat-cloudinstances2b=172.16.7.86                | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 084da652-503d-49a7-9ffa-98a0cd5335fd | toolsbeta-sgeexec-10-5                       | ACTIVE        | lan-flat-cloudinstances2b=172.16.2.150               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 1d23fd4f-9b0b-43b1-ad28-099361ddfcd1 | toolsbeta-sgewebgen-10-3                     | ACTIVE        | lan-flat-cloudinstances2b=172.16.4.104               | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 67f5b271-e935-4a5f-a69e-7d368253a548 | toolsbeta-sgewebgen-10-2                     | ACTIVE        | lan-flat-cloudinstances2b=172.16.7.41                | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| 65e2821b-48c4-45e6-9671-a123c9453c18 | toolsbeta-sgewebgen-10-1                     | ACTIVE        | lan-flat-cloudinstances2b=172.16.6.87                | debian-10.0-buster                                                                                                          | g3.cores4.ram8.disk20.swap24.ephem20      |
| b5ee80cf-c839-4c38-a1c0-6d8f53d046f1 | toolsbeta-sgeexec-0902                       | ACTIVE        | lan-flat-cloudinstances2b=172.16.1.239               | debian-9.13-stretch                                                                                                         | g3.cores4.ram8.disk20.swap24.ephem20      |

Related Objects
Search...

Event Timeline

Andrew created this task.Jun 3 2022, 9:35 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJun 3 2022, 9:35 PM
Andrew removed • aborrero as the assignee of this task.Jun 3 2022, 9:35 PM
JJMC89 removed a subscriber: • Phamhi.Jun 3 2022, 9:41 PM
RhinosF1 subscribed.Jun 3 2022, 9:41 PM
taavi claimed this task.Jul 17 2022, 12:37 PM
bd808 moved this task from Backlog to In Progress on the Toolforge board.Aug 10 2022, 5:03 PM
• nskaggs added a comment.Sep 12 2022, 8:31 PM

Posting an updated list of nodes still affected.

cloudcontrol1005:~$ sudo wmcs-openstack server list --all-projects | grep g3.cores4.ram8.disk20.swap24.ephem20
| e3198d16-ec40-485d-b9e0-49c741945df4 | tools-sgeexec-10-15                  | ACTIVE    | lan-flat-cloudinstances2b=172.16.0.70                | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| 8af0a2ad-f119-463b-a78f-591caf6b8e1a | tools-sgeexec-10-14                  | ACTIVE    | lan-flat-cloudinstances2b=172.16.6.55                | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| 4489381b-7273-4e86-a221-3e8293ab8f20 | tools-sgeexec-10-13                  | ACTIVE    | lan-flat-cloudinstances2b=172.16.2.241               | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| 5430b9c8-872d-4a89-ad2b-8b1b460f21fa | tools-sgeexec-10-11                  | ACTIVE    | lan-flat-cloudinstances2b=172.16.5.188               | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| fc0cd3a6-bfeb-4c78-a0e2-229aabd80c3e | tools-sgewebgen-10-2                 | ACTIVE    | lan-flat-cloudinstances2b=172.16.5.75                | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| 31cdc287-83ea-48d8-8569-76317bb1e257 | tools-sgeexec-10-8                   | ACTIVE    | lan-flat-cloudinstances2b=172.16.0.243               | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| 360793b6-64da-4675-9c42-9f5e4d38bab5 | tools-sgeexec-10-9                   | ACTIVE    | lan-flat-cloudinstances2b=172.16.2.247               | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| 8c9a1145-83f7-447c-aaa5-8046414c7a00 | tools-sgeexec-10-7                   | ACTIVE    | lan-flat-cloudinstances2b=172.16.5.42                | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| ae575e76-89d1-4564-910b-e641d081f024 | tools-sgeexec-10-5                   | ACTIVE    | lan-flat-cloudinstances2b=172.16.2.108               | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| f60491b0-b760-4d75-bf6d-8873f3fbd719 | tools-sgeexec-10-1                   | ACTIVE    | lan-flat-cloudinstances2b=172.16.5.106               | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| e30fc3c5-a47d-4edc-92a8-ca1fc67b9534 | toolsbeta-sgeweblight-10-1           | ACTIVE    | lan-flat-cloudinstances2b=172.16.7.86                | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| 084da652-503d-49a7-9ffa-98a0cd5335fd | toolsbeta-sgeexec-10-5               | ACTIVE    | lan-flat-cloudinstances2b=172.16.2.150               | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| 1d23fd4f-9b0b-43b1-ad28-099361ddfcd1 | toolsbeta-sgewebgen-10-3             | ACTIVE    | lan-flat-cloudinstances2b=172.16.4.104               | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| 67f5b271-e935-4a5f-a69e-7d368253a548 | toolsbeta-sgewebgen-10-2             | ACTIVE    | lan-flat-cloudinstances2b=172.16.7.41                | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
| 65e2821b-48c4-45e6-9671-a123c9453c18 | toolsbeta-sgewebgen-10-1             | ACTIVE    | lan-flat-cloudinstances2b=172.16.6.87                | debian-10.0-buster                           | g3.cores4.ram8.disk20.swap24.ephem20      |
bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Sep 27 2022, 9:29 PM
TheresNoTime removed a subscriber: RhinosF1.Dec 15 2022, 11:35 PM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 7:26 PM
fnegri moved this task from Kanban to Doing? (legacy column) on the cloud-services-team board.
fnegri moved this task from Doing? (legacy column) to Inbox on the cloud-services-team board.Jan 19 2023, 12:55 PM
dcaro mentioned this in T335336: [toolschecker] jobs mtime check is flapping.Apr 25 2023, 12:17 PM
taavi moved this task from In Progress to Ready to be worked on on the Toolforge board.Nov 7 2023, 2:21 PM
taavi closed this task as Declined.Dec 13 2023, 11:17 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits