Page MenuHomePhabricator
Create Task
Maniphest T292465

Automate rebuild and rebuild toolsbeta-sgewebgrid-generic-0901
Closed, ResolvedPublic
Actions

Description

fsck of root volume failed, system is inaccessible.

Checking in progress on 1 disk (0.0% complete)Checking in progress on 0 disks (100.0% complete)[[0;1;31mFAILED[0m] Failed to start File System Check o���4970c-1b27-4544-b346-adbfe686feb0.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

• Bstorm created this task.Oct 4 2021, 5:23 PM
dcaro edited projects, added User-dcaro, cloud-services-team (FY2021/2022-Q3); removed cloud-services-team (zz-archived1).Oct 5 2021, 9:05 AM
dcaro moved this task from To refine to Today on the User-dcaro board.Oct 5 2021, 9:10 AM
dcaro claimed this task.Oct 5 2021, 9:13 AM
dcaro moved this task from ToDo to In Progress on the cloud-services-team (FY2021/2022-Q3) board.
dcaro moved this task from Today to Doing on the User-dcaro board.Oct 5 2021, 12:57 PM
dcaro added a comment.Oct 6 2021, 8:09 AM

Using https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/New_grid_engine_exec_host as a guideline to create a
cookbook.

Stashbot added a comment.Oct 6 2021, 10:05 AM

Mentioned in SAL (#wikimedia-cloud) [2021-10-06T10:05:27Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 6 2021, 10:07 AM

Mentioned in SAL (#wikimedia-cloud) [2021-10-06T10:07:58Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 6 2021, 10:08 AM

Mentioned in SAL (#wikimedia-cloud) [2021-10-06T10:08:53Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 6 2021, 10:13 AM

Mentioned in SAL (#wikimedia-cloud) [2021-10-06T10:13:23Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 6 2021, 10:36 AM

Mentioned in SAL (#wikimedia-cloud) [2021-10-06T10:36:40Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

dcaro moved this task from Doing to Blocked on the User-dcaro board.Oct 6 2021, 12:45 PM
gerritbot added a comment.Oct 6 2021, 12:55 PM

Change 726894 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] toolforge: new add_grid_webgrid_generic_node recipe

https://gerrit.wikimedia.org/r/726894

gerritbot added a project: Patch-For-Review.Oct 6 2021, 12:55 PM
Stashbot added a comment.Oct 7 2021, 7:58 AM

Mentioned in SAL (#wikimedia-cloud) [2021-10-07T07:58:03Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 7 2021, 8:04 AM

Mentioned in SAL (#wikimedia-cloud) [2021-10-07T08:04:44Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 7 2021, 12:50 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-07T12:50:10Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 7 2021, 12:50 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-07T12:50:46Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 7 2021, 12:55 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-07T12:55:09Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 7 2021, 1:31 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-07T13:31:42Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 7 2021, 2:06 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-07T14:06:33Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 7 2021, 2:21 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-07T14:21:08Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

dcaro moved this task from Blocked to Today on the User-dcaro board.Oct 8 2021, 7:53 AM
gerritbot added a comment.Oct 11 2021, 8:25 AM

Change 729926 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] cinderutils::ensure: Check falsey instead of empty string

https://gerrit.wikimedia.org/r/729926

Stashbot added a comment.Oct 11 2021, 10:32 AM

Mentioned in SAL (#wikimedia-cloud) [2021-10-11T10:32:00Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

gerritbot added a comment.Oct 11 2021, 1:42 PM

Change 729990 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/software/spicerack@master] puppet.PuppetHost.get_ca_server: use only the last line

https://gerrit.wikimedia.org/r/729990

gerritbot added a comment.Oct 11 2021, 1:51 PM

Change 729994 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmcs::instance: use path to allow different systemctl paths

https://gerrit.wikimedia.org/r/729994

gerritbot added a comment.Oct 11 2021, 1:57 PM

Change 729994 merged by David Caro:

[operations/puppet@production] wmcs::instance: use path to allow different systemctl paths

https://gerrit.wikimedia.org/r/729994

gerritbot added a comment.Oct 11 2021, 2:41 PM

Change 729926 abandoned by David Caro:

[operations/puppet@production] cinderutils::ensure: Check falsey instead of empty string

Reason:

This was not the cause, but the size limits, will add extra verbosity to the messages to avoid falling for this again.

https://gerrit.wikimedia.org/r/729926

gerritbot added a comment.Oct 11 2021, 2:42 PM

Change 730001 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmcs-prepare-cinder-volume.py: chown also after mounting

https://gerrit.wikimedia.org/r/730001

gerritbot added a comment.Oct 11 2021, 2:54 PM

Change 730001 merged by David Caro:

[operations/puppet@production] wmcs-prepare-cinder-volume.py: chown also after mounting

https://gerrit.wikimedia.org/r/730001

Stashbot added a comment.Oct 11 2021, 3:00 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-11T15:00:35Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

gerritbot added a comment.Oct 11 2021, 3:19 PM

Change 730005 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] cinderutils::ensure: give more info when no device found

https://gerrit.wikimedia.org/r/730005

Stashbot added a comment.Oct 11 2021, 3:24 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-11T15:24:57Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

gerritbot added a comment.Oct 11 2021, 4:28 PM

Change 730025 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] p:base: Move the reboot-host script there

https://gerrit.wikimedia.org/r/730025

gerritbot added a comment.Oct 12 2021, 8:18 AM

Change 729990 merged by David Caro:

[operations/software/spicerack@master] puppet.PuppetHost.get_ca_server: use only the last line

https://gerrit.wikimedia.org/r/729990

gerritbot added a comment.Oct 12 2021, 2:07 PM

Change 730025 merged by David Caro:

[operations/puppet@production] cumin::target: Move the reboot-host script there

https://gerrit.wikimedia.org/r/730025

dcaro moved this task from Today to Doing on the User-dcaro board.Oct 12 2021, 2:13 PM
Stashbot added a comment.Oct 12 2021, 2:46 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-12T14:46:18Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 12 2021, 2:52 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-12T14:51:59Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 12 2021, 4:10 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-12T16:10:55Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

gerritbot added a comment.Oct 12 2021, 6:08 PM

Change 730270 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/software/spicerack@master] remote: use only the last line for the uptime

https://gerrit.wikimedia.org/r/730270

dcaro added a comment.Oct 13 2021, 9:06 AM

After manual reboot, the new VM was able to run puppet correctly, but now it seems to have issues with a long hostname xd

root@toolsbeta-sgewebgrid-generic-09-2:~# qconf -sel
reresolve hostname failed: hostname exceeds hostname length(MAXHOSTNAMELEN) on this system

root@toolsbeta-sgewebgrid-generic-09-2:~# getconf HOST_NAME_MAX
64

root@toolsbeta-sgewebgrid-generic-09-2:~# hostname -f | wc -c
67
dcaro added a comment.Oct 13 2021, 9:14 AM

That seems to be hardcoded on the kernel :S

Stashbot added a comment.Oct 13 2021, 10:19 AM

Mentioned in SAL (#wikimedia-cloud) [2021-10-13T10:19:16Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 13 2021, 10:19 AM

Mentioned in SAL (#wikimedia-cloud) [2021-10-13T10:19:46Z] <wm-bot> Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Oct 13 2021, 11:18 AM

Mentioned in SAL (#wikimedia-cloud) [2021-10-13T11:18:56Z] <wm-bot> Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool (T292465) - cookbook ran by dcaro@vulcanus

dcaro renamed this task from Rebuild toolsbeta-sgewebgrid-generic-0901 to Automate rebuild and rebuild toolsbeta-sgewebgrid-generic-0901.Oct 14 2021, 7:34 AM
dcaro triaged this task as High priority.
dcaro moved this task from Doing to Today on the User-dcaro board.
gerritbot added a comment.Oct 14 2021, 6:06 PM

Change 730005 merged by Andrew Bogott:

[operations/puppet@production] cinderutils::ensure: give more info when no device found

https://gerrit.wikimedia.org/r/730005

gerritbot added a comment.Oct 15 2021, 2:30 PM

Change 731111 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] grid_configurator: Added new naming schemes

https://gerrit.wikimedia.org/r/731111

gerritbot added a comment.Oct 15 2021, 2:37 PM

Change 731113 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmcs-srpeadcheck-tools: add new shorter webgrid names

https://gerrit.wikimedia.org/r/731113

gerritbot added a comment.Oct 15 2021, 2:37 PM

Change 731114 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] tools-clush-generator: add the shorter webgrid names

https://gerrit.wikimedia.org/r/731114

gerritbot added a comment.Oct 15 2021, 2:38 PM

Change 731111 merged by David Caro:

[operations/puppet@production] grid_configurator: Added new naming schemes

https://gerrit.wikimedia.org/r/731111

gerritbot added a comment.Oct 18 2021, 8:00 AM

Change 730270 abandoned by David Caro:

[operations/software/spicerack@master] remote: use only the last line for the uptime

Reason:

I'll unblock myself working around this shortcoming on the cookbook code instead.

https://gerrit.wikimedia.org/r/730270

dcaro moved this task from Today to To refine on the User-dcaro board.Oct 19 2021, 7:37 AM
dcaro edited projects, added cloud-services-team (Kanban); removed cloud-services-team (FY2021/2022-Q3).Oct 19 2021, 9:23 AM
gerritbot added a comment.Oct 19 2021, 9:28 AM

Change 731911 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] start_instance_with_prefix: fix next instance counter

https://gerrit.wikimedia.org/r/731911

gerritbot added a comment.Oct 19 2021, 9:28 AM

Change 731912 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] start_instance_with_prefix: add tries parameter

https://gerrit.wikimedia.org/r/731912

dcaro removed dcaro as the assignee of this task.Oct 19 2021, 1:47 PM
dcaro removed a project: User-dcaro.
gerritbot added a comment.Oct 29 2021, 3:22 PM

Change 731911 abandoned by David Caro:

[operations/cookbooks@wmcs] start_instance_with_prefix: fix next instance counter

Reason:

Merged into https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/731885

https://gerrit.wikimedia.org/r/731911

gerritbot added a comment.Nov 1 2021, 5:06 PM

Change 731912 merged by jenkins-bot:

[operations/cookbooks@wmcs] start_instance_with_prefix: add tries parameter

https://gerrit.wikimedia.org/r/731912

gerritbot added a comment.Dec 15 2021, 11:21 AM

Change 731113 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] wmcs-srpeadcheck-tools: add new shorter webgrid names

https://gerrit.wikimedia.org/r/731113

gerritbot added a comment.Dec 17 2021, 9:21 AM

Change 726894 merged by Arturo Borrero Gonzalez:

[operations/cookbooks@wmcs] toolforge: new add_grid_webgrid_generic_node recipe

https://gerrit.wikimedia.org/r/726894

gerritbot added a comment.Mar 3 2022, 2:13 PM

Change 731114 abandoned by David Caro:

[operations/puppet@production] tools-clush-generator: add the shorter webgrid names

Reason:

This was redone in some other patch

https://gerrit.wikimedia.org/r/731114

taavi closed this task as Resolved.Sep 26 2022, 10:55 AM
taavi claimed this task.
dcaro mentioned this in rCCKBcc011b8cac5c: start_instance_with_prefix: add tries parameter.Dec 14 2022, 3:32 PM
dcaro mentioned this in rCCKBe3fd4f80a901: toolforge: new add_grid_webgrid_generic_node recipe.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL