Page MenuHomePhabricator
Create Task
Maniphest T216030

cloudvps: evaluate draining cloudvirt1018
Closed, ResolvedPublic
Actions

Description

Because T216004: Degraded RAID on cloudvirt1018.

For the record, all the following VMs are currently unreachable:

root@cloudcontrol1004:~# nova list --all-tenants --host cloudvirt1018
+--------------------------------------+----------------------------------+---------------------------+---------+------------+-------------+------------------------------------------------------+
| ID                                   | Name                             | Tenant ID                 | Status  | Task State | Power State | Networks                                             |
+--------------------------------------+----------------------------------+---------------------------+---------+------------+-------------+------------------------------------------------------+
| f866beab-86c4-406b-9daf-8840343497b6 | a11y                             | reading-web-staging       | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.58                |
| ecef497d-4b1f-4217-a4fc-2c6b7b12cf22 | abogott-scapserver               | testlabs                  | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.7.181               |
| 0b180503-67c2-44ed-ad69-07614a7e1339 | af-puppetdb01                    | automation-framework      | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.4.74                |
| 1f14c95b-c347-41ce-9518-f225c17929ff | api                              | openocr                   | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.118               |
| 6b0fed2a-34a6-48d1-ba2f-472a3f22c5c7 | asdf                             | quotatest                 | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.196               |
| 1144cb61-f7ab-436f-bd02-003e40667be6 | bastion-eqiad1-02                | bastion                   | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.3.155, 185.15.56.16 |
| 46ee95a5-04d3-4490-b334-692a92d2fc85 | clm-test-01                      | community-labs-monitoring | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.10                |
| b65c0c7d-280f-407b-bf13-fb4b16523c2c | compiler1002                     | puppet-diffs              | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.4.137               |
| f14867c7-d6d1-4ffc-bcf7-0674a9114733 | cyberbot-exec-iabot-01           | cyberbot                  | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.22, 185.15.56.22  |
| 27460e9d-5548-4cd6-9472-548db6402294 | deployment-db03                  | deployment-prep           | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.23                |
| 73f33f94-7cc3-4015-a9e1-59bb7ed81cb3 | deployment-db04                  | deployment-prep           | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.5                 |
| 52864d30-c5c1-420a-bcb7-b0dd14c5ef7a | deployment-memc05                | deployment-prep           | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.17                |
| c5959b75-64b2-4203-98ed-ccb4bbcf0a79 | deployment-pdfrender02           | deployment-prep           | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.43                |
| 3982270f-e578-4a39-809a-eb5c9656f13a | deployment-sca01                 | deployment-prep           | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.13                |
| 30c4c394-d8cf-49de-8169-94be79b2d8bb | design-lsg3                      | design                    | SHUTOFF | -          | Shutdown    | lan-flat-cloudinstances2b=172.16.7.136               |
| 9b42bf2e-dda6-4898-b2d5-09383d6891ea | eventmetrics-dev01               | eventmetrics              | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.51                |
| 13911639-b8a4-4144-ab12-c43eab9ad8e7 | fridolin                         | catgraph                  | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.8                 |
| c7bed3eb-decf-4a95-8ed5-a13703f0284d | gtirloni-puppetmaster-01         | testlabs                  | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.3                 |
| 86d34401-0a21-443c-8fb4-6bd2fc62069b | hadoop-master-3                  | analytics                 | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.239               |
| 7d65bc85-5bc6-4055-a6d8-62658805d6d8 | ign                              | ign2commons               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.7.30                |
| f6fdd3f2-ee6a-4002-8231-92f1921ce4a7 | integration-castor03             | integration               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.146               |
| 9bd9c836-cead-43c4-bc16-c0f06d7b83c3 | integration-slave-docker-1017    | integration               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.220               |
| d9c6bb3c-40d9-4331-b88c-06c94bf63f6e | integration-slave-docker-1033    | integration               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.221               |
| 8a9838cf-f05d-41fe-ba6a-1f975f6ae0ba | integration-slave-docker-1038    | integration               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.7.169               |
| 40ceef97-ed07-48bc-8636-8a9dfc6b7ec9 | integration-slave-jessie-1003    | integration               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.227               |
| dae7f0c4-524a-4a90-bd2a-683c73042ac2 | integration-slave-jessie-android | integration               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.6.183               |
| 5a339ad6-fd19-43b7-85ab-23f62153e47a | k8s-master-01                    | general-k8s               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.202               |
| 91a2c950-2590-4e17-88fb-88d8d60ffbdf | k8s-node-03                      | general-k8s               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.200               |
| eb4ab189-eef2-4998-992e-9e028b19f0a0 | k8s-node-05                      | general-k8s               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.205               |
| b25b3e23-ab4f-4844-ab89-9fe17db298da | k8s-node-06                      | general-k8s               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.206               |
| b84f48a5-6e9b-46bf-b0e8-4c1744346ab3 | kdc                              | analytics                 | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.235               |
| 28880c4d-3a07-4a9e-8c56-3f3b26241172 | labstash-jessie1                 | logging                   | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.32                |
| cead6ba0-bc5b-4b6a-85a2-6480cd9227bd | language-mleb-legacy             | language                  | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.102               |
| 5f162b81-465f-49f5-baf6-eb90c3ebe020 | login-test                       | catgraph                  | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.7                 |
| 2f567056-a6e9-4d25-b346-f72cbc7522e7 | lsg-01                           | design                    | SHUTOFF | -          | Shutdown    | lan-flat-cloudinstances2b=172.16.7.131               |
| 2600f5bc-dad7-4ccc-9c6a-679819627703 | mathosphere                      | math                      | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.60, 185.15.56.41  |
| 3d70188e-749a-4f21-9780-74990b90ca91 | mc-clusterA-1                    | test-twemproxy            | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.74                |
| 1022031a-7557-444e-8d88-27866a65dfe9 | mwoffliner5                      | mwoffliner                | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.7.222               |
| fccb403f-c9c2-4b85-b918-e841a7dfc190 | novaadminmadethis-4              | quotatest                 | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.202               |
| fc896d7a-bbf6-431e-9f4d-3d13a9dc21fc | ntp-01                           | cloudinfra                | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.81, 185.15.56.3   |
| 484f1583-2288-4be0-8c34-bb130a172124 | ntp-02                           | cloudinfra                | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.82, 185.15.56.27  |
| 1ae4ec8f-6bad-42cc-a646-58ede7f143ff | ogvjs-testing                    | ogvjs-integration         | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.149               |
| 3c3cb7b8-568b-4441-974a-cc0221756ca9 | phragile-pro                     | phragile                  | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.6.143               |
| 6fa321b2-9e00-49b3-bc45-74183658fc1f | planet-hotdog                    | planet                    | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.208               |
| 035e1323-825a-486e-b355-2a296ae7f540 | pub2                             | wikiapiary                | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.0.87, 185.15.56.32  |
| b0497c5a-04da-4b5c-8db2-7c101fd56686 | puppenmeister                    | planet                    | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.207               |
| baa5fee9-c34c-44d7-915c-554588f422ac | puppet-compiler-v4-other         | testlabs                  | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.253               |
| 357ce0c0-31a4-4590-95f5-935132a9e370 | puppet-compiler-v4-tools         | testlabs                  | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.252               |
| d6a6c911-feb1-4a3b-aed4-0d8d81489073 | quarry-beta-01                   | quarry                    | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.7.7                 |
| 715afc3e-6092-4800-bfc9-32b7d8f338d5 | signwriting-swis                 | signwriting               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.145               |
| c8f78a3f-dd7c-4e8f-95a0-46fb427af633 | signwriting-swserver             | signwriting               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.5.147               |
| 137ee7e2-62f7-4117-b34f-959b484b3c80 | social-tools3                    | social-tools              | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.3.196               |
| bca530a9-888b-438b-94ca-5c0b854cac5c | striker-deploy04                 | striker                   | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.209               |
| 60dfcf9c-558f-4a1b-b02b-36299741d573 | striker-puppet01                 | striker                   | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.214               |
| 03d53e02-5084-46ea-8c1c-e0aeffb1f61a | t166878                          | otrs                      | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.6.147               |
| 5e15682b-63db-438b-96aa-2ba77b09fb9a | togetherjs                       | visualeditor              | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.4.216               |
| 6f86abc0-9786-4f72-80fd-ec1fe7117d41 | tools-sgebastion-06              | tools                     | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.4.22, 185.15.56.48  |
| bad6cee2-2748-4e14-9dcc-3ccdc28f279e | tools-sgeexec-0902               | tools                     | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.4.228               |
| c6fb7a36-21da-40f9-b872-d354957ec55a | tools-sgeexec-0903               | tools                     | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.7.105               |
| 91dab9d7-4e88-427e-a02b-961b705d4355 | tools-sgewebgrid-generic-0901    | tools                     | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.4.209               |
| 9a27a423-3bdf-42da-bd89-52d5b4ad2417 | tools-sgewebgrid-lighttpd-0901   | tools                     | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.4.211               |
| 0934e6d1-8e58-497b-a1d3-d16d015fdcda | ve-font                          | design                    | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.7.130               |
| 91d34ba6-80cc-4c4f-9dbe-76a2ba5f5ac6 | wikibase1                        | sciencesource             | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.4.30                |
| 02a2bd3e-cf28-4e7a-8567-50736099027b | wikicitevis-prod                 | wikicitevis               | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.4.220               |
| e8ac63f4-eba6-42c1-bb10-32a3f5ded1f4 | wikifarm                         | pluggableauth             | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.41                |
| 8e928fd4-a83a-4917-ac43-f5a770503491 | women-in-red                     | globaleducation           | ACTIVE  | -          | Running     | lan-flat-cloudinstances2b=172.16.2.93                |
+--------------------------------------+----------------------------------+---------------------------+---------+------------+-------------+------------------------------------------------------+

Related Objects

Event Timeline

aborrero created this task.Feb 13 2019, 12:19 PM
Restricted Application added subscribers: Cyberpower678, Aklapper. · View Herald TranscriptFeb 13 2019, 12:19 PM
aborrero triaged this task as High priority.Feb 13 2019, 12:19 PM
aborrero added a project: cloud-services-team (Kanban).
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
Stashbot added a subscriber: Stashbot.Feb 13 2019, 12:32 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-13T12:32:17Z] <arturo> T216030 T216004 rebooting cloudvirt1018

Stashbot mentioned this in T216004: Degraded RAID on cloudvirt1018.Feb 13 2019, 12:32 PM
Stashbot added a comment.Feb 13 2019, 12:34 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-13T12:34:42Z] <arturo> T216030 icinga downtime cloudvirt1018 for 2 weeks

Stashbot added a comment.Feb 13 2019, 1:03 PM

Mentioned in SAL (#wikimedia-cloud) [2019-02-13T13:03:26Z] <arturo> T216030 switch login-stretch.tools.wmflabs.org floating IP to tools-sgebastion-07

Cyberpower678 added a comment.Feb 13 2019, 1:06 PM

*cough* what? My instances were literally just moved several months ago. How can the RAID be failing there already?

aborrero updated the task description. (Show Details)Feb 13 2019, 1:11 PM
Cyberpower678 added a comment.EditedFeb 13 2019, 2:19 PM

@aborrero are you saying that my VM is down, or just not reachable? Because cyberbot-exec-iabot-01 is InternetArchiveBot, which I kind of need access to.

Based on the bot's contributions, I'd say the VM is down.

jcrespo mentioned this in T216045: Beta cluster is broken: No working replica DB server: Unknown error (172.16.5.5:3306)).Feb 13 2019, 3:20 PM
jcrespo mentioned this in T216039: jenkins / zuul backing up due to jenkins slaves down.Feb 13 2019, 3:26 PM
Framawiki added a subscriber: Framawiki.Feb 13 2019, 5:45 PM
Lucas_Werkmeister_WMDE mentioned this in T216053: Jenkins failing everything due to npm being screwed up.Feb 13 2019, 6:10 PM
Andrew closed this task as Resolved.Feb 14 2019, 3:11 PM
Andrew added a subscriber: Andrew.

It's drained -- updates have been sent to cloud-announce, e.g. https://lists.wikimedia.org/pipermail/cloud-announce/2019-February/000133.html

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL