Page MenuHomePhabricator

Several hosts on virt1002 are down
Closed, ResolvedPublic

Description

This includes deployment-db1, which means you can't login into beta cluster.

Other downed hosts include:

  • Estest1003.search.eqiad.wmflabs
  • integration-slave-precise-1014
  • tools-exec-1221

Event Timeline

Luke081515 raised the priority of this task from to Unbreak Now!.
Luke081515 updated the task description. (Show Details)
Luke081515 added a subscriber: Luke081515.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 5 2015, 5:05 PM
coren added a subscriber: coren.Nov 5 2015, 5:12 PM

labvirt1002 seems ill; not sure why yet. Probably affected instances are:

| e68c80c9-e6de-4be7-83b8-e2b04fc89308 | accounts-application3                               | labvirt1002           | stwalkerster    |
| e7c2f66a-8891-4285-a0b5-d610b382c215 | agent3                                              | labvirt1002           | csteipp         |
| 1b55a7e0-760c-4b85-8338-631f3748c183 | appservice                                          | labvirt1002           | gwicke          |
| a75daea4-ebbb-488d-938e-c81eb8dc9959 | chadwick                                            | labvirt1002           | cristiancantoro |
| 6e398d2a-5512-44d5-9735-81e6ea77fd55 | commonsarchive-test                                 | labvirt1002           | zhuyifei1999    |
| c6f1aa6d-a52b-4234-9835-630658a71940 | compiler                                            | labvirt1002           | akosiaris       |
| f7780509-31ac-4d54-9ced-e5c752e077dc | compiler02                                          | labvirt1002           | oblivian        |
| 4c2d38b7-0104-4033-bed5-4cbbd7a7fb0b | contactpage-test                                    | labvirt1002           | anomie          |
| b52e2a1e-bb61-4819-b2f5-d552c1cfc825 | cvn-apache8                                         | labvirt1002           | krinkle         |
| fb90cee7-df88-404c-9170-936c451c8eb3 | dan-druid                                           | labvirt1002           | milimetric      |
| 7eca64b6-8300-4be2-a2a5-f7b7fe4b8364 | dashiki1                                            | labvirt1002           | milimetric      |
| 88c5b221-ea9e-4c93-ba4e-c1ebecb4c887 | deployment-cache-upload04                           | labvirt1002           | demon           |
| aec2245c-e965-4f06-8496-a0d9abf519ee | deployment-db1                                      | labvirt1002           | hashar          |
| 1411d0ec-e934-4bfa-8327-81bfbbe4df32 | deployment-elastic06                                | labvirt1002           | demon           |
| 9de62e0c-3af1-4bf5-8550-8708efb9d8d0 | deployment-eventlogging03                           | labvirt1002           | otto            |
| dfacf7e3-d60c-4990-9681-30610df4ae3d | deployment-kafka02                                  | labvirt1002           | otto            |
| e8cdee8b-d4b9-4ccb-8be5-944093ae3bf3 | deployment-mathoid                                  | labvirt1002           | akosiaris       |
| 2cfaf18c-e6ea-4c2d-b96f-df7f50b6bc9a | deployment-mediawiki03                              | labvirt1002           | yuvipanda       |
| 507ed00f-b7fc-42e8-803c-53224646598d | deployment-memc04                                   | labvirt1002           | ori             |
| 54c66f88-4c39-487b-802b-2eec751f4300 | deployment-pdf02                                    | labvirt1002           | jeremyb         |
| fcaa135e-3fce-4fae-afe0-ded789fe6f6a | deployment-sca01                                    | labvirt1002           | johnflewis      |
| 9efc6176-c5bc-45e5-b628-65251a0da9ef | dolly                                               | labvirt1002           | smalyshev       |
| 17d0e2e1-6578-47bd-a6e1-22445891608e | ee-jenkins-test                                     | labvirt1002           | ebernhardson    |
| 49623320-ac69-4f99-be9f-08aaffb3cf34 | estest1003                                          | labvirt1002           | ebernhardson    |
| 500fb0b6-1b11-4072-80e8-74b0b650a167 | flow1                                               | labvirt1002           | jkrauska        |
| 357202f9-9a57-4824-8265-6dad86869fdf | graphtest                                           | labvirt1002           | yurik           |
| 3e6075e3-a3c3-441f-8af9-69d9107586c7 | integration-dev                                     | labvirt1002           | krinkle         |
| f0476096-196f-4851-bb64-6af50c358396 | integration-puppetmaster                            | labvirt1002           | hashar          |
| 78c56d53-1770-466b-9ad2-6955a539561c | integration-saltmaster                              | labvirt1002           | hashar          |
| 9fe75408-9feb-4e31-a528-90e8180d5ff6 | integration-slave-precise-1014                      | labvirt1002           | zfilipin        |
| 6605a415-9558-4184-88ea-2b9f337b519b | integration-slave-trusty-1016                       | labvirt1002           | krinkle         |
| 754414bc-aff7-4c88-8ac7-c369e9ea20e7 | keystone                                            | labvirt1002           | csteipp         |
| f1e76268-3d29-426d-b854-b313085e03a8 | labs-vmbuilder-lucid                                | labvirt1002           | novaadmin       |
| a1d7df04-a56b-4daa-ae1e-e85bb2a0024f | language-replag-wiki                                | labvirt1002           | novaadmin       |
| 54c885b2-2d46-4427-bac8-a3a04de2bb0c | languagetool                                        | labvirt1002           | kumari          |
| dc64300a-0954-4e12-a96c-bd733f28dc4a | maps-tiles3                                         | labvirt1002           | apmon           |
| 4d7671cb-cc20-4f89-aa09-dc7289fa91bb | math                                                | labvirt1002           | physikerwelt    |
| 272866eb-9d95-4008-a19a-f39f25ab11c4 | megachron-three                                     | labvirt1002           | marc            |
| b7fb3543-519d-46e7-a3f6-6f6946ea9495 | metamorphosis-kafka-01                              | labvirt1002           | yuvipanda       |
| 79897857-fad6-48fd-b4eb-24a99762fe89 | mlp                                                 | labvirt1002           | physikerwelt    |
| 1c2083e4-0db5-473c-8d30-c486f477adea | mobile-video                                        | labvirt1002           | brion           |
| 9930528e-c824-48e0-932e-862ffe582412 | novaproxy-02                                        | labvirt1002           | yuvipanda       |
| 29fa8a12-597e-4088-9b43-c8927eea9c2b | oppenheimer                                         | labvirt1002           | tomaluca95      |
| 25e7f424-85bf-46d7-b401-85a8c4597cf8 | osrm                                                | labvirt1002           | sbiribizio      |
| 7a11e7d7-0d20-47c6-b8bd-b07579044afa | otrs-test2                                          | labvirt1002           | jgreen          |
| c2fb6c05-10b8-4baf-bf42-db29e4eaf276 | phab-02                                             | labvirt1002           | negative24      |
| abcb156c-640b-4e17-a5c0-fb8eefbfbd42 | staging-mc1                                         | labvirt1002           | demon           |
| bf30c55f-c79f-43c5-8ead-871955a5237f | staging-ms-fe01                                     | labvirt1002           | thcipriani      |
| 140c242c-4e96-4552-9603-6d45f2d13439 | staging-rdb1                                        | labvirt1002           | twentyafterfour |
| 8bdf3d13-6020-4f8e-a1e1-783c5559f257 | stashbot-deploy                                     | labvirt1002           | bd808           |
| 7b09e8b3-79ca-46eb-86da-edfc15f31e77 | szilard                                             | labvirt1002           | cristiancantoro |
| 3de640e0-03e5-40d9-931a-0950a37a75f2 | test-servicestats2                                  | labvirt1002           | filippo         |
| 1e2f67cd-4d6b-4378-991d-0ad4d79c3929 | test-servicestats3                                  | labvirt1002           | filippo         |
| fba8e0e4-eaab-47c2-b655-2d11afb75867 | tools-exec-1203                                     | labvirt1002           | yuvipanda       |
| 8a6b2533-f3a8-46da-82ac-d28b74138ca8 | tools-exec-1210                                     | labvirt1002           | yuvipanda       |
| 478bf0ba-439b-4ac7-aea4-94a9694f9985 | tools-exec-1214                                     | labvirt1002           | yuvipanda       |
| d09fe2aa-c7ad-48ea-b573-1caed518b9ab | tools-exec-1221                                     | labvirt1002           | yuvipanda       |
| 307878dd-edf3-469d-804d-971553f2fb82 | tools-exec-1405                                     | labvirt1002           | yuvipanda       |
| 7e79e49d-b540-459b-a59e-3faa934d730e | tools-exec-gift                                     | labvirt1002           | marc            |
| c9030a35-4475-4581-a84c-1728d27bcf9b | tools-services-01                                   | labvirt1002           | yuvipanda       |
| 96c37c36-970b-4cc7-a7ba-d1ee90a225b5 | tools-submit                                        | labvirt1002           | marc            |
| d1ae7038-1e97-4ba0-a3ac-b0357578166a | tools-web-static-02                                 | labvirt1002           | yuvipanda       |
| 324a528c-da1f-48b3-b572-81dfebf15812 | tools-webgrid-generic-1403                          | labvirt1002           | yuvipanda       |
| e46ff66c-15e2-4a6c-a410-37b8f06c32b0 | tools-webgrid-lighttpd-1204                         | labvirt1002           | yuvipanda       |
| bee7f41e-da4b-45b5-8a8c-a3b7214ea2d1 | tools-webgrid-lighttpd-1209                         | labvirt1002           | yuvipanda       |
| 259f842e-a6e8-40e3-9e78-a58c69ef19e9 | tools-webgrid-lighttpd-1401                         | labvirt1002           | yuvipanda       |
| 44fd49a0-ebc0-4d9d-97a8-6c55ad57fdea | tools-webgrid-lighttpd-1405                         | labvirt1002           | yuvipanda       |
| f4c05842-1a7f-4cae-8991-d4e45febfaf4 | tools-worker-02                                     | labvirt1002           | yuvipanda       |
| 3e203dc1-f1c7-42be-89c7-d625d09830d8 | wdm-rdf                                             | labvirt1002           | chjohnson       |
| 4e9f8d4f-67bb-4d64-877c-aa12a14b6211 | wdq-redirect                                        | labvirt1002           | smalyshev       |
| 75f972e3-4735-4c29-8342-fc3a1cf8d8ec | wikibrain0                                          | labvirt1002           | shiladsen       |
| 33f2911d-fc68-404f-b454-0f5d294ce8bf | wikistats-southpark                                 | labvirt1002           | dzahn           |
| d9395210-edd4-4638-9d61-80376d9f05d4 | zerotests                                           | labvirt1002           | jhobs           |
valhallasw renamed this task from deployment-db1 is down to Several hosts on virt1002 are down.Nov 5 2015, 5:14 PM
valhallasw updated the task description. (Show Details)
valhallasw set Security to None.
valhallasw added a subscriber: valhallasw.
dduvall added a subscriber: dduvall.Nov 5 2015, 6:15 PM

deployment-db1 is back up and replication to deployment-db2 is running.

yuvipanda closed this task as Resolved.Nov 5 2015, 10:02 PM
yuvipanda claimed this task.
yuvipanda added a subscriber: yuvipanda.

It just ran out of space again.

(some instances were migrated off, has space now, and there's a critical alert for disk space on these hosts)

Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptJun 7 2017, 6:46 PM