Page MenuHomePhabricator
Create Task
Maniphest T120586

Instance creation fails
Closed, ResolvedPublic
Actions

Description

I created two instances in project "services", but both are still listed as building / scheduling after 10+ / 20+ minutes.

  • Instance names: docker-testing, docker-testing01
  • Instance types: small, Debian Jessie

Details

SubjectRepoBranchLines +/-
Labs: make labs-ip-alias-dump ignore addressless hostsoperations/puppetproduction+16 -11
Customize query in gerrit

Related Objects

Event Timeline

• GWicke created this task.Dec 6 2015, 10:55 PM
• GWicke raised the priority of this task from to Needs Triage.
• GWicke updated the task description. (Show Details)
• GWicke added a project: Cloud-Services.
• GWicke added subscribers: • GWicke, yuvipanda.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 6 2015, 10:55 PM
hashar added projects: Continuous-Integration-Scaling, Continuous-Integration-Infrastructure.Dec 7 2015, 10:05 AM
hashar subscribed.

Nodepool (the CI process that spawn instances on labs) has been broken since Dec 6th 23:00 UTC at least. It is no more able to spawn instances :-(

hashar triaged this task as Unbreak Now! priority.Dec 7 2015, 10:08 AM

On the contintcloud labs project, I can get a list of servers and they are reachable:

nodepool@labnodepool1001:/var/log/nodepool$ openstack server list
+--------------------------------------+------------------------------------------------------+--------+---------------------+
| ID                                   | Name                                                 | Status | Networks            |
+--------------------------------------+------------------------------------------------------+--------+---------------------+
| 5aee0f17-4a78-4f0a-a858-b6949890a85c | ci-jessie-wikimedia-10700.contintcloud.eqiad.wmflabs | ACTIVE | public=10.68.16.24  |
| e1bb0cee-d2f7-4e31-8b94-9224f11a5a80 | ci-jessie-wikimedia-10699.contintcloud.eqiad.wmflabs | ACTIVE | public=10.68.21.46  |
| 177b3202-309b-4406-b1bd-9b9a6e6a10a1 | ci-jessie-wikimedia-10698.contintcloud.eqiad.wmflabs | ACTIVE | public=10.68.16.4   |
| 8b25c33a-836d-4e88-9a4a-3b261a54299f | ci-jessie-wikimedia-10697.contintcloud.eqiad.wmflabs | ACTIVE | public=10.68.21.43  |
| b7539c2f-af95-495c-8815-81637b93e5f7 | ci-jessie-wikimedia-10695.contintcloud.eqiad.wmflabs | ACTIVE | public=10.68.21.41  |
| ac52b0cf-5c26-42f1-ae87-2a591002d7f9 | ci-jessie-wikimedia-10694.contintcloud.eqiad.wmflabs | ACTIVE | public=10.68.21.42  |
| c3291858-696c-445c-8b11-d5a667a6e52d | ci-jessie-wikimedia-10692.contintcloud.eqiad.wmflabs | ACTIVE | public=10.68.16.159 |
| fc015081-e269-48b6-9f7a-631f6618c62b | ci-jessie-wikimedia-10687.contintcloud.eqiad.wmflabs | ACTIVE | public=10.68.21.33  |
| ca41bf98-feba-4e5a-97a9-67ff1abb3820 | ci-jessie-wikimedia-10667.contintcloud.eqiad.wmflabs | ACTIVE | public=10.68.21.31  |
+--------------------------------------+------------------------------------------------------+--------+---------------------+
nodepool@labnodepool1001:/var/log/nodepool$

Attempt to delete one seems to work client side, or at least no error is yield back:

...
$ openstack -v -v server delete ci-jessie-wikimedia-10667.contintcloud.eqiad.wmflabs
...
DEBUG: keystoneclient.session REQ: curl -g -i -X DELETE http://labnet1002.eqiad.wmnet:8774/v2/contintcloud/servers/ca41bf98-feba-4e5a-97a9-67ff1abb3820 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}XXXX"
DEBUG: urllib3.connectionpool "DELETE /v2/contintcloud/servers/ca41bf98-feba-4e5a-97a9-67ff1abb3820 HTTP/1.1" 204 0
DEBUG: keystoneclient.session RESP: [204] date: Mon, 07 Dec 2015 10:07:04 GMT content-length: 0 content-type: application/json connection: keep-alive
DEBUG: openstackclient.shell clean_up DeleteServer
hashar added a comment.Dec 7 2015, 10:13 AM

The last instances spawned were apparently on 2015-12-06T14:00:17 UTC.

yuvipanda added a comment.Dec 7 2015, 10:15 AM

There is a bunch of VMs that are stuck in the building/scheduling state:

root@labcontrol1001:/home/yuvipanda# nova show 9bb7842a-4580-4743-b3e8-571f32899a61
+--------------------------------------+-----------------------------------------------------------------------+
| Property                             | Value                                                                 |
+--------------------------------------+-----------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                                |
| OS-EXT-AZ:availability_zone          | nova                                                                  |
| OS-EXT-SRV-ATTR:host                 | -                                                                     |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                                     |
| OS-EXT-SRV-ATTR:instance_name        | i-00003f3f                                                            |
| OS-EXT-STS:power_state               | 0                                                                     |
| OS-EXT-STS:task_state                | scheduling                                                            |
| OS-EXT-STS:vm_state                  | building                                                              |
| OS-SRV-USG:launched_at               | -                                                                     |
| OS-SRV-USG:terminated_at             | -                                                                     |
| accessIPv4                           |                                                                       |
| accessIPv6                           |                                                                       |
| config_drive                         |                                                                       |
| created                              | 2015-12-07T10:13:15Z                                                  |
| flavor                               | m1.medium (3)                                                         |
| hostId                               |                                                                       |
| id                                   | 9bb7842a-4580-4743-b3e8-571f32899a61                                  |
| image                                | ci-jessie-wikimedia-1449324840 (6b841169-b205-4764-9b82-b449f9fe8299) |
| key_name                             | -                                                                     |
| metadata                             | {}                                                                    |
| name                                 | ci-jessie-wikimedia-11008.contintcloud.eqiad.wmflabs                  |
| os-extended-volumes:volumes_attached | []                                                                    |
| progress                             | 0                                                                     |
| security_groups                      | default                                                               |
| status                               | BUILD                                                                 |
| tenant_id                            | contintcloud                                                          |
| updated                              | 2015-12-07T10:15:24Z                                                  |
| user_id                              | nodepoolmanager                                                       |
+--------------------------------------+-----------------------------------------------------------------------+
yuvipanda added a comment.Dec 7 2015, 10:27 AM

Ok, I restarted nova-conductor and nova-scheduler, and that seems to have fixed it. There is also a labsservices1001 puppet failure that myteriously co-incided with this. Needs investigation!

yuvipanda lowered the priority of this task from Unbreak Now! to High.Dec 7 2015, 10:27 AM
yuvipanda added a comment.Dec 7 2015, 10:49 AM

The puppet failures were because the ip alias generator found instances that had no address (because they were stuck in the scheduler):

ESC[0;32mInfo: Applying configuration version '1449481984'ESC[0m
ESC[mNotice: /Stage[main]/Dnsrecursor::Labsaliaser/Exec[/usr/local/bin/labs-ip-alias-dump.py]/returns: Traceback (most recent call last):ESC[0m
ESC[mNotice: /Stage[main]/Dnsrecursor::Labsaliaser/Exec[/usr/local/bin/labs-ip-alias-dump.py]/returns:   File "/usr/local/bin/labs-ip-alias-dump.py", line 41, in <module>ESC[0m
ESC[mNotice: /Stage[main]/Dnsrecursor::Labsaliaser/Exec[/usr/local/bin/labs-ip-alias-dump.py]/returns:     str(ip['addr']) for ip in server.addresses['public'] if ip['OS-EXT-IPS:type'] == 'fixed'ESC[0m
ESC[mNotice: /Stage[main]/Dnsrecursor::Labsaliaser/Exec[/usr/local/bin/labs-ip-alias-dump.py]/returns: KeyError: 'public'ESC[0m
ESC[1;31mError: /usr/local/bin/labs-ip-alias-dump.py returned 1 instead of one of [0]ESC[0m
ESC[1;31mError: /Stage[main]/Dnsrecursor::Labsaliaser/Exec[/usr/local/bin/labs-ip-alias-dump.py]/returns: change from notrun to 0 failed: /usr/local/bin/labs-ip-alias-dump.py returned 1 instead of one of [0]ESC[0m
ESC[mNotice: /Stage[main]/Dnsrecursor/Service[pdns-recursor]: Dependency Exec[/usr/local/bin/labs-ip-alias-dump.py] has failures: trueESC[0m
ESC[1;31mWarning: /Stage[main]/Dnsrecursor/Service[pdns-recursor]: Skipping because of failed dependenciesESC[0m
ESC[mNotice: Finished catalog run in 23.67 secondsESC[0m
ESC[0;32mInfo: Sleeping for 25 seconds (splay is enabled)ESC[0m
ESC[0;32mInfo: Retrieving pluginESC[0m

So this was caused by the scheduling related stuff, but isn't otherwise related.

Luke081515 subscribed.EditedDec 7 2015, 11:29 AM

hm, creation of my instance rcm-5 worked. I start the creation at 21UTC, and one and a half hour later, it finished, so this is a long time, but it works...

coren claimed this task.Dec 7 2015, 2:09 PM
gerritbot subscribed.Dec 7 2015, 2:35 PM

Change 257323 had a related patch set uploaded (by coren):
Labs: make labs-ip-alias-dump ignore addressless hosts

https://gerrit.wikimedia.org/r/257323

gerritbot added a project: Patch-For-Review.Dec 7 2015, 2:35 PM
gerritbot added a comment.Dec 7 2015, 2:50 PM

Change 257323 merged by coren:
Labs: make labs-ip-alias-dump ignore addressless hosts

https://gerrit.wikimedia.org/r/257323

coren closed this task as Resolved.Dec 7 2015, 2:52 PM

@yuvipanda was correct that the puppet error was caused by the stall in the scheduler; the patch above prevents that edge case from breaking things.

hashar added a comment.Dec 7 2015, 2:57 PM

Since people mentioned issue with the DNS resolver aliasing, it is working again:

deployment-bastion:~$ dig +short ALL en.wikipedia.beta.wmflabs.org
10.68.18.103

Thanks!

yuvipanda reopened this task as Open.Dec 7 2015, 7:12 PM

The edge case was unrelated to anything here, let's keep this open for investigation.

hashar added a comment.Dec 7 2015, 7:53 PM

Nodepool uses the OpenStack API to delete instances. The first error occurred at 2015-12-06 08:00:30,030 (nothing earlier nor on 2015-12-05). Most due to timeout.

Lame grep for ERROR:

P2377 nodepool errors 2015-12-06
1hashar@labnodepool1001:/var/log/nodepool$ grep -h ERROR debug.log.2015-12-06*
22015-12-06 08:00:30,030 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
32015-12-06 08:11:40,142 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
42015-12-06 08:22:23,287 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
52015-12-06 08:33:22,133 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
62015-12-06 08:44:25,374 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
72015-12-06 08:55:37,121 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
82015-12-06 09:06:22,673 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
92015-12-06 09:17:23,313 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
102015-12-06 09:28:54,441 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
112015-12-06 09:39:37,792 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
122015-12-06 09:50:24,027 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
132015-12-06 10:01:23,846 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
142015-12-06 10:13:06,130 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
152015-12-06 10:24:24,131 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
162015-12-06 10:35:22,464 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
172015-12-06 10:46:22,697 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
182015-12-06 10:57:23,262 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
192015-12-06 11:08:23,106 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
202015-12-06 11:19:22,696 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
212015-12-06 11:30:23,200 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
222015-12-06 11:41:23,226 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
232015-12-06 11:52:24,065 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
242015-12-06 12:03:23,180 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
252015-12-06 12:14:23,363 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
262015-12-06 12:25:23,356 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
272015-12-06 12:36:23,104 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
282015-12-06 12:47:23,701 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
292015-12-06 12:58:23,181 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
302015-12-06 13:09:22,739 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
312015-12-06 13:20:23,121 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
322015-12-06 13:31:23,658 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
332015-12-06 13:42:22,558 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
342015-12-06 13:53:23,844 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
352015-12-06 14:04:22,385 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
362015-12-06 14:12:22,242 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
372015-12-06 14:13:03,837 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
382015-12-06 14:13:18,223 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
392015-12-06 14:15:26,324 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
402015-12-06 14:20:23,574 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
412015-12-06 14:23:32,828 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
422015-12-06 14:24:29,299 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
432015-12-06 14:24:49,743 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
442015-12-06 14:26:33,892 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
452015-12-06 14:31:35,324 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
462015-12-06 14:34:32,921 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
472015-12-06 14:35:25,390 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
482015-12-06 14:35:46,486 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
492015-12-06 14:37:28,380 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
502015-12-06 14:42:35,520 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
512015-12-06 14:43:07,780 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
522015-12-06 14:43:43,878 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10711 in provider: wmflabs-eqiad error:
532015-12-06 14:44:04,357 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10712 in provider: wmflabs-eqiad error:
542015-12-06 14:44:34,834 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10713 in provider: wmflabs-eqiad error:
552015-12-06 14:45:06,189 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10714 in provider: wmflabs-eqiad error:
562015-12-06 14:45:26,729 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10715 in provider: wmflabs-eqiad error:
572015-12-06 14:45:29,559 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
582015-12-06 14:45:47,162 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10716 in provider: wmflabs-eqiad error:
592015-12-06 14:46:07,540 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10717 in provider: wmflabs-eqiad error:
602015-12-06 14:46:32,849 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
612015-12-06 14:46:38,142 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10718 in provider: wmflabs-eqiad error:
622015-12-06 14:46:53,844 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
632015-12-06 14:47:26,328 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
642015-12-06 14:48:26,364 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
652015-12-06 14:48:31,178 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10720 in provider: wmflabs-eqiad error:
662015-12-06 14:48:38,401 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
672015-12-06 14:49:18,414 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
682015-12-06 14:53:34,343 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
692015-12-06 14:54:57,068 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
702015-12-06 14:56:49,039 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
712015-12-06 14:57:30,768 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
722015-12-06 14:57:40,731 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
732015-12-06 14:58:54,305 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
742015-12-06 14:59:24,334 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
752015-12-06 14:59:44,873 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
762015-12-06 15:00:46,802 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
772015-12-06 15:02:12,073 ERROR nodepool.NodeLauncher: Exception launching node id: 10705 in provider: wmflabs-eqiad error:
782015-12-06 15:02:12,076 ERROR nodepool.NodeLauncher: Exception in run method:
792015-12-06 15:04:33,056 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
802015-12-06 15:05:45,442 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
812015-12-06 15:05:48,334 ERROR nodepool.NodeLauncher: Exception launching node id: 10706 in provider: wmflabs-eqiad error:
822015-12-06 15:05:48,335 ERROR nodepool.NodeLauncher: Exception in run method:
832015-12-06 15:07:36,937 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
842015-12-06 15:08:27,623 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
852015-12-06 15:08:59,435 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
862015-12-06 15:09:39,396 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
872015-12-06 15:10:21,589 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
882015-12-06 15:10:41,131 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
892015-12-06 15:11:33,127 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
902015-12-06 15:15:01,581 ERROR nodepool.SnapshotImageUpdater: Exception updating image ci-jessie-wikimedia in wmflabs-eqiad:
912015-12-06 15:15:01,583 ERROR nodepool.NodePool: Could not delete image 385
922015-12-06 15:15:29,524 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
932015-12-06 15:16:34,315 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
942015-12-06 15:18:28,346 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
952015-12-06 15:19:18,260 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
962015-12-06 15:19:40,675 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
972015-12-06 15:20:30,083 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
982015-12-06 15:21:22,323 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
992015-12-06 15:21:42,122 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
1002015-12-06 15:22:35,557 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
1012015-12-06 15:26:29,632 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
1022015-12-06 15:27:31,873 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
1032015-12-06 15:28:20,701 ERROR nodepool.NodeLauncher: Exception launching node id: 10707 in provider: wmflabs-eqiad error:
1042015-12-06 15:28:20,703 ERROR nodepool.NodeLauncher: Exception in run method:
1052015-12-06 15:29:34,743 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
1062015-12-06 15:30:25,748 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
1072015-12-06 15:30:46,905 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
1082015-12-06 15:31:26,662 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
1092015-12-06 15:32:19,170 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
1102015-12-06 15:32:39,997 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
1112015-12-06 15:33:29,503 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
1122015-12-06 15:35:41,597 ERROR nodepool.NodeLauncher: Exception launching node id: 10708 in provider: wmflabs-eqiad error:
1132015-12-06 15:35:41,599 ERROR nodepool.NodeLauncher: Exception in run method:
1142015-12-06 15:35:51,474 ERROR nodepool.NodeLauncher: Exception launching node id: 10709 in provider: wmflabs-eqiad error:
1152015-12-06 15:35:51,477 ERROR nodepool.NodeLauncher: Exception in run method:
1162015-12-06 15:37:37,809 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
1172015-12-06 15:38:28,567 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
1182015-12-06 15:40:33,615 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
1192015-12-06 15:41:26,700 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
1202015-12-06 15:41:46,243 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
1212015-12-06 15:42:26,271 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
1222015-12-06 15:43:28,601 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
1232015-12-06 15:43:48,375 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
1242015-12-06 15:44:30,271 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
1252015-12-06 15:48:22,844 ERROR nodepool.NodeLauncher: Exception launching node id: 10719 in provider: wmflabs-eqiad error:
1262015-12-06 15:48:22,847 ERROR nodepool.NodeLauncher: Exception in run method:
1272015-12-06 15:48:32,979 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
1282015-12-06 15:49:33,299 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
1292015-12-06 15:51:35,610 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
1302015-12-06 15:52:18,020 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
1312015-12-06 15:52:38,953 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
1322015-12-06 15:53:30,218 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
1332015-12-06 15:54:22,008 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
1342015-12-06 15:54:42,895 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
1352015-12-06 15:55:44,159 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
1362015-12-06 15:59:44,359 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
1372015-12-06 16:00:34,364 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
1382015-12-06 16:02:29,365 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
1392015-12-06 16:03:20,846 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
1402015-12-06 16:03:40,693 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
1412015-12-06 16:04:32,949 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
1422015-12-06 16:05:22,711 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
1432015-12-06 16:05:44,480 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
1442015-12-06 16:06:36,565 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
1452015-12-06 16:10:31,185 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
1462015-12-06 16:11:31,235 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
1472015-12-06 16:13:34,724 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
1482015-12-06 16:14:24,655 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
1492015-12-06 16:14:46,627 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
1502015-12-06 16:15:28,304 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
1512015-12-06 16:16:19,583 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
1522015-12-06 16:16:41,101 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
1532015-12-06 16:17:32,632 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
1542015-12-06 16:21:26,378 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
1552015-12-06 16:22:29,173 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
1562015-12-06 16:24:29,803 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
1572015-12-06 16:25:23,360 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
1582015-12-06 16:25:43,226 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
1592015-12-06 16:26:33,644 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
1602015-12-06 16:26:44,539 ERROR nodepool.NodeLauncher: Exception launching node id: 10723 in provider: wmflabs-eqiad error:
1612015-12-06 16:26:44,541 ERROR nodepool.NodeLauncher: Exception in run method:
1622015-12-06 16:27:46,982 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
1632015-12-06 16:27:55,722 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
1642015-12-06 16:28:29,140 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
1652015-12-06 16:32:33,445 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
1662015-12-06 16:33:34,539 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
1672015-12-06 16:35:36,407 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
1682015-12-06 16:36:26,718 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
1692015-12-06 16:36:38,651 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
1702015-12-06 16:36:49,152 ERROR nodepool.NodeLauncher: Exception launching node id: 10726 in provider: wmflabs-eqiad error:
1712015-12-06 16:36:49,155 ERROR nodepool.NodeLauncher: Exception in run method:
1722015-12-06 16:37:45,031 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
1732015-12-06 16:38:25,878 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
1742015-12-06 16:38:46,498 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
1752015-12-06 16:39:27,152 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
1762015-12-06 16:43:23,087 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
1772015-12-06 16:44:42,849 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
1782015-12-06 16:46:27,429 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
1792015-12-06 16:47:27,641 ERROR nodepool.NodeLauncher: Exception launching node id: 10729 in provider: wmflabs-eqiad error:
1802015-12-06 16:47:27,644 ERROR nodepool.NodeLauncher: Exception in run method:
1812015-12-06 16:47:41,838 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
1822015-12-06 16:48:13,902 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
1832015-12-06 16:48:43,544 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
1842015-12-06 16:48:50,837 ERROR nodepool.NodeLauncher: Exception launching node id: 10730 in provider: wmflabs-eqiad error:
1852015-12-06 16:48:50,840 ERROR nodepool.NodeLauncher: Exception in run method:
1862015-12-06 16:49:24,642 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
1872015-12-06 16:49:46,394 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
1882015-12-06 16:50:26,578 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
1892015-12-06 16:54:30,715 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
1902015-12-06 16:54:36,491 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10744 in provider: wmflabs-eqiad error:
1912015-12-06 16:55:06,929 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10745 in provider: wmflabs-eqiad error:
1922015-12-06 16:55:33,097 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
1932015-12-06 16:57:36,074 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
1942015-12-06 16:59:01,624 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
1952015-12-06 16:59:43,282 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
1962015-12-06 16:59:53,681 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
1972015-12-06 17:00:37,621 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
1982015-12-06 17:00:59,203 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
1992015-12-06 17:01:39,875 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
2002015-12-06 17:06:10,036 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
2012015-12-06 17:06:40,464 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
2022015-12-06 17:08:32,344 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
2032015-12-06 17:10:26,012 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
2042015-12-06 17:10:35,820 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
2052015-12-06 17:11:02,089 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
2062015-12-06 17:11:26,106 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
2072015-12-06 17:11:47,639 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
2082015-12-06 17:12:28,630 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
2092015-12-06 17:17:24,784 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
2102015-12-06 17:17:47,069 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
2112015-12-06 17:19:29,433 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
2122015-12-06 17:21:21,495 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
2132015-12-06 17:21:42,211 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
2142015-12-06 17:22:25,155 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
2152015-12-06 17:22:35,798 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
2162015-12-06 17:22:56,056 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
2172015-12-06 17:23:37,431 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
2182015-12-06 17:28:23,947 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
2192015-12-06 17:28:45,022 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
2202015-12-06 17:30:28,184 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
2212015-12-06 17:30:32,956 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10747 in provider: wmflabs-eqiad error:
2222015-12-06 17:30:53,471 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10748 in provider: wmflabs-eqiad error:
2232015-12-06 17:31:24,105 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10749 in provider: wmflabs-eqiad error:
2242015-12-06 17:31:54,988 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10750 in provider: wmflabs-eqiad error:
2252015-12-06 17:32:15,482 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10751 in provider: wmflabs-eqiad error:
2262015-12-06 17:32:19,288 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
2272015-12-06 17:32:35,882 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10752 in provider: wmflabs-eqiad error:
2282015-12-06 17:32:40,452 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
2292015-12-06 17:32:56,324 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10753 in provider: wmflabs-eqiad error:
2302015-12-06 17:33:22,873 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
2312015-12-06 17:33:32,597 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
2322015-12-06 17:33:37,250 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10754 in provider: wmflabs-eqiad error:
2332015-12-06 17:33:53,112 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
2342015-12-06 17:34:08,548 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10755 in provider: wmflabs-eqiad error:
2352015-12-06 17:34:34,902 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
2362015-12-06 17:35:01,101 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10756 in provider: wmflabs-eqiad error:
2372015-12-06 17:35:53,008 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10757 in provider: wmflabs-eqiad error:
2382015-12-06 17:36:24,379 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10758 in provider: wmflabs-eqiad error:
2392015-12-06 17:39:21,227 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
2402015-12-06 17:39:43,589 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
2412015-12-06 17:41:46,623 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
2422015-12-06 17:43:27,420 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
2432015-12-06 17:43:59,743 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
2442015-12-06 17:44:39,910 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
2452015-12-06 17:44:52,500 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
2462015-12-06 17:45:23,410 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
2472015-12-06 17:45:43,555 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
2482015-12-06 17:50:21,469 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
2492015-12-06 17:50:42,427 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
2502015-12-06 17:52:35,524 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
2512015-12-06 17:53:24,575 ERROR nodepool.NodeLauncher: Exception launching node id: 10736 in provider: wmflabs-eqiad error:
2522015-12-06 17:53:24,577 ERROR nodepool.NodeLauncher: Exception in run method:
2532015-12-06 17:53:53,514 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10760 in provider: wmflabs-eqiad error:
2542015-12-06 17:53:56,464 ERROR nodepool.NodeLauncher: Exception launching node id: 10740 in provider: wmflabs-eqiad error:
2552015-12-06 17:53:56,467 ERROR nodepool.NodeLauncher: Exception in run method:
2562015-12-06 17:54:16,354 ERROR nodepool.NodeLauncher: Exception launching node id: 10742 in provider: wmflabs-eqiad error:
2572015-12-06 17:54:16,357 ERROR nodepool.NodeLauncher: Exception in run method:
2582015-12-06 17:54:17,933 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
2592015-12-06 17:54:38,716 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
2602015-12-06 17:55:20,378 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
2612015-12-06 17:55:40,225 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
2622015-12-06 17:55:49,497 ERROR nodepool.NodeLauncher: Exception launching node id: 10746 in provider: wmflabs-eqiad error:
2632015-12-06 17:55:49,500 ERROR nodepool.NodeLauncher: Exception in run method:
2642015-12-06 17:56:23,411 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
2652015-12-06 17:56:42,822 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
2662015-12-06 17:56:49,050 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10764 in provider: wmflabs-eqiad error:
2672015-12-06 18:01:22,936 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
2682015-12-06 18:01:40,355 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
2692015-12-06 18:03:33,104 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
2702015-12-06 18:05:36,681 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
2712015-12-06 18:06:40,079 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
2722015-12-06 18:07:00,048 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
2732015-12-06 18:07:21,010 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
2742015-12-06 18:07:50,506 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10766 in provider: wmflabs-eqiad error:
2752015-12-06 18:08:07,471 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
2762015-12-06 18:08:10,972 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10767 in provider: wmflabs-eqiad error:
2772015-12-06 18:08:27,019 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
2782015-12-06 18:09:02,696 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10768 in provider: wmflabs-eqiad error:
2792015-12-06 18:09:43,461 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10769 in provider: wmflabs-eqiad error:
2802015-12-06 18:10:14,872 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10770 in provider: wmflabs-eqiad error:
2812015-12-06 18:10:35,430 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10771 in provider: wmflabs-eqiad error:
2822015-12-06 18:10:56,040 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10772 in provider: wmflabs-eqiad error:
2832015-12-06 18:11:16,544 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10773 in provider: wmflabs-eqiad error:
2842015-12-06 18:11:47,056 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10774 in provider: wmflabs-eqiad error:
2852015-12-06 18:12:17,680 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10775 in provider: wmflabs-eqiad error:
2862015-12-06 18:12:23,130 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
2872015-12-06 18:12:43,373 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
2882015-12-06 18:12:59,435 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10777 in provider: wmflabs-eqiad error:
2892015-12-06 18:13:30,056 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10778 in provider: wmflabs-eqiad error:
2902015-12-06 18:14:11,629 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10779 in provider: wmflabs-eqiad error:
2912015-12-06 18:14:32,136 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10780 in provider: wmflabs-eqiad error:
2922015-12-06 18:14:36,047 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
2932015-12-06 18:14:52,552 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10781 in provider: wmflabs-eqiad error:
2942015-12-06 18:15:22,981 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10782 in provider: wmflabs-eqiad error:
2952015-12-06 18:15:53,630 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10783 in provider: wmflabs-eqiad error:
2962015-12-06 18:16:14,010 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10784 in provider: wmflabs-eqiad error:
2972015-12-06 18:16:29,365 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
2982015-12-06 18:16:34,488 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10785 in provider: wmflabs-eqiad error:
2992015-12-06 18:16:54,988 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10786 in provider: wmflabs-eqiad error:
3002015-12-06 18:17:15,421 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10787 in provider: wmflabs-eqiad error:
3012015-12-06 18:17:30,116 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
3022015-12-06 18:17:45,871 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10788 in provider: wmflabs-eqiad error:
3032015-12-06 18:18:16,629 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10789 in provider: wmflabs-eqiad error:
3042015-12-06 18:18:32,190 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
3052015-12-06 18:18:52,899 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
3062015-12-06 18:18:58,165 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10790 in provider: wmflabs-eqiad error:
3072015-12-06 18:19:34,687 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
3082015-12-06 18:20:06,468 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
3092015-12-06 18:23:27,317 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10792 in provider: wmflabs-eqiad error:
3102015-12-06 18:23:40,790 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
3112015-12-06 18:24:01,600 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
3122015-12-06 18:25:44,973 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
3132015-12-06 18:27:57,960 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
3142015-12-06 18:28:39,612 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
3152015-12-06 18:29:27,950 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10795 in provider: wmflabs-eqiad error:
3162015-12-06 18:29:52,276 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
3172015-12-06 18:30:09,727 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10796 in provider: wmflabs-eqiad error:
3182015-12-06 18:30:13,666 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
3192015-12-06 18:30:43,943 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
3202015-12-06 18:31:34,899 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
3212015-12-06 18:34:28,792 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
3222015-12-06 18:34:59,173 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10798 in provider: wmflabs-eqiad error:
3232015-12-06 18:35:30,632 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
3242015-12-06 18:36:32,717 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
3252015-12-06 18:38:46,585 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
3262015-12-06 18:40:21,156 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
3272015-12-06 18:40:30,013 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10802 in provider: wmflabs-eqiad error:
3282015-12-06 18:41:00,531 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10803 in provider: wmflabs-eqiad error:
3292015-12-06 18:41:14,789 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
3302015-12-06 18:41:35,075 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
3312015-12-06 18:41:54,923 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
3322015-12-06 18:42:37,114 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
3332015-12-06 18:45:41,206 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
3342015-12-06 18:46:31,752 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
3352015-12-06 18:47:43,562 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
3362015-12-06 18:49:27,556 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
3372015-12-06 18:52:02,584 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
3382015-12-06 18:52:34,502 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
3392015-12-06 18:52:44,452 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
3402015-12-06 18:53:06,446 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
3412015-12-06 18:53:46,628 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
3422015-12-06 18:55:47,802 ERROR nodepool.NodeLauncher: Exception launching node id: 10761 in provider: wmflabs-eqiad error:
3432015-12-06 18:55:47,803 ERROR nodepool.NodeLauncher: Exception in run method:
3442015-12-06 18:56:30,510 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
3452015-12-06 18:57:31,620 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
3462015-12-06 18:57:46,326 ERROR nodepool.NodeLauncher: Exception launching node id: 10765 in provider: wmflabs-eqiad error:
3472015-12-06 18:57:46,328 ERROR nodepool.NodeLauncher: Exception in run method:
3482015-12-06 18:58:10,532 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10806 in provider: wmflabs-eqiad error:
3492015-12-06 18:58:34,406 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
3502015-12-06 19:00:36,746 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
3512015-12-06 19:03:20,858 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
3522015-12-06 19:03:30,888 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
3532015-12-06 19:03:50,979 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
3542015-12-06 19:04:23,352 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
3552015-12-06 19:04:43,276 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
3562015-12-06 19:07:29,868 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
3572015-12-06 19:08:33,476 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
3582015-12-06 19:09:37,860 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
3592015-12-06 19:11:29,982 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
3602015-12-06 19:12:40,500 ERROR nodepool.NodeLauncher: Exception launching node id: 10776 in provider: wmflabs-eqiad error:
3612015-12-06 19:12:40,502 ERROR nodepool.NodeLauncher: Exception in run method:
3622015-12-06 19:14:22,625 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
3632015-12-06 19:14:35,002 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
3642015-12-06 19:14:55,077 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
3652015-12-06 19:15:26,648 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
3662015-12-06 19:15:46,334 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
3672015-12-06 19:18:31,903 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
3682015-12-06 19:19:32,920 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
3692015-12-06 19:19:40,924 ERROR nodepool.NodeLauncher: Exception launching node id: 10791 in provider: wmflabs-eqiad error:
3702015-12-06 19:19:40,927 ERROR nodepool.NodeLauncher: Exception in run method:
3712015-12-06 19:20:35,112 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
3722015-12-06 19:22:26,247 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
3732015-12-06 19:25:25,479 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
3742015-12-06 19:25:37,684 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
3752015-12-06 19:25:58,654 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
3762015-12-06 19:26:29,086 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
3772015-12-06 19:26:50,395 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
3782015-12-06 19:29:19,382 ERROR nodepool.NodeLauncher: Exception launching node id: 10794 in provider: wmflabs-eqiad error:
3792015-12-06 19:29:19,385 ERROR nodepool.NodeLauncher: Exception in run method:
3802015-12-06 19:29:32,841 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
3812015-12-06 19:29:43,019 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10810 in provider: wmflabs-eqiad error:
3822015-12-06 19:30:53,369 ERROR nodepool.NodeLauncher: Exception launching node id: 10797 in provider: wmflabs-eqiad error:
3832015-12-06 19:30:53,371 ERROR nodepool.NodeLauncher: Exception in run method:
3842015-12-06 19:30:59,639 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
3852015-12-06 19:31:40,635 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10812 in provider: wmflabs-eqiad error:
3862015-12-06 19:31:41,816 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
3872015-12-06 19:33:34,329 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
3882015-12-06 19:36:17,977 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
3892015-12-06 19:36:38,563 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
3902015-12-06 19:36:52,655 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
3912015-12-06 19:37:19,182 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
3922015-12-06 19:37:40,247 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
3932015-12-06 19:40:09,777 ERROR nodepool.NodeLauncher: Exception launching node id: 10800 in provider: wmflabs-eqiad error:
3942015-12-06 19:40:09,780 ERROR nodepool.NodeLauncher: Exception in run method:
3952015-12-06 19:40:37,305 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10814 in provider: wmflabs-eqiad error:
3962015-12-06 19:41:00,943 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
3972015-12-06 19:41:30,825 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
3982015-12-06 19:42:54,354 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10816 in provider: wmflabs-eqiad error:
3992015-12-06 19:43:06,595 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
4002015-12-06 19:43:14,831 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10817 in provider: wmflabs-eqiad error:
4012015-12-06 19:43:35,488 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10818 in provider: wmflabs-eqiad error:
4022015-12-06 19:44:27,381 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
4032015-12-06 19:44:57,821 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10820 in provider: wmflabs-eqiad error:
4042015-12-06 19:46:31,171 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10822 in provider: wmflabs-eqiad error:
4052015-12-06 19:47:01,686 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10823 in provider: wmflabs-eqiad error:
4062015-12-06 19:47:21,942 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
4072015-12-06 19:47:32,439 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10824 in provider: wmflabs-eqiad error:
4082015-12-06 19:47:33,573 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
4092015-12-06 19:47:52,670 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
4102015-12-06 19:48:25,488 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
4112015-12-06 19:48:45,297 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10826 in provider: wmflabs-eqiad error:
4122015-12-06 19:48:45,595 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
4132015-12-06 19:49:16,594 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10827 in provider: wmflabs-eqiad error:
4142015-12-06 19:49:57,286 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10828 in provider: wmflabs-eqiad error:
4152015-12-06 19:50:28,716 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10829 in provider: wmflabs-eqiad error:
4162015-12-06 19:50:49,292 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10830 in provider: wmflabs-eqiad error:
4172015-12-06 19:51:09,833 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10831 in provider: wmflabs-eqiad error:
4182015-12-06 19:51:30,329 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10832 in provider: wmflabs-eqiad error:
4192015-12-06 19:51:51,108 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10833 in provider: wmflabs-eqiad error:
4202015-12-06 19:52:11,564 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10834 in provider: wmflabs-eqiad error:
4212015-12-06 19:52:23,003 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
4222015-12-06 19:52:44,813 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
4232015-12-06 19:54:25,086 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10836 in provider: wmflabs-eqiad error:
4242015-12-06 19:54:27,493 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
4252015-12-06 19:54:45,477 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10837 in provider: wmflabs-eqiad error:
4262015-12-06 19:55:05,864 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10838 in provider: wmflabs-eqiad error:
4272015-12-06 19:55:28,993 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
4282015-12-06 19:55:36,303 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10839 in provider: wmflabs-eqiad error:
4292015-12-06 19:56:07,095 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10840 in provider: wmflabs-eqiad error:
4302015-12-06 19:56:30,232 ERROR nodepool.NodeLauncher: Exception launching node id: 10805 in provider: wmflabs-eqiad error:
4312015-12-06 19:56:30,235 ERROR nodepool.NodeLauncher: Exception in run method:
4322015-12-06 19:56:37,523 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10841 in provider: wmflabs-eqiad error:
4332015-12-06 19:58:24,247 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
4342015-12-06 19:58:35,653 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
4352015-12-06 19:58:55,006 ERROR nodepool.NodeLauncher: Exception launching node id: 10807 in provider: wmflabs-eqiad error:
4362015-12-06 19:58:55,008 ERROR nodepool.NodeLauncher: Exception in run method:
4372015-12-06 19:59:16,402 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
4382015-12-06 19:59:45,087 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10844 in provider: wmflabs-eqiad error:
4392015-12-06 19:59:47,759 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
4402015-12-06 20:00:08,947 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
4412015-12-06 20:03:33,850 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
4422015-12-06 20:03:55,866 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
4432015-12-06 20:05:48,316 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
4442015-12-06 20:06:49,980 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
4452015-12-06 20:09:24,768 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
4462015-12-06 20:09:36,729 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
4472015-12-06 20:10:39,496 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
4482015-12-06 20:11:00,487 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
4492015-12-06 20:11:41,127 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
4502015-12-06 20:13:22,686 ERROR nodepool.NodeLauncher: Exception launching node id: 10808 in provider: wmflabs-eqiad error:
4512015-12-06 20:13:22,688 ERROR nodepool.NodeLauncher: Exception in run method:
4522015-12-06 20:14:25,087 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
4532015-12-06 20:14:45,012 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
4542015-12-06 20:16:27,325 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
4552015-12-06 20:17:38,171 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
4562015-12-06 20:20:25,485 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
4572015-12-06 20:20:29,277 ERROR nodepool.NodeLauncher: Exception launching node id: 10809 in provider: wmflabs-eqiad error:
4582015-12-06 20:20:29,280 ERROR nodepool.NodeLauncher: Exception in run method:
4592015-12-06 20:20:45,712 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
4602015-12-06 20:21:37,663 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
4612015-12-06 20:22:17,356 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
4622015-12-06 20:22:38,066 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
4632015-12-06 20:25:24,873 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
4642015-12-06 20:25:45,763 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
4652015-12-06 20:27:27,565 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
4662015-12-06 20:28:28,591 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
4672015-12-06 20:30:17,862 ERROR nodepool.NodeLauncher: Exception launching node id: 10811 in provider: wmflabs-eqiad error:
4682015-12-06 20:30:17,865 ERROR nodepool.NodeLauncher: Exception in run method:
4692015-12-06 20:31:36,542 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
4702015-12-06 20:31:57,640 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
4712015-12-06 20:32:14,695 ERROR nodepool.NodeLauncher: Exception launching node id: 10813 in provider: wmflabs-eqiad error:
4722015-12-06 20:32:14,698 ERROR nodepool.NodeLauncher: Exception in run method:
4732015-12-06 20:32:36,720 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
4742015-12-06 20:33:28,462 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
4752015-12-06 20:33:40,984 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
4762015-12-06 20:36:22,398 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
4772015-12-06 20:36:45,087 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
4782015-12-06 20:38:35,803 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
4792015-12-06 20:39:28,707 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
4802015-12-06 20:41:12,747 ERROR nodepool.NodeLauncher: Exception launching node id: 10815 in provider: wmflabs-eqiad error:
4812015-12-06 20:41:12,749 ERROR nodepool.NodeLauncher: Exception in run method:
4822015-12-06 20:42:22,665 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
4832015-12-06 20:42:43,811 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
4842015-12-06 20:43:37,465 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
4852015-12-06 20:44:19,827 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
4862015-12-06 20:44:40,744 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
4872015-12-06 20:47:22,498 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
4882015-12-06 20:47:44,356 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
4892015-12-06 20:47:54,857 ERROR nodepool.NodeLauncher: Exception launching node id: 10825 in provider: wmflabs-eqiad error:
4902015-12-06 20:47:54,860 ERROR nodepool.NodeLauncher: Exception in run method:
4912015-12-06 20:49:37,103 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
4922015-12-06 20:50:27,339 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
4932015-12-06 20:53:23,205 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
4942015-12-06 20:53:45,227 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
4952015-12-06 20:54:35,586 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
4962015-12-06 20:55:27,896 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
4972015-12-06 20:55:47,361 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
4982015-12-06 20:58:03,539 ERROR nodepool.NodeLauncher: Exception launching node id: 10843 in provider: wmflabs-eqiad error:
4992015-12-06 20:58:03,540 ERROR nodepool.NodeLauncher: Exception in run method:
5002015-12-06 20:58:22,020 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
5012015-12-06 20:59:02,109 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
5022015-12-06 21:00:18,934 ERROR nodepool.NodeLauncher: Exception launching node id: 10845 in provider: wmflabs-eqiad error:
5032015-12-06 21:00:18,935 ERROR nodepool.NodeLauncher: Exception in run method:
5042015-12-06 21:00:36,793 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
5052015-12-06 21:01:28,001 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
5062015-12-06 21:04:21,087 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
5072015-12-06 21:04:42,058 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
5082015-12-06 21:05:32,918 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
5092015-12-06 21:06:23,907 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
5102015-12-06 21:06:58,404 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
5112015-12-06 21:09:31,071 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
5122015-12-06 21:10:32,480 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
5132015-12-06 21:11:56,358 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
5142015-12-06 21:12:36,637 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
5152015-12-06 21:16:15,624 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
5162015-12-06 21:16:37,161 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
5172015-12-06 21:16:47,603 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
5182015-12-06 21:17:17,415 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
5192015-12-06 21:17:39,404 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
5202015-12-06 21:20:33,611 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
5212015-12-06 21:21:35,990 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
5222015-12-06 21:22:36,492 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
5232015-12-06 21:23:26,966 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
5242015-12-06 21:27:22,828 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
5252015-12-06 21:27:34,909 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
5262015-12-06 21:27:54,929 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
5272015-12-06 21:28:26,665 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
5282015-12-06 21:28:47,645 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
5292015-12-06 21:31:30,620 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
5302015-12-06 21:32:32,436 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
5312015-12-06 21:33:34,315 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
5322015-12-06 21:34:35,297 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
5332015-12-06 21:38:21,202 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
5342015-12-06 21:38:32,348 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
5352015-12-06 21:38:54,541 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
5362015-12-06 21:39:24,433 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
5372015-12-06 21:39:45,064 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
5382015-12-06 21:42:29,652 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
5392015-12-06 21:43:31,104 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
5402015-12-06 21:44:33,431 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
5412015-12-06 21:45:09,157 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10860 in provider: wmflabs-eqiad error:
5422015-12-06 21:45:35,015 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
5432015-12-06 21:45:49,945 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10861 in provider: wmflabs-eqiad error:
5442015-12-06 21:46:10,577 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10862 in provider: wmflabs-eqiad error:
5452015-12-06 21:46:51,459 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10863 in provider: wmflabs-eqiad error:
5462015-12-06 21:49:19,826 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
5472015-12-06 21:49:31,671 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
5482015-12-06 21:49:51,623 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
5492015-12-06 21:50:21,517 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
5502015-12-06 21:50:44,342 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
5512015-12-06 21:53:28,098 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
5522015-12-06 21:54:29,833 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
5532015-12-06 21:55:41,327 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
5542015-12-06 21:56:31,118 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
5552015-12-06 22:00:19,265 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
5562015-12-06 22:00:35,935 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10865 in provider: wmflabs-eqiad error:
5572015-12-06 22:00:48,159 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
5582015-12-06 22:00:50,326 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
5592015-12-06 22:00:56,384 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10866 in provider: wmflabs-eqiad error:
5602015-12-06 22:01:17,052 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10867 in provider: wmflabs-eqiad error:
5612015-12-06 22:01:22,947 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
5622015-12-06 22:01:43,178 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
5632015-12-06 22:02:08,102 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10868 in provider: wmflabs-eqiad error:
5642015-12-06 22:02:59,859 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10869 in provider: wmflabs-eqiad error:
5652015-12-06 22:03:30,626 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10870 in provider: wmflabs-eqiad error:
5662015-12-06 22:03:51,071 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10871 in provider: wmflabs-eqiad error:
5672015-12-06 22:04:11,787 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10872 in provider: wmflabs-eqiad error:
5682015-12-06 22:04:21,821 ERROR nodepool.NodeLauncher: Exception launching node id: 10851 in provider: wmflabs-eqiad error:
5692015-12-06 22:04:21,823 ERROR nodepool.NodeLauncher: Exception in run method:
5702015-12-06 22:04:27,220 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
5712015-12-06 22:04:32,213 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10873 in provider: wmflabs-eqiad error:
5722015-12-06 22:05:13,848 ERROR nodepool.NodeLauncher: Exception launching node id: 10856 in provider: wmflabs-eqiad error:
5732015-12-06 22:05:13,851 ERROR nodepool.NodeLauncher: Exception in run method:
5742015-12-06 22:05:23,813 ERROR nodepool.NodeLauncher: Exception launching node id: 10857 in provider: wmflabs-eqiad error:
5752015-12-06 22:05:23,815 ERROR nodepool.NodeLauncher: Exception in run method:
5762015-12-06 22:05:28,562 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
5772015-12-06 22:07:32,239 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
5782015-12-06 22:07:38,725 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
5792015-12-06 22:07:57,119 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10878 in provider: wmflabs-eqiad error:
5802015-12-06 22:08:38,623 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10879 in provider: wmflabs-eqiad error:
5812015-12-06 22:09:10,005 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10880 in provider: wmflabs-eqiad error:
5822015-12-06 22:09:30,413 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10881 in provider: wmflabs-eqiad error:
5832015-12-06 22:09:50,765 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10882 in provider: wmflabs-eqiad error:
5842015-12-06 22:10:11,134 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10883 in provider: wmflabs-eqiad error:
5852015-12-06 22:10:31,542 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10884 in provider: wmflabs-eqiad error:
5862015-12-06 22:10:52,041 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10885 in provider: wmflabs-eqiad error:
5872015-12-06 22:11:12,481 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10886 in provider: wmflabs-eqiad error:
5882015-12-06 22:11:39,134 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
5892015-12-06 22:11:42,971 ERROR nodepool.NodeLauncher: Forbidden launching node id: 10887 in provider: wmflabs-eqiad error:
5902015-12-06 22:11:48,770 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
5912015-12-06 22:12:26,183 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
5922015-12-06 22:12:57,393 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
5932015-12-06 22:13:10,765 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
5942015-12-06 22:15:36,924 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
5952015-12-06 22:16:31,667 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
5962015-12-06 22:18:31,310 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
5972015-12-06 22:19:01,567 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
5982015-12-06 22:22:47,123 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
5992015-12-06 22:23:07,458 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
6002015-12-06 22:23:39,410 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
6012015-12-06 22:23:49,189 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
6022015-12-06 22:24:41,038 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
6032015-12-06 22:26:37,030 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
6042015-12-06 22:27:29,596 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
6052015-12-06 22:29:31,253 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
6062015-12-06 22:30:33,234 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
6072015-12-06 22:33:27,394 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
6082015-12-06 22:34:20,847 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
6092015-12-06 22:34:31,545 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
6102015-12-06 22:34:51,960 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
6112015-12-06 22:35:33,566 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
6122015-12-06 22:37:36,831 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
6132015-12-06 22:38:27,322 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
6142015-12-06 22:40:31,211 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
6152015-12-06 22:41:54,140 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
6162015-12-06 22:44:28,879 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
6172015-12-06 22:45:21,856 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
6182015-12-06 22:45:31,649 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
6192015-12-06 22:45:51,913 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
6202015-12-06 22:46:33,980 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
6212015-12-06 22:47:24,108 ERROR nodepool.NodeLauncher: Exception launching node id: 10864 in provider: wmflabs-eqiad error:
6222015-12-06 22:47:24,111 ERROR nodepool.NodeLauncher: Exception in run method:
6232015-12-06 22:48:36,785 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
6242015-12-06 22:49:26,608 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
6252015-12-06 22:51:29,808 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
6262015-12-06 22:52:31,313 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
6272015-12-06 22:55:47,608 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
6282015-12-06 22:56:38,050 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
6292015-12-06 22:56:48,719 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
6302015-12-06 22:57:20,219 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
6312015-12-06 22:57:32,327 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
6322015-12-06 22:59:27,816 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
6332015-12-06 23:00:28,637 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
6342015-12-06 23:02:42,095 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
6352015-12-06 23:03:24,107 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
6362015-12-06 23:06:29,608 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
6372015-12-06 23:06:55,631 ERROR nodepool.NodeLauncher: Exception launching node id: 10874 in provider: wmflabs-eqiad error:
6382015-12-06 23:06:55,634 ERROR nodepool.NodeLauncher: Exception in run method:
6392015-12-06 23:07:16,014 ERROR nodepool.NodeLauncher: Exception launching node id: 10876 in provider: wmflabs-eqiad error:
6402015-12-06 23:07:16,017 ERROR nodepool.NodeLauncher: Exception in run method:
6412015-12-06 23:07:20,602 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
6422015-12-06 23:07:42,565 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
6432015-12-06 23:08:22,901 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
6442015-12-06 23:08:43,953 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
6452015-12-06 23:10:39,570 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
6462015-12-06 23:11:26,471 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
6472015-12-06 23:12:14,564 ERROR nodepool.NodeLauncher: Exception launching node id: 10888 in provider: wmflabs-eqiad error:
6482015-12-06 23:12:14,567 ERROR nodepool.NodeLauncher: Exception in run method:
6492015-12-06 23:13:30,472 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
6502015-12-06 23:14:32,459 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
6512015-12-06 23:17:26,461 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
6522015-12-06 23:18:32,868 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
6532015-12-06 23:18:53,591 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
6542015-12-06 23:19:47,689 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
6552015-12-06 23:20:08,637 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
6562015-12-06 23:21:14,092 ERROR nodepool.NodeLauncher: Exception launching node id: 10889 in provider: wmflabs-eqiad error:
6572015-12-06 23:21:14,095 ERROR nodepool.NodeLauncher: Exception in run method:
6582015-12-06 23:21:31,350 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
6592015-12-06 23:22:31,048 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
6602015-12-06 23:24:28,050 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
6612015-12-06 23:25:30,150 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
6622015-12-06 23:28:32,876 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
6632015-12-06 23:29:24,416 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
6642015-12-06 23:29:45,645 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
6652015-12-06 23:30:36,257 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
6662015-12-06 23:31:28,361 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
6672015-12-06 23:32:35,830 ERROR nodepool.NodeLauncher: Exception launching node id: 10890 in provider: wmflabs-eqiad error:
6682015-12-06 23:32:35,834 ERROR nodepool.NodeLauncher: Exception in run method:
6692015-12-06 23:32:44,075 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
6702015-12-06 23:32:46,326 ERROR nodepool.NodeLauncher: Exception launching node id: 10891 in provider: wmflabs-eqiad error:
6712015-12-06 23:32:46,328 ERROR nodepool.NodeLauncher: Exception in run method:
6722015-12-06 23:33:35,067 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
6732015-12-06 23:35:27,653 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
6742015-12-06 23:36:29,651 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
6752015-12-06 23:39:33,572 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
6762015-12-06 23:40:24,136 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
6772015-12-06 23:40:45,895 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
6782015-12-06 23:41:36,615 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
6792015-12-06 23:42:28,184 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
6802015-12-06 23:42:53,047 ERROR nodepool.NodeLauncher: Exception launching node id: 10893 in provider: wmflabs-eqiad error:
6812015-12-06 23:42:53,049 ERROR nodepool.NodeLauncher: Exception in run method:
6822015-12-06 23:43:53,841 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
6832015-12-06 23:44:35,771 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
6842015-12-06 23:46:07,749 ERROR nodepool.NodeLauncher: Exception launching node id: 10894 in provider: wmflabs-eqiad error:
6852015-12-06 23:46:07,755 ERROR nodepool.NodeLauncher: Exception in run method:
6862015-12-06 23:46:27,241 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
6872015-12-06 23:47:39,901 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
6882015-12-06 23:48:45,781 ERROR nodepool.NodeLauncher: Exception launching node id: 10895 in provider: wmflabs-eqiad error:
6892015-12-06 23:48:45,784 ERROR nodepool.NodeLauncher: Exception in run method:
6902015-12-06 23:50:32,171 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
6912015-12-06 23:51:24,677 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
6922015-12-06 23:51:44,789 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
6932015-12-06 23:52:01,156 ERROR nodepool.NodeLauncher: Exception launching node id: 10896 in provider: wmflabs-eqiad error:
6942015-12-06 23:52:01,159 ERROR nodepool.NodeLauncher: Exception in run method:
6952015-12-06 23:52:35,373 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
6962015-12-06 23:53:16,710 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
6972015-12-06 23:54:31,482 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
6982015-12-06 23:55:33,810 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
6992015-12-06 23:57:28,784 ERROR nodepool.NodeDeleter: Exception deleting node 10699:
7002015-12-06 23:58:30,035 ERROR nodepool.NodeDeleter: Exception deleting node 10692:
7012015-12-07 00:01:28,017 ERROR nodepool.NodeDeleter: Exception deleting node 10700:
7022015-12-07 00:02:18,592 ERROR nodepool.NodeDeleter: Exception deleting node 10697:
7032015-12-07 00:02:52,946 ERROR nodepool.NodeDeleter: Exception deleting node 10694:
7042015-12-07 00:03:44,961 ERROR nodepool.NodeDeleter: Exception deleting node 10695:
7052015-12-07 00:04:35,311 ERROR nodepool.NodeDeleter: Exception deleting node 10667:
7062015-12-07 00:05:22,775 ERROR nodepool.NodeLauncher: Exception launching node id: 10897 in provider: wmflabs-eqiad error:
7072015-12-07 00:05:22,777 ERROR nodepool.NodeLauncher: Exception in run method:
7082015-12-07 00:05:26,623 ERROR nodepool.NodeDeleter: Exception deleting node 10687:
7092015-12-07 00:06:29,261 ERROR nodepool.NodeDeleter: Exception deleting node 10698:
710hashar@labnodepool1001:/var/log/nodepool$

hashar added a comment.Dec 21 2015, 2:41 PM

So this task is about Nova getting weird back on Sunday December 6th. Got fixed by Yuvi by restarting nova-conductor and nova-scheduler. We kept this task open for later investigation, is there anyone willing to look at it or should we just close the task?

hashar lowered the priority of this task from High to Medium.Dec 21 2015, 2:41 PM
hashar moved this task from Backlog to Backlog - ops related on the Continuous-Integration-Scaling board.

Lowering priority since the immediate issue has been fixed.

hashar moved this task from Untriaged to Done on the Continuous-Integration-Infrastructure board.Dec 21 2015, 2:42 PM
coren removed coren as the assignee of this task.Dec 21 2015, 6:17 PM
coren subscribed.
• Peter601980 renamed this task from Instance creation fails to h.Dec 28 2015, 8:05 AM
• Peter601980 lowered the priority of this task from Medium to Lowest.
• Peter601980 updated the task description. (Show Details)
• Peter601980 removed projects: Patch-For-Review, Continuous-Integration-Infrastructure, Continuous-Integration-Scaling, Cloud-Services.
• Peter601980 set Security to None.
• Peter601980 removed subscribers: coren, gerritbot, Luke081515 and 5 others.
Aklapper renamed this task from h to Instance creation fails.Dec 28 2015, 9:23 AM
Aklapper raised the priority of this task from Lowest to Medium.
Aklapper updated the task description. (Show Details)
Aklapper added projects: Patch-For-Review, Continuous-Integration-Infrastructure, Continuous-Integration-Scaling, Cloud-Services.
Aklapper added subscribers: coren, gerritbot, Luke081515 and 5 others.
hashar mentioned this in T122731: Nodepool deadlocks when querying unresponsive OpenStack API (was: rake-jessie jobs stuck due to no ci-jessie-wikimedia slaves being attached to Jenkins).Jan 4 2016, 1:21 PM
hashar closed this task as Resolved.Feb 19 2016, 8:26 PM
hashar claimed this task.

The issue got solved by restarting the scheduler. After two months there is not much left to investigate.

The patch Labs: make labs-ip-alias-dump ignore addressless hosts https://gerrit.wikimedia.org/r/257323 is supposed to fix the corner case.

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:45 PM
Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptJun 7 2017, 6:45 PM
• Phabricator_maintenance edited projects, added RelEng-Archive-FY201718-Q1; removed Release-Engineering-Team (Kanban).Sep 26 2017, 11:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL