We are migrating some projects from main to eqiad1.
This task is for tracking and coordinating this work, and also issues we may find.
Docs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron_Migration
We are migrating some projects from main to eqiad1.
This task is for tracking and coordinating this work, and also issues we may find.
Docs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron_Migration
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Open | None | T53494 Use Beta cluster as a true canary for code deployments (epic) | |||
Open | None | T87220 Minimize infrastructure differences between Beta Cluster and production | |||
Open | None | T196662 Set up LVS in beta like prod | |||
Resolved | bd808 | T166396 Program 1 Outcome 4: VPS hosting | |||
Resolved | None | T167293 Nova-network to Neutron migration | |||
Resolved | Andrew | T204745 cloudvps: migrate projects from main to eqiad1 | |||
Resolved | None | T207677 Migrate 'Quarry' project to eqiad1 | |||
Resolved | Andrew | T209632 Move 'video' project to eqiad1-r | |||
Declined | None | T209938 CloudVPS: diamond report some metrics with the host IP address instead of host name | |||
Resolved | Andrew | T213085 Delete eqiad region bastion hosts | |||
Resolved | Andrew | T213087 Stray user files on old eqiad bastions | |||
Resolved | Andrew | T213540 Migrate nova proxies to eqiad1 |
Change 462887 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: horizon: move wikimania-support to eqiad1
Mentioned in SAL (#wikimedia-cloud) [2018-09-26T09:18:04Z] <arturo> T204745 migrating project to eqiad1
Change 462887 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: horizon: move wikimania-support to eqiad1
The region-migrate script failed:
root@labcontrol1001:~# openstack --os-tenant-name=wikimania-support server list +--------------------------------------+-----------------+--------+---------------------+ | ID | Name | Status | Networks | +--------------------------------------+-----------------+--------+---------------------+ | 8f1618ee-fc93-45a5-878c-35d252fba016 | scholarships-02 | ACTIVE | public=10.68.16.149 | +--------------------------------------+-----------------+--------+---------------------+ root@labcontrol1001:~# ./region-migrate 8f1618ee-fc93-45a5-878c-35d252fba016 Source instance 8f1618ee-fc93-45a5-878c-35d252fba016 is now on host labvirt1017 with state ACTIVE Warning: Permanently added '10.68.16.149' (ECDSA) to the list of known hosts. verified ssh for 10.68.16.149, returned hostname scholarships-02 activating image af63165e-cf13-404e-bffe-ed7a86e97594 Current status for 24de5332-237d-463d-a7cd-f42d1ffdf74b is BUILD; waiting for it to change to ACTIVE. Current status for 24de5332-237d-463d-a7cd-f42d1ffdf74b is ERROR; waiting for it to change to ACTIVE.
This is the message in the instance status:
u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 24de5332-237d-463d-a7cd-f42d1ffdf74b. Last exception: Binding failed for port 2f178795-434b-4649-9bf3-5853def226d6, please check neutron logs for more information.', u'code': 500, u'created': u'2018-09-26T09:20:53Z'}
Issue seems to be that neutron is having a hard time with agents:
+--------------------------------------+--------------------+---------------+-------------------+-------+----------------+---------------------------+ | id | agent_type | host | availability_zone | alive | admin_state_up | binary | +--------------------------------------+--------------------+---------------+-------------------+-------+----------------+---------------------------+ | 468aef2a-8eb6-4382-abba-bc284efd9fa5 | DHCP agent | cloudnet1004 | nova | xxx | True | neutron-dhcp-agent | | 601bef99-b53c-4e6a-b384-65d1feebedff | Metadata agent | cloudnet1003 | | :-) | True | neutron-metadata-agent | | 8af5d8a1-2e29-40e6-baf0-3cd79a7ac77b | L3 agent | cloudnet1003 | nova | :-) | True | neutron-l3-agent | | 970df1d1-505d-47a4-8d35-1b13c0dfe098 | L3 agent | cloudnet1004 | nova | xxx | True | neutron-l3-agent | | 9f8833de-11a4-4395-8da5-f57fe8326659 | Linux bridge agent | cloudnet1003 | | xxx | True | neutron-linuxbridge-agent | | ad3461d7-b79e-4279-921d-5a476e296767 | Linux bridge agent | cloudnet1004 | | xxx | True | neutron-linuxbridge-agent | | b0f1cdf2-8d03-4f7b-978c-201ecea69b84 | Linux bridge agent | cloudvirt1020 | | xxx | True | neutron-linuxbridge-agent | | b2f9da63-2f16-4aa5-9400-ae708a733f91 | Linux bridge agent | cloudvirt1021 | | xxx | True | neutron-linuxbridge-agent | | d475e07d-52b3-476e-9a4f-e63b21e1075e | Metadata agent | cloudnet1004 | | :-) | True | neutron-metadata-agent | | e382a233-e6a0-422e-9d2e-5651082783fc | Linux bridge agent | cloudvirt1022 | | xxx | True | neutron-linuxbridge-agent | | fc45a34d-d8a4-45fe-982d-5b4b7a8fcde1 | Linux bridge agent | cloudvirt1019 | | xxx | True | neutron-linuxbridge-agent | | ff2a8228-3748-4588-927b-4b6563da9ca0 | DHCP agent | cloudnet1003 | nova | xxx | True | neutron-dhcp-agent | +--------------------------------------+--------------------+---------------+-------------------+-------+----------------+---------------------------+
Mentioned in SAL (#wikimedia-cloud) [2018-09-26T15:48:44Z] <arturo> T204745 actually migrating project to eqiad1
Mentioned in SAL (#wikimedia-cloud) [2018-09-26T16:24:03Z] <arturo> T204745 project migrated to eqiad1 but the webservice isn't working (lxc issue)
Mentioned in SAL (#wikimedia-cloud) [2018-09-27T11:06:11Z] <arturo> T204745 migrating project to eqiad1
This machine migration had a disk failure when copying which apparently solved by itself:
root@labcontrol1001:~# ./region-migrate ea390a38-3df0-427d-8109-8a88e1b74e5f Source instance ea390a38-3df0-427d-8109-8a88e1b74e5f is now on host labvirt1008 with state ACTIVE Warning: Permanently added '10.68.16.143' (ECDSA) to the list of known hosts. verified ssh for 10.68.16.143, returned hostname mwv-apt-01 activating image b7c4fc02-433f-4b49-97c2-b2492012b742 Current status for 6d6ee374-a987-403d-8e26-611eab9ce7da is BUILD; waiting for it to change to ACTIVE. dest_instance_id: 6d6ee374-a987-403d-8e26-611eab9ce7da on cloudvirt1021 deactivating image b7c4fc02-433f-4b49-97c2-b2492012b742 Updating proxy record {u'domain': u'mwv-apt.wmflabs.org.', u'backends': [u'http://10.68.16.143:80']} Current status for ea390a38-3df0-427d-8109-8a88e1b74e5f is ACTIVE; waiting for it to change to SHUTOFF. Current status for 6d6ee374-a987-403d-8e26-611eab9ce7da is ACTIVE; waiting for it to change to SHUTOFF. ssh -i /root/.ssh/compute-hosts-key nova@labvirt1008.eqiad.wmnet /usr/bin/rsync -S -avz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/nova/.ssh/id_rsa" --progress --exclude=console.log /var/lib/nova/instances/ea390a38-3df0-427d-8109-8a88e1b74e5f/* nova@cloudvirt1021.eqiad.wmnet:/var/lib/nova/instances/6d6ee374-a987-403d-8e26-611eab9ce7da sending incremental file list disk 18,677,760,000 100% 13.67MB/s 0:21:43 (xfr#1, to-chk=2/3) disk.info 79 100% 0.10kB/s 0:00:00 (xfr#2, to-chk=1/3) libvirt.xml 2,685 100% 3.40kB/s 0:00:00 (xfr#3, to-chk=0/3) WARNING: disk failed verification -- update discarded (will try again). disk 18,677,760,000 100% 14.13MB/s 0:21:00 (xfr#4, to-chk=2/3) sent 14,235,922,819 bytes received 95,812 bytes 5,546,860.95 bytes/sec total size is 18,677,762,764 speedup is 1.31 Instance copied. Now updating nova db... Current status for 6d6ee374-a987-403d-8e26-611eab9ce7da is SHUTOFF; waiting for it to change to ACTIVE. Waiting for the copied VM to boot and adjust to the new ip Rebooting to acquire the new hostname Warning: Permanently added '172.16.1.240' (ECDSA) to the list of known hosts. verified ssh for 172.16.1.240, returned hostname mwv-apt-01 Got multiple IPs for mwv-apt-01.mwv-apt.eqiad.wmflabs waiting to recheck DNS channel 0: open failed: connect failed: Connection refused ssh_exchange_identification: Connection closed by remote host exception caught while attempting ssh to 172.16.1.240
But I checked the machine afterwards, and all seems fine.
Mentioned in SAL (#wikimedia-cloud) [2018-09-27T12:04:33Z] <arturo> T204745 migration completed
Change 463279 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] region-migrate: take a nap before we start the rsync
Change 463279 merged by Andrew Bogott:
[operations/puppet@production] region-migrate: take a nap before we start the rsync
Change 463282 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: horizon: mwv-apt is now in eqiad1
Change 463282 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: horizon: mwv-apt is now in eqiad1
Change 468629 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] horizon: move 'planet' and 'general-k8s' to eqiad1
Change 468629 merged by Andrew Bogott:
[operations/puppet@production] horizon: move 'planet' and 'general-k8s' to eqiad1
Change 468634 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] horizon: move striker to eqiad1
Change 468634 merged by Andrew Bogott:
[operations/puppet@production] horizon: move striker to eqiad1
Change 469020 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] horizon: move 'suggestbot' to the new neutron region
Change 469020 merged by Andrew Bogott:
[operations/puppet@production] horizon: move 'suggestbot' to the new neutron region
Change 470380 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move some projects to eqiad1: antiharassment, catgraph, codereview, cvn
Change 470380 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move some projects to eqiad1: antiharassment, catgraph, codereview, cvn
Change 470596 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] horizon: enable eqiad1 for puppet-diffs, fastcci, cyberbot
Change 470596 merged by Andrew Bogott:
[operations/puppet@production] horizon: enable eqiad1 for puppet-diffs, fastcci, cyberbot
Change 470855 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1: huggle, mwstake, logging, mobile
Change 470855 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1: huggle, mwstake, logging, mobile
Change 471007 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1: privpol-captcha, qna, pluggableauth
Change 471007 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1: privpol-captcha, qna, pluggableauth
Change 471263 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move more projects to eqiad1
Change 471263 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move more projects to eqiad1
Change 471731 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: moved more projects to eqiad1.
Change 471731 merged by Andrew Bogott:
[operations/puppet@production] Horizon: moved more projects to eqiad1.
Change 471963 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move more projects to eqiad1-r:
Change 471963 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move more projects to eqiad1-r:
Change 472023 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: enable eqiad1-r for toolsbeta project
Change 472023 merged by Bstorm:
[operations/puppet@production] Horizon: enable eqiad1-r for toolsbeta project
Change 472155 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move 'ores' and 'puppet' to eqiad1-r
Change 472155 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move 'ores' and 'puppet' to eqiad1-r
Change 472448 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r: recommendatation-api, sciencesource, sentry
Change 472448 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r: recommendatation-api, sciencesource, sentry
Change 472653 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 472653 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 473215 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 473215 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 473531 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 473531 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 473570 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move wmf-research-tools to eqiad1-r
Change 473570 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move wmf-research-tools to eqiad1-r
Change 474180 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move three projects to eqiad1:
Change 474180 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move three projects to eqiad1:
Change 475760 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move more projects to eqiad1
Change 475760 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move more projects to eqiad1
Change 476022 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move more projects to eqiad1-r
Change 476022 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move more projects to eqiad1-r
Change 476181 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move utrs project to eqiad1-r
Change 476181 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move utrs project to eqiad1-r
Change 476269 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 476269 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 476507 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 476507 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 476858 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 476858 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 477715 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move 'incubator' project to eqiad1-r
Change 477715 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move 'incubator' project to eqiad1-r
Change 478951 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r: maps and wm-bot
Change 478951 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r: maps and wm-bot
Change 479153 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move mwoffliner to eqiad1-r
Change 479153 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move mwoffliner to eqiad1-r
Change 479277 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: enable projects in eqiad1-r
Change 479277 merged by Andrew Bogott:
[operations/puppet@production] Horizon: enable projects in eqiad1-r
Change 479690 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1
Change 479690 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1
Change 479844 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move 'openstack' project to eqiad1-r
Change 479844 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move 'openstack' project to eqiad1-r
Change 480082 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move more projects to eqiad1:
Change 480082 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move more projects to eqiad1:
Change 480496 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 480496 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r
Change 480513 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: enable more projects in eqiad1
Change 480513 merged by Andrew Bogott:
[operations/puppet@production] Horizon: enable more projects in eqiad1
Change 480819 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move more projects to eqiad1-r:
Change 480819 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move more projects to eqiad1-r:
Change 483792 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: disable 'bastion' and 'redirects' in eqiad region
Change 483792 merged by Andrew Bogott:
[operations/puppet@production] Horizon: disable 'bastion' and 'redirects' in eqiad region