Page MenuHomePhabricator

cloudvps: migrate projects from main to eqiad1
Closed, ResolvedPublic

Description

We are migrating some projects from main to eqiad1.

This task is for tracking and coordinating this work, and also issues we may find.

Docs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron_Migration

Details

SubjectRepoBranchLines +/-
operations/puppetproduction+1 -2
operations/puppetproduction+4 -4
operations/puppetproduction+6 -6
operations/puppetproduction+4 -4
operations/puppetproduction+4 -4
operations/puppetproduction+1 -1
operations/puppetproduction+3 -4
operations/puppetproduction+4 -4
operations/puppetproduction+1 -1
operations/puppetproduction+2 -2
operations/puppetproduction+1 -1
operations/puppetproduction+3 -3
operations/puppetproduction+4 -4
operations/puppetproduction+4 -4
operations/puppetproduction+1 -1
operations/puppetproduction+7 -7
operations/puppetproduction+7 -7
operations/puppetproduction+3 -3
operations/puppetproduction+1 -1
operations/puppetproduction+4 -4
operations/puppetproduction+5 -5
operations/puppetproduction+6 -6
operations/puppetproduction+3 -3
operations/puppetproduction+2 -2
operations/puppetproduction+1 -0
operations/puppetproduction+8 -8
operations/puppetproduction+4 -4
operations/puppetproduction+5 -5
operations/puppetproduction+3 -3
operations/puppetproduction+4 -4
operations/puppetproduction+3 -3
operations/puppetproduction+4 -4
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+2 -2
operations/puppetproduction+1 -1
operations/puppetproduction+5 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+2 -2
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 462887 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: horizon: move wikimania-support to eqiad1

https://gerrit.wikimedia.org/r/462887

Mentioned in SAL (#wikimedia-cloud) [2018-09-26T09:18:04Z] <arturo> T204745 migrating project to eqiad1

Change 462887 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: horizon: move wikimania-support to eqiad1

https://gerrit.wikimedia.org/r/462887

The region-migrate script failed:

root@labcontrol1001:~# openstack --os-tenant-name=wikimania-support server list
+--------------------------------------+-----------------+--------+---------------------+
| ID                                   | Name            | Status | Networks            |
+--------------------------------------+-----------------+--------+---------------------+
| 8f1618ee-fc93-45a5-878c-35d252fba016 | scholarships-02 | ACTIVE | public=10.68.16.149 |
+--------------------------------------+-----------------+--------+---------------------+
root@labcontrol1001:~# ./region-migrate 8f1618ee-fc93-45a5-878c-35d252fba016
Source instance 8f1618ee-fc93-45a5-878c-35d252fba016 is now on host labvirt1017 with state ACTIVE
Warning: Permanently added '10.68.16.149' (ECDSA) to the list of known hosts.
verified ssh for 10.68.16.149, returned hostname scholarships-02

activating image af63165e-cf13-404e-bffe-ed7a86e97594
Current status for 24de5332-237d-463d-a7cd-f42d1ffdf74b is BUILD; waiting for it to change to ACTIVE.
Current status for 24de5332-237d-463d-a7cd-f42d1ffdf74b is ERROR; waiting for it to change to ACTIVE.

This is the message in the instance status:

u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 24de5332-237d-463d-a7cd-f42d1ffdf74b. Last exception: Binding failed for port 2f178795-434b-4649-9bf3-5853def226d6, please check neutron logs for more information.', u'code': 500, u'created': u'2018-09-26T09:20:53Z'}

Issue seems to be that neutron is having a hard time with agents:

+--------------------------------------+--------------------+---------------+-------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host          | availability_zone | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+---------------+-------------------+-------+----------------+---------------------------+
| 468aef2a-8eb6-4382-abba-bc284efd9fa5 | DHCP agent         | cloudnet1004  | nova              | xxx   | True           | neutron-dhcp-agent        |
| 601bef99-b53c-4e6a-b384-65d1feebedff | Metadata agent     | cloudnet1003  |                   | :-)   | True           | neutron-metadata-agent    |
| 8af5d8a1-2e29-40e6-baf0-3cd79a7ac77b | L3 agent           | cloudnet1003  | nova              | :-)   | True           | neutron-l3-agent          |
| 970df1d1-505d-47a4-8d35-1b13c0dfe098 | L3 agent           | cloudnet1004  | nova              | xxx   | True           | neutron-l3-agent          |
| 9f8833de-11a4-4395-8da5-f57fe8326659 | Linux bridge agent | cloudnet1003  |                   | xxx   | True           | neutron-linuxbridge-agent |
| ad3461d7-b79e-4279-921d-5a476e296767 | Linux bridge agent | cloudnet1004  |                   | xxx   | True           | neutron-linuxbridge-agent |
| b0f1cdf2-8d03-4f7b-978c-201ecea69b84 | Linux bridge agent | cloudvirt1020 |                   | xxx   | True           | neutron-linuxbridge-agent |
| b2f9da63-2f16-4aa5-9400-ae708a733f91 | Linux bridge agent | cloudvirt1021 |                   | xxx   | True           | neutron-linuxbridge-agent |
| d475e07d-52b3-476e-9a4f-e63b21e1075e | Metadata agent     | cloudnet1004  |                   | :-)   | True           | neutron-metadata-agent    |
| e382a233-e6a0-422e-9d2e-5651082783fc | Linux bridge agent | cloudvirt1022 |                   | xxx   | True           | neutron-linuxbridge-agent |
| fc45a34d-d8a4-45fe-982d-5b4b7a8fcde1 | Linux bridge agent | cloudvirt1019 |                   | xxx   | True           | neutron-linuxbridge-agent |
| ff2a8228-3748-4588-927b-4b6563da9ca0 | DHCP agent         | cloudnet1003  | nova              | xxx   | True           | neutron-dhcp-agent        |
+--------------------------------------+--------------------+---------------+-------------------+-------+----------------+---------------------------+

I'm tracking neutron agent/server issues in T205524

Mentioned in SAL (#wikimedia-cloud) [2018-09-26T15:48:44Z] <arturo> T204745 actually migrating project to eqiad1

Mentioned in SAL (#wikimedia-cloud) [2018-09-26T16:24:03Z] <arturo> T204745 project migrated to eqiad1 but the webservice isn't working (lxc issue)

Mentioned in SAL (#wikimedia-cloud) [2018-09-27T11:06:11Z] <arturo> T204745 migrating project to eqiad1

This machine migration had a disk failure when copying which apparently solved by itself:

root@labcontrol1001:~# ./region-migrate ea390a38-3df0-427d-8109-8a88e1b74e5f
Source instance ea390a38-3df0-427d-8109-8a88e1b74e5f is now on host labvirt1008 with state ACTIVE
Warning: Permanently added '10.68.16.143' (ECDSA) to the list of known hosts.
verified ssh for 10.68.16.143, returned hostname mwv-apt-01

activating image b7c4fc02-433f-4b49-97c2-b2492012b742
Current status for 6d6ee374-a987-403d-8e26-611eab9ce7da is BUILD; waiting for it to change to ACTIVE.
dest_instance_id: 6d6ee374-a987-403d-8e26-611eab9ce7da on cloudvirt1021
deactivating image b7c4fc02-433f-4b49-97c2-b2492012b742
Updating proxy record {u'domain': u'mwv-apt.wmflabs.org.', u'backends': [u'http://10.68.16.143:80']}
Current status for ea390a38-3df0-427d-8109-8a88e1b74e5f is ACTIVE; waiting for it to change to SHUTOFF.
Current status for 6d6ee374-a987-403d-8e26-611eab9ce7da is ACTIVE; waiting for it to change to SHUTOFF.
ssh -i /root/.ssh/compute-hosts-key nova@labvirt1008.eqiad.wmnet /usr/bin/rsync -S -avz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/nova/.ssh/id_rsa" --progress --exclude=console.log /var/lib/nova/instances/ea390a38-3df0-427d-8109-8a88e1b74e5f/* nova@cloudvirt1021.eqiad.wmnet:/var/lib/nova/instances/6d6ee374-a987-403d-8e26-611eab9ce7da
sending incremental file list
disk
 18,677,760,000 100%   13.67MB/s    0:21:43 (xfr#1, to-chk=2/3)
disk.info
             79 100%    0.10kB/s    0:00:00 (xfr#2, to-chk=1/3)
libvirt.xml
          2,685 100%    3.40kB/s    0:00:00 (xfr#3, to-chk=0/3)
WARNING: disk failed verification -- update discarded (will try again).
disk
 18,677,760,000 100%   14.13MB/s    0:21:00 (xfr#4, to-chk=2/3)

sent 14,235,922,819 bytes  received 95,812 bytes  5,546,860.95 bytes/sec
total size is 18,677,762,764  speedup is 1.31
Instance copied.  Now updating nova db...
Current status for 6d6ee374-a987-403d-8e26-611eab9ce7da is SHUTOFF; waiting for it to change to ACTIVE.
Waiting for the copied VM to boot and adjust to the new ip
Rebooting to acquire the new hostname
Warning: Permanently added '172.16.1.240' (ECDSA) to the list of known hosts.
verified ssh for 172.16.1.240, returned hostname mwv-apt-01

Got multiple IPs for mwv-apt-01.mwv-apt.eqiad.wmflabs
waiting to recheck DNS
channel 0: open failed: connect failed: Connection refused
ssh_exchange_identification: Connection closed by remote host
exception caught while attempting ssh to 172.16.1.240

But I checked the machine afterwards, and all seems fine.

Change 463279 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] region-migrate: take a nap before we start the rsync

https://gerrit.wikimedia.org/r/463279

Change 463279 merged by Andrew Bogott:
[operations/puppet@production] region-migrate: take a nap before we start the rsync

https://gerrit.wikimedia.org/r/463279

Change 463282 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: horizon: mwv-apt is now in eqiad1

https://gerrit.wikimedia.org/r/463282

Change 463282 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: horizon: mwv-apt is now in eqiad1

https://gerrit.wikimedia.org/r/463282

Change 468629 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] horizon: move 'planet' and 'general-k8s' to eqiad1

https://gerrit.wikimedia.org/r/468629

Change 468629 merged by Andrew Bogott:
[operations/puppet@production] horizon: move 'planet' and 'general-k8s' to eqiad1

https://gerrit.wikimedia.org/r/468629

Change 468634 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] horizon: move striker to eqiad1

https://gerrit.wikimedia.org/r/468634

Change 468634 merged by Andrew Bogott:
[operations/puppet@production] horizon: move striker to eqiad1

https://gerrit.wikimedia.org/r/468634

Change 469020 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] horizon: move 'suggestbot' to the new neutron region

https://gerrit.wikimedia.org/r/469020

Change 469020 merged by Andrew Bogott:
[operations/puppet@production] horizon: move 'suggestbot' to the new neutron region

https://gerrit.wikimedia.org/r/469020

Change 470380 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move some projects to eqiad1: antiharassment, catgraph, codereview, cvn

https://gerrit.wikimedia.org/r/470380

Change 470380 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move some projects to eqiad1: antiharassment, catgraph, codereview, cvn

https://gerrit.wikimedia.org/r/470380

Change 470596 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] horizon: enable eqiad1 for puppet-diffs, fastcci, cyberbot

https://gerrit.wikimedia.org/r/470596

Change 470596 merged by Andrew Bogott:
[operations/puppet@production] horizon: enable eqiad1 for puppet-diffs, fastcci, cyberbot

https://gerrit.wikimedia.org/r/470596

Change 470855 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1: huggle, mwstake, logging, mobile

https://gerrit.wikimedia.org/r/470855

Change 470855 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1: huggle, mwstake, logging, mobile

https://gerrit.wikimedia.org/r/470855

Change 471007 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1: privpol-captcha, qna, pluggableauth

https://gerrit.wikimedia.org/r/471007

Change 471007 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1: privpol-captcha, qna, pluggableauth

https://gerrit.wikimedia.org/r/471007

Change 471263 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move more projects to eqiad1

https://gerrit.wikimedia.org/r/471263

Change 471263 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move more projects to eqiad1

https://gerrit.wikimedia.org/r/471263

Change 471731 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: moved more projects to eqiad1.

https://gerrit.wikimedia.org/r/471731

Change 471731 merged by Andrew Bogott:
[operations/puppet@production] Horizon: moved more projects to eqiad1.

https://gerrit.wikimedia.org/r/471731

Change 471963 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move more projects to eqiad1-r:

https://gerrit.wikimedia.org/r/471963

Change 471963 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move more projects to eqiad1-r:

https://gerrit.wikimedia.org/r/471963

Change 472023 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: enable eqiad1-r for toolsbeta project

https://gerrit.wikimedia.org/r/472023

Change 472023 merged by Bstorm:
[operations/puppet@production] Horizon: enable eqiad1-r for toolsbeta project

https://gerrit.wikimedia.org/r/472023

Change 472155 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move 'ores' and 'puppet' to eqiad1-r

https://gerrit.wikimedia.org/r/472155

Change 472155 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move 'ores' and 'puppet' to eqiad1-r

https://gerrit.wikimedia.org/r/472155

Change 472448 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r: recommendatation-api, sciencesource, sentry

https://gerrit.wikimedia.org/r/472448

Change 472448 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r: recommendatation-api, sciencesource, sentry

https://gerrit.wikimedia.org/r/472448

Change 472653 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/472653

Change 472653 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/472653

Change 473215 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/473215

Change 473215 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/473215

Change 473531 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/473531

Change 473531 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/473531

Change 473570 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move wmf-research-tools to eqiad1-r

https://gerrit.wikimedia.org/r/473570

Change 473570 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move wmf-research-tools to eqiad1-r

https://gerrit.wikimedia.org/r/473570

Change 474180 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move three projects to eqiad1:

https://gerrit.wikimedia.org/r/474180

Change 474180 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move three projects to eqiad1:

https://gerrit.wikimedia.org/r/474180

Change 475760 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move more projects to eqiad1

https://gerrit.wikimedia.org/r/475760

Change 475760 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move more projects to eqiad1

https://gerrit.wikimedia.org/r/475760

Change 476022 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move more projects to eqiad1-r

https://gerrit.wikimedia.org/r/476022

Change 476022 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move more projects to eqiad1-r

https://gerrit.wikimedia.org/r/476022

Change 476181 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move utrs project to eqiad1-r

https://gerrit.wikimedia.org/r/476181

Change 476181 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move utrs project to eqiad1-r

https://gerrit.wikimedia.org/r/476181

Change 476269 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/476269

Change 476269 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/476269

Change 476507 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/476507

Change 476507 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/476507

Change 476858 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/476858

Change 476858 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/476858

Change 477715 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move 'incubator' project to eqiad1-r

https://gerrit.wikimedia.org/r/477715

Change 477715 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move 'incubator' project to eqiad1-r

https://gerrit.wikimedia.org/r/477715

Change 478951 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r: maps and wm-bot

https://gerrit.wikimedia.org/r/478951

Change 478951 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r: maps and wm-bot

https://gerrit.wikimedia.org/r/478951

Change 479153 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move mwoffliner to eqiad1-r

https://gerrit.wikimedia.org/r/479153

Change 479153 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move mwoffliner to eqiad1-r

https://gerrit.wikimedia.org/r/479153

Change 479277 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: enable projects in eqiad1-r

https://gerrit.wikimedia.org/r/479277

Change 479277 merged by Andrew Bogott:
[operations/puppet@production] Horizon: enable projects in eqiad1-r

https://gerrit.wikimedia.org/r/479277

Change 479690 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1

https://gerrit.wikimedia.org/r/479690

Change 479690 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1

https://gerrit.wikimedia.org/r/479690

Change 479844 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move 'openstack' project to eqiad1-r

https://gerrit.wikimedia.org/r/479844

Change 479844 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move 'openstack' project to eqiad1-r

https://gerrit.wikimedia.org/r/479844

Change 480082 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move more projects to eqiad1:

https://gerrit.wikimedia.org/r/480082

Change 480082 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move more projects to eqiad1:

https://gerrit.wikimedia.org/r/480082

Change 480496 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/480496

Change 480496 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move projects to eqiad1-r

https://gerrit.wikimedia.org/r/480496

Change 480513 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: enable more projects in eqiad1

https://gerrit.wikimedia.org/r/480513

Change 480513 merged by Andrew Bogott:
[operations/puppet@production] Horizon: enable more projects in eqiad1

https://gerrit.wikimedia.org/r/480513

Change 480819 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move more projects to eqiad1-r:

https://gerrit.wikimedia.org/r/480819

Change 480819 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move more projects to eqiad1-r:

https://gerrit.wikimedia.org/r/480819

Change 483792 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: disable 'bastion' and 'redirects' in eqiad region

https://gerrit.wikimedia.org/r/483792

Change 483792 merged by Andrew Bogott:
[operations/puppet@production] Horizon: disable 'bastion' and 'redirects' in eqiad region

https://gerrit.wikimedia.org/r/483792

The old region is gone.