Page MenuHomePhabricator

Create 4 new VMs to replace schema[12]00[12]
Closed, ResolvedPublic

Description

In T219556 we created 4 little VMs for schema[12]00[12], and now we'd like to create 4 copies but with Buster. The end goal so to decommission the actual ones with Stretch of course.

Event Timeline

elukey renamed this task from Create 4 new VMs to replace schema[12 to Create 4 new VMs to replace schema[12]00[12].Aug 13 2020, 12:51 PM
elukey claimed this task.
elukey triaged this task as Medium priority.
elukey updated the task description. (Show Details)
elukey added a subscriber: Ottomata.

As described in T255026#6276301:

When you do that, please use row B/D in eqiad and row C/D in codfw to better balance out our Ganeti groups.

Change 619998 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Add schema[12]00[34] A/PTR records

https://gerrit.wikimedia.org/r/619998

Change 619998 merged by Elukey:
[operations/dns@master] Add schema[12]00[34] A/PTR records

https://gerrit.wikimedia.org/r/619998

Mentioned in SAL (#wikimedia-operations) [2020-08-13T14:00:35Z] <elukey> create schema[12]00[34] in ganeti - T260347

Change 620016 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add schema[12]00[34] dhcp + role config

https://gerrit.wikimedia.org/r/620016

Change 620016 merged by Elukey:
[operations/puppet@production] Add schema[12]00[34] dhcp + role config

https://gerrit.wikimedia.org/r/620016

elukey@cumin1001:~$ sudo cookbook sre.ganeti.makevm eqiad_B schema1003.eqiad.wmnet --vcpus 2 --memory 2 --disk 10
START - Cookbook sre.ganeti.makevm
/usr/lib/python3/dist-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for ganeti01.svc.eqiad.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
Ready to create Ganeti VM schema1003.eqiad.wmnet in the ganeti01.svc.eqiad.wmnet cluster on row B with 2 vCPUs, 2GB of RAM, 10GB of disk in the private network.
Is this correct?
Type "done" to proceed
> done
The command output will be printed at the end.
Creating VM schema1003.eqiad.wmnet in cluster ganeti01.svc.eqiad.wmnet with row=B vcpus=2 memory=2GB disk=10GB link=private. This may take a few minutes.
Thu Aug 13 13:39:55 2020  - INFO: No-installation mode selected, disabling startup
Thu Aug 13 13:40:08 2020  - INFO: Selected nodes for instance schema1003.eqiad.wmnet via iallocator hail: ganeti1015.eqiad.wmnet, ganeti1017.eqiad.wmnet
Thu Aug 13 13:40:09 2020 * creating instance disks...
Thu Aug 13 13:40:13 2020 adding instance schema1003.eqiad.wmnet to cluster config
Thu Aug 13 13:40:13 2020 adding disks to cluster config
Thu Aug 13 13:40:13 2020  - INFO: Waiting for instance schema1003.eqiad.wmnet to sync disks
Thu Aug 13 13:40:14 2020  - INFO: - device disk/0:  0.20% done, 20m 47s remaining (estimated)
Thu Aug 13 13:41:14 2020  - INFO: - device disk/0: 21.70% done, 3m 37s remaining (estimated)
Thu Aug 13 13:42:14 2020  - INFO: - device disk/0: 43.20% done, 2m 33s remaining (estimated)
Thu Aug 13 13:43:14 2020  - INFO: - device disk/0: 64.80% done, 1m 36s remaining (estimated)
Thu Aug 13 13:44:14 2020  - INFO: - device disk/0: 86.30% done, 37s remaining (estimated)
Thu Aug 13 13:44:52 2020  - INFO: - device disk/0: 99.60% done, 1s remaining (estimated)
Thu Aug 13 13:44:53 2020  - INFO: - device disk/0: 100.00% done, 0s remaining (estimated)
Thu Aug 13 13:44:53 2020  - INFO: Instance schema1003.eqiad.wmnet's disks are in sync
Thu Aug 13 13:44:53 2020  - INFO: Waiting for instance schema1003.eqiad.wmnet to sync disks
Thu Aug 13 13:44:53 2020  - INFO: Instance schema1003.eqiad.wmnet's disks are in sync
MAC address for schema1003.eqiad.wmnet is: aa:00:00:88:83:8a
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)

elukey@cumin1001:~$ sudo cookbook sre.ganeti.makevm eqiad_D schema1004.eqiad.wmnet --vcpus 2 --memory 2 --disk 10
START - Cookbook sre.ganeti.makevm
/usr/lib/python3/dist-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for ganeti01.svc.eqiad.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
Ready to create Ganeti VM schema1004.eqiad.wmnet in the ganeti01.svc.eqiad.wmnet cluster on row D with 2 vCPUs, 2GB of RAM, 10GB of disk in the private network.
Is this correct?
Type "done" to proceed
> done
The command output will be printed at the end.
Creating VM schema1004.eqiad.wmnet in cluster ganeti01.svc.eqiad.wmnet with row=D vcpus=2 memory=2GB disk=10GB link=private. This may take a few minutes.
Thu Aug 13 13:46:06 2020  - INFO: No-installation mode selected, disabling startup
Thu Aug 13 13:46:19 2020  - INFO: Selected nodes for instance schema1004.eqiad.wmnet via iallocator hail: ganeti1019.eqiad.wmnet, ganeti1021.eqiad.wmnet
Thu Aug 13 13:46:20 2020 * creating instance disks...
Thu Aug 13 13:46:24 2020 adding instance schema1004.eqiad.wmnet to cluster config
Thu Aug 13 13:46:25 2020 adding disks to cluster config
Thu Aug 13 13:46:25 2020  - INFO: Waiting for instance schema1004.eqiad.wmnet to sync disks
Thu Aug 13 13:46:25 2020  - INFO: - device disk/0:  0.20% done, 18m 11s remaining (estimated)
Thu Aug 13 13:47:25 2020  - INFO: - device disk/0: 21.70% done, 3m 37s remaining (estimated)
Thu Aug 13 13:48:25 2020  - INFO: - device disk/0: 43.20% done, 2m 33s remaining (estimated)
Thu Aug 13 13:49:25 2020  - INFO: - device disk/0: 64.80% done, 1m 36s remaining (estimated)
Thu Aug 13 13:50:26 2020  - INFO: - device disk/0: 86.30% done, 37s remaining (estimated)
Thu Aug 13 13:51:03 2020  - INFO: - device disk/0: 99.50% done, 1s remaining (estimated)
Thu Aug 13 13:51:04 2020  - INFO: - device disk/0: 100.00% done, 0s remaining (estimated)
Thu Aug 13 13:51:04 2020  - INFO: - device disk/0: 100.00% done, 0s remaining (estimated)
Thu Aug 13 13:51:04 2020  - INFO: Instance schema1004.eqiad.wmnet's disks are in sync
Thu Aug 13 13:51:04 2020  - INFO: Waiting for instance schema1004.eqiad.wmnet to sync disks
Thu Aug 13 13:51:04 2020  - INFO: Instance schema1004.eqiad.wmnet's disks are in sync
MAC address for schema1004.eqiad.wmnet is: aa:00:00:cf:b0:47
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)

elukey@cumin1001:~$ sudo cookbook sre.ganeti.makevm codfw_C schema2003.codfw.wmnet --vcpus 2 --memory 2 --disk 10
START - Cookbook sre.ganeti.makevm
/usr/lib/python3/dist-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for ganeti01.svc.codfw.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
Ready to create Ganeti VM schema2003.codfw.wmnet in the ganeti01.svc.codfw.wmnet cluster on row C with 2 vCPUs, 2GB of RAM, 10GB of disk in the private network.
Is this correct?
Type "done" to proceed
> done
The command output will be printed at the end.
Creating VM schema2003.codfw.wmnet in cluster ganeti01.svc.codfw.wmnet with row=C vcpus=2 memory=2GB disk=10GB link=private. This may take a few minutes.
Thu Aug 13 13:53:18 2020  - INFO: No-installation mode selected, disabling startup
Thu Aug 13 13:53:31 2020  - INFO: Selected nodes for instance schema2003.codfw.wmnet via iallocator hail: ganeti2012.codfw.wmnet, ganeti2013.codfw.wmnet
Thu Aug 13 13:53:32 2020 * creating instance disks...
Thu Aug 13 13:53:36 2020 adding instance schema2003.codfw.wmnet to cluster config
Thu Aug 13 13:53:36 2020 adding disks to cluster config
Thu Aug 13 13:53:36 2020  - INFO: Waiting for instance schema2003.codfw.wmnet to sync disks
Thu Aug 13 13:53:36 2020  - INFO: - device disk/0:  0.20% done, 20m 47s remaining (estimated)
Thu Aug 13 13:54:36 2020  - INFO: - device disk/0: 21.70% done, 3m 37s remaining (estimated)
Thu Aug 13 13:55:37 2020  - INFO: - device disk/0: 43.20% done, 2m 32s remaining (estimated)
Thu Aug 13 13:56:37 2020  - INFO: - device disk/0: 64.80% done, 1m 36s remaining (estimated)
Thu Aug 13 13:57:37 2020  - INFO: - device disk/0: 86.30% done, 37s remaining (estimated)
Thu Aug 13 13:58:14 2020  - INFO: - device disk/0: 99.50% done, 1s remaining (estimated)
Thu Aug 13 13:58:15 2020  - INFO: - device disk/0: 100.00% done, 0s remaining (estimated)
Thu Aug 13 13:58:16 2020  - INFO: - device disk/0: 100.00% done, 0s remaining (estimated)
Thu Aug 13 13:58:16 2020  - INFO: Instance schema2003.codfw.wmnet's disks are in sync
Thu Aug 13 13:58:16 2020  - INFO: Waiting for instance schema2003.codfw.wmnet to sync disks
Thu Aug 13 13:58:16 2020  - INFO: Instance schema2003.codfw.wmnet's disks are in sync
MAC address for schema2003.codfw.wmnet is: aa:00:00:6c:bf:94
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)

elukey@cumin1001:~$ sudo cookbook sre.ganeti.makevm codfw_D schema2004.codfw.wmnet --vcpus 2 --memory 2 --disk 10
START - Cookbook sre.ganeti.makevm
/usr/lib/python3/dist-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for ganeti01.svc.codfw.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
Ready to create Ganeti VM schema2004.codfw.wmnet in the ganeti01.svc.codfw.wmnet cluster on row D with 2 vCPUs, 2GB of RAM, 10GB of disk in the private network.
Is this correct?
Type "done" to proceed
> done
The command output will be printed at the end.
Creating VM schema2004.codfw.wmnet in cluster ganeti01.svc.codfw.wmnet with row=D vcpus=2 memory=2GB disk=10GB link=private. This may take a few minutes.
Thu Aug 13 13:59:59 2020  - INFO: No-installation mode selected, disabling startup
Thu Aug 13 14:00:12 2020  - INFO: Selected nodes for instance schema2004.codfw.wmnet via iallocator hail: ganeti2017.codfw.wmnet, ganeti2015.codfw.wmnet
Thu Aug 13 14:00:13 2020 * creating instance disks...
Thu Aug 13 14:00:18 2020 adding instance schema2004.codfw.wmnet to cluster config
Thu Aug 13 14:00:18 2020 adding disks to cluster config
Thu Aug 13 14:00:18 2020  - INFO: Waiting for instance schema2004.codfw.wmnet to sync disks
Thu Aug 13 14:00:18 2020  - INFO: - device disk/0:  0.20% done, 15m 2s remaining (estimated)
Thu Aug 13 14:01:18 2020  - INFO: - device disk/0: 21.70% done, 3m 37s remaining (estimated)
Thu Aug 13 14:02:18 2020  - INFO: - device disk/0: 43.20% done, 2m 33s remaining (estimated)
Thu Aug 13 14:03:19 2020  - INFO: - device disk/0: 64.80% done, 1m 36s remaining (estimated)
Thu Aug 13 14:04:19 2020  - INFO: - device disk/0: 86.30% done, 37s remaining (estimated)
Thu Aug 13 14:04:56 2020  - INFO: - device disk/0: 99.60% done, 1s remaining (estimated)
Thu Aug 13 14:04:57 2020  - INFO: - device disk/0: 100.00% done, 0s remaining (estimated)
Thu Aug 13 14:04:57 2020  - INFO: Instance schema2004.codfw.wmnet's disks are in sync
Thu Aug 13 14:04:57 2020  - INFO: Waiting for instance schema2004.codfw.wmnet to sync disks
Thu Aug 13 14:04:58 2020  - INFO: Instance schema2004.codfw.wmnet's disks are in sync
MAC address for schema2004.codfw.wmnet is: aa:00:00:3d:8a:63
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)

All vms created and bootstrapped!