Page MenuHomePhabricator
Create Task
Maniphest T239151

Gerrit VM to test data migration
Closed, ResolvedPublic
Actions

Description

We need to test migration of data from Gerrit schema 2.15 to Gerrit schema 2.16 using "real" data. Since the data is private, I can't do this on a labs machine. This task request a Ganeti VM to be used for those tests. It will be reclaimed once the migration has been completed.

Site/Location: eqiad
Number of systems: 1
Service: Gerrit

Networking Requirements: Ability to easily copy data from gerrit1001, access to gerrit sql
Processor Requirements: 8 (migration is IO bound, but getting an understanding of timing on "production-like" hardware would be ideal)

Memory: 16G
Disks: 80G (32 GB worth of git data + overhead)
Other Requirements: None
Project Duration: 3 weeks (hopefully less)

Details

SubjectRepoBranchLines +/-
mariadb: remove ferm firewall hole for gerrit serversoperations/puppetproduction+0 -7
remove gerrit1002.wikimedia.orgoperations/dnsmaster+0 -4
remove gerrit-test.wikimedia.orgoperations/dnsmaster+1 -4
site/DHCP/partman: decom gerrit1002operations/puppetproduction+0 -36
site: remove gerrit role from gerrit1002operations/puppetproduction+1 -1
gerrit: remove absented host key file for gerrit-testoperations/puppetproduction+0 -9
acme_chief: remove gerrit-testoperations/puppetproduction+0 -2
zuul: remove gerrit-test connection and setupoperations/puppetproduction+1 -11
gerrit: stop rsyncing to gerrit1002operations/puppetproduction+0 -1
profile,gerrit: add enable_monitoring flag for gerrit-testoperations/puppetproduction+56 -44
gerrit: allow multiple rsync destination hosts in migration classoperations/puppetproduction+7 -5
ferm_misc/db: allow connections from gerrit1002 in fermoperations/puppetproduction+1 -1
gerrit: assign host gerrit1002 role::gerritoperations/puppetproduction+2 -1
gerrit: set gerrit host name and server list for gerrit1002/gerrit-testoperations/puppetproduction+6 -0
acme_chief/gerrit: remove gerrit-new, add gerrit1002operations/puppetproduction+2 -1
install_server: update MAC address of gerrit1002operations/puppetproduction+1 -1
add IPs for gerrit1002 in row Coperations/dnsmaster+9 -9
site: replace gerrit-test with gerrit1002operations/puppetproduction+1 -1
install_server: rename gerrit-test to gerrit1002operations/puppetproduction+2 -3
install_server: add entries for gerrit-testoperations/puppetproduction+7 -0
gerrit: use 'gerritro' readonly db user on test serveroperations/puppetproduction+3 -0
gerrit: make db_user configurable in Hieraoperations/puppetproduction+5 -1
gerrit: adjust bacula backup behaviour to deal with multiple hostsoperations/puppetproduction+11 -5
Initially assing spare role to gerrit-test.wikimedia.orgoperations/puppetproduction+4 -0
base/icinga: disable notifications and some monitoring for gerrit-testoperations/puppetproduction+8 -1
add IPv6 records for gerrit-test.wikimedia.orgoperations/dnsmaster+2 -0
dns: move gerrit-test to public1-c-eqiadoperations/dnsmaster+2 -5
assign IPs for gerrit1002 and gerrit-testoperations/dnsmaster+8 -1
installserver: add gerrit1002 with flat/VM partman recipeoperations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jan 8 2020, 9:20 PM

Change 562965 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ferm_misc/db: allow connections from gerrit-test in ferm

https://gerrit.wikimedia.org/r/562965

gerritbot added a comment.Jan 9 2020, 10:10 PM

Change 563284 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] gerrit: make db_user configurable in Hiera

https://gerrit.wikimedia.org/r/563284

gerritbot added a comment.Jan 9 2020, 10:38 PM

Change 563284 merged by Dzahn:
[operations/puppet@production] gerrit: make db_user configurable in Hiera

https://gerrit.wikimedia.org/r/563284

gerritbot added a comment.Jan 9 2020, 11:15 PM

Change 563302 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] gerrit: use 'gerritro' readonly db user on test server

https://gerrit.wikimedia.org/r/563302

gerritbot added a comment.Jan 9 2020, 11:39 PM

Change 563302 merged by Dzahn:
[operations/puppet@production] gerrit: use 'gerritro' readonly db user on test server

https://gerrit.wikimedia.org/r/563302

herron mentioned this in T242885: Expand Eqiad Ganeti row_A capacity.Jan 15 2020, 4:30 PM
gerritbot added a comment.Jan 16 2020, 9:35 PM

Change 565392 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: rename gerrit-test to gerrit1002

https://gerrit.wikimedia.org/r/565392

gerritbot added a comment.Jan 16 2020, 9:42 PM

Change 565392 merged by Dzahn:
[operations/puppet@production] install_server: rename gerrit-test to gerrit1002

https://gerrit.wikimedia.org/r/565392

gerritbot added a comment.Jan 16 2020, 9:59 PM

Change 565395 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: replace gerrit-test with gerrit1002

https://gerrit.wikimedia.org/r/565395

gerritbot added a comment.Jan 16 2020, 10:00 PM

Change 565395 merged by Dzahn:
[operations/puppet@production] site: replace gerrit-test with gerrit1002

https://gerrit.wikimedia.org/r/565395

gerritbot added a comment.Jan 16 2020, 10:28 PM

Change 565399 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add IPs for gerrit1002 in row C

https://gerrit.wikimedia.org/r/565399

Stashbot added a comment.Jan 16 2020, 10:38 PM

Mentioned in SAL (#wikimedia-operations) [2020-01-16T22:38:41Z] <mutante> ganeti1003 - deleting VM gerrit-test (T239151)

ops-monitoring-bot added a comment.Jan 17 2020, 6:19 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: gerrit-test.wikimedia.org

  • gerrit-test.wikimedia.org (FAIL)
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Unable to connect to the host, wipe of bootloaders will not be performed: Cumin execution failed (exit_code=2)
    • Failed to shutdown, manual intervention required: Cumin execution failed (exit_code=2)
    • Set Netbox status on VM not yet supported: manual intervention required
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

gerritbot added a comment.Jan 17 2020, 8:33 PM

Change 565399 merged by Dzahn:
[operations/dns@master] add IPs for gerrit1002 in row C

https://gerrit.wikimedia.org/r/565399

Dzahn added a comment.Jan 17 2020, 8:35 PM

IP situation fixed!

server:

gerrit1002.wikimedia.org has address 208.80.154.75
gerrit1002.wikimedia.org has IPv6 address 2620:0:861:3:208:80:154:75

service:

gerrit-test.wikimedia.org has address 208.80.154.78
gerrit-test.wikimedia.org has IPv6 address 2620:0:861:3:208:80:154:78

Dzahn added a comment.Jan 17 2020, 8:54 PM

recreating VM as gerrit1002 so that we can use gerrit-test as service name:

Creating new VM named gerrit1002.wikimedia.org in eqiad with row=C vcpu=1 memory=16 gigabytes disk=80 gigabytes link=public

gerritbot added a comment.Jan 18 2020, 12:30 AM

Change 565708 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: update MAC address of gerrit1002

https://gerrit.wikimedia.org/r/565708

gerritbot added a comment.Jan 18 2020, 12:45 AM

Change 565708 merged by Dzahn:
[operations/puppet@production] install_server: update MAC address of gerrit1002

https://gerrit.wikimedia.org/r/565708

gerritbot added a comment.Jan 18 2020, 1:12 AM

Change 565715 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] gerrit: set gerrit host name and server list for gerrit1002/gerrit-test

https://gerrit.wikimedia.org/r/565715

gerritbot added a comment.Jan 18 2020, 1:22 AM

Change 565716 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] acme_chief/gerrit: remove gerrit-new, add gerrit1002

https://gerrit.wikimedia.org/r/565716

gerritbot added a comment.Jan 21 2020, 7:26 PM

Change 565716 merged by Dzahn:
[operations/puppet@production] acme_chief/gerrit: remove gerrit-new, add gerrit1002

https://gerrit.wikimedia.org/r/565716

gerritbot added a comment.Jan 21 2020, 7:28 PM

Change 565715 merged by Dzahn:
[operations/puppet@production] gerrit: set gerrit host name and server list for gerrit1002/gerrit-test

https://gerrit.wikimedia.org/r/565715

gerritbot added a comment.Jan 21 2020, 8:14 PM

Change 562587 merged by Dzahn:
[operations/puppet@production] gerrit: assign host gerrit1002 role::gerrit

https://gerrit.wikimedia.org/r/562587

Dzahn added a comment.Jan 21 2020, 8:25 PM

https://gerrit.wikimedia.org/r/q/topic:%22gerrit-test%22+(status:open%20OR%20status:merged)

https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit1002.wikimedia.org

Dzahn added a comment.Jan 21 2020, 8:29 PM

The VM is now usable. It has the role(gerrit) on it and no more puppet errors. It uses its own service name/IP:

https://gerrit-test.wikimedia.org

Shell access is automatically granted by the role to the same people who have it on the prod server.

Monitoring and backups should be disabled.

Gerrit is configured to only know about itself and not the other Gerrit servers.

There are 63G free on / including /srv

Dzahn added a comment.Jan 21 2020, 8:31 PM

The mysql user has also been made configurable (along with backups / monitoring) and it is using:

104     hostname = m2-master.eqiad.wmnet
105     database = reviewdb
106     username = gerritro

Note the 'gerritro' read-only user.

Dzahn added a comment.Jan 21 2020, 8:33 PM
 94     heapLimit = 5g
 95     slave = false

116     canonicalWebUrl = https://gerrit-test.wikimedia.org/r

218 [sshd]
219     listenAddress = gerrit-test.wikimedia.org:29418
220 
221     listenAddress = [2620:0:861:3:208:80:154:78]:29418
Dzahn closed this task as Resolved.Jan 21 2020, 8:34 PM
Dzahn claimed this task.
Dzahn added a comment.Jan 21 2020, 8:41 PM

The gerrit acmechief TLS cert has been updated to contain "gerrit-test" in addition to gerrit and gerrit-replica. The "gerrit-new" name has been removed from it. This affected all Gerrit servers, including prod gerrit1001 which has the new cert now.

gerritbot added a comment.Jan 21 2020, 8:42 PM

Change 562965 merged by Dzahn:
[operations/puppet@production] ferm_misc/db: allow connections from gerrit1002 in ferm

https://gerrit.wikimedia.org/r/562965

Maintenance_bot removed a project: Patch-For-Review.Jan 21 2020, 9:11 PM
gerritbot added a comment.Jan 21 2020, 9:34 PM

Change 566367 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] gerrit: allow multiple rsync destination hosts in migration class

https://gerrit.wikimedia.org/r/566367

gerritbot added a project: Patch-For-Review.Jan 21 2020, 9:34 PM
gerritbot added a comment.Jan 21 2020, 9:43 PM

Change 566367 merged by Dzahn:
[operations/puppet@production] gerrit: allow multiple rsync destination hosts in migration class

https://gerrit.wikimedia.org/r/566367

Stashbot added a comment.Jan 21 2020, 9:48 PM

Mentioned in SAL (#wikimedia-releng) [2020-01-21T21:48:46Z] <mutante> gerrit - rsyncing git data from gerrit1001 to gerrit1002 (T239151)

Stashbot added a comment.Jan 21 2020, 10:09 PM

Mentioned in SAL (#wikimedia-releng) [2020-01-21T22:09:48Z] <mutante> gerrit - rsyncing 'git' and 'plugin' data dirs and /var/lib/gerrit2/review_site/ from gerrit1001 to gerrit1002 WITH --delete T239151

Maintenance_bot removed a project: Patch-For-Review.Jan 21 2020, 10:11 PM
Marostegui mentioned this in T243800: gerritro user getting access denied from gerrit1002.Jan 27 2020, 11:09 PM
Stashbot added a comment.Feb 3 2020, 11:21 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-03T23:21:49Z] <mutante> gerrit1002 - deleting gerrit.log and gerrit.json files from January to free about 4GB of space (T239151 T243983)

Stashbot mentioned this in T243983: Add second virtual hard disk to ganeti gerrit test instance.Feb 3 2020, 11:21 PM
Stashbot added a comment.Feb 3 2020, 11:26 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-03T23:26:32Z] <mutante> ganeti1003 - sudo gnt-instance modify --disk add:size=10G gerrit1002.wikimedia.org (T239151 T243983)

Stashbot added a comment.Feb 4 2020, 12:09 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-04T00:09:21Z] <mutante> gerrit1002 - replaced ens5 with ens6 in /etc/network/interfaces (IP and row had changed in the past, needed manual fix after reboot and now came back) ; mkfs.ext4 /dev/vdb on new additional 10GB disk. (T239151 T243983)

Dzahn added a subtask: T243808: gerrit1002 running out of space.Feb 5 2020, 7:58 PM
thcipriani closed subtask T243808: gerrit1002 running out of space as Resolved.Feb 25 2020, 7:50 PM
jbond reopened subtask T243808: gerrit1002 running out of space as Open.Apr 1 2020, 10:35 AM
Dzahn closed subtask T243808: gerrit1002 running out of space as Resolved.Apr 2 2020, 12:35 PM
jbond reopened subtask T243808: gerrit1002 running out of space as Open.Apr 2 2020, 2:36 PM
Dzahn closed subtask T243808: gerrit1002 running out of space as Resolved.Apr 29 2020, 8:56 AM
Dzahn reassigned this task from Dzahn to QChris.Apr 29 2020, 8:59 AM
Dzahn added a subscriber: QChris.

@QChris fyi, this is the dedicated test machine for the gerrit upgrade, you can feel free to use it. I confirmed your shell user exists.

also see T243808#6025787 for some reason logs are not gzipped there but they are on the prod gerrit server, even though this uses the same role of course

gerritbot added a comment.May 4 2020, 8:34 PM

Change 594293 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile,gerrit: add enable_monitoring flag for gerrit-test

https://gerrit.wikimedia.org/r/594293

gerritbot added a project: Patch-For-Review.May 4 2020, 8:34 PM
gerritbot added a comment.May 5 2020, 8:51 AM

Change 594293 merged by Dzahn:
[operations/puppet@production] profile,gerrit: add enable_monitoring flag for gerrit-test

https://gerrit.wikimedia.org/r/594293

Dzahn added a comment.May 5 2020, 9:03 AM

Icinga monitoring for gerrit1002 has been removed. Thanks Cole.

Maintenance_bot removed a project: Patch-For-Review.May 5 2020, 9:11 AM
Dzahn reopened this task as Open.May 28 2020, 9:52 AM

Was blocked by T243800 which is now fixed.

Though gerrit service does not keep running due to other issues, like file permissions.

Reopening

Stashbot added a comment.May 28 2020, 10:02 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-28T10:02:39Z] <mutante> gerrit1002 (test server) - chown -R gerrit2:gerrit2 /var/lib/gerrit/review_site ; restarted gerrit service, now the service is not in restart loop anymore, gerrit-ssh is listening too, just not accepting publickey (T239151)

Dzahn closed this task as Resolved.May 28 2020, 10:06 AM
● gerrit.service - Gerrit code review tool
   Loaded: loaded (/lib/systemd/system/gerrit.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-05-28 09:51:08 UTC; 12min ago
[contint2001:~] $ sudo -u zuul ssh -p 29418 jenkins-bot@gerrit-test.wikimedia.org

  ****    Welcome to Gerrit Code Review    ****

  Hi jenkins-bot, you have successfully connected over SSH.

  Unfortunately, interactive shells are disabled.
  To clone a hosted Git repository, use:

  git clone ssh://jenkins-bot@gerrit-test.wikimedia.org:29418/REPOSITORY_NAME.git

Connection to gerrit-test.wikimedia.org closed.

cc: @hashar @QChris ^

Dzahn reopened this task as Open.Jun 5 2020, 11:46 AM
Dzahn claimed this task.

As Chris points out this VM only has 1 vcpu. But requested were 8 and he needs 8. That was my mistake it seems. Reopening.

QChris added a parent task: T254158: Gerrit 3.2 upgrade.Jun 5 2020, 11:56 AM
Stashbot added a comment.Jun 5 2020, 12:41 PM

Mentioned in SAL (#wikimedia-operations) [2020-06-05T12:41:16Z] <mutante> rebooting gerrit1002 to add more vCPUs, after [ganeti1009:~] $ sudo gnt-instance modify -B vcpus=8 gerrit1002.wikimedia.org T239151

Dzahn added a comment.Jun 5 2020, 12:50 PM
[gerrit1002:~] $ lscpu
...
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           8
...
CPU MHz:             2499.998
BogoMIPS:            4999.99
Dzahn closed this task as Resolved.Jun 5 2020, 12:51 PM

@QChris ^ fixed

QChris added a comment.Jun 5 2020, 1:04 PM

Wooohooo \o/ Thanks!

Dzahn mentioned this in T254644: Set up test instance for the community to test new version.Jun 27 2020, 8:50 PM

T254644 has some comments about removing this again now that Gerrit prod is on 3.2. We can also use this ticket to decom it and feel free to open it.

Dzahn reopened this task as Open.Jun 27 2020, 8:53 PM
gerritbot added a comment.Jul 7 2020, 12:42 AM

Change 609875 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] zuul: remove gerrit-test connection and setup

https://gerrit.wikimedia.org/r/609875

gerritbot added a project: Patch-For-Review.Jul 7 2020, 12:42 AM
gerritbot added a comment.Jul 7 2020, 12:51 AM

Change 609878 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] acme_chief: remove gerrit-test

https://gerrit.wikimedia.org/r/609878

gerritbot added a comment.Jul 7 2020, 12:54 AM

Change 609879 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/DHCP/partman: decom gerrit1002

https://gerrit.wikimedia.org/r/609879

gerritbot added a comment.Jul 7 2020, 12:57 AM

Change 609883 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] gerrit: stop rsyncing to gerrit1002

https://gerrit.wikimedia.org/r/609883

gerritbot added a comment.Jul 7 2020, 1:04 AM

Change 609884 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mariadb: remove ferm firewall hole for gerrit servers

https://gerrit.wikimedia.org/r/609884

gerritbot added a comment.Jul 7 2020, 1:07 AM

Change 609886 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove gerrit-test.wikimedia.org

https://gerrit.wikimedia.org/r/609886

gerritbot added a comment.Jul 7 2020, 1:07 AM

Change 609887 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove gerrit1002.wikimedia.org

https://gerrit.wikimedia.org/r/609887

gerritbot added a comment.Jul 7 2020, 6:31 PM

Change 609883 merged by Dzahn:
[operations/puppet@production] gerrit: stop rsyncing to gerrit1002

https://gerrit.wikimedia.org/r/609883

gerritbot added a comment.Jul 7 2020, 6:41 PM

Change 609875 merged by Dzahn:
[operations/puppet@production] zuul: remove gerrit-test connection and setup

https://gerrit.wikimedia.org/r/609875

gerritbot added a comment.Jul 7 2020, 6:48 PM

Change 609878 merged by Dzahn:
[operations/puppet@production] acme_chief: remove gerrit-test

https://gerrit.wikimedia.org/r/609878

gerritbot added a comment.Jul 7 2020, 6:54 PM

Change 610139 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] gerrit: remove absented host key file for gerrit-test

https://gerrit.wikimedia.org/r/610139

gerritbot added a comment.Jul 7 2020, 7:02 PM

Change 610139 merged by Dzahn:
[operations/puppet@production] gerrit: remove absented host key file for gerrit-test

https://gerrit.wikimedia.org/r/610139

gerritbot added a comment.Jul 7 2020, 7:11 PM

Change 610142 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: remove gerrit role from gerrit1002

https://gerrit.wikimedia.org/r/610142

gerritbot added a comment.Jul 7 2020, 7:16 PM

Change 610142 merged by Dzahn:
[operations/puppet@production] site: remove gerrit role from gerrit1002

https://gerrit.wikimedia.org/r/610142

ops-monitoring-bot added a comment.Jul 7 2020, 7:29 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: gerrit1002.wikimedia.org

  • gerrit1002.wikimedia.org (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
gerritbot added a comment.Jul 7 2020, 7:33 PM

Change 609879 merged by Dzahn:
[operations/puppet@production] site/DHCP/partman: decom gerrit1002

https://gerrit.wikimedia.org/r/609879

gerritbot added a comment.Jul 7 2020, 7:35 PM

Change 609886 merged by Dzahn:
[operations/dns@master] remove gerrit-test.wikimedia.org

https://gerrit.wikimedia.org/r/609886

gerritbot added a comment.Jul 7 2020, 7:39 PM

Change 609887 merged by Dzahn:
[operations/dns@master] remove gerrit1002.wikimedia.org

https://gerrit.wikimedia.org/r/609887

Dzahn closed this task as Resolved.Jul 7 2020, 7:43 PM

The VM has been removed from all places in the repos, puppet and DNS.

The decom cookbook destroyed it and removed it from monitoring, puppetdb etc.

gerritbot added a comment.Jul 10 2020, 6:55 PM

Change 609884 merged by Dzahn:
[operations/puppet@production] mariadb: remove ferm firewall hole for gerrit servers

https://gerrit.wikimedia.org/r/609884

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL