Page MenuHomePhabricator

Create test Gerrit instance
Open, In Progress, LowPublic

Description

I'm in need for testing against the Gerrit API and have noticed that the test instances have been deleted. I'd like for a new instance so I can test, pretty please!

Event Timeline

Recently both the old buster Gerrit instance in devtools AND the newer bullseye test instance were deleted in T360964#9718714.

Deleting the buster instance was a positive to me, deleting the buster instance as well wasn't intended originally.

LSobanski moved this task from Incoming to Backlog on the collaboration-services board.

@BCornwall what specific features do you need as it may influence how quickly this can happen?

Mentioned in SAL (#wikimedia-cloud) [2024-05-01T23:59:03Z] <mutante> creating instance gerrit-bullseye (T363196)

Good news: We didn't lose all the Hiera data because most of it is in the repo (as opposed to web Hiera which gets deleted when an instance gets deleted if it's applied on role level).

Needs a few adjustments only.

~/repos/puppet/hieradata/cloud/eqiad1/devtools$ grep gerrit common.yaml 
  gerrit:
  gerrit/gerrit:
    repository: operations/software/gerrit
    repository: operations/software/gerrit/tools/gervert/deploy
profile::gerrit::config: 'gerrit.config.erb'
profile::gerrit::host: 'gerrit.devtools.wmcloud.org'
profile::gerrit::mask_service: false
profile::gerrit::ssh_allowed_hosts:
- gerrit-prod-1001.devtools.eqiad.wmflabs
profile::gerrit::replica_hosts: []
profile::gerrit::ipv6: ~
profile::gerrit::replication: {}
profile::gerrit::git_dir: /srv/gerrit/git
profile::gerrit::ssh_host_key: ssh_host_key
profile::gerrit::bacula: gerrit-repo-data
profile::gerrit::java_version: 8
profile::gerrit::daemon_user: 'gerrit2'
profile::gerrit::gerrit_site: "/var/lib/gerrit2/review_site"
profile::gerrit::scap_user: 'gerrit-deploy'
profile::gerrit::manage_scap_user: true
profile::gerrit::scap_key_name: 'gerrit'
  - '/etc/ssh/userkeys/%u.d/gerrit-scap'
profile::gerrit::use_acmechief: false
profile::gerrit::backups_enabled: false
profile::gerrit::backup_set: 'gerrit-repo-data'
profile::gerrit::active_host: 'gerrit-prod-1001.devtools.eqiad.wmflabs'
profile::gerrit::migration::data_dir: /srv/gerrit
          - 'gerrit-prod-1001.devtools.eqiad.wmflabs'

And the remaining keys on web Hiera can be retrieved from:

https://phabricator.wikimedia.org/rCLIP0817761ddbf3b926f8992d43a21210f82c9ec247

Change #1026195 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: set java_home and migration user in repo Hiera

https://gerrit.wikimedia.org/r/1026195

Change #1026197 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] devtools: update gerrit and phab instance names in default Hiera

https://gerrit.wikimedia.org/r/1026197

Change #1026195 merged by Dzahn:

[operations/puppet@production] gerrit: set java_home and migration user in repo Hiera

https://gerrit.wikimedia.org/r/1026195

@BCornwall what specific features do you need as it may influence how quickly this can happen?

I'm working on automated CR submission and didn't want to spam the active gerrit/channels with tests. For now I can just use the WIP feature to avoid notifications, I imagine. Thanks!

Change #1026197 merged by Dzahn:

[operations/puppet@production] devtools: update gerrit and phab instance names in default Hiera

https://gerrit.wikimedia.org/r/1026197

Change #1036764 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] devtools: rename hieradata host data, match new instance name

https://gerrit.wikimedia.org/r/1036764

Change #1036764 merged by Dzahn:

[operations/puppet@production] devtools: rename hieradata host data, match new instance name

https://gerrit.wikimedia.org/r/1036764

Change #1036765 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] devtools: update IP for gerrit test instance

https://gerrit.wikimedia.org/r/1036765

Change #1036765 merged by Dzahn:

[operations/puppet@production] devtools: update IP for gerrit test instance

https://gerrit.wikimedia.org/r/1036765

Change #1036767 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] devtools: update host name for new gerrit test instance

https://gerrit.wikimedia.org/r/1036767

Change #1036767 merged by Dzahn:

[operations/puppet@production] devtools: update host name for new gerrit test instance

https://gerrit.wikimedia.org/r/1036767

Change #1036771 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: add parameter to toggle lfs_replica_sync

https://gerrit.wikimedia.org/r/1036771

Change #1037574 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit/test: set lfs sync dest host to itself

https://gerrit.wikimedia.org/r/1037574

Change #1036771 merged by Dzahn:

[operations/puppet@production] gerrit: add parameter to toggle lfs_replica_sync

https://gerrit.wikimedia.org/r/1036771

Deploying the change above fixed a major puppet issue related to lfs data sync on the re-created test instance - while it was noop on prod machines

We are now skipping the lfs_replica_sync if in testing. It caused errors because we don't have multiple machines there like in production.

A good step forward.

Now we are on to the (expected) scap deploy problem on hosts that didn't have a deploy yet.

Error: Execution of '/usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False' returned 70: 
Error: /Stage[main]/Gerrit/Scap::Target[gerrit/gerrit]/Package[gerrit/gerrit]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False' returned 70: 

Error: Execution of '/usr/bin/scap deploy-local --repo gervert/deploy -D log_json:False' returned 70: 
Error: /Stage[main]/Gerrit/Scap::Target[gervert/deploy]/Package[gervert/deploy]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/scap deploy-local --repo gervert/deploy -D log_json:False' returned 70:

It's probably T257317.

The manual fix should have been sudo -u gerrit-deploy scap deploy --init -Dblock_deployments:False in /srv/deployment/gerrit (and gervert) as we found last time (T257317#9762601).

But:

ERROR:deploy:deploy failed: <FileNotFoundError> [Errno 2] No such file or directory: '/srv/deployment/gerrit/.git/config-files'

Change #1037574 abandoned by Dzahn:

[operations/puppet@production] gerrit/test: set lfs sync dest host to itself

Reason:

solved by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1036771 instead

https://gerrit.wikimedia.org/r/1037574

Dzahn changed the task status from Open to In Progress.Fri, May 31, 10:21 PM