Page MenuHomePhabricator

Get a writable reviewdb clone to test Gerrit upgrade with
Closed, ResolvedPublic

Description

While the whole Gerrit upgrade process seems to work fine, but at some points Gerrit wants to write to the database. Like increasing the database schema version. That of course fails with the current read-only user.

Could we get a writable copy of the current reviewdb on m2 that we can test against to make sure the upgrade on gerrit1001 does not get blocked by the (then writable) database?

(@jcrespo: Sorry for brashly assigning to you, but I'm not sure what the correct project would be, and you're a DBA and we've had contact around these databases already. Feel free to push back/re-assign/or simply tell what the correct tag would be)

Related Objects

Event Timeline

QChris created this task.Jun 4 2020, 7:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 4 2020, 7:56 PM
QChris assigned this task to jcrespo.Jun 4 2020, 7:57 PM
QChris added a parent task: T254158: Gerrit 3.2 upgrade.
QChris updated the task description. (Show Details)
Paladox added a subscriber: Paladox.Jun 4 2020, 8:02 PM
jcrespo added a subscriber: Marostegui.EditedJun 5 2020, 2:47 PM

Thanks for the ticket, otherwise we would surely forget about the request with so many things going on.

I can take care fully of this, but @Marostegui should be on the loop because of new db and grants will be needed and he should be aware of the final state of misc dbs. Specially as next week I may be unavailable for a few days.

@QChris Let me understand the request, because I think I got the general idea, but I need the details. Questions:

  • You need a copy of the "current" db. With current, you mean that you don't need it to be updated all the time, just like "a snapshot" (a dump and a reload to same other place) , correct? You don't need it to have production application writes on the copy, right? We could provide the second option, but it would take more time to setup.
  • Does it need to be on the same server as the regular DB? Again, we can provide both the same and a separate one. If the db is small, it is easier to do it on the same, if it is very large, is easier and safer to do it on a separate one
  • From which server will you be accessing the new db/testing the upgrade? There are some security considerations if it is from another datacenter, as wan connections require encryption to be enabled on client. Asking specifically because in the past many misc services were tested from codfw, and in that case it would be better to create the test db in the same datacenter.
  • I am guessing the copy is temporary, for how long do you think you will be using it?
  • You mention "the current read-only user", but AFAIK there is not such a thing as a gerrit read only issues, and I wonder if you are currently working with a read-only replica (for which no read-write users can be effectively created). Let me know if you want to use the same credential that for production.

Let's sync on understanding what you are trying to do, so I can provide you the copy in the best way for what you are trying to achieve.

jcrespo triaged this task as Medium priority.Jun 5 2020, 2:50 PM
QChris added a comment.Jun 5 2020, 7:03 PM

I can take care fully of this, but @Marostegui should be on the loop because of new db and grants will be needed and he should be aware of the final state of misc dbs.

Thanks!

  • You need a copy of the "current" db. With current, you mean that you don't need it to be updated all the time, just like "a snapshot" (a dump and a reload to same other place) , correct?

Correct. A snapshot is all I need.

You don't need it to have production application writes on the copy, right?

Correct. Gerrit production writes need not be replicated to the new database.

  • Does it need to be on the same server as the regular DB?

No, it does not need to be on the same server. Whatever is easier to set up. (gerrit1002 should be able to access the DB, that's all I care about.)

  • From which server will you be accessing the new db/testing the upgrade?

Only from gerrit1002 (no other hosts).

  • I am guessing the copy is temporary, for how long do you think you will be using it?

Yes, it'll be temporary. I need it for at least one full upgrade run. It'll be a few hours. So let's say I need it for 2 days (to avoid frictions because we're in different timezones).

  • You mention "the current read-only user", but AFAIK there is not such a thing as a gerrit read only issues, and I wonder if you are currently working with a read-only replica (for which no read-write users can be effectively created).

I think both gerrit1001 and gerrit1002 are using the same database. gerrit1001 with a user that has read/write access. gerrit1002 with a user that has read only access. I'm not sure which aspects of the connection settings we consider public and private, so I'm not posting them. But compare the output of grep -A 5 ^.database /etc/gerrit/gerrit.config on gerrit1001 and gerrit1002.

Let me know if you want to use the same credential that for production.

I'm fine if we re-use either credentials. But I'm also fine if we pick new ones. Whatever is in-line with WMF policies and is easier for you to set up.

(If the above seems too vague, let's have a user gerrit-test with random password. That user name makes the intention clear.)

Thank you for your help!

Thanks Jaime for taking care of this.
Just a clarification, there is a RO gerrit user called: gerritro which just SELECT grant.

GRANT SELECT ON `reviewdb`.* TO 'gerritro'@'10.64.0.135'
Dzahn added a subscriber: Dzahn.Jun 8 2020, 9:42 AM

Access for the gerritro user was fixed on May 28th in T243800#6171861. And that made it possible to run gerrit on the test server.

I think that's an entirely separate request now, though.

Marostegui moved this task from Triage to In progress on the DBA board.Jun 8 2020, 11:04 AM

Thanks, now that I understood the actual requirements (I wasn't aware of the existence of a ro account), I will take care of creating a test upgrade database from a reviewdb snapshot and provide an additional user with the same grants as the current gerrit app r/w user.

The new, temporary user must have access to the new db from gerrit1002.

Not knowing how large gerrit db is and how much time it will take to clone it, I am guessing this will be done on Wednesday, as I will be unavailable tomorrow, and not sure I will be able to have it finished by the end of today. Rise priority and ping Manuel if this needs more urgency than that.

Mentioned in SAL (#wikimedia-operations) [2020-06-10T10:03:50Z] <jynus> cloning reviewdb into reviewdb-test at db1132 with replication enabled T254516

jcrespo closed this task as Resolved.Jun 10 2020, 10:39 AM

A reviewdb dump from 2020-06-09 00:00:01 has been loaded into m2, under the reviewdb-test database name. A separate pair of user & password has been created with read/write access to the reviewdb-test tables only. Please note that while all data there can be modified in anyway and it won't affect the real db, it lives in the same server as the real production reviewdb database, so please do not create excesive load or hour-running transactions it may affect other services there.

Also please be careful to not confuse reviewdb with reviewdb-test.

Credentials for access have been written to gerrit1002 host at /etc/gerrit/gerrit_test. I tested them and they work well.

Please reopen or file a new ticket when done so we can remove the data & credentials.

Thank you for handling this.
Is it worth adding that user to the .sql files we have in puppet to track users?

Is it worth adding that user to the .sql files we have in puppet to track users?

I intended not to do that as I was told this would only need a week.

This also needs a puppet change to allow changing the database name in Hiera and the password needs to be added to the private repo.

Otherwise @QChris would have to have puppet disabled the entire time.

Uploading a change for that.

Change 604343 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] gerrit: add parameter for db_name, let gerrit1002 use test db

https://gerrit.wikimedia.org/r/604343

Change 604343 merged by Dzahn:
[operations/puppet@production] gerrit: add parameter for db_name, let gerrit1002 use test db

https://gerrit.wikimedia.org/r/604343

Dzahn added a comment.Jun 12 2020, 9:26 AM

After the merge above:

on gerrit1002:

- database = reviewdb
-  username = gerritro 

+    database = reviewdb-test
+    username = gerrittest

no change on prod servers.

The db_pass was updated in private/hieradata in hosts/gerrit1002.yaml.