Thanks for the ticket, otherwise we would surely forget about the request with so many things going on.

I can take care fully of this, but @Marostegui should be on the loop because of new db and grants will be needed and he should be aware of the final state of misc dbs. Specially as next week I may be unavailable for a few days.

@QChris Let me understand the request, because I think I got the general idea, but I need the details. Questions:

• You need a copy of the "current" db. With current, you mean that you don't need it to be updated all the time, just like "a snapshot" (a dump and a reload to same other place) , correct? You don't need it to have production application writes on the copy, right? We could provide the second option, but it would take more time to setup.

• Does it need to be on the same server as the regular DB? Again, we can provide both the same and a separate one. If the db is small, it is easier to do it on the same, if it is very large, is easier and safer to do it on a separate one

• From which server will you be accessing the new db/testing the upgrade? There are some security considerations if it is from another datacenter, as wan connections require encryption to be enabled on client. Asking specifically because in the past many misc services were tested from codfw, and in that case it would be better to create the test db in the same datacenter.

• I am guessing the copy is temporary, for how long do you think you will be using it?

• You mention "the current read-only user", but AFAIK there is not such a thing as a gerrit read only issues, and I wonder if you are currently working with a read-only replica (for which no read-write users can be effectively created). Let me know if you want to use the same credential that for production.

Let's sync on understanding what you are trying to do, so I can provide you the copy in the best way for what you are trying to achieve.

In T254516#6196583, @jcrespo wrote:

I can take care fully of this, but @Marostegui should be on the loop because of new db and grants will be needed and he should be aware of the final state of misc dbs.

Thanks!

• You need a copy of the "current" db. With current, you mean that you don't need it to be updated all the time, just like "a snapshot" (a dump and a reload to same other place) , correct?

Correct. A snapshot is all I need.

You don't need it to have production application writes on the copy, right?

Correct. Gerrit production writes need not be replicated to the new database.

• Does it need to be on the same server as the regular DB?

No, it does not need to be on the same server. Whatever is easier to set up. (gerrit1002 should be able to access the DB, that's all I care about.)

• From which server will you be accessing the new db/testing the upgrade?

Only from gerrit1002 (no other hosts).

• I am guessing the copy is temporary, for how long do you think you will be using it?

Yes, it'll be temporary. I need it for at least one full upgrade run. It'll be a few hours. So let's say I need it for 2 days (to avoid frictions because we're in different timezones).

• You mention "the current read-only user", but AFAIK there is not such a thing as a gerrit read only issues, and I wonder if you are currently working with a read-only replica (for which no read-write users can be effectively created).

I think both gerrit1001 and gerrit1002 are using the same database. gerrit1001 with a user that has read/write access. gerrit1002 with a user that has read only access. I'm not sure which aspects of the connection settings we consider public and private, so I'm not posting them. But compare the output of grep -A 5 ^.database /etc/gerrit/gerrit.config on gerrit1001 and gerrit1002.

Let me know if you want to use the same credential that for production.

I'm fine if we re-use either credentials. But I'm also fine if we pick new ones. Whatever is in-line with WMF policies and is easier for you to set up.

(If the above seems too vague, let's have a user gerrit-test with random password. That user name makes the intention clear.)

Thank you for your help!

Thanks Jaime for taking care of this.
Just a clarification, there is a RO gerrit user called: gerritro which just SELECT grant.

GRANT SELECT ON `reviewdb`.* TO 'gerritro'@'10.64.0.135'

Access for the gerritro user was fixed on May 28th in T243800#6171861. And that made it possible to run gerrit on the test server.

I think that's an entirely separate request now, though.

Thanks, now that I understood the actual requirements (I wasn't aware of the existence of a ro account), I will take care of creating a test upgrade database from a reviewdb snapshot and provide an additional user with the same grants as the current gerrit app r/w user.

The new, temporary user must have access to the new db from gerrit1002.

Not knowing how large gerrit db is and how much time it will take to clone it, I am guessing this will be done on Wednesday, as I will be unavailable tomorrow, and not sure I will be able to have it finished by the end of today. Rise priority and ping Manuel if this needs more urgency than that.

Mentioned in SAL (#wikimedia-operations) [2020-06-10T10:03:50Z] <jynus> cloning reviewdb into reviewdb-test at db1132 with replication enabled T254516

Thank you for handling this.
Is it worth adding that user to the .sql files we have in puppet to track users?

Is it worth adding that user to the .sql files we have in puppet to track users?

I intended not to do that as I was told this would only need a week.

Sounds good!

This also needs a puppet change to allow changing the database name in Hiera and the password needs to be added to the private repo.

Otherwise @QChris would have to have puppet disabled the entire time.

Uploading a change for that.

Change 604343 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] gerrit: add parameter for db_name, let gerrit1002 use test db

https://gerrit.wikimedia.org/r/604343

Change 604343 merged by Dzahn:
[operations/puppet@production] gerrit: add parameter for db_name, let gerrit1002 use test db

https://gerrit.wikimedia.org/r/604343

After the merge above:

on gerrit1002:

- database = reviewdb
-  username = gerritro 

+    database = reviewdb-test
+    username = gerrittest

no change on prod servers.

The db_pass was updated in private/hieradata in hosts/gerrit1002.yaml.