@Jelto I tested this on gitlab1004.

@Jelto Thanks for the heads up. I had down timed alerts on this host since it's still a work in progress. I think the downtime expired today but I extended it.

I have some concerns because the docs state The chosen previous backup is overwritten.. So we might see quite a lot of IO here for extracting the tar archive and writing it again. But tests will show that.

Thanks @Marostegui I can confirm that this works on otrs1001. I trust that it will work on the other host but I have to install the mysql-client. I will test on vrts2001 after making the necessary Puppet changes.

Actually, just scratch everything I've said so far... We might not need a new database user. Could we just have the current user otrs be able to connnect to m2-slave in both data centers?

aokoth@otrs1001:~$ mysql -h m2-slave.eqiad.wmnet -u otrs -p otrs -P 3323
ERROR 1045 (28000): Access denied for user 'otrs'@'10.64.16.39' (using password: YES)

Hi @Marostegui We did some testing and the outcome was we found we could connect to the m2-master on both hosts but can't connect to the m2-slave on either server. Despite the fact that the new user has read-only grants, we would prefer that the non-active host connects to the secondary database server as well.

@Marostegui Yeah, that's okay. I'd like the username vrts.

Resolved: https://netbox.wikimedia.org/virtualization/virtual-machines/539/

@Marostegui Yeah, that's exactly what I need but not on the primary database host. We are setting up a new failover VM for VRTS in codfw (vrts2001). The current VRTS instance (otrs1001) connects to the m2-master host in eqiad. So we need a new user that can connect to m2-slave in codfw and said user should only have SELECT grants (I'm not sure if this is implied by the fact that it connects to the secondary host).

As an idea, I was combing through the docs (https://docs.gitlab.com/ee/raketasks/backup_gitlab.html#incremental-repository-backups) and noticed Gitlab supports incremental backups:

@Marostegui Yeah, that would be okay.

Hey @jbond Seems like the merge broke Puppet on otrs1001.eqiad.wmnet. It fails with the following error:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Failed to parse template vrts/exim4.conf.vrts.erb:
  Filepath: /etc/puppet/modules/vrts/templates/exim4.conf.vrts.erb
  Line: 22
  Detail: undefined method `unwrap' for "<redacted>":String
 (file: /etc/puppet/modules/vrts/manifests/mail.pp, line: 41, column: 20) on node otrs1001.eqiad.wmnet
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Hey @Lucas_Werkmeister_WMDE Yeah, I was actually debating whether to remove that checkbox but I'll just leave it unchecked since it's not needed in this case. It's resolved now. Feel free to close the ticket.

Hey @AnnWF Kindly sign this https://phabricator.wikimedia.org/L3

I have added to the wmf-nda group as well. Thanks @Aklapper

Ooh, sorry I missed that step. I have added you to the wmf-nda group as well. Thanks @Aklapper

Hey @Ottomata @karapayneWMDE Kindly approve.

@greg This is resolved. Feel free to close the ticket if everything is good on your side.

@Slst2020 This is resolved. Feel free to close the ticket if everything is good on your end.

Hey @greg Yeah, Lisa G is your manager (confirmed on Namely). So will need approval from her (@Lgruwell-WMF ) as well as @Ottomata or @odimitrijevic

Hey, @Ottomata You are listed as one of the approvers for members joining this list. Kindly approve.

Hey @Dzahn Wasn't this patch reverted?

Hi @rook Sorry. I hadn't noticed that the number of instances should also be increased as part of the quota. Given the resources that were added, I think we need to increase that by two since it fits the two smallest instance flavors.

@taavi Scratch that. I just confirmed we don't need the floating IP. Thanks.

Hey @Kormat @jbond I was wondering if this was resolved. I noticed the file now references some vrts passwords.

# less modules/profile/templates/mariadb/grants/production-m2.sql.erb  | grep "vrts"
IDENTIFIED BY '<%= @vrts_exim_database_pass %>'
IDENTIFIED BY '<%= @vrts_database_pw %>'
IDENTIFIED BY '<%= @vrts_exim_database_pass %>'
IDENTIFIED BY '<%= @vrts_database_pw %>'
IDENTIFIED BY '<%= @vrts_exim_database_pass %>'
IDENTIFIED BY '<%= @vrts_database_pw %>'

https://gerrit.wikimedia.org/r/c/operations/puppet/+/820563 @LSobanski This was the fix for the SSL issue. The record gitlab-replica-old had been used before for this purpose but it was removed when testing was complete.

Thanks to @Dzahn the SSL issue is now resolved.

@Jelto I managed to switch over the hosts i.e. gitlab1003 and gitlab2002. The current replica is now gitlab2002. Though the old replica is currently experiencing some SSL issues and I'm not entirely sure why. Perhaps I missed a configuration step but didn't do much on the old host other than disabling and re-enabling puppet.