- User Since
- Aug 10 2021, 2:29 PM (84 w, 6 d)
- LDAP User
- MediaWiki User
- AOkoth (WMF) [ Global Accounts ]
@Jelto I tested this on gitlab1004.
Tue, Mar 21
Mon, Mar 13
Wed, Mar 1
Feb 14 2023
@Jelto Thanks for the heads up. I had down timed alerts on this host since it's still a work in progress. I think the downtime expired today but I extended it.
Jan 31 2023
I have some concerns because the docs state The chosen previous backup is overwritten.. So we might see quite a lot of IO here for extracting the tar archive and writing it again. But tests will show that.
Jan 23 2023
Dec 19 2022
Dec 15 2022
Thanks @Marostegui I can confirm that this works on otrs1001. I trust that it will work on the other host but I have to install the mysql-client. I will test on vrts2001 after making the necessary Puppet changes.
Dec 14 2022
Actually, just scratch everything I've said so far... We might not need a new database user. Could we just have the current user otrs be able to connnect to m2-slave in both data centers?
aokoth@otrs1001:~$ mysql -h m2-slave.eqiad.wmnet -u otrs -p otrs -P 3323 ERROR 1045 (28000): Access denied for user 'otrs'@'10.64.16.39' (using password: YES)
Hi @Marostegui We did some testing and the outcome was we found we could connect to the m2-master on both hosts but can't connect to the m2-slave on either server. Despite the fact that the new user has read-only grants, we would prefer that the non-active host connects to the secondary database server as well.
Dec 9 2022
@Marostegui Yeah, that's okay. I'd like the username vrts.
Dec 8 2022
Dec 7 2022
@Marostegui Yeah, that's exactly what I need but not on the primary database host. We are setting up a new failover VM for VRTS in codfw (vrts2001). The current VRTS instance (otrs1001) connects to the m2-master host in eqiad. So we need a new user that can connect to m2-slave in codfw and said user should only have SELECT grants (I'm not sure if this is implied by the fact that it connects to the secondary host).
Dec 5 2022
As an idea, I was combing through the docs (https://docs.gitlab.com/ee/raketasks/backup_gitlab.html#incremental-repository-backups) and noticed Gitlab supports incremental backups:
@Marostegui Yeah, that would be okay.
Nov 30 2022
Ready to create Ganeti VM vrts2001.codfw.wmnet in the codfw cluster on group C with 4 vCPUs, 8GB of RAM, 25GB of disk in the private network.
Nov 29 2022
Nov 21 2022
Nov 15 2022
Oct 24 2022
Hey @jbond Seems like the merge broke Puppet on otrs1001.eqiad.wmnet. It fails with the following error:
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Failed to parse template vrts/exim4.conf.vrts.erb: Filepath: /etc/puppet/modules/vrts/templates/exim4.conf.vrts.erb Line: 22 Detail: undefined method `unwrap' for "<redacted>":String (file: /etc/puppet/modules/vrts/manifests/mail.pp, line: 41, column: 20) on node otrs1001.eqiad.wmnet Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run
Oct 7 2022
Hey @Lucas_Werkmeister_WMDE Yeah, I was actually debating whether to remove that checkbox but I'll just leave it unchecked since it's not needed in this case. It's resolved now. Feel free to close the ticket.
Oct 6 2022
Hey @AnnWF Kindly sign this https://phabricator.wikimedia.org/L3
I have added to the wmf-nda group as well. Thanks @Aklapper
Ooh, sorry I missed that step. I have added you to the wmf-nda group as well. Thanks @Aklapper
Oct 5 2022
Hey @Ottomata @karapayneWMDE Kindly approve.
@greg This is resolved. Feel free to close the ticket if everything is good on your side.
@Slst2020 This is resolved. Feel free to close the ticket if everything is good on your end.
aokoth@mwmaint1002:~$ ldapsearch -x cn=wmf | grep "kindrobot" member: uid=kindrobot,ou=people,dc=wikimedia,dc=org
aokoth@mwmaint1002:~$ ldapsearch -x cn=wmf | grep "mhorsey" member: uid=mhorsey,ou=people,dc=wikimedia,dc=org
Hey @greg Yeah, Lisa G is your manager (confirmed on Namely). So will need approval from her (@Lgruwell-WMF ) as well as @Ottomata or @odimitrijevic
Oct 4 2022
Hey, @Ottomata You are listed as one of the approvers for members joining this list. Kindly approve.
Sep 20 2022
Sep 5 2022
Sep 4 2022
Sep 1 2022
Sep 01 20:19:06 otrs1001 systemd: spamassassin_updates.service: Succeeded
Hey @Dzahn Wasn't this patch reverted?
Aug 25 2022
Hi @rook Sorry. I hadn't noticed that the number of instances should also be increased as part of the quota. Given the resources that were added, I think we need to increase that by two since it fits the two smallest instance flavors.
Aug 24 2022
@taavi Scratch that. I just confirmed we don't need the floating IP. Thanks.
Aug 19 2022
Hey @Kormat @jbond I was wondering if this was resolved. I noticed the file now references some vrts passwords.
# less modules/profile/templates/mariadb/grants/production-m2.sql.erb | grep "vrts" IDENTIFIED BY '<%= @vrts_exim_database_pass %>' IDENTIFIED BY '<%= @vrts_database_pw %>' IDENTIFIED BY '<%= @vrts_exim_database_pass %>' IDENTIFIED BY '<%= @vrts_database_pw %>' IDENTIFIED BY '<%= @vrts_exim_database_pass %>' IDENTIFIED BY '<%= @vrts_database_pw %>'
Aug 8 2022
https://gerrit.wikimedia.org/r/c/operations/puppet/+/820563 @LSobanski This was the fix for the SSL issue. The record gitlab-replica-old had been used before for this purpose but it was removed when testing was complete.
Aug 4 2022
Thanks to @Dzahn the SSL issue is now resolved.
@Jelto I managed to switch over the hosts i.e. gitlab1003 and gitlab2002. The current replica is now gitlab2002. Though the old replica is currently experiencing some SSL issues and I'm not entirely sure why. Perhaps I missed a configuration step but didn't do much on the old host other than disabling and re-enabling puppet.
# host gitlab-replica-old.wikimedia.org gitlab-replica-old.wikimedia.org has address 220.127.116.11 gitlab-replica-old.wikimedia.org has IPv6 address 2620:0:861:1:208:80:154:15 # host gitlab-replica.wikimedia.org gitlab-replica.wikimedia.org has address 18.104.22.168 gitlab-replica.wikimedia.org has IPv6 address 2620:0:860:1:208:80:153:8