Page MenuHomePhabricator

jcrespo (Jaime Crespo)
Sr Database Administrator

Projects (12)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
May 11 2015, 8:31 AM (462 w, 21 h)
Availability
Available
IRC Nick
jynus
LDAP User
Jcrespo
MediaWiki User
JCrespo (WMF) [ Global Accounts ]

Recent Activity

Wed, Mar 6

jcrespo added a comment to T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).

Meanwhile, I am going to merge and deploy https://gitlab.wikimedia.org/repos/sre/mediabackups/-/merge_requests/1 in any case (workaround for media backups), as backups should be compatible an be able to recover files with illegal names, independently of being a mw bug, as I mention in the review.

Wed, Mar 6, 12:07 PM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
jcrespo added a comment to T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).

Thanks a lot, @Bawolff, this is the kind of take I really needed- as my initial reaction was to just make the field larger.

Wed, Mar 6, 11:56 AM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups

Tue, Mar 5

jcrespo added a comment to T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).

Looking good, backups unblocked:

Tue, Mar 5, 4:25 PM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
jcrespo added a comment to T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).

1SELECT *
2FROM oldimage
3WHERE oi_name = 'ДАЖО_127-1-68.1897._Геодезичний_опис_ділянки_землі_вічного_чиншовика_Антона_Станіслава_Гарбовських_села_Рудня-Старики_Овруцького_повіту.pdf'
4AND oi_archive_name like '20231203130229!ДАЖО_127-1-68.1897._Геодезичний_опис_ділянки_землі_вічного_чиншовика_Антона_Станіслава_Гарбовських_села_Рудня-Старики_Овруцького_повіт%'
5ORDER BY oi_name
6LIMIT 1;
7
8SELECT *
9FROM oldimage
10WHERE oi_name = 'Алфавітно-предметний_покажчик_за_1938_рік_до_Збірника_постанов_і_розпоряджень_Уряду_Української_Радянської_Соціалістичної_Республіки.pdf'
11AND oi_archive_name like '20240116211741!Алфавітно-предметний_покажчик_за_1938_рік_до_Збірника_постанов_і_розпоряджень_Уряду_Української_Радянської_Соціалістичної_Республ%'
12ORDER BY oi_name
13LIMIT 1;
14
15UPDATE oldimage
16SET oi_archive_name = '20231203130229!ДАЖО_127-1-68.1897._Геодезичний_опис_ділянки_землі_вічного_чиншовика_Антона_Станіслава_Гарбовських_села_Рудня-Старики_Овруцького_повіт'
17WHERE oi_name = 'ДАЖО_127-1-68.1897._Геодезичний_опис_ділянки_землі_вічного_чиншовика_Антона_Станіслава_Гарбовських_села_Рудня-Старики_Овруцького_повіту.pdf'
18AND oi_archive_name like '20231203130229!ДАЖО_127-1-68.1897._Геодезичний_опис_ділянки_землі_вічного_чиншовика_Антона_Станіслава_Гарбовських_села_Рудня-Старики_Овруцького_повіт%'
19ORDER BY oi_name
20LIMIT 1;
21
22UPDATE oldimage
23SET oi_archive_name = '20240116211741!Алфавітно-предметний_покажчик_за_1938_рік_до_Збірника_постанов_і_розпоряджень_Уряду_Української_Радянської_Соціалістичної_Республ'
24WHERE oi_name = 'Алфавітно-предметний_покажчик_за_1938_рік_до_Збірника_постанов_і_розпоряджень_Уряду_Української_Радянської_Соціалістичної_Республіки.pdf'
25AND oi_archive_name like '20240116211741!Алфавітно-предметний_покажчик_за_1938_рік_до_Збірника_постанов_і_розпоряджень_Уряду_Української_Радянської_Соціалістичної_Республ%'
26ORDER BY oi_name
27LIMIT 1;
28
29select count(*) FROM oldimage WHERE NOT (oi_archive_name = CONVERT(oi_archive_name USING utf8mb4)); -- should return 0
30

Tue, Mar 5, 4:13 PM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
jcrespo edited P58491 Database fix plan To run on s4 primary (db2179 with replication enabled).
Tue, Mar 5, 4:12 PM
jcrespo created P58491 Database fix plan To run on s4 primary (db2179 with replication enabled).
Tue, Mar 5, 4:10 PM
jcrespo added a comment to T358417: Inbound interface errors.

I didn't ask for a cable change, and so far I haven't observed any problem with the host, TBH, it was @ayounsi who requested it, but I wonder if the metrics are too sensitive- we do the backup as fast as possible, and we trust on TCP (well, and the checksum we do later, to be fair) to fix potential errors when doing the regular backups.

Tue, Mar 5, 3:51 PM · SRE, ops-codfw
jcrespo added a comment to T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).

My suggestion would be, for a short-ish term, to make the field longer (the archival table - oldimage- will be very small) and fill-in the data from the original name, when possible. Although I haven't checked the deleted one, it may suffer from the same issue with deleted, archived files.

Tue, Mar 5, 3:35 PM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
jcrespo updated subscribers of T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).

Based on the above, I would like to mostly unblock media backups by fixing the UTF8 invalid characters first on production. For that, It would like to just remove the invalid characters on the first 2 files, and then we can maybe discuss how to move forward for the main bug (code fixes, a schema change on oldimage to make oi_archive_name larger?).

Tue, Mar 5, 3:24 PM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
jcrespo added a comment to T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).

After investigating, it seems the issue goes beyond invalid UTF-8 characters, there is metadata loss due to the pseudo-path stored on the database being silently truncated. When the path is calcualted, the date + an admiration sign is added as prefix, leading to the full path being cut at the end, sometimes, if cut on multi-byte characters, to invalid data (but on any case truncated).

Tue, Mar 5, 3:15 PM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
jcrespo renamed T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata) from Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orfan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata) to Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).
Tue, Mar 5, 3:09 PM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
jcrespo renamed T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata) from Two files on commons have invalid UTF-8 characters in path metadata to Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orfan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).
Tue, Mar 5, 3:09 PM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
jcrespo created P58482 Commons truncated metadata on archival.
Tue, Mar 5, 3:04 PM
jcrespo added a comment to T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).

The theoretical container and path of these 2 files should be, in theory:

Tue, Mar 5, 2:21 PM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
jcrespo created T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).
Tue, Mar 5, 2:08 PM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
jcrespo updated the task description for T289996: Media storage metadata inconsistent with Swift or corrupted in general.
Tue, Mar 5, 2:05 PM · Commons, MediaWiki-File-management, media-backups, SRE-swift-storage
jcrespo added projects to T289996: Media storage metadata inconsistent with Swift or corrupted in general: media-backups, MediaWiki-File-management.
Tue, Mar 5, 2:03 PM · Commons, MediaWiki-File-management, media-backups, SRE-swift-storage
jcrespo added a comment to T358417: Inbound interface errors.

Replacing the cable can be done any time between 6:00 and 23:55 UTC. Let me know if it will be for a period of extended time so I can downtime it.

Tue, Mar 5, 11:31 AM · SRE, ops-codfw

Fri, Mar 1

jcrespo added a comment to T355353: Q3:rack/setup/install dbprov100[56].

@jcrespo are the Raid instructions backwards os is usually on ssd's RAID 0?

Fri, Mar 1, 6:03 PM · SRE, Data-Persistence, ops-eqiad, DC-Ops
jcrespo added a comment to T355353: Q3:rack/setup/install dbprov100[56].

Sure, let me know what can I do for you, and I will get it done next week. The only custom stuff compared to other hosts is the HW RAID partitioning, the rest is a rather standard db-like install. If it is only adding it to netconf and the default role (staging), it shouldn't take me long, but I will get it done on Monday, as I have finished my work week already.

Fri, Mar 1, 6:00 PM · SRE, Data-Persistence, ops-eqiad, DC-Ops

Thu, Feb 29

jcrespo added a comment to P58241 Slides for data persistence.

SRE Data Persistence 2024 State of the Union - Jaime's version(1).png (540×960 px, 61 KB)

Thu, Feb 29, 2:30 PM
jcrespo added a comment to P58241 Slides for data persistence.

SRE Data Persistence 2024 State of the Union - Jaime's version.png (540×960 px, 56 KB)

Thu, Feb 29, 12:23 PM
jcrespo created P58241 Slides for data persistence.
Thu, Feb 29, 12:22 PM

Wed, Feb 28

jcrespo added a comment to T358642: Upgrade x1 to MariaDB 10.6.

As I prepared beforehand for a previous upgrade, s6, x1 and s2 are already producing 10.6-compatible backups, and backup sources should be, at least partially, upgraded- we can just drop the 10.4 ones when fully upgraded and move 10.4 sections instead (I will handle that). So I should not be a blocker for this ticket. If you have a roadmap of future upgrades beyond those sections, I can start working on preparing those now, so I am ready already like in this case.

Wed, Feb 28, 9:03 AM · DBA

Thu, Feb 22

jcrespo closed T269108: Create a read-only swift identity for backup taking as Resolved.

It took some time to confirm it live, because the number of new deleted files don't grow as fast as the "latest" versions, and the algorithm starts backing up the non-deleted ones, but I can confirm the new deleted files backups count is growing up with the new user:

Thu, Feb 22, 5:34 PM · Data-Persistence-Backup, SRE, SRE-swift-storage
jcrespo closed T269108: Create a read-only swift identity for backup taking, a subtask of T327157: Create and deploy the logic to generate incremental backups of MediaWiki media files, to keep its file storage backup up to date, automatically, as Resolved.
Thu, Feb 22, 5:34 PM · Data-Persistence, media-backups, Data-Persistence-Backup, Goal
jcrespo added a comment to T269108: Create a read-only swift identity for backup taking.

Thank you a lot, as I mentioned in private, I will try to run the automatic downloads back again with the new user, if it works we will be very close to resolve this. Thank you again!

Thu, Feb 22, 1:48 PM · Data-Persistence-Backup, SRE, SRE-swift-storage
jcrespo placed T269108: Create a read-only swift identity for backup taking up for grabs.
Thu, Feb 22, 11:28 AM · Data-Persistence-Backup, SRE, SRE-swift-storage

Wed, Feb 21

jcrespo closed T357878: db2097 rebooted itself as Resolved.

Thanks for the reply, the reboot happened on the 17, so no relation to that.

Wed, Feb 21, 2:28 PM · SRE, ops-codfw, DC-Ops, Data-Persistence-Backup
jcrespo added a comment to T316655: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals.

an-db backups looking good:

✔️ root@backup1001:~$ check_bacula.py an-db1001.eqiad.wmnet-Daily-productionEqiad-data-engineering-postgres
id: 552007, ts: 2024-02-08 04:05:00, type: F, status: T, bytes: 643284416
id: 552157, ts: 2024-02-09 04:05:01, type: F, status: T, bytes: 645145616
id: 552309, ts: 2024-02-10 04:05:00, type: F, status: T, bytes: 646974416
id: 552458, ts: 2024-02-11 04:05:00, type: F, status: T, bytes: 648762976
id: 552609, ts: 2024-02-12 04:05:00, type: F, status: T, bytes: 650570592
id: 552760, ts: 2024-02-13 04:05:00, type: F, status: T, bytes: 652389680
id: 552918, ts: 2024-02-14 04:05:01, type: F, status: T, bytes: 654207808
id: 553088, ts: 2024-02-15 04:05:00, type: F, status: T, bytes: 655958880
id: 553260, ts: 2024-02-16 04:05:01, type: F, status: T, bytes: 657687120
id: 553428, ts: 2024-02-17 04:05:00, type: F, status: T, bytes: 659326944
id: 553597, ts: 2024-02-18 04:05:00, type: F, status: T, bytes: 660873424
id: 553760, ts: 2024-02-19 04:05:00, type: F, status: T, bytes: 662617920
id: 553924, ts: 2024-02-20 04:05:00, type: F, status: T, bytes: 665207840
id: 554109, ts: 2024-02-21 04:05:01, type: F, status: T, bytes: 630091008
✔️ root@backup1001:~$ check_bacula.py an-db1002.eqiad.wmnet-Daily-productionEqiad-data-engineering-postgres
id: 552008, ts: 2024-02-08 04:05:00, type: F, status: T, bytes: 182492896
id: 552158, ts: 2024-02-09 04:05:01, type: F, status: T, bytes: 193882176
id: 552310, ts: 2024-02-10 04:05:01, type: F, status: T, bytes: 178497152
id: 552459, ts: 2024-02-11 04:05:00, type: F, status: T, bytes: 171867232
id: 552610, ts: 2024-02-12 04:05:00, type: F, status: T, bytes: 182933552
id: 552761, ts: 2024-02-13 04:05:00, type: F, status: T, bytes: 177721968
id: 552919, ts: 2024-02-14 04:05:01, type: F, status: T, bytes: 178077264
id: 553089, ts: 2024-02-15 04:05:08, type: F, status: T, bytes: 193632432
id: 553261, ts: 2024-02-16 04:05:01, type: F, status: T, bytes: 179027088
id: 553429, ts: 2024-02-17 04:05:00, type: F, status: T, bytes: 179340992
id: 553598, ts: 2024-02-18 04:05:00, type: F, status: T, bytes: 189788240
id: 553761, ts: 2024-02-19 04:05:00, type: F, status: T, bytes: 179704320
id: 553925, ts: 2024-02-20 04:05:08, type: F, status: T, bytes: 180750272
id: 554110, ts: 2024-02-21 04:05:02, type: F, status: T, bytes: 180212720
Wed, Feb 21, 7:44 AM · Data-Engineering, Infrastructure-Foundations, bacula, Data-Persistence-Backup, netbox

Tue, Feb 20

jcrespo claimed T357878: db2097 rebooted itself.

Papaul is on vacations, so it couldn't be him. :-(

Tue, Feb 20, 11:08 AM · SRE, ops-codfw, DC-Ops, Data-Persistence-Backup
jcrespo triaged T357878: db2097 rebooted itself as High priority.
Tue, Feb 20, 11:08 AM · SRE, ops-codfw, DC-Ops, Data-Persistence-Backup

Mon, Feb 19

jcrespo assigned T357878: db2097 rebooted itself to Papaul.

Hey, @Papaul I saw a login at 02/17/2024 05:24:45, barely a minute after a flash reset. Maybe some dc op saw something else when logging in, or there was some ongoing maintenance on that host or a close one?

Mon, Feb 19, 11:41 AM · SRE, ops-codfw, DC-Ops, Data-Persistence-Backup
jcrespo added a comment to T357878: db2097 rebooted itself.
1736	
Server power restored.	02/17/2024 05:23:51	1	Maintenance, Administration
1735	
	Server reset.	02/17/2024 05:23:51	1	Maintenance, Administration
1734	
Server power removed.	02/17/2024 05:23:46	1	Maintenance, Administration
1733	
Power on request received by: Automatic Power Recovery.	02/17/2024 05:23:45	1	Maintenance, Administration
1732	
Server power restored.	02/17/2024 05:23:27	1	Maintenance, Administration
1731	
	Server reset.	02/17/2024 05:23:27	1	Maintenance, Administration
1730	
Server power removed.	02/17/2024 05:23:26	1	Maintenance, Administration
1729	
Embedded Flash: Restarted	02/17/2024 05:23:18	1	Firmware
Mon, Feb 19, 11:25 AM · SRE, ops-codfw, DC-Ops, Data-Persistence-Backup
jcrespo added a comment to T357878: db2097 rebooted itself.

I will take it from here- at the very least, recover it from backups.

Mon, Feb 19, 10:00 AM · SRE, ops-codfw, DC-Ops, Data-Persistence-Backup

Feb 14 2024

jcrespo added a comment to T355980: [vrts] Move attachment storage out of mysql.

After performing the article migration on the test instance, we have some doubts as to whether this is going to be the best solution long-term for the following reasons:

Feb 14 2024, 1:26 PM · Znuny, collaboration-services

Feb 13 2024

jcrespo awarded T357291: mr1-codfw down a Yellow Medal token.
Feb 13 2024, 4:49 PM · SRE, ops-codfw
jcrespo added a comment to T357291: mr1-codfw down.

This is now fixed, right? Or do you prefer to keep it up for followup?

Feb 13 2024, 10:38 AM · SRE, ops-codfw

Feb 12 2024

jcrespo added a comment to T350458: Decommission db11[26-49].

All backups looking good. Please proceed.

Feb 12 2024, 10:01 AM · DBA

Feb 7 2024

jcrespo added a comment to T350458: Decommission db11[26-49].

Technically no blocker for decom of db1139, db1140 or db1145, although I would wait 1 day to make sure backups run correctly on the new hosts at least once.

Feb 7 2024, 4:52 PM · DBA
jcrespo added a project to T344036: Productionize db12[26-49]: database-backups.
Feb 7 2024, 4:51 PM · database-backups, DBA
jcrespo added a comment to T344036: Productionize db12[26-49].

New servers are in use, the old backup sources are idle (no service uses them), but still replicating/available. All work on my side is done, unless for some reason the backups on the new host start to fail.

Feb 7 2024, 4:50 PM · database-backups, DBA
jcrespo updated the task description for T344036: Productionize db12[26-49].
Feb 7 2024, 4:49 PM · database-backups, DBA
jcrespo awarded T316655: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals a Party Time token.
Feb 7 2024, 12:57 PM · Data-Engineering, Infrastructure-Foundations, bacula, Data-Persistence-Backup, netbox
jcrespo added a comment to T316655: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals.

This is looking good- I will check in the following days that an db backups are being produced (the data copy for Bacula, the dump correctness at the moment should be the service owner responsability 0:-) ).

Feb 7 2024, 12:02 PM · Data-Engineering, Infrastructure-Foundations, bacula, Data-Persistence-Backup, netbox

Feb 6 2024

jcrespo added a comment to T356788: thanos-query probedown due to OOM of both eqiad titan frontends.

Potentially a similar issue (request/traffic related high load) happened around the past 28 of September, when I added this TODO: https://wikitech.wikimedia.org/w/index.php?title=Thanos&diff=prev&oldid=2115837 I was confused back then even what this service was.

Feb 6 2024, 4:43 PM · SRE Observability (FY2023/2024-Q3), Sustainability (Incident Followup), SRE, observability
jcrespo awarded T355530: Cannot delete files with non-standard extensions on MySQL in a strict mode a Yellow Medal token.
Feb 6 2024, 3:52 PM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), MW-1.41-notes, MW-1.39-notes, MW-1.40-notes, MediaWiki-Uploading

Feb 5 2024

jcrespo updated subscribers of P55975 (An Untitled Masterwork).
✔️ root@backup1001:~$ check_bacula.py netboxdb1002.eqiad.wmnet-Daily-productionEqiad-netbox-postgres
id: 550937, ts: 2024-02-01 07:28:36, type: F, status: T, bytes: 46654944
id: 551106, ts: 2024-02-02 04:52:07, type: F, status: T, bytes: 46611456
id: 551272, ts: 2024-02-03 06:04:44, type: F, status: T, bytes: 46521712
id: 551439, ts: 2024-02-04 10:32:37, type: F, status: T, bytes: 46399072
id: 551600, ts: 2024-02-05 05:36:16, type: F, status: T, bytes: 46267136
✔️ root@backup1001:~$ check_bacula.py netboxdb2002.codfw.wmnet-Daily-productionEqiad-netbox-postgres
id: 550938, ts: 2024-02-01 07:28:39, type: F, status: T, bytes: 46176368
id: 551107, ts: 2024-02-02 04:52:09, type: F, status: T, bytes: 46081440
id: 551273, ts: 2024-02-03 06:04:47, type: F, status: T, bytes: 46001888
id: 551440, ts: 2024-02-04 10:32:41, type: F, status: T, bytes: 45952240
id: 551601, ts: 2024-02-05 05:36:18, type: F, status: T, bytes: 45729200

@Volans 👍

Feb 5 2024, 9:53 AM

Feb 2 2024

jcrespo added a comment to T344036: Productionize db12[26-49].

FYI: I've recovered from backups db1239 (s1, s2), db1240 (s1) and db1245 (s4, s5)

Feb 2 2024, 6:18 PM · database-backups, DBA
jcrespo renamed T334069: Evaluate and decide the future of MinIO for media backups given the upgrade requirements and increase the available storage space from Evaluate and decide the future of MinIO for media backups given the upgrade requirements to Evaluate and decide the future of MinIO for media backups given the upgrade requirements and increase the available storage space.
Feb 2 2024, 10:29 AM · Patch-For-Review, Data-Persistence-Backup, media-backups
jcrespo added a comment to P55975 (An Untitled Masterwork).
root@backup1001:~$ check_bacula.py netboxdb2002.codfw.wmnet-Daily-productionEqiad-netbox-postgres
id: 550938, ts: 2024-02-01 07:28:39, type: F, status: T, bytes: 46176368
id: 551107, ts: 2024-02-02 04:52:09, type: F, status: T, bytes: 46081440
✔️ root@backup1001:~$ check_bacula.py netboxdb1002.eqiad.wmnet-Daily-productionEqiad-netbox-postgres
id: 550937, ts: 2024-02-01 07:28:36, type: F, status: T, bytes: 46654944
id: 551106, ts: 2024-02-02 04:52:07, type: F, status: T, bytes: 46611456
✔️
Feb 2 2024, 9:00 AM

Feb 1 2024

jcrespo added a comment to P55975 (An Untitled Masterwork).
*list jobname=netboxdb1002.eqiad.wmnet-Monthly-1st-Wed-productionEqiad-netbox-postgres
+---------+--------------------------------------------------------------------------+---------------------+------+-------+----------+-------------+-----------+
| JobId   | Name                                                                     | StartTime           | Type | Level | JobFiles | JobBytes    | JobStatus |
+---------+--------------------------------------------------------------------------+---------------------+------+-------+----------+-------------+-----------+
| 536,832 | netboxdb1002.eqiad.wmnet-Monthly-1st-Wed-productionEqiad-netbox-postgres | 2023-11-04 06:28:49 | B    | I     |        2 |  48,881,440 | T         |
Feb 1 2024, 10:14 AM
jcrespo added a comment to P55975 (An Untitled Masterwork).
root@backup1001:~$ check_bacula.py netboxdb1002.eqiad.wmnet-Daily-productionEqiad-netbox-postgres
id: 550937, ts: 2024-02-01 07:28:36, type: F, status: T, bytes: 46654944
✔️ root@backup1001:~$ check_bacula.py netboxdb2002.codfw.wmnet-Daily-productionEqiad-netbox-postgres
id: 550938, ts: 2024-02-01 07:28:39, type: F, status: T, bytes: 46176368
✔️
Feb 1 2024, 9:58 AM

Jan 31 2024

jcrespo added a comment to P55975 (An Untitled Masterwork).
root@backup1001:~$ check_bacula.py netboxdb1002.eqiad.wmnet-Monthly-1st-Wed-productionEqiad-netbox-postgres
id: 536667, ts: 2023-11-03 05:16:05, type: I, status: T, bytes: 48667264
id: 536832, ts: 2023-11-04 06:28:49, type: I, status: T, bytes: 48881440
id: 536997, ts: 2023-11-05 10:28:53, type: I, status: T, bytes: 49098288
id: 537157, ts: 2023-11-06 05:04:41, type: I, status: T, bytes: 49285552
id: 537316, ts: 2023-11-07 06:24:22, type: I, status: T, bytes: 49513840
id: 537472, ts: 2023-11-08 04:56:28, type: I, status: T, bytes: 49746272
id: 537621, ts: 2023-11-09 04:56:18, type: I, status: T, bytes: 49892240
id: 537769, ts: 2023-11-10 04:53:37, type: I, status: T, bytes: 50039280
id: 537916, ts: 2023-11-11 04:44:50, type: I, status: T, bytes: 50245888
id: 538063, ts: 2023-11-12 04:33:23, type: I, status: T, bytes: 50376480
id: 538212, ts: 2023-11-13 04:53:46, type: I, status: T, bytes: 50548784
id: 538361, ts: 2023-11-14 05:11:29, type: I, status: T, bytes: 50676496
id: 538442, ts: 2023-11-15 03:07:18, type: D, status: T, bytes: 451827680
id: 538544, ts: 2023-11-15 05:00:24, type: I, status: T, bytes: 0
id: 538711, ts: 2023-11-16 05:33:09, type: I, status: T, bytes: 50964816
id: 538878, ts: 2023-11-17 05:06:32, type: I, status: T, bytes: 51023920
id: 539043, ts: 2023-11-18 04:43:40, type: I, status: T, bytes: 50837616
id: 539208, ts: 2023-11-19 04:50:40, type: I, status: T, bytes: 50675840
id: 539368, ts: 2023-11-20 05:05:16, type: I, status: T, bytes: 50499072
id: 539527, ts: 2023-11-21 19:22:14, type: I, status: T, bytes: 50283632
id: 539685, ts: 2023-11-22 04:52:01, type: I, status: T, bytes: 50152992
id: 539834, ts: 2023-11-23 05:01:30, type: I, status: T, bytes: 49899936
id: 539982, ts: 2023-11-24 05:00:31, type: I, status: T, bytes: 49704800
id: 540129, ts: 2023-11-25 04:41:16, type: I, status: T, bytes: 49533520
id: 540276, ts: 2023-11-26 04:34:46, type: I, status: T, bytes: 49303216
id: 540425, ts: 2023-11-27 04:40:35, type: I, status: T, bytes: 49105536
id: 540574, ts: 2023-11-28 05:29:51, type: I, status: T, bytes: 48861232
id: 540730, ts: 2023-11-29 05:02:19, type: I, status: T, bytes: 48646816
id: 540879, ts: 2023-11-30 05:35:15, type: I, status: T, bytes: 96952608
id: 541046, ts: 2023-12-01 05:01:20, type: I, status: T, bytes: 48688640
id: 541211, ts: 2023-12-02 06:32:13, type: I, status: T, bytes: 48747136
id: 541376, ts: 2023-12-03 10:29:26, type: I, status: T, bytes: 48825792
id: 541536, ts: 2023-12-04 05:05:02, type: I, status: T, bytes: 48805952
id: 541695, ts: 2023-12-05 05:33:17, type: I, status: T, bytes: 48851168
id: 541774, ts: 2023-12-06 02:33:03, type: F, status: T, bytes: 694262768
id: 541877, ts: 2023-12-06 05:20:17, type: I, status: T, bytes: 0
id: 542044, ts: 2023-12-07 06:04:33, type: I, status: T, bytes: 48907520
id: 542192, ts: 2023-12-08 05:00:52, type: I, status: T, bytes: 48854368
id: 542339, ts: 2023-12-09 04:51:26, type: I, status: T, bytes: 48832448
id: 542486, ts: 2023-12-10 04:35:31, type: I, status: T, bytes: 48815328
id: 542635, ts: 2023-12-11 04:41:30, type: I, status: T, bytes: 48777712
id: 542784, ts: 2023-12-12 05:31:34, type: I, status: T, bytes: 48760592
id: 542940, ts: 2023-12-13 04:57:14, type: I, status: T, bytes: 48745440
id: 543089, ts: 2023-12-14 05:09:54, type: I, status: T, bytes: 48734448
id: 543259, ts: 2023-12-15 05:02:15, type: I, status: T, bytes: 48702464
id: 543427, ts: 2023-12-16 04:43:29, type: I, status: T, bytes: 48719600
id: 543595, ts: 2023-12-17 04:41:44, type: I, status: T, bytes: 48715232
id: 543758, ts: 2023-12-18 04:36:42, type: I, status: T, bytes: 48735360
id: 543920, ts: 2023-12-19 05:41:49, type: I, status: T, bytes: 48729312
id: 544000, ts: 2023-12-20 03:06:46, type: D, status: T, bytes: 438541808
id: 544105, ts: 2023-12-20 04:56:18, type: I, status: T, bytes: 0
id: 544278, ts: 2023-12-21 05:07:06, type: I, status: T, bytes: 48680736
id: 544429, ts: 2023-12-22 05:20:10, type: I, status: T, bytes: 48595168
id: 544579, ts: 2023-12-23 04:46:14, type: I, status: T, bytes: 48588896
id: 544729, ts: 2023-12-24 04:46:17, type: I, status: T, bytes: 48580992
id: 544881, ts: 2023-12-25 04:53:23, type: I, status: T, bytes: 48585152
id: 545033, ts: 2023-12-26 05:05:59, type: I, status: T, bytes: 48580576
id: 545192, ts: 2023-12-27 05:01:51, type: I, status: T, bytes: 48506448
id: 545344, ts: 2023-12-28 05:03:59, type: I, status: T, bytes: 48447776
id: 545495, ts: 2023-12-29 04:57:01, type: I, status: T, bytes: 48403152
id: 545645, ts: 2023-12-30 04:47:16, type: I, status: T, bytes: 48348208
id: 545795, ts: 2023-12-31 04:43:20, type: I, status: T, bytes: 48292912
id: 545958, ts: 2024-01-01 05:12:57, type: I, status: T, bytes: 48230352
id: 546120, ts: 2024-01-02 06:02:31, type: I, status: T, bytes: 48210128
id: 546199, ts: 2024-01-03 02:33:04, type: F, status: T, bytes: 690480544
id: 546305, ts: 2024-01-03 05:33:10, type: I, status: T, bytes: 0
id: 546478, ts: 2024-01-04 06:52:32, type: I, status: T, bytes: 48078048
id: 546648, ts: 2024-01-05 04:56:52, type: I, status: T, bytes: 48050336
id: 546816, ts: 2024-01-06 06:55:14, type: I, status: T, bytes: 48057360
id: 546984, ts: 2024-01-07 05:14:48, type: I, status: T, bytes: 48033424
id: 547136, ts: 2024-01-08 04:42:14, type: I, status: T, bytes: 47991664
id: 547288, ts: 2024-01-09 05:19:08, type: I, status: T, bytes: 47922752
id: 547447, ts: 2024-01-10 05:37:54, type: I, status: T, bytes: 47946816
id: 547599, ts: 2024-01-11 05:03:43, type: I, status: T, bytes: 47887856
id: 547750, ts: 2024-01-12 05:10:28, type: I, status: T, bytes: 47768208
id: 547900, ts: 2024-01-13 04:56:52, type: I, status: T, bytes: 47757424
id: 548050, ts: 2024-01-14 04:36:45, type: I, status: T, bytes: 47759536
id: 548213, ts: 2024-01-15 04:57:04, type: I, status: T, bytes: 47726096
id: 548376, ts: 2024-01-16 05:15:40, type: I, status: T, bytes: 47670944
id: 548456, ts: 2024-01-17 03:06:32, type: D, status: T, bytes: 430070016
id: 548561, ts: 2024-01-17 05:28:45, type: I, status: T, bytes: 0
id: 548732, ts: 2024-01-18 05:44:31, type: I, status: T, bytes: 47560048
id: 548901, ts: 2024-01-19 04:54:21, type: I, status: T, bytes: 47488512
id: 549068, ts: 2024-01-20 04:56:54, type: I, status: T, bytes: 47406048
id: 549235, ts: 2024-01-21 05:27:55, type: I, status: T, bytes: 47379184
id: 549386, ts: 2024-01-22 04:35:03, type: I, status: T, bytes: 47290032
id: 549537, ts: 2024-01-23 05:15:24, type: I, status: T, bytes: 47152576
id: 549695, ts: 2024-01-24 05:08:11, type: I, status: T, bytes: 47223856
id: 549847, ts: 2024-01-25 05:06:56, type: I, status: T, bytes: 47114704
id: 549997, ts: 2024-01-26 04:52:36, type: I, status: T, bytes: 47071632
id: 550148, ts: 2024-01-27 04:46:20, type: I, status: T, bytes: 46995824
id: 550297, ts: 2024-01-28 04:35:50, type: I, status: T, bytes: 46947840
id: 550454, ts: 2024-01-29 04:48:36, type: I, status: T, bytes: 46849680
id: 550605, ts: 2024-01-30 05:33:27, type: I, status: T, bytes: 46745792
id: 550763, ts: 2024-01-31 05:14:48, type: I, status: T, bytes: 46700960
Jan 31 2024, 4:21 PM
jcrespo created P55975 (An Untitled Masterwork).
Jan 31 2024, 4:17 PM
jcrespo added a comment to T316655: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals.

Let me do a manual run first and then we can do the followups (I also need to deploy the change backup setting to default to full).

Jan 31 2024, 3:14 PM · Data-Engineering, Infrastructure-Foundations, bacula, Data-Persistence-Backup, netbox
jcrespo added a comment to T316655: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals.

Let me know when you have a new backup in the new format to test and deploy the bacula changes.

Jan 31 2024, 2:27 PM · Data-Engineering, Infrastructure-Foundations, bacula, Data-Persistence-Backup, netbox

Jan 29 2024

jcrespo added a comment to T355863: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw.

Thank you, I will shutdown media backups anyway every time one host is affected, not just this one, to minimize failures.

Jan 29 2024, 3:04 PM · SRE-swift-storage, DBA, ops-codfw, netops, Infrastructure-Foundations, SRE
jcrespo added a comment to T355884: Enable DB backups for ipoid.

Screenshot_20240129_152457.png (1×1 px, 148 KB)

Jan 29 2024, 2:26 PM · database-backups, Data-Persistence-Backup, iPoid-Service
jcrespo closed T355884: Enable DB backups for ipoid as Resolved.

Backup and recovery tested, grants deployed on both datacenters. This is done.

Jan 29 2024, 2:25 PM · database-backups, Data-Persistence-Backup, iPoid-Service
jcrespo added a comment to T355884: Enable DB backups for ipoid.

Data recovery is restricted to roots, it is not self-served. If you are curious how it works it documented at: https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Recovering_the_data_(Mariadb_10.6)

Jan 29 2024, 1:26 PM · database-backups, Data-Persistence-Backup, iPoid-Service

Jan 26 2024

jcrespo created P55738 List of backups from archiva1002.
Jan 26 2024, 1:29 PM

Jan 25 2024

jcrespo renamed T355884: Enable DB backups for ipoid from Consider whether we should enable DB backups for ipoid to Enable DB backups for ipoid.
Jan 25 2024, 1:10 PM · database-backups, Data-Persistence-Backup, iPoid-Service
jcrespo added a comment to T355884: Enable DB backups for ipoid.

We will keep them for 3 months, the (actually small, in comparison) size doesn't justify treating them differently than the other backups. :-)

Jan 25 2024, 1:10 PM · database-backups, Data-Persistence-Backup, iPoid-Service
jcrespo added a comment to T355884: Enable DB backups for ipoid.

Proof:

proof.png (1×2 px, 151 KB)

Jan 25 2024, 1:08 PM · database-backups, Data-Persistence-Backup, iPoid-Service
jcrespo added a project to T355884: Enable DB backups for ipoid: database-backups.
Jan 25 2024, 1:05 PM · database-backups, Data-Persistence-Backup, iPoid-Service
jcrespo triaged T355884: Enable DB backups for ipoid as High priority.
Jan 25 2024, 1:04 PM · database-backups, Data-Persistence-Backup, iPoid-Service
jcrespo claimed T355884: Enable DB backups for ipoid.

This was already in motion before being filed :-D - backups are currently being generated now. I need to finish some productionization of the grants, but the most pressing thing has already been completed.

Jan 25 2024, 1:04 PM · database-backups, Data-Persistence-Backup, iPoid-Service
jcrespo added a comment to T355814: VRTS Database Growth Rate.

Sure, the backup stats were designed to track backup status, not live table metrics, but nonetheless you might be able to get some useful stats by querying the backups and backup_files tables. You can access the data through pampinus ( https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Dashboard ). If that is not enough, you can query it though SQL, the structure is available here: https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Metadata

Jan 25 2024, 9:47 AM · Data-Persistence, collaboration-services

Jan 24 2024

jcrespo reassigned T349360: Clean up dbbackups.backup_files table from jcrespo to Ladsgroup.

I did a long term archival backup of records from 2022 and before into the Archival section of backups:

Jan 24 2024, 5:10 PM · database-backups, Data-Persistence-Backup

Jan 18 2024

jcrespo updated the task description for T355313: db2146 started being behind on replication.
Jan 18 2024, 12:51 PM · DBA
jcrespo updated subscribers of T355313: db2146 started being behind on replication.

@Ladsgroup @ABran-WMF @Marostegui

Jan 18 2024, 12:26 PM · DBA
jcrespo created T355313: db2146 started being behind on replication.
Jan 18 2024, 12:25 PM · DBA
jcrespo added a comment to T355246: Respond to 17.01.2024 data import failure.

I've left 2 files on the directory: /home/stran/ipoid_backup @ deploy2002:

Jan 18 2024, 10:57 AM · Trust and Safety Product Sprint, Wikimedia-production-error, iPoid-Service

Jan 15 2024

jcrespo closed T353526: Migrate dbbackups from cumin1001 to cumin1002 as Resolved.

Backups worked over the weekend with no issues. Resolving.

Jan 15 2024, 11:18 AM · database-backups, Data-Persistence, Data-Persistence-Backup, Infrastructure-Foundations, SRE
jcrespo closed T353526: Migrate dbbackups from cumin1001 to cumin1002, a subtask of T353419: Setup cumin1002 and eventually decom cumin1001, as Resolved.
Jan 15 2024, 11:18 AM · Infrastructure-Foundations, SRE

Jan 12 2024

jcrespo added a comment to T353526: Migrate dbbackups from cumin1001 to cumin1002.

The custom x1 backup worked well from cumin1002. Waiting now on the regular daily backup.

Jan 12 2024, 4:23 PM · database-backups, Data-Persistence, Data-Persistence-Backup, Infrastructure-Foundations, SRE
jcrespo moved T353526: Migrate dbbackups from cumin1001 to cumin1002 from Triage to Done on the Data-Persistence-Backup board.
Jan 12 2024, 2:19 PM · database-backups, Data-Persistence, Data-Persistence-Backup, Infrastructure-Foundations, SRE
jcrespo added a comment to T353526: Migrate dbbackups from cumin1001 to cumin1002.

Doing a test backup now.

Jan 12 2024, 2:18 PM · database-backups, Data-Persistence, Data-Persistence-Backup, Infrastructure-Foundations, SRE
jcrespo triaged T353526: Migrate dbbackups from cumin1001 to cumin1002 as High priority.
Jan 12 2024, 12:46 PM · database-backups, Data-Persistence, Data-Persistence-Backup, Infrastructure-Foundations, SRE
jcrespo added projects to T353526: Migrate dbbackups from cumin1001 to cumin1002: Data-Persistence-Backup, Data-Persistence, database-backups.
Jan 12 2024, 12:46 PM · database-backups, Data-Persistence, Data-Persistence-Backup, Infrastructure-Foundations, SRE
jcrespo changed the status of T353526: Migrate dbbackups from cumin1001 to cumin1002 from Open to In Progress.
Jan 12 2024, 12:46 PM · database-backups, Data-Persistence, Data-Persistence-Backup, Infrastructure-Foundations, SRE
jcrespo changed the status of T353526: Migrate dbbackups from cumin1001 to cumin1002, a subtask of T353419: Setup cumin1002 and eventually decom cumin1001, from Open to In Progress.
Jan 12 2024, 12:45 PM · Infrastructure-Foundations, SRE

Jan 11 2024

jcrespo closed T327157: Create and deploy the logic to generate incremental backups of MediaWiki media files, to keep its file storage backup up to date, automatically as Resolved.

This was done long time ago, only the TLS part and puppet/MySQL changes/updates was done last quarter.

Jan 11 2024, 11:36 AM · Data-Persistence, media-backups, Data-Persistence-Backup, Goal
jcrespo closed T327157: Create and deploy the logic to generate incremental backups of MediaWiki media files, to keep its file storage backup up to date, automatically, a subtask of T262668: WMF media storage must be adequately backed up, as Resolved.
Jan 11 2024, 11:36 AM · media-backups, Data-Persistence-Backup, Epic, Goal, SRE, SRE-swift-storage

Jan 8 2024

jcrespo added a comment to T182822: Generate a list of files that are supposed to exist but 404s.

In particular, even if it doesn't affect commons, enwikivoyage has lots of references to old files that were not imported (but probably later re-uploaded with new references).

Jan 8 2024, 1:01 PM · User-ArielGlenn, SRE, Multimedia, SRE-swift-storage, Commons

Dec 20 2023

jcrespo added a project to T353797: Missing original File:Ignatyevo.jpg: media-backups.

The first backup for enwiki media is from 2021, and back then it already failed to be downloaded from swift, so sadly, it is not on the media backups. Here is the log:

[2021-08-19 11:03:43,781] ERROR:swiftclient.service Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.27/2/27/Ignatyevo.jpg 404 Not Found  [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<'
swiftclient.exceptions.ClientException: Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.27/2/27/Ignatyevo.jpg 404 Not Found  [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<'
[2021-08-19 11:03:43,782] ERROR:backup Download of "enwiki Ignatyevo.jpg c336b0a6f224d2a4ad999d37f4b8fdaaeef797b9" failed
[2023-02-15 17:03:11,429] ERROR:backup '2/27/Ignatyevo.jpg' failed to be downloaded
[2023-02-15 17:03:11,429] ERROR:backup Download of "enwiki Ignatyevo.jpg c336b0a6f224d2a4ad999d37f4b8fdaaeef797b9" failed
[2023-02-15 17:23:50,160] ERROR:swiftclient.service Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.27/2/27/Ignatyevo.jpg 404 Not Found  [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<'
swiftclient.exceptions.ClientException: Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.27/2/27/Ignatyevo.jpg 404 Not Found  [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<'
[2023-02-15 17:23:50,160] ERROR:backup '2/27/Ignatyevo.jpg' failed to be downloaded
[2023-02-15 17:23:50,161] ERROR:backup Download of "enwiki Ignatyevo.jpg c336b0a6f224d2a4ad999d37f4b8fdaaeef797b9" failed
[2023-03-22 10:10:44,396] ERROR:swiftclient.service Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.27/2/27/Ignatyevo.jpg 404 Not Found  [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<'
swiftclient.exceptions.ClientException: Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.27/2/27/Ignatyevo.jpg 404 Not Found  [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<'
[2023-03-22 10:10:44,397] ERROR:backup '2/27/Ignatyevo.jpg' failed to be downloaded
[2023-03-22 10:10:44,397] ERROR:backup Download of "enwiki Ignatyevo.jpg c336b0a6f224d2a4ad999d37f4b8fdaaeef797b9 2006-12-05 21:19:48" failed
Dec 20 2023, 1:05 PM · media-backups, Data-Persistence, SRE-swift-storage

Dec 12 2023

jcrespo added a comment to T350020: Access request to deleted image files in the production Swift cluster.

There is ongoing conversations with legal, which doesn't write here. Don't worry- deployment of this should be handled by data persistence team (I think), at least any changes on Swift that may be needed.

Dec 12 2023, 8:14 PM · Data-Persistence, UploadWizard, Structured-Data-Backlog
jcrespo added a comment to T343707: Migrate SRE repositories to GitLab - Archiving unused Gerrit repositories.

I did operations/software/pampinus

Dec 12 2023, 11:38 AM · Projects-Cleanup, Release-Engineering-Team (Priority Backlog 📥), collaboration-services
jcrespo added a comment to T316655: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals.

If the idea is to create full backups, it should be compressed- deltas won't work unless files are byte-identical. That is why I want to switch to full backups, because generally each dump will be different.

Dec 12 2023, 10:15 AM · Data-Engineering, Infrastructure-Foundations, bacula, Data-Persistence-Backup, netbox
jcrespo added a comment to T316655: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals.

File or directory is the same- but it has to have an exact name, contain no other file and not use soft links. Then we will switch to full backups rather than incrementals. I suggested the directory model because at some point we would like to integrate that into wmfbackups- but that is out of scope here.

Dec 12 2023, 10:03 AM · Data-Engineering, Infrastructure-Foundations, bacula, Data-Persistence-Backup, netbox

Dec 11 2023

jcrespo added a comment to T316655: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals.

This is the list of backups we keep:

Dec 11 2023, 4:39 PM · Data-Engineering, Infrastructure-Foundations, bacula, Data-Persistence-Backup, netbox

Dec 5 2023

jcrespo added a comment to T352695: MediaWiki PHPUnit test suite can result in non-test database being modified.

Could it be https://jira.mariadb.org/browse/MDEV-28334 ?

Dec 5 2023, 6:06 PM · MW-1.42-notes (1.42.0-wmf.9; 2023-12-12), Upstream, Data-Persistence, MediaWiki-libs-Rdbms, MediaWiki-Core-Tests
jcrespo added a comment to T352695: MediaWiki PHPUnit test suite can result in non-test database being modified.

Apparently a temporary table with the name _ can match any SHOW TABLES LIKE query.

MariaDB [wiki1]> CREATE TEMPORARY TABLE _ (a INT);
Query OK, 0 rows affected (0.001 sec)

MariaDB [wiki1]> SHOW TABLES LIKE 'T352695';
+---------------------------+
| Tables_in_wiki1 (T352695) |
+---------------------------+
| _                         |
+---------------------------+
1 row in set (0.001 sec)
Dec 5 2023, 5:55 PM · MW-1.42-notes (1.42.0-wmf.9; 2023-12-12), Upstream, Data-Persistence, MediaWiki-libs-Rdbms, MediaWiki-Core-Tests
jcrespo added a comment to T343707: Migrate SRE repositories to GitLab - Archiving unused Gerrit repositories.

The default setting is to allow maintainers to push and merge and set force push to off. All SREs are owners of /repos/sre so in an emergency situation would be able to enable force push for the main branch of a repository if needed - would that be sufficient from your point of view (with additional documentation on how to do it)?

Dec 5 2023, 4:55 PM · Projects-Cleanup, Release-Engineering-Team (Priority Backlog 📥), collaboration-services
jcrespo added a comment to T352655: Automate backup file deletion by using a batch file.

Docs: https://wikitech.wikimedia.org/wiki/Media_storage/Backups#Batch_query%2C_recovery_and_deletion

Dec 5 2023, 2:52 PM · Data-Persistence, Data-Persistence-Backup, media-backups
jcrespo closed T352655: Automate backup file deletion by using a batch file as Resolved.

This is now down (with further improvements down the line), will now document it on wikitech.

Dec 5 2023, 2:30 PM · Data-Persistence, Data-Persistence-Backup, media-backups
jcrespo added a comment to T343707: Migrate SRE repositories to GitLab - Archiving unused Gerrit repositories.

I migrated:

Dec 5 2023, 1:12 PM · Projects-Cleanup, Release-Engineering-Team (Priority Backlog 📥), collaboration-services
jcrespo committed rOSMB900eb03bf85b: Implement batch deletion, restoration and query of files (authored by jcrespo).
Implement batch deletion, restoration and query of files
Dec 5 2023, 11:48 AM
jcrespo committed rOSMBd3f54f903346: Prepare for 0.3.0 release (authored by jcrespo).
Prepare for 0.3.0 release
Dec 5 2023, 11:48 AM