User Details
- User Since
- May 11 2015, 8:31 AM (462 w, 21 h)
- Availability
- Available
- IRC Nick
- jynus
- LDAP User
- Jcrespo
- MediaWiki User
- JCrespo (WMF) [ Global Accounts ]
Wed, Mar 6
Meanwhile, I am going to merge and deploy https://gitlab.wikimedia.org/repos/sre/mediabackups/-/merge_requests/1 in any case (workaround for media backups), as backups should be compatible an be able to recover files with illegal names, independently of being a mw bug, as I mention in the review.
Thanks a lot, @Bawolff, this is the kind of take I really needed- as my initial reaction was to just make the field larger.
Tue, Mar 5
Looking good, backups unblocked:
I didn't ask for a cable change, and so far I haven't observed any problem with the host, TBH, it was @ayounsi who requested it, but I wonder if the metrics are too sensitive- we do the backup as fast as possible, and we trust on TCP (well, and the checksum we do later, to be fair) to fix potential errors when doing the regular backups.
My suggestion would be, for a short-ish term, to make the field longer (the archival table - oldimage- will be very small) and fill-in the data from the original name, when possible. Although I haven't checked the deleted one, it may suffer from the same issue with deleted, archived files.
Based on the above, I would like to mostly unblock media backups by fixing the UTF8 invalid characters first on production. For that, It would like to just remove the invalid characters on the first 2 files, and then we can maybe discuss how to move forward for the main bug (code fixes, a schema change on oldimage to make oi_archive_name larger?).
After investigating, it seems the issue goes beyond invalid UTF-8 characters, there is metadata loss due to the pseudo-path stored on the database being silently truncated. When the path is calcualted, the date + an admiration sign is added as prefix, leading to the full path being cut at the end, sometimes, if cut on multi-byte characters, to invalid data (but on any case truncated).
The theoretical container and path of these 2 files should be, in theory:
Replacing the cable can be done any time between 6:00 and 23:55 UTC. Let me know if it will be for a period of extended time so I can downtime it.
Fri, Mar 1
Sure, let me know what can I do for you, and I will get it done next week. The only custom stuff compared to other hosts is the HW RAID partitioning, the rest is a rather standard db-like install. If it is only adding it to netconf and the default role (staging), it shouldn't take me long, but I will get it done on Monday, as I have finished my work week already.
Thu, Feb 29
Wed, Feb 28
As I prepared beforehand for a previous upgrade, s6, x1 and s2 are already producing 10.6-compatible backups, and backup sources should be, at least partially, upgraded- we can just drop the 10.4 ones when fully upgraded and move 10.4 sections instead (I will handle that). So I should not be a blocker for this ticket. If you have a roadmap of future upgrades beyond those sections, I can start working on preparing those now, so I am ready already like in this case.
Thu, Feb 22
It took some time to confirm it live, because the number of new deleted files don't grow as fast as the "latest" versions, and the algorithm starts backing up the non-deleted ones, but I can confirm the new deleted files backups count is growing up with the new user:
Thank you a lot, as I mentioned in private, I will try to run the automatic downloads back again with the new user, if it works we will be very close to resolve this. Thank you again!
Wed, Feb 21
Thanks for the reply, the reboot happened on the 17, so no relation to that.
an-db backups looking good:
✔️ root@backup1001:~$ check_bacula.py an-db1001.eqiad.wmnet-Daily-productionEqiad-data-engineering-postgres id: 552007, ts: 2024-02-08 04:05:00, type: F, status: T, bytes: 643284416 id: 552157, ts: 2024-02-09 04:05:01, type: F, status: T, bytes: 645145616 id: 552309, ts: 2024-02-10 04:05:00, type: F, status: T, bytes: 646974416 id: 552458, ts: 2024-02-11 04:05:00, type: F, status: T, bytes: 648762976 id: 552609, ts: 2024-02-12 04:05:00, type: F, status: T, bytes: 650570592 id: 552760, ts: 2024-02-13 04:05:00, type: F, status: T, bytes: 652389680 id: 552918, ts: 2024-02-14 04:05:01, type: F, status: T, bytes: 654207808 id: 553088, ts: 2024-02-15 04:05:00, type: F, status: T, bytes: 655958880 id: 553260, ts: 2024-02-16 04:05:01, type: F, status: T, bytes: 657687120 id: 553428, ts: 2024-02-17 04:05:00, type: F, status: T, bytes: 659326944 id: 553597, ts: 2024-02-18 04:05:00, type: F, status: T, bytes: 660873424 id: 553760, ts: 2024-02-19 04:05:00, type: F, status: T, bytes: 662617920 id: 553924, ts: 2024-02-20 04:05:00, type: F, status: T, bytes: 665207840 id: 554109, ts: 2024-02-21 04:05:01, type: F, status: T, bytes: 630091008 ✔️ root@backup1001:~$ check_bacula.py an-db1002.eqiad.wmnet-Daily-productionEqiad-data-engineering-postgres id: 552008, ts: 2024-02-08 04:05:00, type: F, status: T, bytes: 182492896 id: 552158, ts: 2024-02-09 04:05:01, type: F, status: T, bytes: 193882176 id: 552310, ts: 2024-02-10 04:05:01, type: F, status: T, bytes: 178497152 id: 552459, ts: 2024-02-11 04:05:00, type: F, status: T, bytes: 171867232 id: 552610, ts: 2024-02-12 04:05:00, type: F, status: T, bytes: 182933552 id: 552761, ts: 2024-02-13 04:05:00, type: F, status: T, bytes: 177721968 id: 552919, ts: 2024-02-14 04:05:01, type: F, status: T, bytes: 178077264 id: 553089, ts: 2024-02-15 04:05:08, type: F, status: T, bytes: 193632432 id: 553261, ts: 2024-02-16 04:05:01, type: F, status: T, bytes: 179027088 id: 553429, ts: 2024-02-17 04:05:00, type: F, status: T, bytes: 179340992 id: 553598, ts: 2024-02-18 04:05:00, type: F, status: T, bytes: 189788240 id: 553761, ts: 2024-02-19 04:05:00, type: F, status: T, bytes: 179704320 id: 553925, ts: 2024-02-20 04:05:08, type: F, status: T, bytes: 180750272 id: 554110, ts: 2024-02-21 04:05:02, type: F, status: T, bytes: 180212720
Tue, Feb 20
Papaul is on vacations, so it couldn't be him. :-(
Mon, Feb 19
Hey, @Papaul I saw a login at 02/17/2024 05:24:45, barely a minute after a flash reset. Maybe some dc op saw something else when logging in, or there was some ongoing maintenance on that host or a close one?
1736 Server power restored. 02/17/2024 05:23:51 1 Maintenance, Administration 1735 Server reset. 02/17/2024 05:23:51 1 Maintenance, Administration 1734 Server power removed. 02/17/2024 05:23:46 1 Maintenance, Administration 1733 Power on request received by: Automatic Power Recovery. 02/17/2024 05:23:45 1 Maintenance, Administration 1732 Server power restored. 02/17/2024 05:23:27 1 Maintenance, Administration 1731 Server reset. 02/17/2024 05:23:27 1 Maintenance, Administration 1730 Server power removed. 02/17/2024 05:23:26 1 Maintenance, Administration 1729 Embedded Flash: Restarted 02/17/2024 05:23:18 1 Firmware
I will take it from here- at the very least, recover it from backups.
Feb 14 2024
Feb 13 2024
This is now fixed, right? Or do you prefer to keep it up for followup?
Feb 12 2024
All backups looking good. Please proceed.
Feb 7 2024
Technically no blocker for decom of db1139, db1140 or db1145, although I would wait 1 day to make sure backups run correctly on the new hosts at least once.
New servers are in use, the old backup sources are idle (no service uses them), but still replicating/available. All work on my side is done, unless for some reason the backups on the new host start to fail.
This is looking good- I will check in the following days that an db backups are being produced (the data copy for Bacula, the dump correctness at the moment should be the service owner responsability 0:-) ).
Feb 6 2024
Potentially a similar issue (request/traffic related high load) happened around the past 28 of September, when I added this TODO: https://wikitech.wikimedia.org/w/index.php?title=Thanos&diff=prev&oldid=2115837 I was confused back then even what this service was.
Feb 5 2024
✔️ root@backup1001:~$ check_bacula.py netboxdb1002.eqiad.wmnet-Daily-productionEqiad-netbox-postgres id: 550937, ts: 2024-02-01 07:28:36, type: F, status: T, bytes: 46654944 id: 551106, ts: 2024-02-02 04:52:07, type: F, status: T, bytes: 46611456 id: 551272, ts: 2024-02-03 06:04:44, type: F, status: T, bytes: 46521712 id: 551439, ts: 2024-02-04 10:32:37, type: F, status: T, bytes: 46399072 id: 551600, ts: 2024-02-05 05:36:16, type: F, status: T, bytes: 46267136 ✔️ root@backup1001:~$ check_bacula.py netboxdb2002.codfw.wmnet-Daily-productionEqiad-netbox-postgres id: 550938, ts: 2024-02-01 07:28:39, type: F, status: T, bytes: 46176368 id: 551107, ts: 2024-02-02 04:52:09, type: F, status: T, bytes: 46081440 id: 551273, ts: 2024-02-03 06:04:47, type: F, status: T, bytes: 46001888 id: 551440, ts: 2024-02-04 10:32:41, type: F, status: T, bytes: 45952240 id: 551601, ts: 2024-02-05 05:36:18, type: F, status: T, bytes: 45729200
@Volans 👍
Feb 2 2024
FYI: I've recovered from backups db1239 (s1, s2), db1240 (s1) and db1245 (s4, s5)
root@backup1001:~$ check_bacula.py netboxdb2002.codfw.wmnet-Daily-productionEqiad-netbox-postgres id: 550938, ts: 2024-02-01 07:28:39, type: F, status: T, bytes: 46176368 id: 551107, ts: 2024-02-02 04:52:09, type: F, status: T, bytes: 46081440 ✔️ root@backup1001:~$ check_bacula.py netboxdb1002.eqiad.wmnet-Daily-productionEqiad-netbox-postgres id: 550937, ts: 2024-02-01 07:28:36, type: F, status: T, bytes: 46654944 id: 551106, ts: 2024-02-02 04:52:07, type: F, status: T, bytes: 46611456 ✔️
Feb 1 2024
*list jobname=netboxdb1002.eqiad.wmnet-Monthly-1st-Wed-productionEqiad-netbox-postgres +---------+--------------------------------------------------------------------------+---------------------+------+-------+----------+-------------+-----------+ | JobId | Name | StartTime | Type | Level | JobFiles | JobBytes | JobStatus | +---------+--------------------------------------------------------------------------+---------------------+------+-------+----------+-------------+-----------+ | 536,832 | netboxdb1002.eqiad.wmnet-Monthly-1st-Wed-productionEqiad-netbox-postgres | 2023-11-04 06:28:49 | B | I | 2 | 48,881,440 | T |
root@backup1001:~$ check_bacula.py netboxdb1002.eqiad.wmnet-Daily-productionEqiad-netbox-postgres id: 550937, ts: 2024-02-01 07:28:36, type: F, status: T, bytes: 46654944 ✔️ root@backup1001:~$ check_bacula.py netboxdb2002.codfw.wmnet-Daily-productionEqiad-netbox-postgres id: 550938, ts: 2024-02-01 07:28:39, type: F, status: T, bytes: 46176368 ✔️
Jan 31 2024
root@backup1001:~$ check_bacula.py netboxdb1002.eqiad.wmnet-Monthly-1st-Wed-productionEqiad-netbox-postgres id: 536667, ts: 2023-11-03 05:16:05, type: I, status: T, bytes: 48667264 id: 536832, ts: 2023-11-04 06:28:49, type: I, status: T, bytes: 48881440 id: 536997, ts: 2023-11-05 10:28:53, type: I, status: T, bytes: 49098288 id: 537157, ts: 2023-11-06 05:04:41, type: I, status: T, bytes: 49285552 id: 537316, ts: 2023-11-07 06:24:22, type: I, status: T, bytes: 49513840 id: 537472, ts: 2023-11-08 04:56:28, type: I, status: T, bytes: 49746272 id: 537621, ts: 2023-11-09 04:56:18, type: I, status: T, bytes: 49892240 id: 537769, ts: 2023-11-10 04:53:37, type: I, status: T, bytes: 50039280 id: 537916, ts: 2023-11-11 04:44:50, type: I, status: T, bytes: 50245888 id: 538063, ts: 2023-11-12 04:33:23, type: I, status: T, bytes: 50376480 id: 538212, ts: 2023-11-13 04:53:46, type: I, status: T, bytes: 50548784 id: 538361, ts: 2023-11-14 05:11:29, type: I, status: T, bytes: 50676496 id: 538442, ts: 2023-11-15 03:07:18, type: D, status: T, bytes: 451827680 id: 538544, ts: 2023-11-15 05:00:24, type: I, status: T, bytes: 0 id: 538711, ts: 2023-11-16 05:33:09, type: I, status: T, bytes: 50964816 id: 538878, ts: 2023-11-17 05:06:32, type: I, status: T, bytes: 51023920 id: 539043, ts: 2023-11-18 04:43:40, type: I, status: T, bytes: 50837616 id: 539208, ts: 2023-11-19 04:50:40, type: I, status: T, bytes: 50675840 id: 539368, ts: 2023-11-20 05:05:16, type: I, status: T, bytes: 50499072 id: 539527, ts: 2023-11-21 19:22:14, type: I, status: T, bytes: 50283632 id: 539685, ts: 2023-11-22 04:52:01, type: I, status: T, bytes: 50152992 id: 539834, ts: 2023-11-23 05:01:30, type: I, status: T, bytes: 49899936 id: 539982, ts: 2023-11-24 05:00:31, type: I, status: T, bytes: 49704800 id: 540129, ts: 2023-11-25 04:41:16, type: I, status: T, bytes: 49533520 id: 540276, ts: 2023-11-26 04:34:46, type: I, status: T, bytes: 49303216 id: 540425, ts: 2023-11-27 04:40:35, type: I, status: T, bytes: 49105536 id: 540574, ts: 2023-11-28 05:29:51, type: I, status: T, bytes: 48861232 id: 540730, ts: 2023-11-29 05:02:19, type: I, status: T, bytes: 48646816 id: 540879, ts: 2023-11-30 05:35:15, type: I, status: T, bytes: 96952608 id: 541046, ts: 2023-12-01 05:01:20, type: I, status: T, bytes: 48688640 id: 541211, ts: 2023-12-02 06:32:13, type: I, status: T, bytes: 48747136 id: 541376, ts: 2023-12-03 10:29:26, type: I, status: T, bytes: 48825792 id: 541536, ts: 2023-12-04 05:05:02, type: I, status: T, bytes: 48805952 id: 541695, ts: 2023-12-05 05:33:17, type: I, status: T, bytes: 48851168 id: 541774, ts: 2023-12-06 02:33:03, type: F, status: T, bytes: 694262768 id: 541877, ts: 2023-12-06 05:20:17, type: I, status: T, bytes: 0 id: 542044, ts: 2023-12-07 06:04:33, type: I, status: T, bytes: 48907520 id: 542192, ts: 2023-12-08 05:00:52, type: I, status: T, bytes: 48854368 id: 542339, ts: 2023-12-09 04:51:26, type: I, status: T, bytes: 48832448 id: 542486, ts: 2023-12-10 04:35:31, type: I, status: T, bytes: 48815328 id: 542635, ts: 2023-12-11 04:41:30, type: I, status: T, bytes: 48777712 id: 542784, ts: 2023-12-12 05:31:34, type: I, status: T, bytes: 48760592 id: 542940, ts: 2023-12-13 04:57:14, type: I, status: T, bytes: 48745440 id: 543089, ts: 2023-12-14 05:09:54, type: I, status: T, bytes: 48734448 id: 543259, ts: 2023-12-15 05:02:15, type: I, status: T, bytes: 48702464 id: 543427, ts: 2023-12-16 04:43:29, type: I, status: T, bytes: 48719600 id: 543595, ts: 2023-12-17 04:41:44, type: I, status: T, bytes: 48715232 id: 543758, ts: 2023-12-18 04:36:42, type: I, status: T, bytes: 48735360 id: 543920, ts: 2023-12-19 05:41:49, type: I, status: T, bytes: 48729312 id: 544000, ts: 2023-12-20 03:06:46, type: D, status: T, bytes: 438541808 id: 544105, ts: 2023-12-20 04:56:18, type: I, status: T, bytes: 0 id: 544278, ts: 2023-12-21 05:07:06, type: I, status: T, bytes: 48680736 id: 544429, ts: 2023-12-22 05:20:10, type: I, status: T, bytes: 48595168 id: 544579, ts: 2023-12-23 04:46:14, type: I, status: T, bytes: 48588896 id: 544729, ts: 2023-12-24 04:46:17, type: I, status: T, bytes: 48580992 id: 544881, ts: 2023-12-25 04:53:23, type: I, status: T, bytes: 48585152 id: 545033, ts: 2023-12-26 05:05:59, type: I, status: T, bytes: 48580576 id: 545192, ts: 2023-12-27 05:01:51, type: I, status: T, bytes: 48506448 id: 545344, ts: 2023-12-28 05:03:59, type: I, status: T, bytes: 48447776 id: 545495, ts: 2023-12-29 04:57:01, type: I, status: T, bytes: 48403152 id: 545645, ts: 2023-12-30 04:47:16, type: I, status: T, bytes: 48348208 id: 545795, ts: 2023-12-31 04:43:20, type: I, status: T, bytes: 48292912 id: 545958, ts: 2024-01-01 05:12:57, type: I, status: T, bytes: 48230352 id: 546120, ts: 2024-01-02 06:02:31, type: I, status: T, bytes: 48210128 id: 546199, ts: 2024-01-03 02:33:04, type: F, status: T, bytes: 690480544 id: 546305, ts: 2024-01-03 05:33:10, type: I, status: T, bytes: 0 id: 546478, ts: 2024-01-04 06:52:32, type: I, status: T, bytes: 48078048 id: 546648, ts: 2024-01-05 04:56:52, type: I, status: T, bytes: 48050336 id: 546816, ts: 2024-01-06 06:55:14, type: I, status: T, bytes: 48057360 id: 546984, ts: 2024-01-07 05:14:48, type: I, status: T, bytes: 48033424 id: 547136, ts: 2024-01-08 04:42:14, type: I, status: T, bytes: 47991664 id: 547288, ts: 2024-01-09 05:19:08, type: I, status: T, bytes: 47922752 id: 547447, ts: 2024-01-10 05:37:54, type: I, status: T, bytes: 47946816 id: 547599, ts: 2024-01-11 05:03:43, type: I, status: T, bytes: 47887856 id: 547750, ts: 2024-01-12 05:10:28, type: I, status: T, bytes: 47768208 id: 547900, ts: 2024-01-13 04:56:52, type: I, status: T, bytes: 47757424 id: 548050, ts: 2024-01-14 04:36:45, type: I, status: T, bytes: 47759536 id: 548213, ts: 2024-01-15 04:57:04, type: I, status: T, bytes: 47726096 id: 548376, ts: 2024-01-16 05:15:40, type: I, status: T, bytes: 47670944 id: 548456, ts: 2024-01-17 03:06:32, type: D, status: T, bytes: 430070016 id: 548561, ts: 2024-01-17 05:28:45, type: I, status: T, bytes: 0 id: 548732, ts: 2024-01-18 05:44:31, type: I, status: T, bytes: 47560048 id: 548901, ts: 2024-01-19 04:54:21, type: I, status: T, bytes: 47488512 id: 549068, ts: 2024-01-20 04:56:54, type: I, status: T, bytes: 47406048 id: 549235, ts: 2024-01-21 05:27:55, type: I, status: T, bytes: 47379184 id: 549386, ts: 2024-01-22 04:35:03, type: I, status: T, bytes: 47290032 id: 549537, ts: 2024-01-23 05:15:24, type: I, status: T, bytes: 47152576 id: 549695, ts: 2024-01-24 05:08:11, type: I, status: T, bytes: 47223856 id: 549847, ts: 2024-01-25 05:06:56, type: I, status: T, bytes: 47114704 id: 549997, ts: 2024-01-26 04:52:36, type: I, status: T, bytes: 47071632 id: 550148, ts: 2024-01-27 04:46:20, type: I, status: T, bytes: 46995824 id: 550297, ts: 2024-01-28 04:35:50, type: I, status: T, bytes: 46947840 id: 550454, ts: 2024-01-29 04:48:36, type: I, status: T, bytes: 46849680 id: 550605, ts: 2024-01-30 05:33:27, type: I, status: T, bytes: 46745792 id: 550763, ts: 2024-01-31 05:14:48, type: I, status: T, bytes: 46700960
Let me do a manual run first and then we can do the followups (I also need to deploy the change backup setting to default to full).
Let me know when you have a new backup in the new format to test and deploy the bacula changes.
Jan 29 2024
Thank you, I will shutdown media backups anyway every time one host is affected, not just this one, to minimize failures.
Backup and recovery tested, grants deployed on both datacenters. This is done.
Data recovery is restricted to roots, it is not self-served. If you are curious how it works it documented at: https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Recovering_the_data_(Mariadb_10.6)
Jan 26 2024
Jan 25 2024
We will keep them for 3 months, the (actually small, in comparison) size doesn't justify treating them differently than the other backups. :-)
Proof:
This was already in motion before being filed :-D - backups are currently being generated now. I need to finish some productionization of the grants, but the most pressing thing has already been completed.
Sure, the backup stats were designed to track backup status, not live table metrics, but nonetheless you might be able to get some useful stats by querying the backups and backup_files tables. You can access the data through pampinus ( https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Dashboard ). If that is not enough, you can query it though SQL, the structure is available here: https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Metadata
Jan 24 2024
I did a long term archival backup of records from 2022 and before into the Archival section of backups:
Jan 18 2024
I've left 2 files on the directory: /home/stran/ipoid_backup @ deploy2002:
Jan 15 2024
Backups worked over the weekend with no issues. Resolving.
Jan 12 2024
The custom x1 backup worked well from cumin1002. Waiting now on the regular daily backup.
Doing a test backup now.
Jan 11 2024
This was done long time ago, only the TLS part and puppet/MySQL changes/updates was done last quarter.
Jan 8 2024
In particular, even if it doesn't affect commons, enwikivoyage has lots of references to old files that were not imported (but probably later re-uploaded with new references).
Dec 20 2023
The first backup for enwiki media is from 2021, and back then it already failed to be downloaded from swift, so sadly, it is not on the media backups. Here is the log:
[2021-08-19 11:03:43,781] ERROR:swiftclient.service Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.27/2/27/Ignatyevo.jpg 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<' swiftclient.exceptions.ClientException: Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.27/2/27/Ignatyevo.jpg 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<' [2021-08-19 11:03:43,782] ERROR:backup Download of "enwiki Ignatyevo.jpg c336b0a6f224d2a4ad999d37f4b8fdaaeef797b9" failed [2023-02-15 17:03:11,429] ERROR:backup '2/27/Ignatyevo.jpg' failed to be downloaded [2023-02-15 17:03:11,429] ERROR:backup Download of "enwiki Ignatyevo.jpg c336b0a6f224d2a4ad999d37f4b8fdaaeef797b9" failed [2023-02-15 17:23:50,160] ERROR:swiftclient.service Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.27/2/27/Ignatyevo.jpg 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<' swiftclient.exceptions.ClientException: Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.27/2/27/Ignatyevo.jpg 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<' [2023-02-15 17:23:50,160] ERROR:backup '2/27/Ignatyevo.jpg' failed to be downloaded [2023-02-15 17:23:50,161] ERROR:backup Download of "enwiki Ignatyevo.jpg c336b0a6f224d2a4ad999d37f4b8fdaaeef797b9" failed [2023-03-22 10:10:44,396] ERROR:swiftclient.service Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.27/2/27/Ignatyevo.jpg 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<' swiftclient.exceptions.ClientException: Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.27/2/27/Ignatyevo.jpg 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<' [2023-03-22 10:10:44,397] ERROR:backup '2/27/Ignatyevo.jpg' failed to be downloaded [2023-03-22 10:10:44,397] ERROR:backup Download of "enwiki Ignatyevo.jpg c336b0a6f224d2a4ad999d37f4b8fdaaeef797b9 2006-12-05 21:19:48" failed
Dec 12 2023
There is ongoing conversations with legal, which doesn't write here. Don't worry- deployment of this should be handled by data persistence team (I think), at least any changes on Swift that may be needed.
I did operations/software/pampinus
If the idea is to create full backups, it should be compressed- deltas won't work unless files are byte-identical. That is why I want to switch to full backups, because generally each dump will be different.
File or directory is the same- but it has to have an exact name, contain no other file and not use soft links. Then we will switch to full backups rather than incrementals. I suggested the directory model because at some point we would like to integrate that into wmfbackups- but that is out of scope here.
Dec 11 2023
This is the list of backups we keep:
Dec 5 2023
Could it be https://jira.mariadb.org/browse/MDEV-28334 ?
This is now down (with further improvements down the line), will now document it on wikitech.
I migrated: