Page MenuHomePhabricator

Create and deploy the logic to generate incremental backups of MediaWiki media files, to keep its file storage backup up to date, automatically
Closed, ResolvedPublic

Description

After numerous planning and improvements, we have to finalize the backup workflow as initially designed, implementing incremental updates to existing media backups, while keeping history intact- not only to add the newly uploaded files to the backups, but to also make sure its metadata (file status: deleted, archived, latest; renames, etc.) is kept -as good as possible- archived so later it can be used for recoveries to Swift storage.

This work should conclude the initial scope planned for backups so that they can now function completely autonomously, without needing a human manually scheduling runs. While certainly later improvements should be made: fully automated mass recovery, streaming backups, better UI, etc. those should be later enhancements not in the initial scope, and implemented on a later iteration, allowing to have a feature-complete self-sustained backup workflow for media.

Details

Show related patches Customize query in gerrit

Event Timeline

Change 889534 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackups: Test backup update on testwiki for eqiad

https://gerrit.wikimedia.org/r/889534

Change 889534 merged by Jcrespo:

[operations/puppet@production] mediabackups: Test backup update on testwiki for eqiad

https://gerrit.wikimedia.org/r/889534

Change 889646 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Preparing for release 0.1.6

https://gerrit.wikimedia.org/r/889646

Currently running a metadata backup update of commonswiki on eqiad. 🤞

This is the main kind of unhandled error I have: https://commons.wikimedia.org/wiki/File:3_castillos.jpg

2 identical files uploaded in the same second- and not having any unique identifier makes it impossible to update its metadata.

Change 889646 merged by Jcrespo:

[operations/software/mediabackups@master] Preparing for release 0.1.6

https://gerrit.wikimedia.org/r/889646

Change 892891 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Update sql to add newly history table files_history

https://gerrit.wikimedia.org/r/892891

Change 911943 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Create new script to read recent logs and update backups metadata

https://gerrit.wikimedia.org/r/911943

Change 912818 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Update indexes for latest queries needed for mediabackups

https://gerrit.wikimedia.org/r/912818

Change 912819 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Add functionality to detect last uploaded time for backup start

https://gerrit.wikimedia.org/r/912819

Change 912926 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] Set both eqiad and codfw mediabackups to by default, backup Commons

https://gerrit.wikimedia.org/r/912926

Change 912926 merged by Jcrespo:

[operations/puppet@production] mediabackups: Set both dcs to backup Commons by default

https://gerrit.wikimedia.org/r/912926

Change 912935 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] recentuploads: Set custom headers for querying the mediawiki api

https://gerrit.wikimedia.org/r/912935

Mentioned in SAL (#wikimedia-operations) [2023-04-28T15:23:28Z] <jynus> update schema for backup1-codfw (mediabackups) T327157

Change 892891 merged by Jcrespo:

[operations/software/mediabackups@master] Update sql to add newly history table file_history

https://gerrit.wikimedia.org/r/892891

Change 911943 merged by Jcrespo:

[operations/software/mediabackups@master] Create new script to read recent logs and update backups metadata

https://gerrit.wikimedia.org/r/911943

Change 912818 merged by Jcrespo:

[operations/software/mediabackups@master] Update indexes for latest queries needed for mediabackups

https://gerrit.wikimedia.org/r/912818

Change 912819 merged by Jcrespo:

[operations/software/mediabackups@master] Add functionality to detect last uploaded time for backup start

https://gerrit.wikimedia.org/r/912819

Change 912935 merged by Jcrespo:

[operations/software/mediabackups@master] recentuploads: Set custom headers for querying the mediawiki api

https://gerrit.wikimedia.org/r/912935

This unexpected behavior bit me:

>>> cursor.execute("SELECT STR_TO_DATE('20230216083531', '%%Y%%m%%d%%H%%i%%s')", tuple());
1
>>> cursor.fetchall()
((datetime.datetime(2023, 2, 16, 8, 35, 31),),)
>>> cursor.execute("SELECT STR_TO_DATE('20230216083531', '%%Y%%m%%d%%H%%i%%s')");
1
>>> cursor.fetchall()
((None,),)

Change 914321 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Fix bug by which %% wasn't adequately escaped on sql queries

https://gerrit.wikimedia.org/r/914321

Change 914321 merged by Jcrespo:

[operations/software/mediabackups@master] Fix bug by which %% wasn't adequately escaped on sql queries

https://gerrit.wikimedia.org/r/914321

Change 922830 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Add tmpdir removal, now that upload is stable

https://gerrit.wikimedia.org/r/922830

Change 922907 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Increase unit test coverage for File, MySQLMedia and MySQLMetadata

https://gerrit.wikimedia.org/r/922907

Change 922830 merged by Jcrespo:

[operations/software/mediabackups@master] Add tmpdir removal, now that upload is stable

https://gerrit.wikimedia.org/r/922830

Change 922907 merged by Jcrespo:

[operations/software/mediabackups@master] Increase unit test coverage for File, MySQLMedia and MySQLMetadata

https://gerrit.wikimedia.org/r/922907

Change 978042 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackup: Update mediabackups to use new TLS CA and prepare for 0.2.0

https://gerrit.wikimedia.org/r/978042

Change 978133 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Migrate TLS configuration to separate file and prepare for puppet call

https://gerrit.wikimedia.org/r/978133

Change 978643 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Prepare for 0.2.0 release

https://gerrit.wikimedia.org/r/978643

Change 978042 merged by Jcrespo:

[operations/puppet@production] mediabackup: Update mediabackups to use new TLS CA and prepare for 0.2.0

https://gerrit.wikimedia.org/r/978042

Change 979367 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackups: Fix mw_db.cnf template yaml justification

https://gerrit.wikimedia.org/r/979367

Change 979367 merged by Jcrespo:

[operations/puppet@production] mediabackups: Fix mw_db.cnf template yaml justification

https://gerrit.wikimedia.org/r/979367

This is now ready for Puppet 7 and the new CA- only pending thing is to add it to configuration. (this was already done)

Also make sure sigint is treated correctly and puppetize systemd timers / daemons.

Change 978133 merged by Jcrespo:

[operations/software/mediabackups@master] Migrate TLS configuration to separate file and prepare for puppet call

https://gerrit.wikimedia.org/r/978133

Change 978643 merged by Jcrespo:

[operations/software/mediabackups@master] Prepare for 0.2.0 release

https://gerrit.wikimedia.org/r/978643

This is the main kind of unhandled error I have: https://commons.wikimedia.org/wiki/File:3_castillos.jpg

2 identical files uploaded in the same second- and not having any unique identifier makes it impossible to update its metadata.

Moreover, I concluded that Wikimedia Commons sha1 hashes seem to have an invalid format, so I cannot compare the files either.

This was done long time ago, only the TLS part and puppet/MySQL changes/updates was done last quarter.