Page MenuHomePhabricator
Paste P30677

test run of deletions and recoveries with the extra messages
ActivePublic
Actions

Authored by jcrespo on Jun 30 2022, 4:50 PM.
Tags
None
Referenced Files
F35289851: Screenshot_20220630_185109.png
Jun 30 2022, 4:52 PM
F35289850: Screenshot_20220630_185038.png
Jun 30 2022, 4:52 PM
F35289849: test run of deletions and recoveries with the extra messages
Jun 30 2022, 4:50 PM
Subscribers
None
root@ms-backup1001:/srv$ delete-media-file
This is a dry run deletion- no actual file or metadata will be affected, even if the script will follow the same steps and confirmation.
[2022-06-30 16:49:12,575] INFO:deletion Starting an interactive deletion session
Wiki for deletion [default: commonswiki]: testwiki
0) Title of the file on upload (or after rename)
1) sha1sum hash of the file contents, in hexadecimal
2) sha1sum hash of the file contents, in Mediawiki's base 36
3) Original container name and full path as was stored on Swift
4) sha256sum hash of the file contents, in hexadecimal
5) Exact date of the original file upload, as registered on the metadata
6) Exact date of the latest file archival, as registered on the metadata
7) Exact date of the latest file deletion, as registered on the metadata
Chose method to identify the media file to deletion (0-7) [default: 0]: 0
Title (spaces will be converted to underscores, first letter normally in uppercase): Tiff_test.tiff
This is the list of 1 files found with the given criteria:
0)
wiki | testwiki
title | Tiff_test.tiff
production_container | wikipedia-test-local-public
production_path | 9/9b/Tiff_test.tiff
sha1 | f5bc9986a1391b7d371ef980650c9cac27b194b9
sha256 | 1ed815d5ea0073ede2bf18d4c83a3590f2e5918572f90e3cbe9b4eb96349865c
size | 227712
production_status | public
production_url | https://upload.wikimedia.org/wikipedia/test/9/9b/Tiff_test.tiff
upload_date | 2021-09-24 10:35:06
archive_date | None
delete_date | None
backup_status | backedup
backup_date | 2021-12-14 21:53:26
backup_location | https://backup1004.eqiad.wmnet:9000
backup_container | mediabackups
backup_path | testwiki/1ed/1ed815d5ea0073ede2bf18d4c83a3590f2e5918572f90e3cbe9b4eb96349865c
Executing deletion in dry mode, so files will not be actually deleted
Confirm deletion of 1 file(s)? (y/n) y
[2022-06-30 16:49:27,643] INFO:deletion About to delete 1 files...
[2022-06-30 16:49:27,922] INFO:deletion All files were queried from production and none were found publicly available: https://upload.wikimedia.org/wikipedia/test/9/9b/Tiff_test.tiff
[2022-06-30 16:49:27,923] INFO:deletion Attempting to delete "testwiki/1ed/1ed815d5ea0073ede2bf18d4c83a3590f2e5918572f90e3cbe9b4eb96349865c" from "https://backup1004.eqiad.wmnet:9000"
[2022-06-30 16:49:27,953] WARNING:deletion Not actually deleting testwiki/1ed/1ed815d5ea0073ede2bf18d4c83a3590f2e5918572f90e3cbe9b4eb96349865c from https://backup1004.eqiad.wmnet:9000 because we are in DRY MODE - but otherwise succesful.
[2022-06-30 16:49:27,953] INFO:deletion 1 out of 1 files were successfully deleted from backup storage.
[2022-06-30 16:49:27,959] INFO:backup Reading foreign key values for the files table from the database
[2022-06-30 16:49:28,132] WARNING:deletion Metadata update completed correctly, but database not actually touched because we are in DRY MODE!
Remember to perform the same deletion on the other datacenter too (only data from one site was affected for the current session!).
[2022-06-30 16:49:28,132] INFO:deletion Finishing the interactive deletion session
✔️
root@ms-backup1001:~$ delete-media-file
This is a dry run deletion- no actual file or metadata will be affected, even if the script will follow the same steps and confirmation.
[2022-06-30 16:44:11,150] INFO:deletion Starting an interactive deletion session
Wiki for deletion [default: commonswiki]:
0) Title of the file on upload (or after rename)
1) sha1sum hash of the file contents, in hexadecimal
2) sha1sum hash of the file contents, in Mediawiki's base 36
3) Original container name and full path as was stored on Swift
4) sha256sum hash of the file contents, in hexadecimal
5) Exact date of the original file upload, as registered on the metadata
6) Exact date of the latest file archival, as registered on the metadata
7) Exact date of the latest file deletion, as registered on the metadata
Chose method to identify the media file to deletion (0-7) [default: 0]:
Title (spaces will be converted to underscores, first letter normally in uppercase): Baffled_LIGO_Scientists.jpg
This is the list of 1 files found with the given criteria:
0)
wiki | commonswiki
title | Baffled_LIGO_Scientists.jpg
production_container | wikipedia-commons-local-public.cf
production_path | c/cf/Baffled_LIGO_Scientists.jpg
sha1 | 848d03eff611ba1a54479b8e78041d407b099a40
sha256 | c515fa40c3c8b95b5f6d44b9bf186f91ec8cc4c57c5939919b1c8d918b0b6466
size | 11757543
production_status | public
production_url | https://upload.wikimedia.org/wikipedia/commons/c/cf/Baffled_LIGO_Scientists.jpg
upload_date | 2019-12-02 13:48:09
archive_date | None
delete_date | None
backup_status | backedup
backup_date | 2021-09-04 06:09:24
backup_location | https://backup1007.eqiad.wmnet:9000
backup_container | mediabackups
backup_path | commonswiki/c51/c515fa40c3c8b95b5f6d44b9bf186f91ec8cc4c57c5939919b1c8d918b0b6466
Executing deletion in dry mode, so files will not be actually deleted
Confirm deletion of 1 file(s)? (y/n) y
[2022-06-30 16:44:21,944] INFO:deletion About to delete 1 files...
[2022-06-30 16:44:22,127] ERROR:deletion We got an HTTP status code of 200 when we tried querying https://upload.wikimedia.org/wikipedia/commons/c/cf/Baffled_LIGO_Scientists.jpg from production, we expected a 404.
[2022-06-30 16:44:22,127] ERROR:deletion Aborting deletion process.
❌ root@ms-backup1001:~$ delete-media-file --execute
An actual backup file deletion will be performed- these actions are undoable- although you will be given the chance of a final confirmation.
[2022-06-30 16:44:38,671] INFO:deletion Starting an interactive deletion session
Wiki for deletion [default: commonswiki]:
0) Title of the file on upload (or after rename)
1) sha1sum hash of the file contents, in hexadecimal
2) sha1sum hash of the file contents, in Mediawiki's base 36
3) Original container name and full path as was stored on Swift
4) sha256sum hash of the file contents, in hexadecimal
5) Exact date of the original file upload, as registered on the metadata
6) Exact date of the latest file archival, as registered on the metadata
7) Exact date of the latest file deletion, as registered on the metadata
Chose method to identify the media file to deletion (0-7) [default: 0]:
Title (spaces will be converted to underscores, first letter normally in uppercase): Baffled_LIGO_Scientists.jpg
This is the list of 1 files found with the given criteria:
0)
wiki | commonswiki
title | Baffled_LIGO_Scientists.jpg
production_container | wikipedia-commons-local-public.cf
production_path | c/cf/Baffled_LIGO_Scientists.jpg
sha1 | 848d03eff611ba1a54479b8e78041d407b099a40
sha256 | c515fa40c3c8b95b5f6d44b9bf186f91ec8cc4c57c5939919b1c8d918b0b6466
size | 11757543
production_status | public
production_url | https://upload.wikimedia.org/wikipedia/commons/c/cf/Baffled_LIGO_Scientists.jpg
upload_date | 2019-12-02 13:48:09
archive_date | None
delete_date | None
backup_status | backedup
backup_date | 2021-09-04 06:09:24
backup_location | https://backup1007.eqiad.wmnet:9000
backup_container | mediabackups
backup_path | commonswiki/c51/c515fa40c3c8b95b5f6d44b9bf186f91ec8cc4c57c5939919b1c8d918b0b6466
WARNING! File deletion cannot be reverted
Confirm deletion of 1 file(s)? (y/n) n
[2022-06-30 16:44:48,960] WARNING:deletion deletion aborted due to user input
❌
root@ms-backup1001:/srv$ restore-media-file
[2022-06-30 16:46:05,231] INFO:recovery Starting an interactive recovery session
Wiki for recovery [default: commonswiki]:
0) Title of the file on upload (or after rename)
1) sha1sum hash of the file contents, in hexadecimal
2) sha1sum hash of the file contents, in Mediawiki's base 36
3) Original container name and full path as was stored on Swift
4) sha256sum hash of the file contents, in hexadecimal
5) Exact date of the original file upload, as registered on the metadata
6) Exact date of the latest file archival, as registered on the metadata
7) Exact date of the latest file deletion, as registered on the metadata
Chose method to identify the media file to recovery (0-7) [default: 0]:
Title (spaces will be converted to underscores, first letter normally in uppercase): Baffled_LIGO_Scientists.jpg
This is the list of 1 files found with the given criteria:
0)
wiki | commonswiki
title | Baffled_LIGO_Scientists.jpg
production_container | wikipedia-commons-local-public.cf
production_path | c/cf/Baffled_LIGO_Scientists.jpg
sha1 | 848d03eff611ba1a54479b8e78041d407b099a40
sha256 | c515fa40c3c8b95b5f6d44b9bf186f91ec8cc4c57c5939919b1c8d918b0b6466
size | 11757543
production_status | public
production_url | https://upload.wikimedia.org/wikipedia/commons/c/cf/Baffled_LIGO_Scientists.jpg
upload_date | 2019-12-02 13:48:09
archive_date | None
delete_date | None
backup_status | backedup
backup_date | 2021-09-04 06:09:24
backup_location | https://backup1007.eqiad.wmnet:9000
backup_container | mediabackups
backup_path | commonswiki/c51/c515fa40c3c8b95b5f6d44b9bf186f91ec8cc4c57c5939919b1c8d918b0b6466
Confirm recovery of 1 file(s)? (y/n) y
[2022-06-30 16:46:15,329] INFO:recovery About to recover 1 files...
[2022-06-30 16:46:15,488] INFO:recovery Attempting to recover "commonswiki/c51/c515fa40c3c8b95b5f6d44b9bf186f91ec8cc4c57c5939919b1c8d918b0b6466" from "https://backup1007.eqiad.wmnet:9000" into "Baffled_LIGO_Scientists.jpg"...
[2022-06-30 16:46:15,972] INFO:recovery "commonswiki/c51/c515fa40c3c8b95b5f6d44b9bf186f91ec8cc4c57c5939919b1c8d918b0b6466" was successfully downloaded from "https://backup1007.eqiad.wmnet:9000" and saved into "Baffled_LIGO_Scientists.jpg"
[2022-06-30 16:46:15,972] INFO:recovery 1 out of 1 files were successfully written to the local filesystem.
Remember to perform the same recovery on the other datacenter too (only data from one site was affected for the current session!).
[2022-06-30 16:46:15,977] INFO:recovery Finishing the interactive recovery session
✔️
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL