Page MenuHomePhabricator
Create Task
Maniphest T438961

rclone cannot handle objects with 0x201B / SINGLE HIGH-REVERSED-9 QUOTATION MARK / ‛ in the name
Open, Needs TriagePublic
Actions

Description

We've been seeing rclone errors like this in the weekly sync job:

P96481 rclone output 2026-09-20
12026-09-21T03:00:00.208552+00:00 ms-be1069 systemd[1]: Starting swift_rclone_sync.service - Swift rclone-based sync...
22026-09-21T04:19:43.100314+00:00 ms-be1069 swift-rclone-sync[2345604]: ERROR : wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛‛_al-ḤIKĀYĀT_VA_LAVĀMI‛‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg: Failed to copy: failed to open source object: Object Not Found
32026-09-21T04:20:02.381540+00:00 ms-be1069 swift-rclone-sync[2345604]: ERROR : wikipedia-commons-local-public.4d/4/4d/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(142_of_580).jpg: Failed to copy: failed to open source object: Object Not Found
42026-09-21T05:41:19.977651+00:00 ms-be1069 swift-rclone-sync[2345604]: ERROR : wikipedia-commons-local-public.a9/a/a9/Ğāmi‛‛_al-tavārīḫ._Rašīd_al-Dīn_Fazl-ullāh_Hamadānī_-_btv1b8427170s_(182_of_597).jpg: Failed to copy: failed to open source object: Object Not Found
52026-09-21T06:21:44.793240+00:00 ms-be1069 swift-rclone-sync[2345604]: ERROR : wikipedia-commons-local-public.c7/c/c7/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(248_of_580).jpg: Failed to copy: failed to open source object: Object Not Found
62026-09-21T07:02:17.318475+00:00 ms-be1069 swift-rclone-sync[2345604]: ERROR : wikipedia-commons-local-public.fa/f/fa/Indiana_Senate_journal,_1867_-_DPLA_-_444ee9c74f7bc49801337a1c5e099dea_(page_455).jpg: Failed to copy: failed to open source object: Object Not Found
72026-09-21T08:46:16.921383+00:00 ms-be1069 swift-rclone-sync[2345604]: ERROR : Attempt 1/1 failed with 5 errors and: failed to open source object: Object Not Found
82026-09-21T08:46:16.921891+00:00 ms-be1069 swift-rclone-sync[2345604]: Failed to sync with 5 errors: last error was: failed to open source object: Object Not Found
92026-09-21T08:46:17.204085+00:00 ms-be1069 systemd[1]: swift_rclone_sync.service: Main process exited, code=exited, status=1/FAILURE
102026-09-21T08:46:17.204336+00:00 ms-be1069 systemd[1]: swift_rclone_sync.service: Failed with result 'exit-code'.
112026-09-21T08:46:17.204695+00:00 ms-be1069 systemd[1]: Failed to start swift_rclone_sync.service - Swift rclone-based sync.
122026-09-21T08:46:17.205302+00:00 ms-be1069 systemd[1]: swift_rclone_sync.service: Consumed 13h 44min 16.409s CPU time, 42G memory peak.

Investigation shows that While, e.g. wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛‛_al-ḤIKĀYĀT_VA_LAVĀMI‛‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg doesn't exist, wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛_al-ḤIKĀYĀT_VA_LAVĀMI‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg does (albeit only in eqiad, hence rclone trying to copy it across).

The problem is that rclone is duplicating the ‛ character, as we can demonstrate by with a smaller reproducer:

P96495 rclone bug reproducer
1mvernon@ms-backup1003:~$ s3cmd ls s3://mcv21test
22024-08-20 13:10 8286485 s3://mcv21test/1-month-old_kittens_32.jpg
32024-08-20 14:21 224878555 s3://mcv21test/Apollo_Butterfly_under_the_microscope,_4k.webm
42024-08-20 13:10 539645 s3://mcv21test/Café_de_Flore.jpg
52024-08-20 12:56 20144 s3://mcv21test/Gnome-edit-delete.svg
62026-09-22 15:09 1484068 s3://mcv21test/Post‛Box.jpg
7
8mvernon@ms-backup1003:~$ ./rclone ls apus:
9 8286485 mcv21test/1-month-old_kittens_32.jpg
10224878555 mcv21test/Apollo_Butterfly_under_the_microscope,_4k.webm
11 539645 mcv21test/Café_de_Flore.jpg
12 20144 mcv21test/Gnome-edit-delete.svg
13 1484068 mcv21test/Post‛‛Box.jpg

This has been reported to rclone upstream, and is a bug in the encoding system. Use of the "Raw" encoding option disables encoding at all, and enables these objects to be successfully retrieved eg

mvernon@ms-backup1003:~/junk$ ../rclone copy --s3-encoding Raw --local-encoding Raw apus:mcv21test/Post‛Box.jpg .
mvernon@ms-backup1003:~/junk$ ls
Post‛Box.jpg

Unfortunately the "Raw" encoder was introduced in rclone v1.68.0, and the version in trixie is 1.60.1. There is currently no version of rclone in testing, as the version in unstable (1.69.3) has a number of security issues fixed in upstream (which is now at 1.75.1); I don't think any of them are particularly impactful for our use case. But it does mean that "update the version of rclone in prod" isn't entirely straightforward.

But we could temporarily use the rclone 1.75.1 binary from upstream (which I used in testing) to copy these 4 impacted objects to codfw from eqiad, which would avoid the risk of their being deleted by rclone once codfw is primary DC.

Related Objects

Event Timeline

MatthewVernon created this task.Wed, Sep 23, 1:45 PM
Herald added a subscriber: Aklapper. · View Herald TranscriptWed, Sep 23, 1:45 PM
Stashbot added a comment.Wed, Sep 23, 3:32 PM

Mentioned in SAL (#wikimedia-operations) [2026-09-23T15:32:05Z] <Emperor> rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛_al-ḤIKĀYĀT_VA_LAVĀMI‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg' codfw:wikipedia-commons-local-public.41/4/41 T438961

Stashbot added a comment.Wed, Sep 23, 3:35 PM

Mentioned in SAL (#wikimedia-operations) [2026-09-23T15:35:23Z] <Emperor> rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.4d/4/4d/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(142_of_580).jpg' codfw:wikipedia-commons-local-public.4d/4/4d T438961

Stashbot added a comment.Wed, Sep 23, 3:38 PM

Mentioned in SAL (#wikimedia-operations) [2026-09-23T15:38:26Z] <Emperor> rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.a9/a/a9/Ğāmi‛_al-tavārīḫ._Rašīd_al-Dīn_Fazl-ullāh_Hamadānī_-_btv1b8427170s_(182_of_597).jpg' codfw:wikipedia-commons-local-public.a9/a/a9/ T438961

Stashbot added a comment.Wed, Sep 23, 3:41 PM

Mentioned in SAL (#wikimedia-operations) [2026-09-23T15:41:38Z] <Emperor> rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.c7/c/c7/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(248_of_580).jpg' codfw:wikipedia-commons-local-public.c7/c/c7 T438961

MatthewVernon added a comment.Wed, Sep 23, 3:47 PM

I replicated the 4 sad items from eqiad to codfw, thus:

  1. Edit /etc/swift/rclone.conf to add encoding = Raw to both config stanzas
  2. copy the rclone 1.75.1 binary to ~
  3. use ./rclone copy to copy each object in turn, checking swift stat in both clusters before and after

rclone runes for each run were logged to SAL.

So that's the immediate issue dealt with, but the question of how to upgrade to a fixed rclone remains open - and when we have that, we'll want to edit the rclone.conf template to add encoding = Raw (we can't do that now, because the version we have installed will reject it as invalid).

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits