Page MenuHomePhabricator

[ftpsync@sodium] ERROR: rsync errors
Closed, ResolvedPublic

Description

Got a few emails to root with the following as attachments.

Mar 27 08:54:26 sodium ftpsync[12462]: Mirrorsync start
Mar 27 08:54:26 sodium ftpsync[12462]: We got pushed from 2001:4f8:1:c::16
Mar 27 08:54:26 sodium ftpsync[12462]: Running mirrorsync, update is required, /srv/mirrors/debian//Archive-Update-Required-mirrors.wikimedia.org exists
Mar 27 08:54:26 sodium ftpsync[12462]: Running stage1: rsync --filter=exclude_/Archive-Update-in-Progress-mirrors.wikimedia.org --filter=protect_/Archive-Update-in-Progress-mirrors.wikimedia.org --filter=exclude_/Archive-Update-Required-mirrors.wikimedia.org --filter=protect_/Archive-Update-Required-mirrors.wikimedia.org --filter=exclude_/project/trace/mirrors.wikimedia.org --filter=protect_/project/trace/mirrors.wikimedia.org --filter=exclude_/project/trace/mirrors.wikimedia.org-stage1 --filter=protect_/project/trace/mirrors.wikimedia.org-stage1 --filter=exclude_/project/trace/_hierarchy --filter=protect_/project/trace/_hierarchy --filter=exclude_/project/trace/_traces --filter=protect_/project/trace/_traces --filter=include_/project/ --filter=protect_/project/ --filter=include_/project/trace/ --filter=protect_/project/trace/ --filter=include_/project/trace/* --exclude=.~tmp~/ mirrors.wikimedia.org@syncproxy2.wna.debian.org::debian /srv/mirrors/debian/ --bwlimit=0 -prltvHSB8192 --safe-links --chmod=D755,F644 --timeout 3600 --stats --no-human-readable --include=*.diff/ --exclude=*.diff/Index --exclude=Packages* --exclude=Sources* --exclude=Release* --exclude=InRelease --include=i18n/by-hash --exclude=i18n/* --exclude=ls-lR*
Mar 27 09:18:51 sodium ftpsync[12462]: Back from rsync with returncode 23
Mar 27 09:18:51 sodium ftpsync[12462]: Latest recorded rsync transfer speed: 7421 KB/s
Mar 27 09:18:51 sodium ftpsync[12462]: Mirrorsync done
rsync: failed to set times on "/srv/mirrors/debian/dists/sid/main/Contents-udeb-arm64.gz": Operation not permitted (1)
rsync: failed to set times on "/srv/mirrors/debian/dists/sid/main/Contents-udeb-ppc64el.gz": Operation not permitted (1)
rsync: failed to set times on "/srv/mirrors/debian/dists/sid/main/Contents-udeb-s390x.gz": Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.2]

CCing Ariel and Daniel based on T224706

Event Timeline

ayounsi triaged this task as Medium priority.Mar 27 2020, 9:28 AM
ayounsi created this task.

For context:

  • exit code 23 on rsync is Partial transfer due to error
  • there are root owned files and symlinks in /srv/mirrors/debian/dists/sid/main
  • the crontab that runs the rsync (AFACIT) is run as the mirror user

Yesterday Icinga was alerting about failed ftpsync on sodium and to debug i ran it manually which then made the original alert recover. But doing that it looks like i messed up permissions causing the secondary issue.

Mentioned in SAL (#wikimedia-operations) [2020-03-27T09:36:15Z] <mutante> sodium fixing root owned files in /srv/mirrors/debian to be owned by mirror:mirror (T248660)

Mentioned in SAL (#wikimedia-operations) [2020-03-27T09:37:56Z] <mutante> sodium - running ftpsync as user mirror (T248660)

I ran

find /srv/mirrors/debian/ -user root -exec chown mirror:mirror {} \;

and afterwards

sudo -u mirror ftpsync

and saw no errors.

Mentioned in SAL (#wikimedia-operations) [2020-03-27T10:03:39Z] <mutante> sodium - find /srv/mirrors/debian/ -user root -exec chown -h mirror:mirror {} \; (-h to also fix symbolic links); sudo -u mirror ftpsync (T248660)

After the last run of ftpsync there was no more error.