Page MenuHomePhabricator

cannot delete non-empty directory: php-1.25wmf14/cache/l10n
Closed, ResolvedPublic

Description

Today during deployment, this happened:

When cleaning up old versions of mediawiki on tin, I ran into files owned by l10nupdate, which I could not remove because of permissions.

I used sudo -u l10nupdate, then manually removed those files... But afterwards, when running scap, I got this:

21:23:57 Started scap: testwiki to php-1.25wmf19 and rebuild l10n cache
21:23:57 Copying to tin.eqiad.wmnet from tin.eqiad.wmnet
21:23:57 Started rsync common
cannot delete non-empty directory: php-1.25wmf14/cache/l10n
cannot delete non-empty directory: php-1.25wmf14/cache/l10n
cannot delete non-empty directory: php-1.25wmf14/cache
cannot delete non-empty directory: php-1.25wmf14/cache
cannot delete non-empty directory: php-1.25wmf14
21:26:04 Finished rsync common (duration: 02m 06s)

So:

  1. it looks like l10nupdate is creating files with the wrong permissions, members of wikidev can't delete them.
  2. when scap runs rsync, it isn't able to override?

Not sure what I need to do to fix this but I have a feeling future scap runs and going to complain about the same files that can't be removed.

@bd808: do you have an insight into this one? it's very low priority so don't bother yourself too much over it.

Event Timeline

mmodell created this task.Feb 25 2015, 9:54 PM
mmodell raised the priority of this task from to Low.
mmodell updated the task description. (Show Details)
mmodell added subscribers: mmodell, bd808.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 25 2015, 9:54 PM

Expected behavior with the process I dreamed up for deleting branches:

https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Remove_left_over_files_from_expired_branches

Because we don't rsync the localization CDB files during scap, some files will be left behind after scap runs for expired branches that were removed. Clean them up across the cluster using dsh

dsh -g mediawiki-installation -M -F 40 -- 'sudo -u mwdeploy -- rm -rf /srv/mediawiki/php-STALE_VERSION'

when I do the dsh command I get a permission denied message.

mmodell closed this task as Invalid.Feb 25 2015, 11:36 PM
mmodell claimed this task.
greg moved this task from To Triage to Done on the Deployments board.Mar 11 2015, 8:48 PM
hashar reopened this task as Open.Oct 26 2016, 1:35 PM
hashar added a subscriber: hashar.

That is happening again with scap 3.3.0-1 on mw2098.codfw.wmnet:

hashar@mw2098:~$ scap pull
13:30:13 Copying to mw2098.codfw.wmnet from deployment.codfw.wmnet
13:30:13 Started rsync common
cannot delete non-empty directory: php-1.28.0-wmf.17/cache/l10n
cannot delete non-empty directory: php-1.28.0-wmf.17/cache/l10n
cannot delete non-empty directory: php-1.28.0-wmf.17/cache
cannot delete non-empty directory: php-1.28.0-wmf.17/cache
cannot delete non-empty directory: php-1.28.0-wmf.17
cannot delete non-empty directory: php-1.28.0-wmf.16/cache/l10n
cannot delete non-empty directory: php-1.28.0-wmf.16/cache/l10n
cannot delete non-empty directory: php-1.28.0-wmf.16/cache
cannot delete non-empty directory: php-1.28.0-wmf.16/cache
cannot delete non-empty directory: php-1.28.0-wmf.16
13:33:21 Finished rsync common (duration: 03m 07s)
mmodell removed mmodell as the assignee of this task.Oct 26 2016, 3:57 PM
mmodell moved this task from Done to Backlog (Tech) on the Deployments board.
mmodell moved this task from Done to Backlog on the Release-Engineering-Team board.
mmodell edited projects, added Scap (Scap3-MediaWiki-MVP); removed Deployments.
mmodell moved this task from Needs Triage to Debt on the Scap (Scap3-MediaWiki-MVP) board.
demon closed this task as Resolved.Mar 14 2017, 5:51 PM
demon claimed this task.
demon added a subscriber: demon.

This should be fixed with scap clean. It'll occasionally happen on the deploy masters themselves (not the targets) but I think we fixed that too?