Page MenuHomePhabricator

When "scap pull" does a (slow) CDB rebuild, it should tell me that that's what it's doing
Closed, ResolvedPublic

Description

catrope@mwdebug1002:~$ scap pull
23:06:46 Copying to mwdebug1002.eqiad.wmnet from deployment.eqiad.wmnet
23:06:46 Started rsync common
cannot delete non-empty directory: php-1.29.0-wmf.8/cache/l10n
cannot delete non-empty directory: php-1.29.0-wmf.8/cache/l10n
cannot delete non-empty directory: php-1.29.0-wmf.8/cache
cannot delete non-empty directory: php-1.29.0-wmf.8/cache
cannot delete non-empty directory: php-1.29.0-wmf.8
23:06:52 Finished rsync common (duration: 00m 06s)
[3 minutes of blinking cursor]
catrope@mwdebug1002:~$

Looking at top in tree mode, it looks like the 3 minutes of silence were caused by scap cdb-rebuild --no-progress. It's one thing that progress reporting is explicitly disabled, but it's a different issue altogether that it doesn't even tell me that it's doing a CDB rebuild, or at what timestamps it started and finished doing this. Instead, it just prints no output for 3 minutes, and then exits.

Event Timeline

Catrope created this task.Apr 4 2017, 11:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 4 2017, 11:15 PM
thcipriani triaged this task as Medium priority.Apr 24 2017, 3:44 PM
thcipriani moved this task from Needs triage to Debt on the Scap board.
thcipriani added a subscriber: thcipriani.

I ran into this when testing the new deployment server (naos) the other day and it was disconcerting.

hashar added a subscriber: hashar.Jun 14 2017, 1:30 PM

Still happening eg today:

1zfilipin@mwdebug1002:~$ scap pull
213:14:27 Copying to mwdebug1002.eqiad.wmnet from tin.eqiad.wmnet
313:14:27 Started rsync common
4cannot delete non-empty directory: php-1.30.0-wmf.3/cache/l10n
5cannot delete non-empty directory: php-1.30.0-wmf.3/cache/l10n
6cannot delete non-empty directory: php-1.30.0-wmf.3/cache
7cannot delete non-empty directory: php-1.30.0-wmf.3/cache
8cannot delete non-empty directory: php-1.30.0-wmf.3
9cannot delete non-empty directory: php-1.30.0-wmf.1/cache/l10n
10cannot delete non-empty directory: php-1.30.0-wmf.2/cache/l10n
1113:14:32 Finished rsync common (duration: 00m 05s)

hashar closed this task as Resolved.Jun 14 2017, 1:37 PM
hashar claimed this task.

Logs are back :-}

hashar reopened this task as Open.Jun 14 2017, 1:37 PM
hashar removed hashar as the assignee of this task.

Wrong task

demon added a subscriber: demon.Jul 5 2017, 4:44 PM

Basically this problem goes away with T99740.

mmodell edited projects, added Scap (Tech Debt Sprint FY201718-Q2); removed Scap.
hashar removed a subscriber: hashar.Oct 2 2017, 7:20 PM

This is annoying to experienced deployers and at least surprising for new deployers, if not scary.

#wikimedia-operations during EU SWAT today
gilles
is it normal for "scap pull" on mwdebug1002 to hang after "14:12:28 Finished rsync common (duration: 00m 03s)"?