Page MenuHomePhabricator

investigate abandoned cirrus search jobs in the jobqueue
Open, NormalPublic

Description

There are an unusual number of cirrussearch jobs abandoned in the job queue. Will attach a list of wikis with their stats once that finishes running

Event Timeline

EBernhardson updated the task description. (Show Details)
EBernhardson raised the priority of this task from to Needs Triage.
EBernhardson added a subscriber: EBernhardson.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 15 2015, 6:59 PM
EBernhardson set Security to None.
Restricted Application added a project: Discovery. · View Herald TranscriptDec 15 2015, 6:59 PM

Current abandoned jobs report, first with only wikis with more than 100 abandoned jobs:

arwiki:  cirrusSearchIncomingLinkCount: 0 queued; 504 claimed (0 active, 504 abandoned); 16 delayed
arwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 505 claimed (0 active, 505 abandoned); 0 delayed
cawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 216 claimed (0 active, 216 abandoned); 0 delayed
cebwiki:  cirrusSearchIncomingLinkCount: 0 queued; 277 claimed (0 active, 277 abandoned); 0 delayed
cebwiki:  cirrusSearchLinksUpdatePrioritized: 4 queued; 157 claimed (3 active, 154 abandoned); 0 delayed
commonswiki:  cirrusSearchIncomingLinkCount: 8 queued; 334 claimed (0 active, 334 abandoned); 57 delayed
commonswiki:  cirrusSearchLinksUpdate: 0 queued; 7501 claimed (1285 active, 6216 abandoned); 0 delayed
commonswiki:  cirrusSearchLinksUpdatePrioritized: 5 queued; 61770 claimed (97 active, 61673 abandoned); 0 delayed
cswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 339 claimed (0 active, 339 abandoned); 0 delayed
dawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 114 claimed (0 active, 114 abandoned); 0 delayed
dewiki:  cirrusSearchIncomingLinkCount: 0 queued; 395 claimed (3 active, 392 abandoned); 31 delayed
enwiki:  cirrusSearchIncomingLinkCount: 3 queued; 3109 claimed (6 active, 3103 abandoned); 245 delayed
enwiki:  cirrusSearchLinksUpdatePrioritized: 18 queued; 26665 claimed (23 active, 26642 abandoned); 0 delayed
enwiki:  cirrusSearchOtherIndex: 1 queued; 306 claimed (0 active, 306 abandoned); 0 delayed
enwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1635 claimed (0 active, 1635 abandoned); 0 delayed
eswiki:  cirrusSearchIncomingLinkCount: 0 queued; 331 claimed (0 active, 331 abandoned); 77 delayed
eswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2306 claimed (0 active, 2306 abandoned); 0 delayed
fawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 358 claimed (0 active, 358 abandoned); 0 delayed
fiwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 188 claimed (0 active, 188 abandoned); 0 delayed
frwiki:  cirrusSearchIncomingLinkCount: 0 queued; 1279 claimed (0 active, 1279 abandoned); 271 delayed
frwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 14102 claimed (0 active, 14102 abandoned); 0 delayed
huwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 468 claimed (0 active, 468 abandoned); 0 delayed
idwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 7158 claimed (0 active, 7158 abandoned); 0 delayed
itwiki:  cirrusSearchIncomingLinkCount: 0 queued; 432 claimed (0 active, 432 abandoned); 2 delayed
itwiki:  cirrusSearchLinksUpdatePrioritized: 2 queued; 609 claimed (1 active, 608 abandoned); 0 delayed
jawiki:  cirrusSearchLinksUpdatePrioritized: 9 queued; 1212 claimed (0 active, 1212 abandoned); 0 delayed
kowiki:  cirrusSearchLinksUpdatePrioritized: 1 queued; 313 claimed (0 active, 313 abandoned); 0 delayed
metawiki:  cirrusSearchIncomingLinkCount: 0 queued; 116 claimed (0 active, 116 abandoned); 2 delayed
metawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 133 claimed (0 active, 133 abandoned); 0 delayed
mnwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1504 claimed (0 active, 1504 abandoned); 0 delayed
nlwiki:  cirrusSearchIncomingLinkCount: 0 queued; 133 claimed (0 active, 133 abandoned); 1 delayed
nlwiki:  cirrusSearchLinksUpdatePrioritized: 2 queued; 827 claimed (1 active, 826 abandoned); 0 delayed
nowiki:  cirrusSearchLinksUpdatePrioritized: 2 queued; 22992 claimed (0 active, 22992 abandoned); 0 delayed
plwiki:  cirrusSearchIncomingLinkCount: 0 queued; 161 claimed (0 active, 161 abandoned); 33 delayed
plwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 169 claimed (1 active, 168 abandoned); 0 delayed
plwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1452 claimed (0 active, 1452 abandoned); 0 delayed
ptwiki:  cirrusSearchIncomingLinkCount: 0 queued; 152 claimed (0 active, 152 abandoned); 40 delayed
ptwiki:  cirrusSearchLinksUpdatePrioritized: 1 queued; 1348 claimed (0 active, 1348 abandoned); 0 delayed
rowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 122 claimed (0 active, 122 abandoned); 0 delayed
ruwiki:  cirrusSearchIncomingLinkCount: 0 queued; 496 claimed (0 active, 496 abandoned); 33 delayed
skwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 112 claimed (0 active, 112 abandoned); 0 delayed
srwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 244 claimed (0 active, 244 abandoned); 0 delayed
thwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 111 claimed (0 active, 111 abandoned); 0 delayed
trwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 237 claimed (0 active, 237 abandoned); 0 delayed
ukwiki:  cirrusSearchIncomingLinkCount: 0 queued; 132 claimed (0 active, 132 abandoned); 26 delayed
ukwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1144 claimed (0 active, 1144 abandoned); 0 delayed
urwiki:  cirrusSearchIncomingLinkCount: 0 queued; 139 claimed (0 active, 139 abandoned); 6 delayed
urwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 103 claimed (1 active, 102 abandoned); 0 delayed
wikidatawiki:  cirrusSearchIncomingLinkCount: 1 queued; 648 claimed (1 active, 647 abandoned); 51 delayed
wikidatawiki:  cirrusSearchLinksUpdatePrioritized: 3 queued; 3036 claimed (0 active, 3036 abandoned); 0 delayed
zhwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2161 claimed (5 active, 2156 abandoned); 0 delayed

Here is the full report for all wikis with 1 or more abandoned jobs:

acewiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
afwiki:  cirrusSearchIncomingLinkCount: 0 queued; 7 claimed (0 active, 7 abandoned); 0 delayed
afwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 16 claimed (0 active, 16 abandoned); 0 delayed
alswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 8 claimed (0 active, 8 abandoned); 0 delayed
amwiki:  cirrusSearchIncomingLinkCount: 0 queued; 8 claimed (0 active, 8 abandoned); 0 delayed
amwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
angwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (1 active, 1 abandoned); 0 delayed
anwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
arbcom_dewiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
arwiki:  cirrusSearchIncomingLinkCount: 0 queued; 504 claimed (0 active, 504 abandoned); 16 delayed
arwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 505 claimed (0 active, 505 abandoned); 0 delayed
arwiki:  cirrusSearchOtherIndex: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
arwikibooks:  cirrusSearchIncomingLinkCount: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
arwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 8 claimed (0 active, 8 abandoned); 0 delayed
arwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
arzwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
astwiki:  cirrusSearchIncomingLinkCount: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
astwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 14 claimed (0 active, 14 abandoned); 0 delayed
aswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
avwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
azwiki:  cirrusSearchIncomingLinkCount: 0 queued; 9 claimed (0 active, 9 abandoned); 2 delayed
azwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 81 claimed (0 active, 81 abandoned); 0 delayed
barwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 22 claimed (0 active, 22 abandoned); 0 delayed
bawiki:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
bawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 27 claimed (0 active, 27 abandoned); 0 delayed
bclwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
be_x_oldwiki:  cirrusSearchIncomingLinkCount: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
be_x_oldwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 11 claimed (0 active, 11 abandoned); 0 delayed
betawikiversity:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
bewiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 24 claimed (0 active, 24 abandoned); 0 delayed
bgwiki:  cirrusSearchIncomingLinkCount: 0 queued; 33 claimed (0 active, 33 abandoned); 0 delayed
bgwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 47 claimed (0 active, 47 abandoned); 0 delayed
bgwikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
bhwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 17 claimed (0 active, 17 abandoned); 0 delayed
bnwiki:  cirrusSearchIncomingLinkCount: 0 queued; 34 claimed (0 active, 34 abandoned); 0 delayed
bnwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
bpywiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 17 claimed (0 active, 17 abandoned); 0 delayed
brwiki:  cirrusSearchIncomingLinkCount: 1 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
brwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
bswiki:  cirrusSearchIncomingLinkCount: 15 queued; 24 claimed (0 active, 24 abandoned); 0 delayed
bswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 17 claimed (0 active, 17 abandoned); 0 delayed
cawiki:  cirrusSearchIncomingLinkCount: 8 queued; 59 claimed (2 active, 57 abandoned); 6 delayed
cawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 216 claimed (0 active, 216 abandoned); 0 delayed
cawikibooks:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
cawikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
cdowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
cebwiki:  cirrusSearchIncomingLinkCount: 0 queued; 277 claimed (0 active, 277 abandoned); 0 delayed
cebwiki:  cirrusSearchLinksUpdatePrioritized: 4 queued; 157 claimed (3 active, 154 abandoned); 0 delayed
cewiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 7 claimed (0 active, 7 abandoned); 0 delayed
ckbwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
commonswiki:  cirrusSearchDeletePages: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
commonswiki:  cirrusSearchIncomingLinkCount: 8 queued; 334 claimed (0 active, 334 abandoned); 57 delayed
commonswiki:  cirrusSearchLinksUpdate: 0 queued; 7501 claimed (1285 active, 6216 abandoned); 0 delayed
commonswiki:  cirrusSearchLinksUpdatePrioritized: 5 queued; 61770 claimed (97 active, 61673 abandoned); 0 delayed
cswiki:  cirrusSearchIncomingLinkCount: 1 queued; 53 claimed (0 active, 53 abandoned); 1 delayed
cswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 339 claimed (0 active, 339 abandoned); 0 delayed
cswikisource:  cirrusSearchIncomingLinkCount: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
cswikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (2 active, 2 abandoned); 0 delayed
cswikiversity:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
cswiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
cvwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
cywiki:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
cywiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 11 claimed (0 active, 11 abandoned); 0 delayed
dawiki:  cirrusSearchIncomingLinkCount: 0 queued; 18 claimed (0 active, 18 abandoned); 22 delayed
dawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 114 claimed (0 active, 114 abandoned); 0 delayed
dawikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
dewiki:  cirrusSearchDeletePages: 0 queued; 8 claimed (0 active, 8 abandoned); 0 delayed
dewiki:  cirrusSearchIncomingLinkCount: 0 queued; 395 claimed (3 active, 392 abandoned); 31 delayed
dewiki:  cirrusSearchOtherIndex: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
dewikibooks:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
dewikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 6 claimed (0 active, 6 abandoned); 0 delayed
dewikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
dewikisource:  cirrusSearchIncomingLinkCount: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
dewikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 46 claimed (0 active, 46 abandoned); 0 delayed
dewikiversity:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
dewikivoyage:  cirrusSearchIncomingLinkCount: 0 queued; 6 claimed (0 active, 6 abandoned); 0 delayed
dewiktionary:  cirrusSearchIncomingLinkCount: 0 queued; 25 claimed (0 active, 25 abandoned); 3 delayed
dewiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 17 claimed (0 active, 17 abandoned); 0 delayed
dsbwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
dvwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
dzwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
elwiki:  cirrusSearchIncomingLinkCount: 0 queued; 31 claimed (0 active, 31 abandoned); 18 delayed
elwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 63 claimed (0 active, 63 abandoned); 0 delayed
elwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
emlwiki:  cirrusSearchIncomingLinkCount: 0 queued; 6 claimed (0 active, 6 abandoned); 0 delayed
emlwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
enwiki:  cirrusSearchDeletePages: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
enwiki:  cirrusSearchIncomingLinkCount: 3 queued; 3109 claimed (6 active, 3103 abandoned); 245 delayed
enwiki:  cirrusSearchLinksUpdatePrioritized: 18 queued; 26665 claimed (23 active, 26642 abandoned); 0 delayed
enwiki:  cirrusSearchOtherIndex: 1 queued; 306 claimed (0 active, 306 abandoned); 0 delayed
enwikinews:  cirrusSearchIncomingLinkCount: 22 queued; 30 claimed (2 active, 28 abandoned); 0 delayed
enwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 59 claimed (0 active, 59 abandoned); 0 delayed
enwikiquote:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
enwikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
enwikisource:  cirrusSearchIncomingLinkCount: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
enwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 31 claimed (0 active, 31 abandoned); 0 delayed
enwikisource:  cirrusSearchOtherIndex: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
enwikiversity:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
enwikivoyage:  cirrusSearchIncomingLinkCount: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
enwikivoyage:  cirrusSearchLinksUpdatePrioritized: 0 queued; 13 claimed (0 active, 13 abandoned); 0 delayed
enwiktionary:  cirrusSearchIncomingLinkCount: 0 queued; 88 claimed (0 active, 88 abandoned); 22 delayed
enwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1635 claimed (0 active, 1635 abandoned); 0 delayed
eowiki:  cirrusSearchIncomingLinkCount: 0 queued; 19 claimed (0 active, 19 abandoned); 0 delayed
eowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 25 claimed (0 active, 25 abandoned); 0 delayed
eowikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
eswiki:  cirrusSearchDeletePages: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
eswiki:  cirrusSearchIncomingLinkCount: 0 queued; 331 claimed (0 active, 331 abandoned); 77 delayed
eswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2306 claimed (0 active, 2306 abandoned); 0 delayed
eswikinews:  cirrusSearchIncomingLinkCount: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
eswikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 7 claimed (0 active, 7 abandoned); 0 delayed
eswikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
eswikivoyage:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
eswiktionary:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 2 delayed
eswiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
etwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 37 claimed (0 active, 37 abandoned); 0 delayed
euwiki:  cirrusSearchIncomingLinkCount: 0 queued; 6 claimed (0 active, 6 abandoned); 0 delayed
euwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 38 claimed (0 active, 38 abandoned); 0 delayed
fawiki:  cirrusSearchIncomingLinkCount: 0 queued; 63 claimed (0 active, 63 abandoned); 0 delayed
fawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 358 claimed (0 active, 358 abandoned); 0 delayed
fawikibooks:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
fawikivoyage:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
fawiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
ffwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
fiu_vrowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
fiwiki:  cirrusSearchIncomingLinkCount: 0 queued; 24 claimed (0 active, 24 abandoned); 0 delayed
fiwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 188 claimed (0 active, 188 abandoned); 0 delayed
fiwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
fiwikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 6 claimed (4 active, 2 abandoned); 0 delayed
fiwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
fiwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
foundationwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 8 claimed (0 active, 8 abandoned); 0 delayed
fowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
frpwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
frrwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
frwiki:  cirrusSearchDeletePages: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
frwiki:  cirrusSearchIncomingLinkCount: 0 queued; 1279 claimed (0 active, 1279 abandoned); 271 delayed
frwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 14102 claimed (0 active, 14102 abandoned); 0 delayed
frwikibooks:  cirrusSearchLinksUpdatePrioritized: 0 queued; 17 claimed (0 active, 17 abandoned); 0 delayed
frwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
frwikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
frwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 24 claimed (0 active, 24 abandoned); 0 delayed
frwikiversity:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
frwikivoyage:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
frwiktionary:  cirrusSearchIncomingLinkCount: 0 queued; 33 claimed (0 active, 33 abandoned); 4 delayed
frwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 46 claimed (0 active, 46 abandoned); 0 delayed
fywiki:  cirrusSearchIncomingLinkCount: 0 queued; 6 claimed (0 active, 6 abandoned); 0 delayed
fywiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
gagwiki:  cirrusSearchIncomingLinkCount: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
ganwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 8 claimed (0 active, 8 abandoned); 0 delayed
gawiki:  cirrusSearchIncomingLinkCount: 0 queued; 15 claimed (0 active, 15 abandoned); 0 delayed
gawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 13 claimed (0 active, 13 abandoned); 0 delayed
gdwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
glwiki:  cirrusSearchIncomingLinkCount: 0 queued; 13 claimed (0 active, 13 abandoned); 0 delayed
glwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 84 claimed (0 active, 84 abandoned); 0 delayed
gnwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
guwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
gvwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
gvwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
hewiki:  cirrusSearchIncomingLinkCount: 0 queued; 89 claimed (0 active, 89 abandoned); 5 delayed
hewiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 64 claimed (0 active, 64 abandoned); 0 delayed
hewikibooks:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
hewikisource:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
hewikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 7 claimed (0 active, 7 abandoned); 0 delayed
hewikivoyage:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
hifwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
hrwiki:  cirrusSearchIncomingLinkCount: 0 queued; 29 claimed (0 active, 29 abandoned); 0 delayed
hrwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 35 claimed (1 active, 34 abandoned); 0 delayed
hrwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
hsbwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
htwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
huwiki:  cirrusSearchIncomingLinkCount: 0 queued; 62 claimed (0 active, 62 abandoned); 2 delayed
huwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 468 claimed (0 active, 468 abandoned); 0 delayed
huwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 11 claimed (0 active, 11 abandoned); 0 delayed
huwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
hywiki:  cirrusSearchIncomingLinkCount: 0 queued; 8 claimed (0 active, 8 abandoned); 0 delayed
hywiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 45 claimed (0 active, 45 abandoned); 0 delayed
hywikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
hywikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
idwiki:  cirrusSearchIncomingLinkCount: 0 queued; 6 claimed (0 active, 6 abandoned); 0 delayed
idwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 7158 claimed (0 active, 7158 abandoned); 0 delayed
idwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 13 claimed (0 active, 13 abandoned); 0 delayed
ilowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
incubatorwiki:  cirrusSearchIncomingLinkCount: 0 queued; 6 claimed (0 active, 6 abandoned); 6 delayed
incubatorwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 28 claimed (0 active, 28 abandoned); 0 delayed
iowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
iswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 12 claimed (0 active, 12 abandoned); 0 delayed
itwiki:  cirrusSearchDeletePages: 1 queued; 2 claimed (1 active, 1 abandoned); 0 delayed
itwiki:  cirrusSearchIncomingLinkCount: 0 queued; 432 claimed (0 active, 432 abandoned); 2 delayed
itwiki:  cirrusSearchLinksUpdatePrioritized: 2 queued; 609 claimed (1 active, 608 abandoned); 0 delayed
itwiki:  cirrusSearchOtherIndex: 0 queued; 19 claimed (0 active, 19 abandoned); 0 delayed
itwikibooks:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
itwikibooks:  cirrusSearchLinksUpdatePrioritized: 0 queued; 6 claimed (0 active, 6 abandoned); 0 delayed
itwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 21 claimed (0 active, 21 abandoned); 0 delayed
itwikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
itwikisource:  cirrusSearchIncomingLinkCount: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
itwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
itwikivoyage:  cirrusSearchLinksUpdatePrioritized: 0 queued; 7 claimed (0 active, 7 abandoned); 0 delayed
itwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
jawiki:  cirrusSearchIncomingLinkCount: 0 queued; 87 claimed (0 active, 87 abandoned); 4 delayed
jawiki:  cirrusSearchLinksUpdatePrioritized: 9 queued; 1212 claimed (0 active, 1212 abandoned); 0 delayed
jawikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
jawikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
jawikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
jawiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
jvwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 16 claimed (0 active, 16 abandoned); 0 delayed
jvwiki:  cirrusSearchOtherIndex: 0 queued; 2 claimed (1 active, 1 abandoned); 0 delayed
kawiki:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
kawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 12 claimed (0 active, 12 abandoned); 0 delayed
kgwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (1 active, 1 abandoned); 0 delayed
kkwiki:  cirrusSearchIncomingLinkCount: 0 queued; 3 claimed (0 active, 3 abandoned); 1 delayed
kkwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 56 claimed (0 active, 56 abandoned); 0 delayed
kmwiki:  cirrusSearchIncomingLinkCount: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
kmwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
knwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 6 claimed (0 active, 6 abandoned); 0 delayed
kowiki:  cirrusSearchIncomingLinkCount: 0 queued; 11 claimed (0 active, 11 abandoned); 1 delayed
kowiki:  cirrusSearchLinksUpdatePrioritized: 1 queued; 313 claimed (0 active, 313 abandoned); 0 delayed
krcwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
kuwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
kuwiktionary:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
kuwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
kvwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
kywiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
labswiki:  cirrusSearchElasticaWrite: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
ladwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
lawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 26 claimed (0 active, 26 abandoned); 0 delayed
lbwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
liwiki:  cirrusSearchIncomingLinkCount: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
liwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
lmowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
ltwiki:  cirrusSearchIncomingLinkCount: 0 queued; 17 claimed (0 active, 17 abandoned); 0 delayed
ltwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 74 claimed (0 active, 74 abandoned); 0 delayed
ltwiktionary:  cirrusSearchIncomingLinkCount: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
ltwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 18 claimed (0 active, 18 abandoned); 0 delayed
lvwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 36 claimed (0 active, 36 abandoned); 0 delayed
map_bmswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
mediawikiwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 17 claimed (0 active, 17 abandoned); 0 delayed
metawiki:  cirrusSearchIncomingLinkCount: 0 queued; 116 claimed (0 active, 116 abandoned); 2 delayed
metawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 133 claimed (0 active, 133 abandoned); 0 delayed
mgwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
mgwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 27 claimed (0 active, 27 abandoned); 0 delayed
mhrwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 15 claimed (0 active, 15 abandoned); 0 delayed
miwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
mkwiki:  cirrusSearchIncomingLinkCount: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
mkwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
mlwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 17 claimed (0 active, 17 abandoned); 0 delayed
mnwiki:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
mnwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
mnwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1504 claimed (0 active, 1504 abandoned); 0 delayed
mrwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
mswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 43 claimed (0 active, 43 abandoned); 0 delayed
mtwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
myvwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
mywiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
mznwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
nahwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
napwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
ndswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 28 claimed (0 active, 28 abandoned); 0 delayed
newiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
nlwiki:  cirrusSearchDeletePages: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
nlwiki:  cirrusSearchIncomingLinkCount: 0 queued; 133 claimed (0 active, 133 abandoned); 1 delayed
nlwiki:  cirrusSearchLinksUpdatePrioritized: 2 queued; 827 claimed (1 active, 826 abandoned); 0 delayed
nlwikibooks:  cirrusSearchLinksUpdatePrioritized: 1 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
nlwikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
nlwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 11 claimed (0 active, 11 abandoned); 0 delayed
nnwiki:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
nnwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 86 claimed (0 active, 86 abandoned); 0 delayed
nowiki:  cirrusSearchIncomingLinkCount: 0 queued; 71 claimed (0 active, 71 abandoned); 0 delayed
nowiki:  cirrusSearchLinksUpdatePrioritized: 2 queued; 22992 claimed (0 active, 22992 abandoned); 0 delayed
nsowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
nvwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (1 active, 2 abandoned); 0 delayed
ocwiki:  cirrusSearchIncomingLinkCount: 0 queued; 7 claimed (0 active, 7 abandoned); 0 delayed
ocwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
orwiki:  cirrusSearchIncomingLinkCount: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
orwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
pamwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
pawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
plwiki:  cirrusSearchDeletePages: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
plwiki:  cirrusSearchIncomingLinkCount: 0 queued; 161 claimed (0 active, 161 abandoned); 33 delayed
plwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 169 claimed (1 active, 168 abandoned); 0 delayed
plwikiquote:  cirrusSearchIncomingLinkCount: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
plwikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 11 claimed (0 active, 11 abandoned); 0 delayed
plwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1452 claimed (0 active, 1452 abandoned); 0 delayed
plwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 17 claimed (0 active, 17 abandoned); 0 delayed
pmswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
pnbwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 13 claimed (0 active, 13 abandoned); 0 delayed
pntwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
pswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
ptwiki:  cirrusSearchIncomingLinkCount: 0 queued; 152 claimed (0 active, 152 abandoned); 40 delayed
ptwiki:  cirrusSearchLinksUpdatePrioritized: 1 queued; 1348 claimed (0 active, 1348 abandoned); 0 delayed
ptwikibooks:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
ptwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
ptwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
ptwikivoyage:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
ptwiktionary:  cirrusSearchIncomingLinkCount: 0 queued; 13 claimed (0 active, 13 abandoned); 0 delayed
ptwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
rowiki:  cirrusSearchIncomingLinkCount: 0 queued; 32 claimed (0 active, 32 abandoned); 0 delayed
rowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 122 claimed (0 active, 122 abandoned); 0 delayed
rowikibooks:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
rowikibooks:  cirrusSearchLinksUpdatePrioritized: 0 queued; 5 claimed (2 active, 3 abandoned); 0 delayed
rowikinews:  cirrusSearchIncomingLinkCount: 0 queued; 7 claimed (1 active, 6 abandoned); 0 delayed
rowikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 8 claimed (0 active, 8 abandoned); 0 delayed
ruewiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
ruwiki:  cirrusSearchIncomingLinkCount: 0 queued; 496 claimed (0 active, 496 abandoned); 33 delayed
ruwiki:  cirrusSearchOtherIndex: 1 queued; 22 claimed (0 active, 22 abandoned); 0 delayed
ruwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
ruwikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
ruwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 37 claimed (0 active, 37 abandoned); 0 delayed
ruwikivoyage:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (1 active, 1 abandoned); 0 delayed
ruwiktionary:  cirrusSearchIncomingLinkCount: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
ruwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 15 claimed (0 active, 15 abandoned); 0 delayed
sahwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
sawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
sawikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
scnwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
scowiki:  cirrusSearchIncomingLinkCount: 0 queued; 49 claimed (0 active, 49 abandoned); 0 delayed
scowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 12 claimed (0 active, 12 abandoned); 0 delayed
scwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
sdwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (1 active, 2 abandoned); 0 delayed
sewiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
shwiki:  cirrusSearchIncomingLinkCount: 0 queued; 34 claimed (0 active, 34 abandoned); 21 delayed
shwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 99 claimed (0 active, 99 abandoned); 0 delayed
simplewiki:  cirrusSearchIncomingLinkCount: 0 queued; 15 claimed (0 active, 15 abandoned); 7 delayed
siwiki:  cirrusSearchIncomingLinkCount: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
siwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
skwiki:  cirrusSearchIncomingLinkCount: 0 queued; 39 claimed (0 active, 39 abandoned); 0 delayed
skwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 112 claimed (0 active, 112 abandoned); 0 delayed
slwiki:  cirrusSearchIncomingLinkCount: 0 queued; 15 claimed (0 active, 15 abandoned); 0 delayed
slwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 34 claimed (0 active, 34 abandoned); 0 delayed
slwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
sourceswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
sowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (1 active, 3 abandoned); 0 delayed
specieswiki:  cirrusSearchIncomingLinkCount: 0 queued; 22 claimed (0 active, 22 abandoned); 0 delayed
specieswiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 35 claimed (0 active, 35 abandoned); 0 delayed
srwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 244 claimed (0 active, 244 abandoned); 0 delayed
srwiki:  cirrusSearchOtherIndex: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
srwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
srwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
strategywiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
suwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
svwiki:  cirrusSearchIncomingLinkCount: 0 queued; 64 claimed (0 active, 64 abandoned); 53 delayed
svwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 6 claimed (0 active, 6 abandoned); 0 delayed
svwikivoyage:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
svwiktionary:  cirrusSearchIncomingLinkCount: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
svwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
swwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 16 claimed (0 active, 16 abandoned); 0 delayed
szlwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
tawiki:  cirrusSearchIncomingLinkCount: 0 queued; 19 claimed (0 active, 19 abandoned); 0 delayed
tawiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 32 claimed (0 active, 32 abandoned); 0 delayed
tawikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
tewiki:  cirrusSearchIncomingLinkCount: 0 queued; 6 claimed (0 active, 6 abandoned); 0 delayed
tgwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
thwiki:  cirrusSearchIncomingLinkCount: 0 queued; 13 claimed (0 active, 13 abandoned); 0 delayed
thwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 111 claimed (0 active, 111 abandoned); 0 delayed
tkwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
tlwiki:  cirrusSearchIncomingLinkCount: 0 queued; 8 claimed (0 active, 8 abandoned); 0 delayed
tlwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 7 claimed (0 active, 7 abandoned); 0 delayed
trwiki:  cirrusSearchIncomingLinkCount: 0 queued; 59 claimed (0 active, 59 abandoned); 0 delayed
trwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 237 claimed (0 active, 237 abandoned); 0 delayed
trwiki:  cirrusSearchOtherIndex: 0 queued; 6 claimed (0 active, 6 abandoned); 0 delayed
trwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 12 claimed (0 active, 12 abandoned); 0 delayed
trwikiquote:  cirrusSearchIncomingLinkCount: 0 queued; 3 claimed (0 active, 3 abandoned); 0 delayed
trwikiquote:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
trwiktionary:  cirrusSearchIncomingLinkCount: 0 queued; 22 claimed (0 active, 22 abandoned); 0 delayed
trwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 5 claimed (0 active, 5 abandoned); 0 delayed
ttwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 38 claimed (0 active, 38 abandoned); 0 delayed
ugwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
ukwiki:  cirrusSearchIncomingLinkCount: 0 queued; 132 claimed (0 active, 132 abandoned); 26 delayed
ukwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1144 claimed (0 active, 1144 abandoned); 0 delayed
ukwiki:  cirrusSearchOtherIndex: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
ukwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 16 claimed (0 active, 16 abandoned); 0 delayed
urwiki:  cirrusSearchIncomingLinkCount: 0 queued; 139 claimed (0 active, 139 abandoned); 6 delayed
urwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 103 claimed (1 active, 102 abandoned); 0 delayed
uzwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 13 claimed (0 active, 13 abandoned); 0 delayed
uzwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
vecwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
vecwikisource:  cirrusSearchIncomingLinkCount: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
vepwiki:  cirrusSearchIncomingLinkCount: 0 queued; 8 claimed (0 active, 8 abandoned); 0 delayed
vepwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
viwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 94 claimed (0 active, 94 abandoned); 0 delayed
viwikibooks:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
vowiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
warwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 28 claimed (0 active, 28 abandoned); 0 delayed
wikidatawiki:  cirrusSearchIncomingLinkCount: 1 queued; 648 claimed (1 active, 647 abandoned); 51 delayed
wikidatawiki:  cirrusSearchLinksUpdatePrioritized: 3 queued; 3036 claimed (0 active, 3036 abandoned); 0 delayed
yiwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 4 claimed (0 active, 4 abandoned); 0 delayed
zh_min_nanwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 27 claimed (0 active, 27 abandoned); 0 delayed
zh_yuewiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 7 claimed (0 active, 7 abandoned); 0 delayed
zhwiki:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2161 claimed (5 active, 2156 abandoned); 0 delayed
zhwikibooks:  cirrusSearchLinksUpdatePrioritized: 0 queued; 9 claimed (0 active, 9 abandoned); 0 delayed
zhwikinews:  cirrusSearchLinksUpdatePrioritized: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed
zhwikisource:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
zhwikivoyage:  cirrusSearchLinksUpdatePrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
zhwiktionary:  cirrusSearchLinksUpdatePrioritized: 0 queued; 6 claimed (0 active, 6 abandoned); 0 delayed

Due to T121568 i'm not able to see what the abandoned jobs actually are. Not sure how to debug this, but will check again tomorrow to see if these were one time failures or something that is still having issues.

the failures spiked on 12/12 and have reduced significantly since then. They don't look to be limited to cirrus jobs, the refreshLinks jobs spiked with abandons at the same time and also settled back down.

I pulled all reported errors from the runJobs.log file for 12/10 through 12/16, there were a total of 941 failures which is much lower than the number of abandoned jobs. For a job to be abandoned it should have failed three times, so that doesn't really make sense :S

921 out of 941 failures were due to T107913. Stas wrote up a fix this and i merged it today.

The other log of failures would be stored in CirrusSearch.log. These failures would not result in job abandonment, we handle them internally and do not expose the failure to the job queue. I'm not sure where else to look :S

aaron added a subscriber: aaron.Dec 17 2015, 8:31 AM

Seems that recycle/abandon sometime have a large spikes together (the together part makes sense):

They often corresponds to exceptions:

...and are even more correlated with redis errors:

The later link is mostly ack() failing due to a connection drop. The spikes match with recycle spikes, which make sense as the jobs could not be confirmed as run so they were recycled. When it happened to the same job (many of them) enough times, the got abandoned.

Deskana triaged this task as Normal priority.Dec 17 2015, 5:38 PM
Deskana added a subscriber: Deskana.

Removing Discovery-Search (Current work) as we've investigated and don't think there's anything for us to do here. Let me know if that's incorrect.

Deskana moved this task from Uncategorised to Technical on the CirrusSearch board.Dec 31 2015, 3:44 AM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptAug 10 2016, 4:23 PM