Page MenuHomePhabricator

Investigate the reason why hewiki_titlesuggest was empty
Closed, ResolvedPublic

Description

The suggester index for hewiki was empty and caused a very bad user experience.
Reading logs from /var/log/mediawiki/cirrus-suggest/hewiki.eqiad.log-20160216.gz on terbium I can see:

Fetching Elasticsearch version...1.7.1...ok
Infering index identifier...hewiki_titlesuggest_1455026477
Recycling index hewiki_titlesuggest
Indexing 185537 documents (185537 in the index) with batchId: 1455584084
        2% done...
        4% done...
        6% done...
        8% done...
[...]
        96% done...
        98% done...
        100% done...
Indexing done.
Purging deleted docs...ok.
Deleting remaining docs from previous batch (184407).
        0% done...
        2% done...
        4% done...
[...]
        98% done...
        100% done...
Done.
Optimizing index...ok.

This is concerning... it means that the recycle process deleted the docs it just created...
I was not able to reproduce the problem by running the process manually multiple times on codfw.

  • One possible (but improbable) cause is that this process was run twice at the same time, both processes could have deleted each other docs because of a batchId mismatch?
  • Another possibility is that the optimize to expunge deleted docs did not run synchronously and the reader was refreshed on the previous state?
  • The reader was not refreshed on the proper state and caused the query to extract the ids to delete to return all docs?

Event Timeline

Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript

There's a lot of duplicated indices in the elastic alias... and most of them are indices created in a very short time period :
(first column is the time in second between index creation, 2nd is the date of the first index creation)

This is extremely confusing, this tend to confirm the first hypothesis: running updateSuggesterIndex on the same index at the same time...
Index being slow to create a second run might be possible...

1	 Tue Feb  9 00:20:07 UTC 2016	aawiktionary_titlesuggest
1	 Tue Feb  9 03:39:18 UTC 2016	biwikibooks_titlesuggest
1	 Tue Feb  9 15:30:40 UTC 2016	hywikiquote_titlesuggest
1	 Tue Feb  9 16:29:57 UTC 2016	ilowiki_titlesuggest
2	 Tue Feb  9 07:33:46 UTC 2016	dawikibooks_titlesuggest
2	 Tue Feb  9 09:24:38 UTC 2016	enwikibooks_titlesuggest
2	 Tue Feb  9 12:03:46 UTC 2016	frwikivoyage_titlesuggest
2	 Tue Feb  9 22:27:46 UTC 2016	lnwikibooks_titlesuggest
2	 Wed Feb 10 06:06:18 UTC 2016	plwikinews_titlesuggest
3	 Tue Feb  9 03:04:36 UTC 2016	bewikimedia_titlesuggest
3	 Tue Feb  9 06:38:23 UTC 2016	cowikimedia_titlesuggest
3	 Tue Feb  9 06:56:55 UTC 2016	cswikiquote_titlesuggest
3	 Tue Feb  9 22:42:00 UTC 2016	lowiktionary_titlesuggest
4	 Tue Feb  9 06:51:25 UTC 2016	csbwiki_titlesuggest
4	 Tue Feb  9 09:45:12 UTC 2016	eowikibooks_titlesuggest
4	 Wed Feb 10 16:00:34 UTC 2016	trwikibooks_titlesuggest
4	 Wed Feb 10 22:01:31 UTC 2016	zhwikiquote_titlesuggest
5	 Wed Feb 10 14:02:12 UTC 2016	tewikiquote_titlesuggest
6	 Tue Feb  9 00:25:58 UTC 2016	abwiktionary_titlesuggest
6	 Tue Feb  9 02:49:09 UTC 2016	be_x_oldwiki_titlesuggest
6	 Tue Feb  9 03:20:31 UTC 2016	bgwikiquote_titlesuggest
6	 Tue Feb  9 12:19:08 UTC 2016	gagwiki_titlesuggest
6	 Tue Feb  9 17:20:11 UTC 2016	itwikiquote_titlesuggest
7	 Tue Feb  9 09:19:47 UTC 2016	emlwiki_titlesuggest
7	 Tue Feb  9 12:02:00 UTC 2016	frwikisource_titlesuggest
7	 Tue Feb  9 17:23:56 UTC 2016	itwikiversity_titlesuggest
7	 Wed Feb 10 13:12:05 UTC 2016	swwiktionary_titlesuggest
7	 Wed Feb 10 19:31:13 UTC 2016	wawiktionary_titlesuggest
8	 Tue Feb  9 00:45:06 UTC 2016	alswikibooks_titlesuggest
8	 Tue Feb  9 05:25:22 UTC 2016	cawiki_titlesuggest
8	 Tue Feb  9 05:31:55 UTC 2016	cawikiquote_titlesuggest
8	 Tue Feb  9 07:54:50 UTC 2016	dewikisource_titlesuggest
8	 Tue Feb  9 19:26:53 UTC 2016	knwikibooks_titlesuggest
8	 Wed Feb 10 17:30:38 UTC 2016	ukwikisource_titlesuggest
8	 Wed Feb 10 20:07:41 UTC 2016	wikimania2016wiki_titlesuggest
8	 Wed Feb 10 20:08:01 UTC 2016	wikimania2017wiki_titlesuggest
8	 Wed Feb 10 22:06:59 UTC 2016	zhwikivoyage_titlesuggest
9	 Wed Feb 10 10:04:43 UTC 2016	simplewikibooks_titlesuggest
9	 Wed Feb 10 21:25:55 UTC 2016	zh_classicalwiki_titlesuggest
10	 Wed Feb 10 01:52:21 UTC 2016	myvwiki_titlesuggest
10	 Wed Feb 10 05:18:29 UTC 2016	pagwiki_titlesuggest
11	 Tue Feb  9 10:26:35 UTC 2016	extwiki_titlesuggest
11	 Wed Feb 10 00:15:52 UTC 2016	mkwikimedia_titlesuggest
11	 Wed Feb 10 19:39:03 UTC 2016	wikimania2006wiki_titlesuggest
12	 Tue Feb  9 00:45:28 UTC 2016	alswikiquote_titlesuggest
12	 Tue Feb  9 06:00:57 UTC 2016	chowiki_titlesuggest
12	 Tue Feb  9 17:08:26 UTC 2016	itwiki_titlesuggest
12	 Wed Feb 10 02:15:18 UTC 2016	napwiki_titlesuggest
12	 Wed Feb 10 08:09:16 UTC 2016	rowiktionary_titlesuggest
12	 Wed Feb 10 13:32:07 UTC 2016	tawikisource_titlesuggest
13	 Tue Feb  9 05:25:49 UTC 2016	cawikimedia_titlesuggest
13	 Tue Feb  9 12:18:56 UTC 2016	fywiktionary_titlesuggest
13	 Tue Feb  9 16:57:21 UTC 2016	iswikibooks_titlesuggest
13	 Tue Feb  9 21:03:16 UTC 2016	kywiktionary_titlesuggest
13	 Wed Feb 10 07:38:50 UTC 2016	rmwiktionary_titlesuggest
13	 Wed Feb 10 20:18:14 UTC 2016	wowikiquote_titlesuggest
14	 Tue Feb  9 02:00:05 UTC 2016	aswikibooks_titlesuggest
14	 Tue Feb  9 05:10:14 UTC 2016	bswikiquote_titlesuggest
14	 Tue Feb  9 17:53:53 UTC 2016	jawikinews_titlesuggest
14	 Tue Feb  9 18:26:37 UTC 2016	kawiki_titlesuggest
14	 Tue Feb  9 23:07:18 UTC 2016	lvwiki_titlesuggest
14	 Wed Feb 10 04:32:45 UTC 2016	officewiki_titlesuggest
15	 Tue Feb  9 02:13:27 UTC 2016	aywikibooks_titlesuggest
15	 Tue Feb  9 03:20:25 UTC 2016	bgwikinews_titlesuggest
15	 Tue Feb  9 07:54:40 UTC 2016	dewikiquote_titlesuggest
15	 Tue Feb  9 18:37:34 UTC 2016	kawiktionary_titlesuggest
15	 Tue Feb  9 23:47:31 UTC 2016	mhrwiki_titlesuggest
15	 Wed Feb 10 00:23:08 UTC 2016	mlwiki_titlesuggest
15	 Wed Feb 10 10:23:17 UTC 2016	skwikiquote_titlesuggest
16	 Tue Feb  9 00:46:17 UTC 2016	alswiktionary_titlesuggest
16	 Tue Feb  9 02:02:56 UTC 2016	aswiktionary_titlesuggest
17	 Wed Feb 10 20:41:11 UTC 2016	xhwiktionary_titlesuggest
18	 Tue Feb  9 03:06:20 UTC 2016	bewikiquote_titlesuggest
18	 Wed Feb 10 04:33:09 UTC 2016	ombudsmenwiki_titlesuggest
18	 Wed Feb 10 04:56:17 UTC 2016	otrs_wikiwiki_titlesuggest
18	 Wed Feb 10 15:50:42 UTC 2016	towiktionary_titlesuggest
19	 Tue Feb  9 07:14:34 UTC 2016	cvwikibooks_titlesuggest
19	 Tue Feb  9 09:48:50 UTC 2016	eowikiquote_titlesuggest
19	 Tue Feb  9 17:09:01 UTC 2016	itwikibooks_titlesuggest
19	 Wed Feb 10 20:25:47 UTC 2016	wuuwiki_titlesuggest
20	 Tue Feb  9 01:27:48 UTC 2016	arwikibooks_titlesuggest
20	 Tue Feb  9 04:50:21 UTC 2016	brwikimedia_titlesuggest
21	 Tue Feb  9 16:19:51 UTC 2016	igwiki_titlesuggest
21	 Tue Feb  9 20:10:35 UTC 2016	kshwiki_titlesuggest
21	 Wed Feb 10 08:50:55 UTC 2016	sahwiki_titlesuggest
21	 Wed Feb 10 20:13:34 UTC 2016	wowiki_titlesuggest
22	 Tue Feb  9 09:00:46 UTC 2016	elwikiversity_titlesuggest
22	 Tue Feb  9 15:40:20 UTC 2016	iawiki_titlesuggest
22	 Tue Feb  9 18:38:13 UTC 2016	kbdwiki_titlesuggest
22	 Wed Feb 10 13:03:35 UTC 2016	swwiki_titlesuggest
22	 Wed Feb 10 19:25:51 UTC 2016	wawiki_titlesuggest
23	 Tue Feb  9 06:16:32 UTC 2016	chwiktionary_titlesuggest
23	 Wed Feb 10 09:41:30 UTC 2016	sdwiktionary_titlesuggest
23	 Wed Feb 10 12:19:47 UTC 2016	stewardwiki_titlesuggest
23	 Wed Feb 10 20:02:48 UTC 2016	wikimania2014wiki_titlesuggest
24	 Tue Feb  9 05:11:32 UTC 2016	bswikisource_titlesuggest
24	 Wed Feb 10 17:55:45 UTC 2016	urwikiquote_titlesuggest
24	 Wed Feb 10 19:53:20 UTC 2016	wikimania2012wiki_titlesuggest
24	 Wed Feb 10 21:08:16 UTC 2016	zawiki_titlesuggest
26	 Tue Feb  9 04:44:40 UTC 2016	bpywiki_titlesuggest
26	 Tue Feb  9 05:31:54 UTC 2016	cawikinews_titlesuggest
26	 Tue Feb  9 19:35:39 UTC 2016	kowiki_titlesuggest
26	 Wed Feb 10 04:31:59 UTC 2016	ocwiktionary_titlesuggest
27	 Tue Feb  9 02:45:11 UTC 2016	bawikibooks_titlesuggest
27	 Tue Feb  9 03:20:12 UTC 2016	bgwikibooks_titlesuggest
27	 Tue Feb  9 03:27:27 UTC 2016	bgwiktionary_titlesuggest
27	 Tue Feb  9 05:03:02 UTC 2016	bswikibooks_titlesuggest
27	 Tue Feb  9 07:34:31 UTC 2016	dawikiquote_titlesuggest
27	 Tue Feb  9 08:48:30 UTC 2016	elwikinews_titlesuggest
27	 Tue Feb  9 22:08:20 UTC 2016	liwikibooks_titlesuggest
27	 Wed Feb 10 04:25:04 UTC 2016	nzwikimedia_titlesuggest
27	 Wed Feb 10 06:31:54 UTC 2016	pswiki_titlesuggest
27	 Wed Feb 10 08:05:53 UTC 2016	rowikisource_titlesuggest
28	 Tue Feb  9 07:33:31 UTC 2016	dawiki_titlesuggest
28	 Tue Feb  9 11:14:41 UTC 2016	fiwikinews_titlesuggest
28	 Wed Feb 10 02:24:42 UTC 2016	nawiktionary_titlesuggest
29	 Tue Feb  9 04:14:52 UTC 2016	boardwiki_titlesuggest
29	 Tue Feb  9 05:05:24 UTC 2016	bswikinews_titlesuggest
29	 Tue Feb  9 06:46:15 UTC 2016	crwikiquote_titlesuggest
29	 Tue Feb  9 12:08:29 UTC 2016	fywiki_titlesuggest
29	 Tue Feb  9 16:37:56 UTC 2016	incubatorwiki_titlesuggest
29	 Wed Feb 10 01:59:08 UTC 2016	mywiktionary_titlesuggest
29	 Wed Feb 10 06:31:42 UTC 2016	pnbwiktionary_titlesuggest
29	 Wed Feb 10 17:57:40 UTC 2016	urwiktionary_titlesuggest
29	 Wed Feb 10 20:03:50 UTC 2016	wikimania2015wiki_titlesuggest
30	 Wed Feb 10 17:43:18 UTC 2016	urwikibooks_titlesuggest
30	 Wed Feb 10 18:57:36 UTC 2016	viwikivoyage_titlesuggest
32	 Tue Feb  9 03:26:53 UTC 2016	bgwikisource_titlesuggest
32	 Tue Feb  9 03:27:33 UTC 2016	bhwiki_titlesuggest
32	 Tue Feb  9 07:42:18 UTC 2016	dawiktionary_titlesuggest
32	 Tue Feb  9 15:42:06 UTC 2016	iawiktionary_titlesuggest
32	 Tue Feb  9 21:39:49 UTC 2016	lbwiktionary_titlesuggest
32	 Tue Feb  9 23:09:07 UTC 2016	lvwikibooks_titlesuggest
32	 Wed Feb 10 02:17:14 UTC 2016	nawiki_titlesuggest
32	 Wed Feb 10 19:53:38 UTC 2016	wikimania2013wiki_titlesuggest
34	 Wed Feb 10 01:40:27 UTC 2016	mwlwiki_titlesuggest
34	 Wed Feb 10 03:47:55 UTC 2016	nowikimedia_titlesuggest
35	 Tue Feb  9 06:01:37 UTC 2016	chrwiki_titlesuggest
35	 Tue Feb  9 19:53:07 UTC 2016	kowikiversity_titlesuggest
35	 Wed Feb 10 21:49:11 UTC 2016	zhwikibooks_titlesuggest
36	 Wed Feb 10 06:03:21 UTC 2016	plwikibooks_titlesuggest
36	 Wed Feb 10 11:01:17 UTC 2016	spcomwiki_titlesuggest
36	 Wed Feb 10 13:31:10 UTC 2016	tawikinews_titlesuggest
37	 Tue Feb  9 12:30:32 UTC 2016	gawiktionary_titlesuggest
37	 Wed Feb 10 05:24:12 UTC 2016	papwiki_titlesuggest
37	 Wed Feb 10 11:03:14 UTC 2016	specieswiki_titlesuggest
37	 Wed Feb 10 11:49:58 UTC 2016	srwikiquote_titlesuggest
38	 Tue Feb  9 02:59:00 UTC 2016	bewikibooks_titlesuggest
38	 Tue Feb  9 05:48:34 UTC 2016	cewiki_titlesuggest
38	 Wed Feb 10 03:29:44 UTC 2016	nnwikiquote_titlesuggest
38	 Wed Feb 10 13:50:58 UTC 2016	testwikidatawiki_titlesuggest
38	 Wed Feb 10 20:20:13 UTC 2016	wowiktionary_titlesuggest
39	 Tue Feb  9 00:54:04 UTC 2016	angwiki_titlesuggest
39	 Tue Feb  9 01:52:56 UTC 2016	astwiktionary_titlesuggest
39	 Wed Feb 10 17:30:21 UTC 2016	ukwikiquote_titlesuggest
40	 Tue Feb  9 03:59:30 UTC 2016	bmwikibooks_titlesuggest
40	 Tue Feb  9 10:41:05 UTC 2016	fawikiquote_titlesuggest
40	 Tue Feb  9 13:53:25 UTC 2016	hawiktionary_titlesuggest
40	 Tue Feb  9 22:40:34 UTC 2016	loginwiki_titlesuggest
40	 Wed Feb 10 17:12:46 UTC 2016	ugwiktionary_titlesuggest
41	 Tue Feb  9 06:15:57 UTC 2016	chwikibooks_titlesuggest
41	 Tue Feb  9 18:14:15 UTC 2016	jvwiki_titlesuggest
41	 Wed Feb 10 00:43:16 UTC 2016	mnwikibooks_titlesuggest
41	 Wed Feb 10 16:20:46 UTC 2016	tswiktionary_titlesuggest
41	 Wed Feb 10 20:57:57 UTC 2016	yiwikisource_titlesuggest
42	 Tue Feb  9 02:46:17 UTC 2016	bclwiki_titlesuggest
42	 Wed Feb 10 21:36:12 UTC 2016	zh_min_nanwikiquote_titlesuggest
43	 Tue Feb  9 08:33:29 UTC 2016	dzwiki_titlesuggest
43	 Wed Feb 10 16:55:10 UTC 2016	uawikimedia_titlesuggest
44	 Wed Feb 10 07:37:51 UTC 2016	rmwikibooks_titlesuggest
44	 Wed Feb 10 16:05:02 UTC 2016	trwikinews_titlesuggest
44	 Wed Feb 10 17:30:05 UTC 2016	ukwikinews_titlesuggest
44	 Wed Feb 10 19:10:13 UTC 2016	vlswiki_titlesuggest
45	 Tue Feb  9 01:27:08 UTC 2016	arwiki_titlesuggest
45	 Tue Feb  9 22:19:31 UTC 2016	liwiktionary_titlesuggest
45	 Wed Feb 10 02:31:33 UTC 2016	ndswikibooks_titlesuggest
45	 Wed Feb 10 03:47:47 UTC 2016	nowikibooks_titlesuggest
45	 Wed Feb 10 13:59:57 UTC 2016	tewiki_titlesuggest
46	 Tue Feb  9 22:41:45 UTC 2016	lowiki_titlesuggest
46	 Wed Feb 10 12:48:06 UTC 2016	svwikinews_titlesuggest
46	 Wed Feb 10 13:42:34 UTC 2016	tenwiki_titlesuggest
47	 Tue Feb  9 05:24:44 UTC 2016	bxrwiki_titlesuggest
47	 Tue Feb  9 23:54:34 UTC 2016	mhwiktionary_titlesuggest
47	 Wed Feb 10 05:50:41 UTC 2016	piwiktionary_titlesuggest
47	 Wed Feb 10 18:54:32 UTC 2016	viwikisource_titlesuggest
48	 Tue Feb  9 16:33:17 UTC 2016	ilwikimedia_titlesuggest
49	 Tue Feb  9 03:20:03 UTC 2016	bgwiki_titlesuggest
49	 Tue Feb  9 07:13:08 UTC 2016	cvwiki_titlesuggest
49	 Tue Feb  9 13:52:48 UTC 2016	hawiki_titlesuggest
49	 Tue Feb  9 15:52:45 UTC 2016	idwikiquote_titlesuggest
49	 Wed Feb 10 04:44:35 UTC 2016	orwiki_titlesuggest
50	 Tue Feb  9 07:23:14 UTC 2016	cywikiquote_titlesuggest
50	 Tue Feb  9 19:00:41 UTC 2016	kkwiktionary_titlesuggest
50	 Tue Feb  9 20:27:34 UTC 2016	kuwikibooks_titlesuggest
51	 Tue Feb  9 19:54:59 UTC 2016	krcwiki_titlesuggest
51	 Wed Feb 10 05:02:00 UTC 2016	pa_uswikimedia_titlesuggest
51	 Wed Feb 10 10:12:39 UTC 2016	siwikibooks_titlesuggest
52	 Tue Feb  9 03:59:15 UTC 2016	bmwiki_titlesuggest
52	 Tue Feb  9 07:23:53 UTC 2016	cywikisource_titlesuggest
52	 Tue Feb  9 19:20:05 UTC 2016	kmwiktionary_titlesuggest
53	 Tue Feb  9 05:12:13 UTC 2016	bswiktionary_titlesuggest
53	 Tue Feb  9 13:30:33 UTC 2016	guwikiquote_titlesuggest
53	 Wed Feb 10 09:05:30 UTC 2016	sawikisource_titlesuggest
53	 Wed Feb 10 17:14:49 UTC 2016	ukwikibooks_titlesuggest
55	 Tue Feb  9 07:11:45 UTC 2016	cuwiki_titlesuggest
55	 Tue Feb  9 22:21:35 UTC 2016	lmowiki_titlesuggest
56	 Tue Feb  9 08:37:16 UTC 2016	eewiki_titlesuggest
56	 Wed Feb 10 05:33:22 UTC 2016	pawiktionary_titlesuggest
58	 Tue Feb  9 03:39:04 UTC 2016	biwiki_titlesuggest
58	 Tue Feb  9 17:31:08 UTC 2016	iuwiki_titlesuggest
59	 Tue Feb  9 20:38:57 UTC 2016	kwwikiquote_titlesuggest
59	 Tue Feb  9 21:41:11 UTC 2016	legalteamwiki_titlesuggest
60	 Tue Feb  9 02:32:57 UTC 2016	barwiki_titlesuggest
60	 Wed Feb 10 00:06:59 UTC 2016	miwiktionary_titlesuggest
61	 Tue Feb  9 23:38:04 UTC 2016	mgwiki_titlesuggest
61	 Wed Feb 10 11:23:07 UTC 2016	sqwikinews_titlesuggest
62	 Tue Feb  9 07:42:07 UTC 2016	dawikisource_titlesuggest
62	 Tue Feb  9 19:22:03 UTC 2016	knwiki_titlesuggest
64	 Tue Feb  9 11:30:49 UTC 2016	fowiki_titlesuggest
64	 Wed Feb 10 04:11:14 UTC 2016	nvwiki_titlesuggest
64	 Wed Feb 10 16:28:19 UTC 2016	ttwikibooks_titlesuggest
65	 Wed Feb 10 13:02:38 UTC 2016	svwiktionary_titlesuggest
66	 Tue Feb  9 04:06:31 UTC 2016	bnwikisource_titlesuggest
66	 Wed Feb 10 16:06:44 UTC 2016	trwikisource_titlesuggest
67	 Tue Feb  9 14:50:20 UTC 2016	hrwiktionary_titlesuggest
67	 Wed Feb 10 14:11:55 UTC 2016	tgwikibooks_titlesuggest
70	 Tue Feb  9 05:51:05 UTC 2016	chairwiki_titlesuggest
70	 Wed Feb 10 12:53:51 UTC 2016	svwikisource_titlesuggest
71	 Tue Feb  9 14:12:17 UTC 2016	hewikisource_titlesuggest
72	 Tue Feb  9 06:55:55 UTC 2016	cswikibooks_titlesuggest
73	 Wed Feb 10 11:51:37 UTC 2016	srwikisource_titlesuggest
73	 Wed Feb 10 21:51:47 UTC 2016	zhwikinews_titlesuggest
74	 Tue Feb  9 19:15:32 UTC 2016	kmwiki_titlesuggest
74	 Wed Feb 10 19:53:16 UTC 2016	wikimania2011wiki_titlesuggest
76	 Wed Feb 10 04:55:25 UTC 2016	oswiki_titlesuggest
77	 Wed Feb 10 06:08:06 UTC 2016	plwikiquote_titlesuggest
77	 Wed Feb 10 16:58:47 UTC 2016	ugwikibooks_titlesuggest
79	 Tue Feb  9 07:21:39 UTC 2016	cywikibooks_titlesuggest
79	 Tue Feb  9 16:38:59 UTC 2016	internalwiki_titlesuggest
80	 Tue Feb  9 06:42:42 UTC 2016	cowiktionary_titlesuggest
80	 Wed Feb 10 07:26:26 UTC 2016	quwikiquote_titlesuggest
82	 Tue Feb  9 02:12:55 UTC 2016	aywiki_titlesuggest
82	 Tue Feb  9 04:56:30 UTC 2016	brwikisource_titlesuggest
82	 Tue Feb  9 14:37:30 UTC 2016	howiki_titlesuggest
82	 Wed Feb 10 12:36:23 UTC 2016	suwikibooks_titlesuggest
83	 Tue Feb  9 20:49:56 UTC 2016	kywikibooks_titlesuggest
85	 Tue Feb  9 02:21:44 UTC 2016	azwikibooks_titlesuggest
85	 Wed Feb 10 16:58:05 UTC 2016	ugwiki_titlesuggest
86	 Wed Feb 10 04:13:45 UTC 2016	nywiki_titlesuggest
89	 Tue Feb  9 01:11:25 UTC 2016	arbcom_fiwiki_titlesuggest
89	 Wed Feb 10 03:03:58 UTC 2016	nlwikiquote_titlesuggest
90	 Tue Feb  9 02:21:54 UTC 2016	azwikiquote_titlesuggest
90	 Tue Feb  9 03:10:43 UTC 2016	bewiktionary_titlesuggest
91	 Wed Feb 10 19:38:56 UTC 2016	wikimania2005wiki_titlesuggest
92	 Tue Feb  9 02:19:32 UTC 2016	azbwiki_titlesuggest
92	 Tue Feb  9 23:20:43 UTC 2016	mediawikiwiki_titlesuggest
94	 Tue Feb  9 00:53:51 UTC 2016	amwikiquote_titlesuggest
94	 Tue Feb  9 21:29:56 UTC 2016	lbewiki_titlesuggest
95	 Tue Feb  9 07:48:33 UTC 2016	dewikibooks_titlesuggest
96	 Wed Feb 10 17:41:14 UTC 2016	ukwikivoyage_titlesuggest
96	 Wed Feb 10 18:19:46 UTC 2016	uzwiktionary_titlesuggest
97	 Tue Feb  9 16:41:50 UTC 2016	iowiki_titlesuggest
98	 Tue Feb  9 05:14:23 UTC 2016	bugwiki_titlesuggest
98	 Tue Feb  9 05:39:52 UTC 2016	cbk_zamwiki_titlesuggest
101	 Tue Feb  9 16:04:21 UTC 2016	iegcomwiki_titlesuggest
103	 Wed Feb 10 11:39:43 UTC 2016	srwikibooks_titlesuggest
104	 Wed Feb 10 04:46:38 UTC 2016	orwikisource_titlesuggest
106	 Wed Feb 10 01:25:54 UTC 2016	mswiktionary_titlesuggest
108	 Tue Feb  9 23:58:34 UTC 2016	miwiki_titlesuggest
112	 Tue Feb  9 22:11:02 UTC 2016	liwikiquote_titlesuggest
117	 Tue Feb  9 15:22:58 UTC 2016	hywiki_titlesuggest
117	 Wed Feb 10 04:59:15 UTC 2016	outreachwiki_titlesuggest
123	 Tue Feb  9 12:13:33 UTC 2016	fywikibooks_titlesuggest
124	 Tue Feb  9 22:53:44 UTC 2016	ltwiki_titlesuggest
128	 Tue Feb  9 01:01:51 UTC 2016	angwikisource_titlesuggest
128	 Tue Feb  9 09:03:56 UTC 2016	elwikivoyage_titlesuggest
130	 Tue Feb  9 00:33:33 UTC 2016	akwiki_titlesuggest
130	 Tue Feb  9 05:40:57 UTC 2016	cdowiki_titlesuggest
133	 Tue Feb  9 01:37:20 UTC 2016	arwiktionary_titlesuggest
136	 Tue Feb  9 03:08:57 UTC 2016	bewikisource_titlesuggest
136	 Wed Feb 10 06:41:50 UTC 2016	pswiktionary_titlesuggest
138	 Tue Feb  9 11:55:21 UTC 2016	frwikinews_titlesuggest
141	 Tue Feb  9 08:02:18 UTC 2016	dewikivoyage_titlesuggest
146	 Tue Feb  9 21:59:34 UTC 2016	liwiki_titlesuggest
147	 Tue Feb  9 21:07:25 UTC 2016	lawiki_titlesuggest
152	 Tue Feb  9 02:55:34 UTC 2016	bewiki_titlesuggest
155	 Tue Feb  9 06:29:31 UTC 2016	commonswiki_titlesuggest
155	 Wed Feb 10 04:02:08 UTC 2016	nsowiki_titlesuggest
159	 Wed Feb 10 15:13:05 UTC 2016	tkwikibooks_titlesuggest
166	 Tue Feb  9 08:07:40 UTC 2016	diqwiki_titlesuggest
172	 Tue Feb  9 03:34:35 UTC 2016	bhwiktionary_titlesuggest
179	 Tue Feb  9 03:45:25 UTC 2016	bjnwiki_titlesuggest

FTR, the command to extract this data:

curl -s elastic1001.eqiad.wmnet:9200/_cat/aliases > /tmp/elastic.aliases
for i in `grep '_titlesuggest' /tmp/elastic.aliases | sort | uniq -d -w 39  | cut -f1 -d' '`; do grep "^$i" /tmp/elastic.aliases | sed -e 's/.*_titlesuggest_\([0-9]\+\).*/\1/' | sort -nr | xargs -I{} -n 2 bash -c 'echo  -en "`expr $1 - $2`\t `date -d @$1`"' {} ; echo -e "\t$i" ; done | sort -n

I have no clue :(
The problem is exactly the same for indices in codfw...

aawiktionary_titlesuggest is one the first indices returned by expanddblist and the creation time (extracted from the index timestamp) is exactly when the cron is scheduled 00:20.

I'd say that the possible causes are :

  1. this crontab is executed on another machine
  2. or expanddblist returned duplicate indices
  3. or a subtle bug in the script...

I think it's nearly impossible that someone launched manually a second run on aawiktionary_titlesuggest at 00:20...

@EBernhardson: any clues?

I looked into this and what happened is there. Is a second machine running the scripts. Mediawiki::maintenance role is installed to two machines but has ensure absent set on the second for each individual script. Somehow our job is not respecting the absent and needs to be fixed in puppet

Thanks Erik, I created T127322.
I'll use this task to cleanup duplicate indices.

Duplicate indices were deleted, I launched another update.

Deskana moved this task from Needs triage to Liaison on the Discovery board.
Deskana moved this task from Liaison to On Sprint Board on the Discovery board.