Page MenuHomePhabricator

Same MassMessage is being sent more than once
Open, HighPublic

Description

https://it.wikipedia.org/w/index.php?title=Discussioni_utente:Evil89&diff=71467952&oldid=20003624

It's the same message, I believe: there are no incoming redirects and both copies say "Evil89". JobQueue issues again?

PLEASE NOTE: this bug hits randomly, and it's unassigned.
Consider appending a preemptive apology to your MassMessage if you really need to use the tool. Having short target lists may also help (if anything, removing any duplicates later will be easier/faster as there should be fewer of them). You can use GUC to figure out if the bug has affected your latest delivery.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Glaisher raised the priority of this task from Medium to High.Jun 30 2015, 5:04 PM

More examples of duplicate massmessage posts:

1https://nl.wikibooks.org/w/index.php?title=Wikibooks:Lerarenkamer&diff=prev&oldid=292392
2https://ro.wikinews.org/w/index.php?title=Wiki%C8%99tiri:Cafenea&diff=prev&oldid=32713
3https://ro.wikinews.org/w/index.php?title=Wiki%C8%99tiri:Cafenea&diff=prev&oldid=31930
4https://ro.wikinews.org/w/index.php?title=Wiki%C8%99tiri:Cafenea&diff=prev&oldid=31937
5https://lo.wikipedia.org/w/index.php?title=%E0%BA%A7%E0%BA%B4%E0%BA%81%E0%BA%B4%E0%BA%9E%E0%BA%B5%E0%BB%80%E0%BA%94%E0%BA%8D:%E0%BA%9B%E0%BA%B0%E0%BA%95%E0%BA%B9%E0%BA%AA%E0%BA%B9%E0%BB%88%E0%BA%8A%E0%BA%B8%E0%BA%A1%E0%BA%8A%E0%BA%BB%E0%BA%99&diff=prev&oldid=51692
6https://lo.wikipedia.org/w/index.php?title=%E0%BA%A7%E0%BA%B4%E0%BA%81%E0%BA%B4%E0%BA%9E%E0%BA%B5%E0%BB%80%E0%BA%94%E0%BA%8D:%E0%BA%9B%E0%BA%B0%E0%BA%95%E0%BA%B9%E0%BA%AA%E0%BA%B9%E0%BB%88%E0%BA%8A%E0%BA%B8%E0%BA%A1%E0%BA%8A%E0%BA%BB%E0%BA%99&diff=prev&oldid=51630
7https://ko.wikisource.org/w/index.php?title=%EC%9C%84%ED%82%A4%EB%AC%B8%ED%97%8C:%EC%82%AC%EB%9E%91%EB%B0%A9&diff=prev&oldid=136684
8https://nso.wikipedia.org/w/index.php?title=Boledi%C5%A1a:Letlakala_la_pele&diff=prev&oldid=15268
9https://nso.wikipedia.org/w/index.php?title=Boledi%C5%A1a:Letlakala_la_pele&diff=prev&oldid=15538
10https://bo.wikipedia.org/w/index.php?title=Wikipedia:Community_Portal&diff=prev&oldid=129861
11https://bo.wikipedia.org/w/index.php?title=Wikipedia:Community_Portal&diff=prev&oldid=129847
12https://crh.wikipedia.org/w/index.php?title=Muzakere:Ba%C5%9F_Saife&diff=prev&oldid=126722
13https://als.wikipedia.org/w/index.php?title=Wikipedia:Technik&diff=prev&oldid=638051
14https://zu.wikipedia.org/w/index.php?title=Wikipedia:Umnyango_wamgwamanda&diff=prev&oldid=38683
15https://es.wikiversity.org/w/index.php?title=Wikiversidad:Claustro_Wikiversitario&diff=prev&oldid=105252
16https://uk.wikiquote.org/w/index.php?title=%D0%92%D1%96%D0%BA%D1%96%D1%86%D0%B8%D1%82%D0%B0%D1%82%D0%B8:%D0%9A%D0%BD%D0%B0%D0%B9%D0%BF%D0%B0&diff=prev&oldid=43675
17https://dv.wiktionary.org/w/index.php?title=%DE%88%DE%A8%DE%86%DE%A8%DE%83%DE%A6%DE%8B%DE%A9%DE%8A%DE%AA:%DE%A2%DE%A7%DE%82%DE%B0%DE%89%DE%AA_%DE%80%DE%AE%DE%85%DE%AA%DE%87%DE%A6%DE%81%DE%A8&diff=prev&oldid=7397
18https://tw.wikipedia.org/w/index.php?title=Wikipedia:Community_Portal&diff=prev&oldid=16506
19https://sq.wikiquote.org/w/index.php?title=Wikiquote:Kuvendi&diff=prev&oldid=8228
20https://sq.wikiquote.org/w/index.php?title=Wikiquote:Kuvendi&diff=prev&oldid=8205
21https://sq.wikiquote.org/w/index.php?title=Wikiquote:Kuvendi&diff=prev&oldid=8198
22https://sq.wikiquote.org/w/index.php?title=Wikiquote:Kuvendi&diff=prev&oldid=8191
23https://yi.wiktionary.org/w/index.php?title=%D7%B0%D7%99%D7%A7%D7%99%D7%B0%D7%A2%D7%A8%D7%98%D7%A2%D7%A8%D7%91%D7%95%D7%9A:%D7%A7%D7%90%D6%B7%D7%B0%D7%A2%D6%BE%D7%94%D7%B1%D7%96&diff=prev&oldid=15212
24https://tyv.wikipedia.org/w/index.php?title=%D0%92%D0%B8%D0%BA%D0%B8%D0%BF%D0%B5%D0%B4%D0%B8%D1%8F:%D0%9A%D0%BE%D0%BB_%D0%B0%D1%80%D1%8B%D0%BD&diff=prev&oldid=15448
25https://wo.wikipedia.org/w/index.php?title=Wikipedia:Buntuw_Askan_wi&diff=prev&oldid=97982
26https://wo.wikipedia.org/w/index.php?title=Wikipedia:P%C3%A9nc_mi&diff=prev&oldid=97981
27https://sr.wikisource.org/w/index.php?title=%D0%92%D0%B8%D0%BA%D0%B8%D0%B7%D0%B2%D0%BE%D1%80%D0%BD%D0%B8%D0%BA:%D0%9F%D0%B8%D1%81%D0%B0%D1%80%D0%BD%D0%B8%D1%86%D0%B0&diff=prev&oldid=59854
28https://pdc.wikipedia.org/w/index.php?title=Wikipedia:Gmeeschafts-Portal&diff=prev&oldid=99130
29https://mt.wikipedia.org/w/index.php?title=Diskussjoni_Wikipedija:VisualEditor&diff=prev&oldid=244555
30https://ha.wiktionary.org/w/index.php?title=Wiktionary:Community_Portal&diff=prev&oldid=6638
31https://it.wikinews.org/w/index.php?title=Wikinotizie:Bar&diff=prev&oldid=930708
32https://da.wikisource.org/w/index.php?title=Wikisource:Skriptoriet&diff=prev&oldid=36753
33https://id.wikisource.org/w/index.php?title=Wikisource:Warung_kopi&diff=prev&oldid=42417
34https://be.wikibooks.org/w/index.php?title=Wikibooks:Community_Portal&diff=prev&oldid=4871
35https://be.wikibooks.org/w/index.php?title=Wikibooks:Community_Portal&diff=prev&oldid=4875
36https://min.wikipedia.org/w/index.php?title=Wikipedia:Kadai_Kopi&diff=prev&oldid=1047665
37https://gd.wikipedia.org/w/index.php?title=Uicipeid:Talla_a%27_Bhaile&diff=prev&oldid=499505
38https://chr.wiktionary.org/w/index.php?title=Wiktionary:Community_portal&diff=prev&oldid=385077
39https://sh.wikipedia.org/w/index.php?title=Wikipedia:Pijaca-%D0%9F%D0%B8%D1%98%D0%B0%D1%86%D0%B0&diff=prev&oldid=20190053
40https://kv.wikipedia.org/w/index.php?title=%D0%92%D0%B8%D0%BA%D0%B8%D0%BF%D0%B5%D0%B4%D0%B8%D1%8F%3A%D0%99%D0%B8%D1%82%D1%87%D3%A7%D0%BC_%D0%BF%D0%BE%D1%80%D1%82%D0%B0%D0%BB&type=revision&diff=121501&oldid=121500
41https://sco.wikipedia.org/w/index.php?title=Wikipedia:Mercat_Cross&diff=prev&oldid=488794
42https://th.wikibooks.org/w/index.php?title=%E0%B8%A7%E0%B8%B4%E0%B8%81%E0%B8%B4%E0%B8%95%E0%B8%B3%E0%B8%A3%E0%B8%B2:%E0%B8%AA%E0%B8%A0%E0%B8%B2%E0%B8%81%E0%B8%B2%E0%B9%81%E0%B8%9F&diff=prev&oldid=33173
43https://th.wikibooks.org/w/index.php?title=%E0%B8%A7%E0%B8%B4%E0%B8%81%E0%B8%B4%E0%B8%95%E0%B8%B3%E0%B8%A3%E0%B8%B2:%E0%B8%AA%E0%B8%A0%E0%B8%B2%E0%B8%81%E0%B8%B2%E0%B9%81%E0%B8%9F&diff=prev&oldid=33173
44https://sk.wikiquote.org/w/index.php?title=Wikicit%C3%A1ty:Kr%C4%8Dma&diff=prev&oldid=83595
45https://sk.wikiquote.org/w/index.php?title=Wikicit%C3%A1ty:Kr%C4%8Dma&diff=prev&oldid=83311
46https://nl.wikiquote.org/w/index.php?title=Wikiquote:De_kantine//archief_Wikimedia-meldingen&diff=prev&oldid=70416
47https://de.wikisource.org/w/index.php?title=Wikisource:Skriptorium&diff=prev&oldid=2368181
48https://dv.wikipedia.org/w/index.php?title=%DE%88%DE%A8%DE%86%DE%A8%DE%95%DE%A9%DE%91%DE%A8%DE%87%DE%A7:%DE%87%DE%A7%DE%82%DE%B0%DE%89%DE%AA_%DE%80%DE%AE%DE%85%DE%AA%DE%87%DE%A6%DE%81&diff=prev&oldid=118352
49https://cu.wikipedia.org/w/index.php?title=%D0%92%D0%B8%D0%BA%D0%B8%D0%BF%D1%94%D0%B4%D1%97%D1%A9_%D0%B1%D1%94%D1%81%D1%A3%CC%81%D0%B4%D0%B0:VisualEditor&diff=prev&oldid=70063
50https://cu.wikipedia.org/w/index.php?title=%D0%92%D0%B8%D0%BA%D0%B8%D0%BF%D1%94%D0%B4%D1%97%D1%A9_%D0%B1%D1%94%D1%81%D1%A3%CC%81%D0%B4%D0%B0:%D0%9E%D0%B1%D1%8C%D1%89%D1%94%D0%BD%D0%B8%EA%99%97_%D1%81%D1%8A%D0%B2%D1%A3%D1%82%D1%8A&diff=prev&oldid=70064
51https://de.wikiquote.org/w/index.php?title=Wikiquote:Fragen_zur_Wikiquote&diff=prev&oldid=490563
52https://de.wikiquote.org/w/index.php?title=Wikiquote:Fragen_zur_Wikiquote&diff=prev&oldid=490485
53https://bg.wikisource.org/w/index.php?title=%D0%A3%D0%B8%D0%BA%D0%B8%D0%B8%D0%B7%D1%82%D0%BE%D1%87%D0%BD%D0%B8%D0%BA:%D0%A0%D0%B0%D0%B7%D0%B3%D0%BE%D0%B2%D0%BE%D1%80%D0%B8&diff=prev&oldid=10204
54https://sq.wikibooks.org/w/index.php?title=Wikibooks:Kuvendi&diff=prev&oldid=31578
55https://sq.wikibooks.org/w/index.php?title=Wikibooks:Kuvendi&diff=prev&oldid=31573
56https://fy.wikibooks.org/w/index.php?title=Wikibooks:Buorren&diff=prev&oldid=4668
57https://fy.wikibooks.org/w/index.php?title=Wikibooks:Buorren&diff=prev&oldid=4658
58https://kaa.wikipedia.org/w/index.php?title=Wikipedia:Ja%27miyet_Portal%C4%B1&diff=prev&oldid=39163
59https://ar.wikisource.org/w/index.php?title=%D9%88%D9%8A%D9%83%D9%8A_%D9%85%D8%B5%D8%AF%D8%B1:%D8%A7%D9%84%D9%85%D9%8A%D8%AF%D8%A7%D9%86&diff=prev&oldid=218751
60https://sq.wiktionary.org/w/index.php?title=Wiktionary:Kuvendi&diff=prev&oldid=102274
61https://li.wikisource.org/w/index.php?title=Wikibr%C3%B3nne:Gebroekersportaol&diff=prev&oldid=6012
62https://en.wikivoyage.org/w/index.php?title=Wikivoyage:Travellers%27_pub&diff=prev&oldid=2839153
63https://en.wikivoyage.org/w/index.php?title=Wikivoyage:Travellers%27_pub&diff=prev&oldid=2832114
64https://he.wikiquote.org/w/index.php?title=%D7%95%D7%99%D7%A7%D7%99%D7%A6%D7%99%D7%98%D7%95%D7%98:%D7%9E%D7%96%D7%A0%D7%95%D7%9F&diff=prev&oldid=122957
65https://eo.wikiquote.org/w/index.php?title=Vikicitaro:Diskutejo&diff=prev&oldid=24105
66https://fy.wikipedia.org/w/index.php?title=Wikipedy:Oerlisside&diff=prev&oldid=757266
67https://ja.wikinews.org/w/index.php?title=%E3%82%A6%E3%82%A3%E3%82%AD%E3%83%8B%E3%83%A5%E3%83%BC%E3%82%B9:%E8%B5%A4%E6%8F%90%E7%81%AF&diff=prev&oldid=141414
68https://bjn.wikipedia.org/w/index.php?title=Wikipidia:Warung_Kupi&diff=prev&oldid=52849
69https://bjn.wikipedia.org/w/index.php?title=Wikipidia:Warung_Kupi&diff=prev&oldid=52466
70https://haw.wikipedia.org/w/index.php?title=Wikipedia:Ka_hui_kaiaulu&diff=prev&oldid=71497
71https://it.wikiquote.org/w/index.php?title=Wikiquote:Bar&diff=prev&oldid=736030
72https://mdf.wikipedia.org/w/index.php?title=%D0%92%D0%B8%D0%BA%D0%B8%D0%BF%D0%B5%D0%B4%D0%B8%D0%B5%D1%81%D1%8C:Community_Portal&diff=prev&oldid=46852
73https://da.wikiquote.org/w/index.php?title=Wikiquote:Caf%C3%A9en&diff=prev&oldid=16394
74https://is.wikipedia.org/w/index.php?title=Wikipedia:Potturinn&diff=prev&oldid=1509294
75https://is.wikipedia.org/w/index.php?title=Wikipediaspjall:Visual_Editor&diff=prev&oldid=1509293
76https://ms.wikibooks.org/w/index.php?title=Wikibooks:Kedai_Kopi&diff=prev&oldid=10114
77https://se.wikipedia.org/w/index.php?title=Wikipedia:G%C3%A1ffestohpu&diff=prev&oldid=274865
78https://se.wikipedia.org/w/index.php?title=Port%C3%A1las%C3%A1gastallan:Ovdasiidu&diff=prev&oldid=274864
79https://kl.wikipedia.org/w/index.php?title=Wikipedia:Saqqaa&diff=prev&oldid=70042
80https://na.wiktionary.org/w/index.php?title=Wiktionary:Embassy&diff=prev&oldid=10348
81https://sl.wikisource.org/w/index.php?title=Wikivir:Pod_lipo&diff=prev&oldid=131611
82https://pcd.wikipedia.org/w/index.php?title=Wikipedia:Accueul_del_conminnité&diff=prev&oldid=51956
83https://eo.wikisource.org/w/index.php?title=Vikifontaro:Sistemkomunika%C4%B5oj&diff=prev&oldid=26408
84https://co.wikipedia.org/w/index.php?title=Wikipedia:Purtone_di_a_cumunit%C3%A0&diff=prev&oldid=351639
85https://jbo.wikipedia.org/w/index.php?title=Wikipedia:bende_ckupau&diff=prev&oldid=106939
86https://no.wikisource.org/w/index.php?title=Wikikilden:Kontoret&diff=prev&oldid=102215
87https://fa.wiktionary.org/w/index.php?title=%D9%88%DB%8C%DA%A9%DB%8C%E2%80%8C%D9%88%D8%A7%DA%98%D9%87:%D9%82%D9%87%D9%88%D9%87%E2%80%8C%D8%AE%D8%A7%D9%86%D9%87&diff=prev&oldid=683563
88https://la.wikisource.org/w/index.php?title=Vicifons:Scriptorium&diff=prev&oldid=72613
89https://bg.wikinews.org/w/index.php?title=%D0%A3%D0%B8%D0%BA%D0%B8%D0%BD%D0%BE%D0%B2%D0%B8%D0%BD%D0%B8:%D0%A0%D0%B0%D0%B7%D0%B3%D0%BE%D0%B2%D0%BE%D1%80%D0%B8&diff=prev&oldid=16678
90https://la.wikibooks.org/w/index.php?title=Vicilibri:Porta_communis&diff=prev&oldid=7700
91https://la.wikibooks.org/w/index.php?title=Vicilibri:Porta_communis&diff=prev&oldid=7695
92https://it.wikibooks.org/w/index.php?title=Wikibooks:Bar&diff=prev&oldid=297444
93https://vi.wikibooks.org/w/index.php?title=Wikibooks:B%C3%A0n_th%E1%BA%A3o_lu%E1%BA%ADn/B%E1%BA%A3ng_th%C3%B4ng_tin&diff=prev&oldid=100320
94https://vo.wikipedia.org/w/index.php?title=V%C3%BCkiped:Kafetar&diff=prev&oldid=3182543
95https://hy.wikisource.org/w/index.php?title=%D5%8E%D5%AB%D6%84%D5%AB%D5%A4%D5%A1%D6%80%D5%A1%D5%B6:%D4%BD%D5%B8%D6%80%D5%B0%D6%80%D5%A4%D5%A1%D6%80%D5%A1%D5%B6&diff=prev&oldid=85699
96https://tum.wikipedia.org/w/index.php?title=Wikipedia:Community_Portal&diff=prev&oldid=19593
97https://el.wikiquote.org/w/index.php?title=%CE%92%CE%B9%CE%BA%CE%B9%CF%86%CE%B8%CE%AD%CE%B3%CE%BC%CE%B1%CF%84%CE%B1:%CE%92%CE%AE%CE%BC%CE%B1&diff=prev&oldid=29640
98https://el.wikiquote.org/w/index.php?title=%CE%92%CE%B9%CE%BA%CE%B9%CF%86%CE%B8%CE%AD%CE%B3%CE%BC%CE%B1%CF%84%CE%B1:%CE%92%CE%AE%CE%BC%CE%B1&diff=prev&oldid=29566
99https://it.wikivoyage.org/w/index.php?title=Wikivoyage:Lounge&diff=prev&oldid=428851
100https://ka.wikiquote.org/w/index.php?title=%E1%83%95%E1%83%98%E1%83%99%E1%83%98%E1%83%AA%E1%83%98%E1%83%A2%E1%83%90%E1%83%A2%E1%83%90:%E1%83%A4%E1%83%9D%E1%83%A0%E1%83%A3%E1%83%9B%E1%83%98&diff=prev&oldid=27257
101https://sa.wiktionary.org/w/index.php?title=Wiktionary:Community_Portal&diff=prev&oldid=29012
102https://ko.wikiquote.org/w/index.php?title=%EC%9C%84%ED%82%A4%EC%9D%B8%EC%9A%A9%EC%A7%91:%EC%82%AC%EB%9E%91%EB%B0%A9&diff=prev&oldid=25135
103https://gv.wiktionary.org/w/index.php?title=Wiktionary:Thie_Oast&diff=prev&oldid=11467
104https://tg.wikipedia.org/w/index.php?title=%D0%92%D0%B8%D0%BA%D0%B8%D0%BF%D0%B5%D0%B4%D0%B8%D0%B0:%D0%92%D1%83%D1%80%D1%83%D0%B4%D0%B8_%D0%BA%D0%BE%D1%80%D0%B1%D0%B0%D1%80%D0%BE%D0%BD&diff=prev&oldid=653305
105https://hy.wikiquote.org/w/index.php?title=%D5%8E%D5%AB%D6%84%D5%AB%D6%84%D5%A1%D5%B2%D5%BE%D5%A1%D5%AE%D6%84:%D4%BD%D5%B8%D6%80%D5%B0%D6%80%D5%A4%D5%A1%D6%80%D5%A1%D5%B6&diff=prev&oldid=27106
106https://mg.wiktionary.org/w/index.php?title=Wiktionary:Community_Portal&diff=prev&oldid=21373262
107https://vi.wikisource.org/w/index.php?title=Wikisource:Th%E1%BA%A3o_lu%E1%BA%ADn&diff=prev&oldid=51474
108https://fi.wikiquote.org/w/index.php?title=Wikisitaatit:Kahvihuone&diff=prev&oldid=54603
109https://fi.wikiquote.org/w/index.php?title=Wikisitaatit:Kahvihuone&diff=prev&oldid=53831
110https://af.wikiquote.org/w/index.php?title=Wikiquote:Geselshoekie&diff=prev&oldid=7928
111https://ca.wiktionary.org/w/index.php?title=Viccionari:La_taverna&diff=prev&oldid=571182
112https://uk.wikivoyage.org/w/index.php?title=%D0%92%D1%96%D0%BA%D1%96%D0%BC%D0%B0%D0%BD%D0%B4%D1%80%D0%B8:%D0%9A%D0%BD%D0%B0%D0%B9%D0%BF%D0%B0&diff=prev&oldid=14609
113https://uk.wikivoyage.org/w/index.php?title=%D0%92%D1%96%D0%BA%D1%96%D0%BC%D0%B0%D0%BD%D0%B4%D1%80%D0%B8:%D0%9A%D0%BD%D0%B0%D0%B9%D0%BF%D0%B0&diff=prev&oldid=14583
114https://sn.wikipedia.org/w/index.php?title=Wikipedia:Community_Portal&diff=prev&oldid=41055
115https://gl.wikibooks.org/w/index.php?title=Wikibooks:A_Taberna&diff=prev&oldid=14959
116https://ba.wikipedia.org/w/index.php?title=%D0%92%D0%B8%D0%BA%D0%B8%D0%BF%D0%B5%D0%B4%D0%B8%D1%8F:Bar&diff=prev&oldid=525416
117https://jv.wiktionary.org/w/index.php?title=Wiktionary:Warung_Kopi&diff=prev&oldid=103647
118https://id.wiktionary.org/w/index.php?title=Wiktionary:Warung_kopi&diff=prev&oldid=765313
119https://kn.wikisource.org/w/index.php?title=Wikisource:Community_Portal&diff=prev&oldid=17790
120https://ast.wiktionary.org/w/index.php?title=Uiccionariu:Chigre&diff=prev&oldid=149847
121https://ny.wikipedia.org/w/index.php?title=Wikipedia:Community_Portal&diff=prev&oldid=16189
122https://vi.wikiquote.org/w/index.php?title=Wikiquote:Th%E1%BA%A3o_lu%E1%BA%ADn&diff=prev&oldid=30762
123https://vi.wikiquote.org/w/index.php?title=Wikiquote:Th%E1%BA%A3o_lu%E1%BA%ADn&diff=prev&oldid=30693
124https://hi.wikibooks.org/w/index.php?title=Wikibooks:Community_Portal&diff=prev&oldid=7793
125https://ar.wikiversity.org/w/index.php?title=%D9%88%D9%8A%D9%83%D9%8A_%D8%A7%D9%84%D8%AC%D8%A7%D9%85%D8%B9%D8%A9:%D8%A7%D9%84%D9%85%D9%8A%D8%AF%D8%A7%D9%86&diff=prev&oldid=48729
126https://tpi.wikipedia.org/w/index.php?title=Wikipedia:Haus_bilong_toktok&diff=prev&oldid=82577
127https://cbk-zam.wikipedia.org/w/index.php?title=Wikipedia:Portal_del_Comunidad&diff=prev&oldid=92498
128https://bs.wikibooks.org/w/index.php?title=Wikiknjige:%C4%8Car%C5%A1ija&diff=prev&oldid=6336
129https://bs.wikibooks.org/w/index.php?title=Wikiknjige:%C4%8Car%C5%A1ija&diff=prev&oldid=6332
130https://ff.wikipedia.org/w/index.php?title=Wikipedia:Accueil&diff=prev&oldid=19994
131https://ml.wiktionary.org/w/index.php?title=%E0%B4%B5%E0%B4%BF%E0%B4%95%E0%B5%8D%E0%B4%95%E0%B4%BF%E0%B4%A8%E0%B4%BF%E0%B4%98%E0%B4%A3%E0%B5%8D%E0%B4%9F%E0%B5%81:%E0%B4%B5%E0%B4%BF%E0%B4%95%E0%B5%8D%E0%B4%95%E0%B4%BF_%E0%B4%AA%E0%B4%9E%E0%B5%8D%E0%B4%9A%E0%B4%BE%E0%B4%AF%E0%B4%A4%E0%B5%8D%E0%B4%A4%E0%B5%8D_%28%E0%B4%B5%E0%B4%BE%E0%B5%BC%E0%B4%A4%E0%B5%8D%E0%B4%A4%E0%B4%95%E0%B5%BE%29&diff=prev&oldid=451958
132https://sw.wiktionary.org/w/index.php?title=Majadiliano:Mwanzo&diff=prev&oldid=113957
133https://pl.wiktionary.org/w/index.php?title=Wikis%C5%82ownik:Bar&diff=prev&oldid=4746786
134https://sl.wikibooks.org/w/index.php?title=Wikiknjige:Pod_lipo&diff=prev&oldid=14997
135https://ht.wikipedia.org/w/index.php?title=Wikipedya:Kafe&diff=prev&oldid=688083
136https://kbd.wikipedia.org/w/index.php?title=%D0%A3%D0%B8%D0%BA%D0%B8%D0%BF%D0%B5%D0%B4%D0%B8%D1%8D:%D0%A3%D1%8D%D1%80%D1%88%D1%8D%D1%80%D1%8B%D0%BF%D3%80%D1%8D/%D0%90%D0%B4%D1%8B%D0%B3%D1%8D%D0%B1%D0%B7%D1%8D&diff=prev&oldid=39889
137https://ku.wikibooks.org/w/index.php?title=W%C3%AEk%C3%AEpirt%C3%BBk:Portala_kom%C3%AA&diff=prev&oldid=5809
138https://pl.wikibooks.org/w/index.php?title=Wikibooks:Bar&diff=prev&oldid=218564
139https://ku.wikiquote.org/w/index.php?title=W%C3%AEk%C3%AEgotin:Portala_kom%C3%AA&diff=prev&oldid=10869
140https://bar.wikipedia.org/w/index.php?title=Wikipedia:Technik&diff=prev&oldid=477699
141https://fi.wikinews.org/w/index.php?title=Wikiuutiset:Kahvihuone_%28uutiset%29&diff=prev&oldid=51987
142https://fi.wikinews.org/w/index.php?title=Wikiuutiset:Kahvihuone_%28uutiset%29&diff=prev&oldid=50495
143https://el.wikibooks.org/w/index.php?title=%CE%92%CE%B9%CE%BA%CE%B9%CE%B2%CE%B9%CE%B2%CE%BB%CE%AF%CE%B1:%CE%91%CE%B3%CE%BF%CF%81%CE%AC&diff=prev&oldid=37020
144https://el.wikibooks.org/w/index.php?title=%CE%92%CE%B9%CE%BA%CE%B9%CE%B2%CE%B9%CE%B2%CE%BB%CE%AF%CE%B1:%CE%91%CE%B3%CE%BF%CF%81%CE%AC&diff=prev&oldid=37005
145https://sah.wikipedia.org/w/index.php?title=%D0%91%D0%B8%D0%BA%D0%B8%D0%BF%D0%B8%D1%8D%D0%B4%D1%8C%D0%B8%D0%B9%D1%8D:%D0%9A%D1%8D%D0%BF%D1%81%D1%8D%D1%82%D1%8D%D1%80_%D1%81%D0%B8%D1%80/%D0%A2%D0%B5%D1%85%D0%BD%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D1%8D%D0%B9&diff=prev&oldid=319893
146https://sah.wikipedia.org/w/index.php?title=%D0%91%D0%B8%D0%BA%D0%B8%D0%BF%D0%B8%D1%8D%D0%B4%D1%8C%D0%B8%D0%B9%D1%8D:%D0%9A%D1%8D%D0%BF%D1%81%D1%8D%D1%82%D1%8D%D1%80_%D1%81%D0%B8%D1%80/%D0%A2%D0%B5%D1%85%D0%BD%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D1%8D%D0%B9&diff=prev&oldid=318910
147https://fo.wiktionary.org/w/index.php?title=Wiktionary:Undirh%C3%BAsi%C3%B0&diff=prev&oldid=15976
148https://is.wikiquote.org/w/index.php?title=Wikivitnun:Potturinn&diff=prev&oldid=12030
149https://is.wikiquote.org/w/index.php?title=Wikivitnun:Potturinn&diff=prev&oldid=11949

It's also a problem on hu.wikisource. This wasn't the first duplication, about the half of messages arrived twice since the creation of the village pump in July.

Legoktm added a subscriber: Legoktm.

(I'm not actively investigating this)

I'm afraid this is back?
RN https://guc.toolforge.org/?by=date&user=MediaWiki_message_delivery says I delivered the same message at 16:42 and at 02:35. I'm going to clean up -_-

Joe added subscribers: Pchelolo, Ottomata, Joe.

@Pchelolo @Ottomata do we have any way to verify how this happened from eventgate and changeprop logs?

I am looking at the GUC log linked above and it appears that first delivery was successful on 145 of the 154 targeted wikis - which is probably normal, as some of those targets are potentially not open to MM delivery - and then, the system re-delivered to 70 of those.

For more quirkiness, I had recently brought up that in a mass message delivery, the GUC log would only feature a few entries even though the message was correctly delivered to a way longer list (and it wasn't a lag issue, this was observed several hours after sending). Please LMK if this is not helpful here - I can report separately. TYVM.

@Joe I don't have full context of how MassMessageJob and JobQueue work here, but at the very least it seems we are able to save the MassMessageJob events to Hive in the event.mediawiki_job_massmessagejob table. Unfortunetly we do run the events in these tables through a deduplicate function, which deduplicates them based on their meta.id field. The raw JSON event data from Kafka is saved in HDFS for 90 days in /wmf/data/raw/event/{eqiad,codfw}_mediawiki_job_MassMessageJob, so we could look there. We also have raw data for /wmf/data/raw/event/{eqiad,codfw}_mediawiki_job_MassMessageSubmitJob (we don't refine MassMessageSubmitJob into Hive because its format has not been consistent enough to do so, so it has been manually excluded from refinement).

Just noting for the record, I had similar problems on Monday, whilst delivering TechNews.
It delivered duplicates to 6 Wiktionary pages, at "ja, la, nl, ml, simple, vi", from this list:
https://meta.wikimedia.org/wiki/User:Johan_(WMF)/Tech_News_target_list_1
The duplicates were delivered ~53 minutes after the first delivery, e.g. at :
https://la.wiktionary.org/w/index.php?title=Victionarium:Nuntii_technici&offset=&limit=20&action=history

AMooney added a subscriber: AMooney.

If Platform team is needed, retag us.

For more quirkiness, I had recently brought up that in a mass message delivery, the GUC log would only feature a few entries even though the message was correctly delivered to a way longer list (and it wasn't a lag issue, this was observed several hours after sending). Please LMK if this is not helpful here - I can report separately. TYVM.

This is now T259454.

I hope this doesn't happen to my delivery as well :(((
I had to contact 588 wikis earlier today. Please cross fingers, toes, whatever you have for me.

[Jul 23, 2020] In T93049#6328594, @Joe added projects: Platform Engineering:

@Pchelolo @Ottomata do we have any way to verify how this happened from eventgate and changeprop logs?

[Jul 28 2020] In T93049#6342727, @AMooney removed a project: Platform Engineering.

If Platform team is needed, retag us.

Yes, per Joe's comment. From what I can see, it was not addressed or answer yet.

At least one message (of 111) I sent out yesterday duplicated, on the Italian Wikipedia (https://it.wikipedia.org/wiki/Discussioni_utente:Keegan_(WMF)#Twice).

Edit: there were a few more duplicates left in the run, the most egregious was three copies being left at https://www.wikidata.org/wiki/User_talk:Pigsonthewing (https://www.wikidata.org/w/index.php?title=User_talk%3APigsonthewing&type=revision&diff=1281386944&oldid=1281276097)

Unfortunately the editing team does not have any expertise is this area.

@Esanders: Is the Editing-team still going to investigate this problem, despite lack of expertise?
If not, then a https://www.mediawiki.org/wiki/Code_stewardship_reviews should be initiated to define code stewards for MassMessage.

Per the page mentioned above, pinging @Legoktm , @wctaiwan and @DAlangi_WMF in case they have anything to contribute here.

While MassMessage is how users see the problem (e.g. no one really notices a page's cache being invalidated twice, unnecessarily), the real issue here is that the job queue is executing a job multiple times, and it really shouldn't be. And unfortunately I no longer really understand the internals of how the job queue works in the new kafka system to be able to debug what's going wrong here.

Ways forward: 1) someone looks into why the job queue is running jobs twice, and/or 2) MassMessage implements double-posting protection.

I think #1 needs to be looked at regardless, because running jobs twice seems problematic even besides MassMessage.

For #2, we could implement something similar to EdwardsBot's counter system. At job submission, we generate a unique Message-ID, like an email would have, include it in the wikitext of the message as a HTML comment, <!-- Message-ID: {...} --> (maybe in the same HTML comment we already add) and then before adding a new section, check for that string. The primary flaw in something like this is race conditions, so the best and probably only complete way to fix this (short of dropping the job queue entirely) is #1.

In T93049#6328594, @Joe wrote:

@Pchelolo @Ottomata do we have any way to verify how this happened from eventgate and changeprop logs?

Ping again?

Are the jobs failing near the end and getting retried? Also, job B can still be enqueued if a duplicate job A is marked as running. This was long-since the semantic logic (and the kafka queue does this too last I checked). If job A fails, it seems like the redis job queue (https://github.com/wikimedia/mediawiki-services-jobrunner/blob/master/redisJobChronService) would then allow duplicate job C, though JobQueueDB would not. Not sure about the Kafka queue in that situation. The behavior there should be standardized in any case.

That said, job-deduplication is just for performance, not for semantics, so the Job::run() code itself should also have a robust check to catch any duplicates (or be otherwise idempotent).

@Pchelolo and I looked at the logs and investigated the most recent incident reported by @Samwalton9 and in that case, we noticed that MassMessage is treating the message as two separate jobs because of the text. Below you can see the difference in the two messages. One appears to be sent by a script from User:Evad37/SPS while the other one is not.

The small change in the text created duplicates for MassMessageSubmitJob and every subscriber of the mailing list received two messages.

Message 1:

<div lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\"><div style=\"-moz-column-count:2; -webkit-column-count:2; column-count:2;\"> {{Wikipedia:Wikipedia Signpost/2020-09-27}} </div><!--Volume 16, Issue 9--> <div class=\"hlist\" style=\"margin-top:10px; font-size:90%; padding-left:5px; font-family:Georgia, Palatino, Palatino Linotype, Times, Times New Roman, serif;\"> * '''[[Wikipedia:Wikipedia Signpost|Read this Signpost in full]]''' * [[Wikipedia:Wikipedia Signpost/Single/2020-09-27|Single-page]] * [[Wikipedia:Wikipedia Signpost/Subscribe|Unsubscribe]] * [[User:MediaWiki message delivery|MediaWiki message delivery]] ([[User talk:MediaWiki message delivery|talk]]) ~~~~~ <!-- Sent via script ([[User:Evad37/SPS]]) --></div></div>

Message 2:

div lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\"><div style=\"-moz-column-count:2; -webkit-column-count:2; column-count:2;\"> {{WP:Wikipedia Signpost/2020-09-27}} </div><!--Volume 16, Issue 09--> <div class=\"hlist\" style=\"margin-top:10px; font-size:90%; padding-left:5px; font-family:Georgia, Palatino, Palatino Linotype, Times, Times New Roman, serif;\"> * '''[[Wikipedia:Wikipedia Signpost|Read this Signpost in full]]''' * [[Wikipedia:Wikipedia Signpost/Single/2020-09-27|Single-page]] * [[Wikipedia:Wikipedia Signpost/Subscribe|Unsubscribe]] * [[User:MediaWiki message delivery|MediaWiki message delivery]] ([[User talk:MediaWiki message delivery|talk]]) ~~~~~ </div></div>

While MassMessage is how users see the problem (e.g. no one really notices a page's cache being invalidated twice, unnecessarily), the real issue here is that the job queue is executing a job multiple times, and it really shouldn't be.

{{cn}}

I see no conclusive proof that's what's happening.

In this case the queue is executing the job multiple times because these are different jobs. The posted text is different. The second one contains a comment which the first one does not. We have deduplication on the jobqueue side, but it just uses whatever Job::getDeduplicationInfo gives it. Message text is a part of the job deduplication hash, so JobQueue operates correctly here.

I think the User:Evad37/SPS bot is a problem here for posting the message twice. I do not know enough about how that bot works, what does it do, or how these messages get into the system, so I can't judge if it's possible to add some better deduplication info there. If it is, we can just override getDeduplicationInfo for this job and dedupe it.

It seems like the Signpost article duplication was a user error, per the discussion at https://en.wikipedia.org/wiki/Wikipedia_talk:Wikipedia_Signpost#Duplicate_delivery%3F. The Books & Bytes example I mentioned further up, however, was definitely only sent once.

The Books & Bytes example I mentioned further up, however, was definitely only sent once.

Unfortunately the detailed logs and jobs for that one has expired by now. In the current system we have only 1 week when all the info is preserved. As soon as this happens again, please add an example here and we will investigate again.

Thanks everyone for your help here.

Ok, it did execute the job twice:

Once on 27th:

2020-10-27 19:52:28 [34499b04-8b9a-4cd1-95dd-9229906705c7] mw1303 huwiki 1.36.0-wmf.14 JobExecutor DEBUG: Beginning job execution {"job":"MassMessageJob Szerkesztővita:Tgr spamlist=Szerkesztő:Samat/MassMessages subject=Októberi online wikitalálkozó page-message= message=Szia!
[[User:Samat|Samat]] <sup>[[User vita:Samat|üzenetrögzítő]]</sup> ~~~~~ • ([[Wikipédia:Wikitalálkozók/Érdeklődők|feliratkozás és leiratkozás]]) comment=[\"Samat\",\"huwiki\",\"https://hu.wikipedia.org/w/index.php?title=Szerkeszt%C5%91:Samat/MassMessages\\u0026oldid=23070985\"] userId=949 originWiki=huwiki isSourceTranslationPage= translationPageSourceLanguage= pageMessageTitle= title=Szerkesztővita:Tgr rootJobSignature=8d2e0b34ef7fe5249c9182c2efb0063261ae50bd rootJobTimestamp=20201027195225 requestId=34499b04-8b9a-4cd1-95dd-9229906705c7","job_type":"MassMessageJob"} 
2020-10-27 19:52:35 [34499b04-8b9a-4cd1-95dd-9229906705c7] mw1303 huwiki 1.36.0-wmf.14 JobExecutor INFO: Finished job execution {"job":"MassMessageJob Szerkesztővita:Tgr spamlist=Szerkesztő:Samat/MassMessages subject=Októberi online wikitalálkozó page-message= message=Szia!
[[User:Samat|Samat]] <sup>[[User vita:Samat|üzenetrögzítő]]</sup> ~~~~~ • ([[Wikipédia:Wikitalálkozók/Érdeklődők|feliratkozás és leiratkozás]]) comment=[\"Samat\",\"huwiki\",\"https://hu.wikipedia.org/w/index.php?title=Szerkeszt%C5%91:Samat/MassMessages\\u0026oldid=23070985\"] userId=949 originWiki=huwiki isSourceTranslationPage= translationPageSourceLanguage= pageMessageTitle= title=Szerkesztővita:Tgr rootJobSignature=8d2e0b34ef7fe5249c9182c2efb0063261ae50bd rootJobTimestamp=20201027195225 requestId=34499b04-8b9a-4cd1-95dd-9229906705c7","job_type":"MassMessageJob","job_status":true,"job_duration":7.977766990661621}

Once on 28th:

2020-10-28 16:06:39 [34499b04-8b9a-4cd1-95dd-9229906705c7] mw1294 huwiki 1.36.0-wmf.14 JobExecutor DEBUG: Beginning job execution {"job":"MassMessageJob Szerkesztővita:Tgr spamlist=Szerkesztő:Samat/MassMessages subject=Októberi online wikitalálkozó page-message= message=Szia!
[[User:Samat|Samat]] <sup>[[User vita:Samat|üzenetrögzítő]]</sup> ~~~~~ • ([[Wikipédia:Wikitalálkozók/Érdeklődők|feliratkozás és leiratkozás]]) comment=[\"Samat\",\"huwiki\",\"https://hu.wikipedia.org/w/index.php?title=Szerkeszt%C5%91:Samat/MassMessages\\u0026oldid=23070985\"] userId=949 originWiki=huwiki isSourceTranslationPage= translationPageSourceLanguage= pageMessageTitle= title=Szerkesztővita:Tgr rootJobSignature=8d2e0b34ef7fe5249c9182c2efb0063261ae50bd rootJobTimestamp=20201027195225 requestId=34499b04-8b9a-4cd1-95dd-9229906705c7","job_type":"MassMessageJob"} 
2020-10-28 16:06:43 [34499b04-8b9a-4cd1-95dd-9229906705c7] mw1294 huwiki 1.36.0-wmf.14 JobExecutor INFO: Finished job execution {"job":"MassMessageJob Szerkesztővita:Tgr spamlist=Szerkesztő:Samat/MassMessages subject=Októberi online wikitalálkozó page-message= message=Szia!
[[User:Samat|Samat]] <sup>[[User vita:Samat|üzenetrögzítő]]</sup> ~~~~~ • ([[Wikipédia:Wikitalálkozók/Érdeklődők|feliratkozás és leiratkozás]]) comment=[\"Samat\",\"huwiki\",\"https://hu.wikipedia.org/w/index.php?title=Szerkeszt%C5%91:Samat/MassMessages\\u0026oldid=23070985\"] userId=949 originWiki=huwiki isSourceTranslationPage= translationPageSourceLanguage= pageMessageTitle= title=Szerkesztővita:Tgr rootJobSignature=8d2e0b34ef7fe5249c9182c2efb0063261ae50bd rootJobTimestamp=20201027195225 requestId=34499b04-8b9a-4cd1-95dd-9229906705c7","job_type":"MassMessageJob","job_status":true,"job_duration":4.2106781005859375}

looking further

Ok, a bit more:

16:06 ppchelko@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .

which corresponds with the time of the second execution.

So, we have found it: the same exact job has been executed twice. I have deployed change-prop for jobqueue right before the second execution. The deduplicator didn't work out in this case since we expire the keys in Redis after 1 hour for by-id and by-sha1 deduplicators.

This suggests a bug in commit logic. Investigation to be continued.

So, we have found it: the same exact job has been executed twice. I have deployed change-prop for jobqueue right before the second execution. The deduplicator didn't work out in this case since we expire the keys in Redis after 1 hour for by-id and by-sha1 deduplicators.

This suggests a bug in commit logic. Investigation to be continued.

Thanks for your work, please let us know how we can assist.

One more new example of a duplication.
first send (22:10) and second send (22:11).
I've checked ~50 other random entries from the distribution list I used, and that was the only duplication I could see.

One more new example of a duplication.
first send (22:10) and second send (22:11).
I've checked ~50 other random entries from the distribution list I used, and that was the only duplication I could see.

@Pchelolo ping in case this new instance has elements that may be helpful to you

@Pchelolo new instance today, sadly. GUC tells me that around 8 messages were delivered again 1h after the first delivery.

@Snaevar Thanks for the report. Unfortunately, (per the comment above in T93049#6512485) the detailed-version of the server-side logs for this system are only retained for a week, so the devs cannot investigate this example.
Note: The logs say this was only sent once (00:22 UTC), so this instance was not user-error.