Page MenuHomePhabricator

CentralNotice: choiceData RL module hashes are flapping
Closed, ResolvedPublic1 Story Points

Description

https://dpaste.de/pfek/raw

These hashes should only change when actual choices for a given RL context change.

This might be a cause of T133765 (or, rather, a contributing issue that caused us to get caught up in the race condition described in T117587).

Event Timeline

Restricted Application added subscribers: Luke081515, TerraCodes, Urbanecm, Aklapper. · View Herald TranscriptApr 29 2016, 4:00 AM

Change 286119 had a related patch set uploaded (by AndyRussG):
Ensure consistent ordering of choiceData for RL module hashes

https://gerrit.wikimedia.org/r/286119

I was able to verify that this is due to ordering. I sent repeated requests for https://en.wikipedia.org/w/load.php?modules=ext.centralNotice.choiceData to appservers.svc.eqiad.wmnet until I got two different responses. I stripped the JavaScript wrapping to leave just the JSON, and expanded !0 and !1 to obtain valid JSON. I then parsed the JSON and reserialized it with the pretty-printing option enabled, so I could get a meaningful diff.

The only differences the diff shows are in the ordering of the keys in the mixin objects. The perflogbot log shows this module flapping between just two distinct hash values, and I found two distinct responses, so that means there aren't any other flapping issues, at least with the configuration we have on enwiki right now. It is possible that there are other lists that are inconsistently ordered, but there are <2 items in those lists right now, so we still want to audit that and ensure everything has a consistent order.

1--- response1-pretty 2016-04-28 17:42:55.424115170 -0700
2+++ response2-pretty 2016-04-28 21:40:09.590187322 -0700
3@@ -76,21 +76,21 @@
4 ],
5 "mixins": {
6 "bannerHistoryLogger": {
7- "maxEntries": 30,
8- "maxEntryAge": 365,
9 "rate": 0.01,
10+ "maxEntryAge": 365,
11+ "maxEntries": 30,
12 "waitLogNoSendBeacon": 200
13 },
14 "legacySupport": {
15- "bannersNotGuaranteedToDisplay": true,
16 "setSRISampleRate": true,
17- "sriSampleRate": 1
18+ "sriSampleRate": 1,
19+ "bannersNotGuaranteedToDisplay": true
20 },
21 "impressionDiet": {
22 "cookieName": "centralnotice_bannercount_fr15",
23+ "skipInitial": 5,
24 "maximumSeen": 1,
25- "restartCycleDelay": 2592000,
26- "skipInitial": 5
27+ "restartCycleDelay": 2592000
28 }
29 }
30 },
31@@ -150,21 +150,21 @@
32 ],
33 "mixins": {
34 "bannerHistoryLogger": {
35- "maxEntries": 30,
36- "maxEntryAge": 365,
37 "rate": 0.01,
38+ "maxEntryAge": 365,
39+ "maxEntries": 30,
40 "waitLogNoSendBeacon": 200
41 },
42 "legacySupport": {
43- "bannersNotGuaranteedToDisplay": true,
44 "setSRISampleRate": true,
45- "sriSampleRate": 1
46+ "sriSampleRate": 1,
47+ "bannersNotGuaranteedToDisplay": true
48 },
49 "impressionDiet": {
50 "cookieName": "centralnotice_bannercount_fr15",
51+ "skipInitial": 5,
52 "maximumSeen": 1,
53- "restartCycleDelay": 2592000,
54- "skipInitial": 5
55+ "restartCycleDelay": 2592000
56 }
57 }
58 },
59@@ -222,21 +222,21 @@
60 ],
61 "mixins": {
62 "bannerHistoryLogger": {
63- "maxEntries": 30,
64- "maxEntryAge": 365,
65 "rate": 0.01,
66+ "maxEntryAge": 365,
67+ "maxEntries": 30,
68 "waitLogNoSendBeacon": 200
69 },
70 "legacySupport": {
71- "bannersNotGuaranteedToDisplay": true,
72 "setSRISampleRate": true,
73- "sriSampleRate": 1
74+ "sriSampleRate": 1,
75+ "bannersNotGuaranteedToDisplay": true
76 },
77 "impressionDiet": {
78 "cookieName": "centralnotice_bannercount_fr15",
79+ "skipInitial": 5,
80 "maximumSeen": 1,
81- "restartCycleDelay": 2592000,
82- "skipInitial": 5
83+ "restartCycleDelay": 2592000
84 }
85 }
86 },
87@@ -299,26 +299,26 @@
88 ],
89 "mixins": {
90 "bannerHistoryLogger": {
91- "maxEntries": 30,
92- "maxEntryAge": 30,
93 "rate": 0.01,
94+ "maxEntryAge": 30,
95+ "maxEntries": 30,
96 "waitLogNoSendBeacon": 200
97 },
98 "legacySupport": {
99- "bannersNotGuaranteedToDisplay": true,
100 "setSRISampleRate": true,
101- "sriSampleRate": 1
102+ "sriSampleRate": 1,
103+ "bannersNotGuaranteedToDisplay": true
104 },
105 "impressionDiet": {
106 "cookieName": "bannercount_fundraiser_2016",
107+ "skipInitial": 0,
108 "maximumSeen": 3,
109- "restartCycleDelay": 2592000,
110- "skipInitial": 0
111+ "restartCycleDelay": 2592000
112 },
113 "largeBannerLimit": {
114 "days": 250,
115- "identifier": "centralnotice-frbanner-seen-fullscreen",
116- "randomize": true
117+ "randomize": true,
118+ "identifier": "centralnotice-frbanner-seen-fullscreen"
119 }
120 }
121 },
122@@ -381,26 +381,26 @@
123 ],
124 "mixins": {
125 "bannerHistoryLogger": {
126- "maxEntries": 30,
127- "maxEntryAge": 30,
128 "rate": 0.01,
129+ "maxEntryAge": 30,
130+ "maxEntries": 30,
131 "waitLogNoSendBeacon": 200
132 },
133 "legacySupport": {
134- "bannersNotGuaranteedToDisplay": true,
135 "setSRISampleRate": true,
136- "sriSampleRate": 1
137+ "sriSampleRate": 1,
138+ "bannersNotGuaranteedToDisplay": true
139 },
140 "impressionDiet": {
141 "cookieName": "bannercount_fundraiser_2016",
142+ "skipInitial": 0,
143 "maximumSeen": 1000,
144- "restartCycleDelay": 2592000,
145- "skipInitial": 0
146+ "restartCycleDelay": 2592000
147 },
148 "largeBannerLimit": {
149 "days": 250,
150- "identifier": "centralnotice-frbanner-seen-fullscreen",
151- "randomize": true
152+ "randomize": true,
153+ "identifier": "centralnotice-frbanner-seen-fullscreen"
154 }
155 }
156 },
157@@ -468,26 +468,26 @@
158 ],
159 "mixins": {
160 "bannerHistoryLogger": {
161- "maxEntries": 30,
162- "maxEntryAge": 30,
163 "rate": 0.01,
164+ "maxEntryAge": 30,
165+ "maxEntries": 30,
166 "waitLogNoSendBeacon": 200
167 },
168 "legacySupport": {
169- "bannersNotGuaranteedToDisplay": true,
170 "setSRISampleRate": true,
171- "sriSampleRate": 1
172+ "sriSampleRate": 1,
173+ "bannersNotGuaranteedToDisplay": true
174 },
175 "impressionDiet": {
176 "cookieName": "bannercount_fundraiser_2016",
177+ "skipInitial": 0,
178 "maximumSeen": 1000,
179- "restartCycleDelay": 2592000,
180- "skipInitial": 0
181+ "restartCycleDelay": 2592000
182 },
183 "largeBannerLimit": {
184 "days": 250,
185- "identifier": "centralnotice-frbanner-seen-fullscreen",
186- "randomize": true
187+ "randomize": true,
188+ "identifier": "centralnotice-frbanner-seen-fullscreen"
189 }
190 }
191 },
192@@ -554,26 +554,26 @@
193 ],
194 "mixins": {
195 "bannerHistoryLogger": {
196- "maxEntries": 30,
197- "maxEntryAge": 30,
198 "rate": 0.01,
199+ "maxEntryAge": 30,
200+ "maxEntries": 30,
201 "waitLogNoSendBeacon": 200
202 },
203 "legacySupport": {
204- "bannersNotGuaranteedToDisplay": true,
205 "setSRISampleRate": true,
206- "sriSampleRate": 1
207+ "sriSampleRate": 1,
208+ "bannersNotGuaranteedToDisplay": true
209 },
210 "impressionDiet": {
211 "cookieName": "bannercount_fundraiser_2016",
212+ "skipInitial": 0,
213 "maximumSeen": 1000,
214- "restartCycleDelay": 2592000,
215- "skipInitial": 0
216+ "restartCycleDelay": 2592000
217 },
218 "largeBannerLimit": {
219 "days": 250,
220- "identifier": "centralnotice-frbanner-seen-fullscreen",
221- "randomize": true
222+ "randomize": true,
223+ "identifier": "centralnotice-frbanner-seen-fullscreen"
224 }
225 }
226 },
227@@ -636,26 +636,26 @@
228 ],
229 "mixins": {
230 "bannerHistoryLogger": {
231- "maxEntries": 30,
232- "maxEntryAge": 30,
233 "rate": 0.01,
234+ "maxEntryAge": 30,
235+ "maxEntries": 30,
236 "waitLogNoSendBeacon": 200
237 },
238 "legacySupport": {
239- "bannersNotGuaranteedToDisplay": true,
240 "setSRISampleRate": true,
241- "sriSampleRate": 1
242+ "sriSampleRate": 1,
243+ "bannersNotGuaranteedToDisplay": true
244 },
245 "impressionDiet": {
246 "cookieName": "bannercount_fundraiser_2016",
247+ "skipInitial": 0,
248 "maximumSeen": 1000,
249- "restartCycleDelay": 2592000,
250- "skipInitial": 0
251+ "restartCycleDelay": 2592000
252 },
253 "largeBannerLimit": {
254 "days": 250,
255- "identifier": "centralnotice-frbanner-seen-fullscreen",
256- "randomize": true
257+ "randomize": true,
258+ "identifier": "centralnotice-frbanner-seen-fullscreen"
259 }
260 }
261 }

Change 286126 had a related patch set uploaded (by AndyRussG):
Ensure consistent ordering of choiceData for RL module hashes

https://gerrit.wikimedia.org/r/286126

Change 286119 abandoned by AndyRussG:
Ensure consistent ordering of choiceData for RL module hashes

Reason:
Superseded by I13ae9c696

https://gerrit.wikimedia.org/r/286119

Change 286126 merged by jenkins-bot:
Ensure consistent ordering of choiceData for RL module hashes

https://gerrit.wikimedia.org/r/286126

Change 286183 had a related patch set uploaded (by Jforrester):
Ensure consistent ordering of choiceData for RL module hashes

https://gerrit.wikimedia.org/r/286183

Change 286183 merged by jenkins-bot:
Ensure consistent ordering of choiceData for RL module hashes

https://gerrit.wikimedia.org/r/286183

Mentioned in SAL [2016-04-29T19:56:14Z] <catrope@tin> Synchronized php-1.27.0-wmf.22/extensions/CentralNotice/: T133971 (duration: 00m 41s)

AndyRussG closed this task as Resolved.May 2 2016, 5:43 PM

@Catrope deployed the fix for this was deployed on Friday, April 29, 2016. Thx!!