After updating the mappings of 8 countries (930293) last week, it was found that the mean first paint time increased from1973 ms to 2054 ms. However, the median first paint time decreased from 1766 ms to 1741 ms. For other countries, both mean and median first paint time decreased (considerably for some).

For US states District of Columbia (DC), Ohio (OH) and Virginia (VA), we are getting abnormally low (0 ms) latency for some probes. We do not know the exact reason for such behaviour (but this could be due to some issues in the MaxmindDB?).

Thanks for the detailed explanation. It was very confusing to find some properties being set as null.

The 'elapsed_time' filed in the NEL report is similar to the 'duration_ms' field in the Probenet report. They are not equal but follow a similar trend. If we ignore the first pulse to each data center (identifier_dc_0), the 'elapsed_time' filed in the NEL report is similar to the 'request_time_ms' field in the Probenet report. Since we mostly depend on the 'request_time_ms' field for analysing the latency data, we can easily rely on the NEL data for that. But, as of now, NEL is not supported on all browsers.

The patch that was submitted for fixing the 'status_code' was later found out would not fix the error. So, a new patch was submitted. But this will fix validation errors for 'status_code' only and will be merged in wmf.15. I will submit a new patch now which do fix the validation errors for all fields .reports[i].pulses[i].*

Update:

• Instead of trimming bottom 10 %, we are trimming bottom 5 % only.
• We are plotting Box plots as well as Violin plots

You can see Box Plots and Violin Plots of the per-country latency data here:

Probenet Results:

For now we are considering only the 'request_time_ms'. We are taking request time for all the probes/pulses and group it based on data centers and countries. We then trim the bottom 10 % and top 10 % values to remove the outliers. Then a Box plot is plotted for each country to visualise the results.

Increase NetworkProbeLimit from 0.0001 (0.01%) to 0.001 (0.1%).

This should also not be allowed.

ctx:
  type: object
  additionalProperties:
    type:
      - string
      - number

Since in every probe, we are making 3 requests to each data center and the first one has artificially high latency. That means around 33 % of the probes will have to be treated differently. So I suggest we try 33 %ile, 50 %ile and 66 %ile also.

Dropped a message in Data-Engineering Slack channel. Waiting for a response.

That's not needed. We can match the identifier then get the pulse number from the Probenet report. But we may add it to the url as well.

Should I add some kind of identifier to match NEL reports and Probenet reports? Like instead of making a request to https://measure-{data-center}.wikimedia.org/measure, we may send a request to https://measure-{data-center}.wikimedia.org/measure?id={some-random-identifier}. This will help us in comparing the both reports in a better way. We don't have to modify in the NEL part. But we have to edit the schema of the report a bit to include the identifier as well and a small update to the JS fetch code patch.

Fixed eslint warnings in WikimediaEvents repo. This patch is not directly related to this task but the warnings were generated after the upgradation of eslint config in WikimediaEvents repo which was necessary for this task.
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/920376

Shouldn't it be "@(wikimedia.org|wikimedia.de)$" instead of "@(wikimedia.org|wikimedia.de)\.com$"?

Not Exactly. I just require access to create a repo. I have to host some code on it.

First of all thank you Timo and Chris for the detailed information.