| • jmatazzoni | |
| Apr 23 2018, 7:41 PM |
| F17287232: Screen Shot 2018-04-24 at 8.32.34 AM.png | |
| Apr 24 2018, 3:36 PM |
| F17279111: Screen Shot 2018-04-23 at 10.37.28 PM.png | |
| Apr 24 2018, 5:39 AM |
https://stats.wikimedia.org/v2/#/en.wikipedia.org/contributing/editors
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | JAllemandou | T192841 Wikistats Bug: all but 2018 data missing? | |||
| Duplicate | Milimetric | T192840 Wikistats: Data Regression |
I re-run indexation for 2018-02 snapshot and now segment sizes in druid are what i would expect, about 2 G per segment.
sudo -u hdfs oozie job --oozie $OOZIE_URL -Drefinery_directory=hdfs://analytics-hadoop$(hdfs dfs -ls -d /wmf/refinery/2018* | tail -n 1 | awk '{print $NF}
') -Dqueue_name=production -Doozie_launcher_queue_name=production -Dstart_time=2018-02-01T00:00Z -Dstop_time=2018-02-02T00:00Z -config /srv/dep
loyment/analytics/refinery/oozie/mediawiki/history/reduced/coordinator.properties -run
Note this indexation did not changed the 2018-03 data, we need to take a second look at that snapshot and see if it is correct cc @Milimetric
We also need totake a closer look to what happened here.
@Nuria actions fixed the problem for data up to 2018-02. I restarted a job ending in 2018-03 as the problem is not related to snapshots but to wrong indexation while testing. Will follow up later today when back from day off.
Job finished, data is up to date. Thanks @Nuria and @Milimetric for having spotted the problem and quick fix it !
Indeed, confirmed all looks good, I'll put this in code review so we can remember to talk about what happened.
@JAllemandou I see the Editor data now. Thanks. But when I split by editor type, Anonymous users come in as zero. See screenshot. So it looks like anon editor data is still missing? Or am I doing something wrong—or is the fix just not on production yet?
hm, indeed https://wikimedia.org/api/rest_v1/metrics/edits/aggregate/all-projects/anonymous/all-page-types/monthly/2016030100/2018042400 shows 0 for all months, and it looks like the anonymous type is broken everywhere:
https://stats.wikimedia.org/v2/#/en.wikipedia.org/contributing/editors
https://stats.wikimedia.org/v2/#/en.wikipedia.org/contributing/edits
https://stats.wikimedia.org/v2/#/en.wikipedia.org/content/edited-pages
https://stats.wikimedia.org/v2/#/en.wikipedia.org/content/absolute-bytes-diff
https://stats.wikimedia.org/v2/#/en.wikipedia.org/content/net-bytes-difference
This seems like a separate problem, but I do remember there being numbers for active editors, so this is also a new problem. I'll look into it.
Reindexing again 2018-02 data, looking into 2018-03 issue with anonymous editors
oozie job -info 0034476-180330093100664-oozie-oozi-W
Change 428922 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Correct mediawiki-history job bugs
Change 428922 merged by jenkins-bot:
[analytics/refinery/source@master] Correct mediawiki-history job bugs
Change 428977 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Add unittest for a Mediawiki-History function already fixed
Change 428977 merged by jenkins-bot:
[analytics/refinery/source@master] Add unittest for a Mediawiki-History function already fixed