Page MenuHomePhabricator
Create Task
Maniphest T214894

Grafana Datamodel References dashboard broken (daily data)
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Problem:
The Grafana graph Reference Snak main properties and other graphs stopped displaying values after 2018-12-31.

The data is generated by https://github.com/wikimedia/analytics-wmde-toolkit-analyzer/blob/master/analyzer/src/main/java/org/wikidata/analyzer/Processor/MetricProcessor.java and sent to graphite using https://github.com/wikimedia/analytics-wmde-scripts/blob/master/src/wikidata/dumpScanProcessing.php

Acceptance criteria:

  • The Grafana graphs on the dashboard show current data again

Details

SubjectRepoBranchLines +/-
New buildanalytics/wmde/toolkit-analyzer-buildproduction+0 -0
New buildanalytics/wmde/toolkit-analyzer-buildmaster+0 -0
Fix port when connecting to WDQSanalytics/wmde/toolkit-analyzermaster+1 -1
New buildanalytics/wmde/toolkit-analyzer-buildproduction+0 -0
New buildanalytics/wmde/toolkit-analyzer-buildmaster+0 -0
Use the internal WDQS endpoint insteadanalytics/wmde/toolkit-analyzermaster+1 -1
Customize query in gerrit

Related Objects

Event Timeline

Michael created this task.Jan 29 2019, 10:27 AM
Restricted Application added a project: Wikidata. · View Herald TranscriptJan 29 2019, 10:27 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Addshore renamed this task from Grafana Reference Snak graph missing current data to Grafana Datamoel Referneces dashboard broken (daily data).Jul 9 2019, 4:49 PM
Addshore updated the task description. (Show Details)
Addshore updated the task description. (Show Details)Jul 9 2019, 4:57 PM
Addshore moved this task from incoming to needs discussion or investigation on the Wikidata board.
Addshore moved this task from Incoming to Needs Work on the Wikidata-Campsite board.
Addshore added a comment.Jul 9 2019, 5:13 PM

From the logs:

addshore@stat1007:~$ sudo -u analytics-wmde cat /srv/analytics-wmde/graphite/log/toolkit-analyzer.log
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
****************************************************************************
***                       Wikidata Toolkit: ToolkitAnalyzer              ***
******************************* Data Directory Layout **********************
* Target storage directory : data/                                         *
* Downloaded dump locations: data/dumpfiles/json-<DATE>/<DATE>-all.json.gz *
* Processor output location: data/<DATE>/                                  *
****************************************************************************
Targeting latest dump: 20190704
Using data directory: /srv/analytics-wmde/graphite/data
MetricProcessor enabled
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Error getting data from https://query.wikidata.org/sparql
Connection timed out (Connection timed out)
java.net.ConnectException: Connection timed out (Connection timed out)
Addshore added a comment.EditedJul 9 2019, 5:17 PM

The endpoint is hardcoded in https://github.com/wikimedia/analytics-wmde-toolkit-analyzer/blob/35ae8f3796f88443ab9bec18c82881b7a0500162/analyzer/src/main/java/org/wikidata/analyzer/Fetcher/WikiDataFetcher.java#L29

Other scripts talk directly to the internal endpoint https://github.com/wikimedia/analytics-wmde-scripts/blob/master/src/wikidata/sparql/instanceof.php#L74

The Java code should do the same.

Note, config is in the process of being added to deifne the hostname https://gerrit.wikimedia.org/r/#/c/analytics/wmde/scripts/+/520903/
Java code should retrieve the hostname from the config once added.

Addshore triaged this task as Medium priority.Jul 9 2019, 5:17 PM
Addshore moved this task from needs discussion or investigation to ready to go on the Wikidata board.
Addshore moved this task from Needs Work to Ready to estimate on the Wikidata-Campsite board.
Ladsgroup merged a task: T227786: broken reference dashboard.Jul 16 2019, 7:59 AM
matej_suchanek renamed this task from Grafana Datamoel Referneces dashboard broken (daily data) to Grafana Datamodel References dashboard broken (daily data).Jul 16 2019, 8:07 AM
• alaa_wmde set the point value for this task to 3.Jul 23 2019, 12:40 PM
• alaa_wmde moved this task from Ready to estimate to Ready to pick up on the Wikidata-Campsite board.
Ladsgroup claimed this task.Jul 30 2019, 4:21 PM
Ladsgroup edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Ladsgroup moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJul 30 2019, 4:21 PM
gerritbot added a comment.Jul 30 2019, 4:23 PM

Change 526471 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[analytics/wmde/toolkit-analyzer@master] Use the internal WDQS endpoint instead

https://gerrit.wikimedia.org/r/526471

gerritbot added a project: Patch-For-Review.Jul 30 2019, 4:23 PM
Ladsgroup moved this task from Peer Review to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 2 2019, 9:49 AM
Ladsgroup moved this task from Done to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
gerritbot added a comment.Aug 6 2019, 4:22 PM

Change 526471 merged by jenkins-bot:
[analytics/wmde/toolkit-analyzer@master] Use the internal WDQS endpoint instead

https://gerrit.wikimedia.org/r/526471

• alaa_wmde moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 7 2019, 9:14 AM
• alaa_wmde subscribed.

is there a way to test this on Beta, or just in production?

Ladsgroup added a comment.Aug 7 2019, 9:17 AM
In T214894#5398716, @alaa_wmde wrote:

is there a way to test this on Beta, or just in production?

It should be only done in production. but I have no idea how to deploy this thing.

Lydia_Pintscher added a comment.Aug 7 2019, 10:36 AM

@Addshore Do you know?

WMDE-leszek moved this task from Test (Verification) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 7 2019, 10:55 AM
WMDE-leszek subscribed.Aug 7 2019, 10:58 AM

https://wikitech.wikimedia.org/wiki/WMDE/Analytics#analytics/wmde/toolkit-analyzer_repo_and_build seems to have some instructions

gerritbot added a comment.Aug 7 2019, 11:27 AM

Change 528767 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[analytics/wmde/toolkit-analyzer-build@master] New build

https://gerrit.wikimedia.org/r/528767

Ladsgroup moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 7 2019, 11:27 AM
Ladsgroup added a comment.Aug 7 2019, 11:40 AM
In T214894#5398977, @WMDE-leszek wrote:

https://wikitech.wikimedia.org/wiki/WMDE/Analytics#analytics/wmde/toolkit-analyzer_repo_and_build seems to have some instructions

Thanks for the hint

gerritbot added a comment.Aug 7 2019, 2:37 PM

Change 528767 merged by jenkins-bot:
[analytics/wmde/toolkit-analyzer-build@master] New build

https://gerrit.wikimedia.org/r/528767

gerritbot added a comment.Aug 7 2019, 3:18 PM

Change 528871 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[analytics/wmde/toolkit-analyzer-build@production] New build

https://gerrit.wikimedia.org/r/528871

gerritbot added a comment.Aug 7 2019, 3:18 PM

Change 528871 merged by jenkins-bot:
[analytics/wmde/toolkit-analyzer-build@production] New build

https://gerrit.wikimedia.org/r/528871

Ladsgroup moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 7 2019, 3:48 PM
Ladsgroup mentioned this in rADTBa0cc2b58d677: New build.Aug 8 2019, 1:32 AM
Ladsgroup mentioned this in rADTB54543a61e8f1: New build.
Ladsgroup added a comment.Aug 8 2019, 11:16 AM

So the graphs don't show up (yet), I confirmed that the new build is on production but the latest log still gives out connection time out to query.wikidata.org. It seems it's ran weekly (the one before that is exactly one week before it). If my assessment is correct, it will show up after 12th of August.

Ladsgroup added a comment.Aug 8 2019, 11:22 AM
In T214894#5402269, @Ladsgroup wrote:

So the graphs don't show up (yet), I confirmed that the new build is on production but the latest log still gives out connection time out to query.wikidata.org. It seems it's ran weekly (the one before that is exactly one week before it). If my assessment is correct, it will show up after 12th of August.

It's actually run daily but at noon, it'll happen in a couple of hours

Ladsgroup added a subscriber: elukey.EditedAug 8 2019, 11:30 AM

It's still failing
we have time java -Dhttp.proxyHost=\"http://webproxy.${::site}.wmnet\" -Dhttp.proxyPort=8080 -Xmx2g -jar ${dir}/src/toolkit-analyzer-build/toolkit-analyzer.jar --processors Metric --store ${dir}/data --latest >> ${log_dir}/toolkit-analyzer.log 2>&1 so it should have not an issue (goes through webproxy which is I'm not sure if needed anymore)

BUT stat is firewalled: https://lists.wikimedia.org/pipermail/analytics/2019-July/006648.html cc: @elukey

Ladsgroup added a comment.Aug 8 2019, 1:18 PM

The port is missing....

gerritbot added a comment.Aug 8 2019, 1:20 PM

Change 529094 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[analytics/wmde/toolkit-analyzer@master] Fix port when connecting to WDQS

https://gerrit.wikimedia.org/r/529094

gerritbot added a comment.Aug 8 2019, 1:30 PM

Change 529094 merged by jenkins-bot:
[analytics/wmde/toolkit-analyzer@master] Fix port when connecting to WDQS

https://gerrit.wikimedia.org/r/529094

gerritbot added a comment.Aug 8 2019, 1:34 PM

Change 529097 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[analytics/wmde/toolkit-analyzer-build@master] New build

https://gerrit.wikimedia.org/r/529097

gerritbot added a comment.Aug 8 2019, 1:36 PM

Change 529097 merged by jenkins-bot:
[analytics/wmde/toolkit-analyzer-build@master] New build

https://gerrit.wikimedia.org/r/529097

Ladsgroup mentioned this in rADTBf502f28de5cb: New build.Aug 8 2019, 1:39 PM
gerritbot added a comment.Aug 8 2019, 1:39 PM

Change 529098 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[analytics/wmde/toolkit-analyzer-build@production] New build

https://gerrit.wikimedia.org/r/529098

gerritbot added a comment.Aug 8 2019, 1:40 PM

Change 529098 merged by jenkins-bot:
[analytics/wmde/toolkit-analyzer-build@production] New build

https://gerrit.wikimedia.org/r/529098

Ladsgroup mentioned this in rADTB300613c85285: New build.Aug 8 2019, 1:42 PM
Lydia_Pintscher added a comment.Aug 12 2019, 9:04 AM

The dashboard now seems to have data for the 5th of August but only that. Is this only tracked weekly?

Ladsgroup added a comment.Aug 12 2019, 10:29 AM
In T214894#5408782, @Lydia_Pintscher wrote:

The dashboard now seems to have data for the 5th of August but only that. Is this only tracked weekly?

It's daily but the date it sends the information is based on the latest json dumps and dumps are ran weekly now it seems.

WMDE-leszek added a comment.Aug 21 2019, 9:20 AM

Random question, maybe you'd be in the know @Ladsgroup: any chance we could get the past missing data back-filled?

Ladsgroup added a comment.Aug 21 2019, 10:25 AM
In T214894#5427682, @WMDE-leszek wrote:

Random question, maybe you'd be in the know @Ladsgroup: any chance we could get the past missing data back-filled?

It's possible and it's not that hard if you need the numbers.

WMDE-leszek closed this task as Resolved.Aug 26 2019, 4:21 PM
WMDE-leszek moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

@Lydia_Pintscher said it is not worth investing effort to restore the past numbers. Closing then.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL