Page MenuHomePhabricator

Update outdated scripts feeding QWERTY-related Grafana boards
Closed, ResolvedPublic8 Story Points

Description

From T213894: Modify analytics/wmde/scripts repo to work with the new analytics multisource db setup, only listing what is relevant for the WMDE QWERTY team:

  • ./advancedsearch/userprops.php
  • ./revslider/userprops.php
  • ./betafeatures/counts.php
  • ./catwatch/userprops.php

Breakdown:

  • Read the comments at T213894.
  • Understand what exactly needs updating (something about scraping data from the source database). Looking at existing patches might help.
  • Share what you learned (as a comment here on this ticket).
  • Update the scripts.
  • Deploy the scripts ( taken care of by @Ladsgroup ) Done by @Addshore
  • Check if (new) numbers are coming in and are visible on the Grafana boards.

Event Timeline

thiemowmde triaged this task as Normal priority.Feb 20 2019, 1:30 PM
thiemowmde updated the task description. (Show Details)
thiemowmde set the point value for this task to 8.

While doing this we might be able to do part of T215891 as well.

WMDE-Fisch moved this task from Sprint Backlog to Doing on the WMDE-QWERTY-Sprint-2019-02-20 board.

Change 493086 had a related patch set uploaded (by WMDE-Fisch; owner: WMDE-Fisch):
[analytics/wmde/scripts@master] Update userprop scripts to new db setup

https://gerrit.wikimedia.org/r/493086

Change 493201 had a related patch set uploaded (by Addshore; owner: WMDE-Fisch):
[analytics/wmde/scripts@production] Update userprop scripts to new db setup

https://gerrit.wikimedia.org/r/493201

Change 493086 merged by jenkins-bot:
[analytics/wmde/scripts@master] Update userprop scripts to new db setup

https://gerrit.wikimedia.org/r/493086

Change 493201 merged by jenkins-bot:
[analytics/wmde/scripts@production] Update userprop scripts to new db setup

https://gerrit.wikimedia.org/r/493201

Understand what exactly needs updating (something about scraping data from the source database). Looking at existing patches might help.
Share what you learned (as a comment here on this ticket).

  • So we're using the scripts to aggregate the user data around our extensions.
  • For that we get data from the analytics replica servers.
  • The setup of these servers are changing ( due to Ubuntu Trusty going EOL and therefor a reorganization of that infrastructure ).
  • Severs are now split up to spread the load and the replica databases are not sitting on one host but on several, depending on the wiki.
  • Scripts getting data from these servers needed to consider the split when connecting to the servers.
    • So before: "Just connect to that one server and get data!"
    • Now: "What wiki do you need data from? Here's the server for it, now get your data!"
  • Almost all our scripts want to have the (usage) data from all wikis, so we obviously of do that for all of these wikis.

Scripts are updated and merged. The deployment will be done step by step though to test if the new setup is working fine. This will be mostly done by @Ladsgroup - We'll leave a note here if that's finished.

WMDE-Fisch updated the task description. (Show Details)Feb 27 2019, 1:06 PM

Change 493214 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[analytics/wmde/scripts@master] Introduce caching in WikimediaDb lib class

https://gerrit.wikimedia.org/r/493214

Change 493212 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[analytics/wmde/scripts@master] Add strict type hints to several scripts and lib classes

https://gerrit.wikimedia.org/r/493212

Change 493452 had a related patch set uploaded (by Addshore; owner: Thiemo Kreuz (WMDE)):
[analytics/wmde/scripts@production] Add strict type hints to several scripts and lib classes

https://gerrit.wikimedia.org/r/493452

Change 493212 merged by jenkins-bot:
[analytics/wmde/scripts@master] Add strict type hints to several scripts and lib classes

https://gerrit.wikimedia.org/r/493212

Change 493452 merged by jenkins-bot:
[analytics/wmde/scripts@production] Add strict type hints to several scripts and lib classes

https://gerrit.wikimedia.org/r/493452

Ladsgroup updated the task description. (Show Details)Feb 28 2019, 7:13 PM
Ladsgroup added a subscriber: Addshore.
Lea_WMDE closed this task as Resolved.Mar 4 2019, 10:36 AM
Lea_WMDE moved this task from Demo to Done on the WMDE-QWERTY-Sprint-2019-02-20 board.