Page MenuHomePhabricator

Implement regular data backups for our R Shiny dashboards for Wikidata
Open, Needs TriagePublic

Description

Main components:

  • Wikidata Analytics

Problem:
The data for our Wikidata Analytics (R Shiny based) dashboards do not get backed up regularly.

Solution:
Let's backup the following data to bacula as described in T294355#7531239.

Acceptance criteria:

  • All of our current and future Wikidata Analytics data is updated at least weekly.

Open questions:

  • Do we really need this, or is all of our source data already backed up by WMF?

Event Timeline

All the data that our dashboards use is accessed in the form of files on our analytics server. These files are already backed up, so no additional backups will be needed. @GoranSMilovanovic, could you please confirm this?

@Manuel

As long as my R production scripts on the Analytics Clients are running the data that feed our analytics will be regularly updated.

The data are copied to the public directory, and also exist on a path in the /srv folder on the Analytics Clients as well as in my home directory there.

I guess the Analytics Clients machines are being regularly backed-up but that is more a question for the Analytics.

@GoranSMilovanovic: Who could verify that there are in fact backups for those systems?

@GoranSMilovanovic: Could you please contact them and find out? Maybe this would also be a good time to arrange an introduction with me. Who would you recommend we talk to?

@Tobi_WMDE_SW Analytics @JAllemandou @Milimetric

Our Wikidata Analytics PM @Manuel would like to get in touch with the WMF Analytics team and find out if the data from the Analytics Clients are being regularly backed-up: T297485#7653025

@Manuel & @GoranSMilovanovic: sorry for the long delay, no we don't back up the stat machines or other analytics clients. From the "Local data storage" section on the Analytics Clients article:

First, note that the Analytics clients store data using redundant RAID configurations, but are not otherwise backed up. Your home directory on HDFS (/user/your-username) is a safer place for important data.

Aklapper added a subscriber: GoranSMilovanovic.

@GoranSMilovanovic: Removing task assignee as this open task has been assigned for more than two years - see the email sent to all task assignees on 2024-04-15.
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome! :)
If this task has been resolved in the meantime, or should not be worked on by anybody ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator. Thanks!