Page MenuHomePhabricator
Create Task
Maniphest T57455

Provide dynamic report of differences between replica databases and production databases
Closed, ResolvedPublic
Actions

Description

Related to bug 54847.

I'd like the ability to regularly read a report that shows the differences between the Labs database replicas and the production databases. This would be helpful for:

  • finding new core or extension table columns or databases that have been deployed that aren't yet exposed via a view; and
  • analyzing what can and can't be additionally exposed (for example, watchlist and archive).

Basically this is a request for a regularly updated status report (or diff) between the two data sets. What's available on production vs. what's available on Labs.

Version: unspecified
Severity: normal

Details

Reference
bz55455

Related Objects
Search...

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusSubtypeAssignedTask
· · ·
ResolvedLadsgroupT104459 Detect object, schema and data drifts between mediawiki HEAD, production masters and replicas
ResolvedNoneT57455 Provide dynamic report of differences between replica databases and production databases
· · ·

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:15 AM
• bzimport added a project: Wikimedia-Labs-General.
• bzimport set Reference to bz55455.
• bzimport added a subscriber: Unknown Object (MLST).
• MZMcBride created this task.Oct 8 2013, 4:01 AM
• MZMcBride added a comment.Oct 13 2013, 7:19 PM

Providing schema-only dumps of the databases from the Labs side and the production would allow for nice-ish diffs, maybe.

• bzimport added a comment.Sep 6 2014, 9:55 PM

rodrigosprimo wrote:

I think that a good start would be to add a list of the missing tables and columns to the page below.

https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access

Aklapper added a project: Cloud-Services.Apr 29 2015, 10:23 AM
Krenair added a project: DBA.Oct 13 2015, 1:07 AM
Krenair subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 13 2015, 1:07 AM
• chasemp triaged this task as Lowest priority.Nov 30 2015, 6:33 PM
• chasemp subscribed.
• AlexMonk-WMF added a parent task: T50930: Database replication problems - production and labs (tracking).Aug 26 2016, 8:10 AM
jcrespo edited parent tasks, added: T104459: Detect object, schema and data drifts between mediawiki HEAD, production masters and replicas; removed: T50930: Database replication problems - production and labs (tracking).Nov 15 2016, 5:33 PM
jcrespo subscribed.

This will be eventually fixed with the parent task at the same time- this ticket will stay for the scope of making a part if it public.

Harej awarded a token.Mar 5 2018, 10:41 PM
Marostegui moved this task from Triage to Backlog on the DBA board.Dec 4 2018, 8:02 AM
Anomie mentioned this in T215445: comment and actor view challenges for Cloud Services.Mar 4 2019, 7:12 PM
RhinosF1 subscribed.Mar 27 2021, 6:36 PM
LSobanski raised the priority of this task from Lowest to Low.Apr 26 2021, 10:09 AM
taavi edited projects, added Data-Services; removed Cloud-Services, Wikimedia-Labs-General.Apr 26 2021, 10:14 AM
bd808 moved this task from Backlog to Wiki replicas on the Data-Services board.Apr 26 2021, 11:31 PM
Marostegui closed this task as Resolved.Nov 5 2021, 7:58 AM
Marostegui subscribed.

I am going to close this as fixed. We've fixed major differences in schema and the new clouddb* hosts were cloned directly from production. We also run ROW based replication there which should ensure consistency on the data (otherwise replication will break).
As mentioned at T57455#2796252 once we've fixed the parent task that means we've fixed the differences and drifts present on the cloudd* hosts.

If someone believe this still needs to stay open, please reopen it.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits