Page MenuHomePhabricator

Compare a few tables per section before the switchover
Closed, ResolvedPublic

Description

In order to avoid issues with possible data drift between eqiad and codfw, let's compare some of the big and most used tables on all the sections between eqiad and codfw hosts to make sure there're no data gaps

Let's use the following tables and columns:

s1-s8

revision rev_id
text old_id
user user_id
change_tag ct_id
actor actor_id
ipblocks ipb_id
comment comment_id
user user_id
watchlist wl_id
text old_id
logging log_id
page page_id
revision rev_id
revision_actor_temp revactor_rev
revision_comment_temp revcomment_rev
slots slot_revision_id
archive ar_id

es4 and es5, let's use enwiki, commonswiki and wikidatawiki
x1 let's use wikishared enwiki, commonswiki and wikidatawiki for echo_event table

Progress

  • s1
  • s2
  • s3
  • s4
  • s5
  • s6
  • s7
  • s8
  • x1
  • es4
  • es5

Event Timeline

Marostegui moved this task from Pending comment to In progress on the DBA board.
Marostegui moved this task from In progress to Pending comment on the DBA board.

Will work on this next week, before the switchover

I am starting the comparison on some sections, which will also help warming up those tables and get them in memory.

Mentioned in SAL (#wikimedia-operations) [2020-08-27T04:53:55Z] <marostegui> Stop db1074 and db2107 in sync to fix drifts on s2 change_tag - T260042

I am doing a few more checks with some more tbales, so the tables are warmed up even more.

Mentioned in SAL (#wikimedia-operations) [2020-08-28T07:07:22Z] <marostegui> Warm up parsercache in codfw - T260042

Do you want me to run the schema drift thingy for codfw too?

With this it now supports codfw: https://github.com/Ladsgroup/db-analyzor-tools/commit/ccbf1e8c53fc407a6a9513a0b1c61dd3f85222ca I started it, and since codfw doesn't have much traffic right now, I made the script a bit fast (0.1 second sleep between every check instead of 1s). It'll be done soon. (I know you didn't ask for it but since it's Christmas, you'll get it)

Marostegui updated the task description. (Show Details)

This is done