Certain wiki databases missing from replicas?
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	Volans
	Apr 16 2016, 11:19 AM

Description

This is a list of databases that are present on s3 shard core DBs but not on dbstores and sanitarium:

bawiktionary
chwikimedia
closed_zh_twwiki
comcomwiki
de_labswikimedia
devwikiinternal
dkwiki
dkwikibooks
dkwiktionary
en_labswikimedia
flaggedrevs_labswikimedia
langcomwiki
liquidthreads_labswikimedia
noboardwiki
ptwikimedia
readerfeedback_labswikimedia
rel13testwiki
ru_sibwiki
sep11wiki
steward      -- created issues on sanitarium only
strategyappswiki
tlhwiki
tlhwiktionary
tokiponawiki
tokiponawikibooks
tokiponawikiquote
tokiponawiktionary
zh_cnwiki

Those needs to be checked to know if they need to exists or not and make sure that they exists on all replicated database or on none of them.

Related Objects
Search...

View Standalone Graph

This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Status	Assigned	Task
		· · ·
Resolved	jcrespo	T50930 Database replication problems - production and labs (tracking)
Resolved	LSobanski	T54921 Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking)
Declined	None	T132838 Certain wiki databases missing from replicas?
		· · ·

Event Timeline

Volans created this task.Apr 16 2016, 11:19 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 16 2016, 11:19 AM

Volans added a parent task: T50930: Database replication problems - production and labs (tracking).Apr 16 2016, 11:20 AM

All but devwikiinternal and zh.cnwiki.hitcounter (but zh_cnwiki is) are on the deleted dbs list

Volans updated the task description. (Show Details)Apr 16 2016, 11:39 AM

Volans updated the task description. (Show Details)

@Peachey88 hitcounter is the table name from the error, I've removed it.
Added also steward that had issues only on sanitarium

I don't know what steward is... all of the tables are empty according to select table_name, table_rows from information_schema.tables where table_schema = 'steward';
Maybe it was a mistake during the creation of stewardwiki?

Volans mentioned this in T130702: Several es20XX servers keep crashing (es2017, es2019, es2015, es2014) since 23 March.Apr 16 2016, 5:38 PM

• MZMcBride subscribed.Apr 17 2016, 12:54 AM

@jcrespo FYI given that I was online I've skipped the errors on dbstore1001 replica

@Krenair @demon : these databases existing, despite being "deleted" created multiple long-to fix replication issues on dbstore (analytics/backups) and labs. The main conceptual problem being that they are logically deleted, but not physically deleted.

Independently of the final decision about these ones in particular, we need to improve the process about extensions creating new tables (and several years later becoming obsolete, see T54921) and its deletion (including whole wikis). I do not think it is your responsibility, but something we should enforce towards devels wanting to deploy new code using them/new wiki creation. I want to figure out a proposal and send it though SOS towards you.

In T132838#2211939, @Krenair wrote:

I don't know what steward is...
Maybe it was a mistake during the creation of stewardwiki?

That'd be my guess.

In T132838#2230390, @jcrespo wrote:

@Krenair @demon : these databases existing, despite being "deleted" created multiple long-to fix replication issues on dbstore (analytics/backups) and labs. The main conceptual problem being that they are logically deleted, but not physically deleted.

If they're creating problems by being left about, I think we can look into dropping them. Just as long as all the data is contained in the dumps and otherwise backed up.

Independently of the final decision about these ones in particular, we need to improve the process about extensions creating new tables (and several years later becoming obsolete, see T54921) and its deletion (including whole wikis). I do not think it is your responsibility, but something we should enforce towards devels wanting to deploy new code using them/new wiki creation. I want to figure out a proposal and send it though SOS towards you.

I think if we properly remove wikis from the general clusters after they're "deleted" then this helps us sidestep the problem for wikis that are horrifically out of date. Thanks for thinking this through though :)

+1. I need to check a general solution for "long term storage/archive".

@jcrespo FYI given that I was online I'm skipping the errors on dbstore1001 replica that are happening today, due to db1038 restart of yesterday.

jcrespo triaged this task as High priority.May 1 2016, 4:18 PM

jcrespo added a parent task: T54921: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking).Nov 7 2016, 3:16 PM

We had this same issue with the dbstore servers while working on: T155605

TerraCodes subscribed.Mar 14 2017, 9:29 PM

greg edited projects, added Release-Engineering-Team (Watching / External); removed Release-Engineering-Team.Jul 6 2017, 8:11 PM

• 1978Gage2001 moved this task from Triage to In progress on the DBA board.Dec 11 2017, 9:46 AM

• Marostegui moved this task from In progress to Backlog on the DBA board.Dec 11 2017, 10:57 AM

Krinkle renamed this task from Missing / Dropped databases? to Certain wiki databases missing from replicas?.Mar 27 2018, 11:57 PM

This is arguably not an issue- those databases are deleted, so of course they are missing from public/analytics replicas. Lowering to low, as there is work to be done for clean up and a better coordination and monitoring, but this is not causing ongoing issues (there is no data loss not any other problem).

It is, however, annoying when performing schema changes, but those should be performed based on dblists, not all databases.

So this is a clean up task, not a data issue. A proposal I suggested some time ago was to create s0, a graveyard for deleted wikis with much lower resources to avoid failures from software like wikidata and others that handle multiple wikis but cannot be trivially cleaned up.

Technically not missing, as this is on purpose. Declining (or resolving) as it is written here - they will continue to be removed from sanitarium/labsdb, but not from new analytics hosts (dbstore hosts rebuilt *have* them-, with no issue with revisiting the current decision, but this is non-actionable right now.

Certain wiki databases missing from replicas?Closed, DeclinedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Certain wiki databases missing from replicas?
Closed, DeclinedPublic
Actions

Related Objects
Search...