Missing / Dropped databases?
Open, HighPublic

Description

This is a list of databases that are present on s3 shard core DBs but not on dbstores and sanitarium:

bawiktionary
chwikimedia
closed_zh_twwiki
comcomwiki
de_labswikimedia
devwikiinternal
dkwiki
dkwikibooks
dkwiktionary
en_labswikimedia
flaggedrevs_labswikimedia
langcomwiki
liquidthreads_labswikimedia
noboardwiki
ptwikimedia
readerfeedback_labswikimedia
rel13testwiki
ru_sibwiki
sep11wiki
steward      -- created issues on sanitarium only
strategyappswiki
tlhwiki
tlhwiktionary
tokiponawiki
tokiponawikibooks
tokiponawikiquote
tokiponawiktionary
zh_cnwiki

Those needs to be checked to know if they need to exists or not and make sure that they exists on all replicated database or on none of them.

Volans created this task.Apr 16 2016, 11:19 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 16 2016, 11:19 AM
Peachey88 added a subscriber: Peachey88.EditedApr 16 2016, 11:27 AM

All but devwikiinternal and zh.cnwiki.hitcounter (but zh_cnwiki is) are on the deleted dbs list

Volans updated the task description. (Show Details)Apr 16 2016, 11:39 AM
Volans updated the task description. (Show Details)
Volans updated the task description. (Show Details)

@Peachey88 hitcounter is the table name from the error, I've removed it.
Added also steward that had issues only on sanitarium

I don't know what steward is... all of the tables are empty according to select table_name, table_rows from information_schema.tables where table_schema = 'steward';
Maybe it was a mistake during the creation of stewardwiki?

@jcrespo FYI given that I was online I've skipped the errors on dbstore1001 replica

jcrespo added a subscriber: demon.

@Krenair @demon : these databases existing, despite being "deleted" created multiple long-to fix replication issues on dbstore (analytics/backups) and labs. The main conceptual problem being that they are logically deleted, but not physically deleted.

Independently of the final decision about these ones in particular, we need to improve the process about extensions creating new tables (and several years later becoming obsolete, see T54921) and its deletion (including whole wikis). I do not think it is your responsibility, but something we should enforce towards devels wanting to deploy new code using them/new wiki creation. I want to figure out a proposal and send it though SOS towards you.

demon added a comment.Apr 22 2016, 3:25 PM

I don't know what steward is...
Maybe it was a mistake during the creation of stewardwiki?

That'd be my guess.

@Krenair @demon : these databases existing, despite being "deleted" created multiple long-to fix replication issues on dbstore (analytics/backups) and labs. The main conceptual problem being that they are logically deleted, but not physically deleted.

If they're creating problems by being left about, I think we can look into dropping them. Just as long as all the data is contained in the dumps and otherwise backed up.

Independently of the final decision about these ones in particular, we need to improve the process about extensions creating new tables (and several years later becoming obsolete, see T54921) and its deletion (including whole wikis). I do not think it is your responsibility, but something we should enforce towards devels wanting to deploy new code using them/new wiki creation. I want to figure out a proposal and send it though SOS towards you.

I think if we properly remove wikis from the general clusters after they're "deleted" then this helps us sidestep the problem for wikis that are horrifically out of date. Thanks for thinking this through though :)

+1. I need to check a general solution for "long term storage/archive".

@jcrespo FYI given that I was online I'm skipping the errors on dbstore1001 replica that are happening today, due to db1038 restart of yesterday.

jcrespo triaged this task as High priority.May 1 2016, 4:18 PM

We had this same issue with the dbstore servers while working on: T155605