Page MenuHomePhabricator

Replicate wikitech wikis to analytics-store.eqiad.wmnet
Closed, DeclinedPublic

Description

I have a script which reads the dblist located at https://noc.wikimedia.org/conf/all.dblist and tries to run a query on each database.

It would seem that labtestwiki now appears in the db list but is not replicated to analytics-store.

Either:

  1. I am making an incorrect assumption that dbs on the list should always be replicated to this servers
  2. Replication needs to be setup / something went wrong?
  3. The wiki is in the dblist but doesn't actually exist!

Event Timeline

Addshore raised the priority of this task from to Needs Triage.
Addshore updated the task description. (Show Details)
Addshore added a subscriber: Addshore.

The wiki exists, it's just not hosted by the normal MySQL servers. It's like labswiki which runs on silver only, except it's on labtestweb2001.

See also T89548 which obviously should be dealt with before this

I am making an incorrect assumption that dbs on the list should always be replicated to this servers

I think replication for labs/analytics/etc. always gets set up after wiki creation.

Krenair renamed this task from labtestwiki appears in the dblist but can not be found on analytics-store.eqiad.wmnet to Replicate wikitech wikis to analytics-store.eqiad.wmnet.Feb 8 2016, 2:21 PM
Krenair set Security to None.

Thanks for all the info @Krenair and poking this ticket into the correct shape ;)

I am making an incorrect assumption that dbs on the list should always be replicated to this servers

There is some separation between labswiki infrastructure and the rest of production for availability reasons. It *may* be posible to do it, but I am not very much for it because a) potential security concerns and b) analytics slaves are already low in resources for now (we cannot even replicate all of x1 for now). Wikitech is not very resource intensive, but it adds I think an extra half a gigabyte of storage to keep updated.

I strongly suggest to iterate over "all - silver.list", if that makes sense. If there is real interest on this, we can try to figure out a safe way to manage this (maybe weekly imports), but please justify it is really needed for your work.

Oh, I missread labtestwiki vs. labswiki. If labstestwiki is on s3, and it is as small as I suppose, it should be already there. I will investigate.

I should have read again, my previous comment apply, then T126218#2008995.

I strongly suggest to iterate over "all - silver.list", if that makes sense. If there is real interest on this, we can try to figure out a safe way to manage this (maybe weekly imports), but please justify it is really needed for your work.

It looks like labstestwiki is not on https://noc.wikimedia.org/conf/silver.dblist !

I have no idea where that is, but if its name is labstestweb*2*XXX, there is a high chance it is on a different datacenter (dallas).

The dblist containing both wikis with DBs hosted externally is wikitech.dblist.

Addshore changed the task status from Open to Stalled.Jun 11 2018, 10:57 AM

Scripts like this https://github.com/wikimedia/analytics-wmde-scripts/blob/9a4c17cd89000eb52c50ed5c50dbad92b7c8adf8/src/betafeatures/counts.php#L64 should instead get the wikitech.dblist and dynamically ignore those wikis rather than also checking them.

Marostegui added a subscriber: Marostegui.

As per our IRC chat