When i follow the links in the "good" column:

http://allthetropes.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=statistics&maxlag=5

http://bus.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=statistics&maxlag=5

i get errors. earlier i saw an "Internal Server Error" with the suggestiong to add "Set $wgShowExceptionDetails = true; and $wgShowDBErrorBacktrace = true"

11:45 <mutante> Internal Server Error
11:45 <mutante> Set $wgShowExceptionDetails = true; and $wgShowDBErrorBacktrace = true; at the bottom of LocalSettings.php to show detailed debugging information

as of right now i see:

503 Backend fetch failed

Our servers are having problems at the moment. Please try again in a few minutes. There may be additional information on our Facebook and Twitter pages.

Please provide the details below if you report this error to the system administrators via https://phabricator.miraheze.org:

Error 503 Backend fetch failed, forwarded for 50.0.124.36, 127.0.0.1 (Varnish XID 495193) via cp2 at Mon, 05 Dec 2016 22:32:54 GMT.

I did not change anything on my side and that error is from your server, so doesn't like on my side.

You chose the worst day to start working on this as our db server decided to crash today. I'll get back to you when we're back online.

@Dzahn you might be able to check now. I'm hoping there's no underlying issues with our database migration (besides one I've noticed).
All of the Miraheze wikis should be back to normal, allthetropes.org has $wgReadOnly set, although the link you sent for the API worked.

allthetropes fails because the API URL redirects, "bus" fails because of an error 503 "backend fetch failed".

A(1/1) - allthetropes.miraheze.org - calling API: https://allthetropes.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=statistics&format=php&maxlag=5

-> http answer: 302 -> parsing answer: 1
---> ERROR - fetching via HTTP failed: id 220 (https://allthetropes.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=statistics&format=php&maxlag=5)

---> update miraheze set http="302" where id="220";

A(1/1) - bus.miraheze.org - calling API: https://bus.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=statistics&format=php&maxlag=5

PHP Warning: file_get_contents(https://bus.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=statistics&format=php&maxlag=5): failed to open stream: HTTP request failed! HTTP/1.1 503 Backend fetch failed

-> http answer: 503 -> parsing answer: 1
---> ERROR - fetching via HTTP failed: id 229 (https://bus.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=statistics&format=php&maxlag=5)

---> update miraheze set http="503" where id="229";

https://bus.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=statistics&maxlag=5

    "servedby": "mw2",
    "error": {
        "code": "readapidenied",
        "info": "You need read permission to use this module",
        "*": "See https://bus.miraheze.org/w/api.php for API usage"
    }
}

so re: allthetropes. yes, the redirect is the problem, using https://allthetropes.org/w/api.php?action=query&meta=siteinfo&siprop=statistics&format=php&maxlag=5 would work, BUT so far the nice thing about the miraheze table was that we could automatically build all the URLs from just the subdomain. In the database and when importing we don't save full URLs, we just save the name and that was nice that way. The only reason to store full URLs would be special cases like that, requiring code changes to handle the special case and saving the full URL in the database just for this one. Nicer would be if you could make the API reachable under their foo.miraheze.org URLs for all wikis despite the custom domains existing.

re: bus.miraheze.org - this just tells me i'm not allowed to get info from API, per configuration

edit: correction, as of right now it went to "503 Backend fetch failed".

Our servers are having problems at the moment. Please try again in a few minutes. There may be additional information on our Facebook and Twitter pages.

Please provide the details below if you report this error to the system administrators via https://phabricator.miraheze.org:

Error 503 Backend fetch failed, forwarded for 50.0.124.36, 127.0.0.1 (Varnish XID 655533) via cp2 at Tue, 06 Dec 2016 16:09:43 GMT.

@Dzahn Sites (at their configured URIs) have been accessible sporadically.

As soon as @Southparkfan came back online, he reinstalled mysql on our old db server and started to migrate back. This is in progress (1150 out of ~1700 databases done).

After we sort out this fiasco I will ask him about making the API available at the *.miraheze.org addresses for all wikis with custom domains.

Edit: Is there a way to purge deleted wikis (database dropped and domain 404's) from the stats list?

@NDKilla sounds good. thanks. Currently the way to purge deleted wikis would be to ask me to drop it from DB, can you provide a list of wikis to be deleted?

@Dzahn Shouldn't be an issue but I copy/pasted MySQL output to be sure I didn't typo the DB name before saying it's deleted. Please only purge links associated with unknown databases below.

ERROR 1049 (42000): Unknown database 'entropediawiki'
ERROR 1049 (42000): Unknown database 'anicarpediawiki'
ERROR 1049 (42000): Unknown database 'd4rkst4rwiki'
ERROR 1049 (42000): Unknown database 'lifelinewiki'
ERROR 1049 (42000): Unknown database 'artelwiki'
ERROR 1049 (42000): Unknown database 'datasciencewiki'
ERROR 1049 (42000): Unknown database 'etpowiki'
ERROR 1049 (42000): Unknown database 'quantixwiki' (this one was mine)
ERROR 1049 (42000): Unknown database 'francisfranckwiki'
ERROR 1049 (42000): Unknown database 'armonianwiki'
ERROR 1049 (42000): Unknown database 'fonehelpwiki'
ERROR 1049 (42000): Unknown database 'atomicpancakeswiki'
ERROR 1049 (42000): Unknown database 'frtitreprovisoirewiki''
ERROR 1049 (42000): Unknown database 'drunkenpeasantswiki'
ERROR 1049 (42000): Unknown database 'isswiki'
ERROR 1049 (42000): Unknown database 'apgwiki'
ERROR 1049 (42000): Unknown database 'lugjuwiki'
ERROR 1049 (42000): Unknown database 'cittanostraunserestadtwiki'
ERROR 1049 (42000): Unknown database 'creedwiki'
ERROR 1049 (42000): Unknown database 'dpwiki'
ERROR 1049 (42000): Unknown database 'drspwiki'
ERROR 1049 (42000): Unknown database 'imeddroneswiki'
ERROR 1049 (42000): Unknown database 'itwelcomewiki'
ERROR 1049 (42000): Unknown database 'moonmanwiki'
ERROR 1049 (42000): Unknown database 'rowelcomewiki'
ERROR 1049 (42000): Unknown database 'rtcsnipwiki'
ERROR 1049 (42000): Unknown database 'theestablishmentmapwiki'
ERROR 1049 (42000): Unknown database 'tuwecleomwiki'
ERROR 1049 (42000): Unknown database 'unnawiki'
ERROR 1049 (42000): Unknown database 'prelliwiki'
ERROR 1049 (42000): Unknown database 'ariawiki'
ERROR 1049 (42000): Unknown database 'atlantisgamemodewiki'
ERROR 1049 (42000): Unknown database 'cogdevsocwiki'
ERROR 1049 (42000): Unknown database 'communityofdissentwiki'
ERROR 1049 (42000): Unknown database 'decrypticwiki'
ERROR 1049 (42000): Unknown database 'diem25ocwiki'
ERROR 1049 (42000): Unknown database 'dishonesttruthwiki'
ERROR 1049 (42000): Unknown database 'diylifewiki'
ERROR 1049 (42000): Unknown database 'dofowikiwiki'
ERROR 1049 (42000): Unknown database 'dpwwiki'
ERROR 1049 (42000): Unknown database 'dzcommunitywiki'
ERROR 1049 (42000): Unknown database 'etec510wiki'
ERROR 1049 (42000): Unknown database 'f14swiki'
ERROR 1049 (42000): Unknown database 'fmu82wiki'
ERROR 1049 (42000): Unknown database 'freedgewiki'
ERROR 1049 (42000): Unknown database 'fvcwiki'
ERROR 1049 (42000): Unknown database 'gappswiki'
ERROR 1049 (42000): Unknown database 'geramaxwiki'
ERROR 1049 (42000): Unknown database 'golanhowtowiki'
ERROR 1049 (42000): Unknown database 'habitatpartageboulonnaiswiki'
ERROR 1049 (42000): Unknown database 'gympawikiwiki'
ERROR 1049 (42000): Unknown database 'instylewiki'
ERROR 1049 (42000): Unknown database 'ioncontainerswiki'

Below are wiki databases who still exits but whose stats havent been updated in over 400 hours
reriawiki
pnpwiki
ronuoswiki
reinventandolasorganizacioneswiki
rwbyfrwiki
rodintestonewiki
rylnbgwiki

In T146712#2856064, @NDKilla wrote

Below are wiki databases who still exits but whose stats havent been updated in over 400 hours
pnpwiki

I took one random example and it does not exist.

404 Wiki Not Found

https://pnpwiki.miraheze.org/wiki/
https://pnpwiki.miraheze.org/wiki/%E7%89%B9%E6%AE%8A:%E8%BF%91%E6%9C%9F%E8%AE%8A%E5%8B%95

In case you say to remove the wiki suffix, pnp is 404 as well.

https://pnp.miraheze.org/wiki/

Maybe just check those URLs in a browser?

@Dzahn all urls are suffixed with wiki (mediawiki grants are to like '%wik%'.* or something).

The databases in the second list still exist, so I'm not exactly sure why those are 404ing.

The most likely situation is that we started the deletion process (removed the wiki from our createwiki table and thus making the domain inaccessible) but never actually dropped the databases.

I'd like to confirm this before the stats for those wikis are purged, but don't have shell access atm :D

@Dzahn Please let me know if you need anything else from us. All APIs should be working at whatever they're $wgServer is set to in our configuration.

I unfortunately haven't gotten to speak to SPF about making API's always accessible from *.miraheze.org/api.php, however all wikis should be working now

@Dzahn Please let me know if you need anything else from us

Just working API URLs under the miraheze.org domain.

So this ticket has become a little too general.

Is it about fixing the special wikis that don't have URLs under miraheze? Well, as explained above either let me know when they are fixed or there would have to be special code for this,if that is necessary i probably won't get to that soon.

I'd like to confirm this before the stats for those wikis are purged, but don't have shell access atm :D

Is it about deleting wikis that don't exist anymore? You said i should wait for you to confirm ^? Is that not the case anymore?

Is it about fixing the version that is displayed? Yet another thing and a general issue for all tables that i have to work on. (script to regularly check versions and be added in cron, for all tables etc). Should be more general than just about miraheze.

@Dzahn
MariaDB [metawiki]> select * from cw_wikis where wiki_dbname="reriawiki" or wiki_dbname="pnpwiki" or wiki_dbname="ronuoswiki" or wiki_dbname="reinventandolasorganizacioneswiki" or wiki_dbname="rwbyfrwiki" or wiki_dbname="rodintestonewiki" or wiki_dbname="rylnbgwiki" or wiki_dbname="ryinbgwiki";
Empty set (0.00 sec)

Please purge the statistics for all wikis mentioned in T146712#2856064

As for fixing the URL's, it's been discussed by us how to go about it, but no progress yet. I'll reply again when we get the *.miraheze.org api's working.

In T146712#2875124, @NDKilla wrote:

Please purge the statistics for all wikis mentioned in T146712#2856064

Done now.

for miraheze in $(cat T146712.fixed); do echo "delete from miraheze where prefix='${miraheze}';"; done | tee T146712.sql
mysql -u root wikistats < T146712.sql

For next time it would be more convenient if you can provide me with just the raw list of names, one per line, without "wiki", like:

entropedia
anicarpedia
d4rkst4r
lifeline
..

As for fixing the URL's, it's been discussed by us how to go about it, but no progress yet. I'll reply again when we get the *.miraheze.org api's working.

Alright.

opened https://phabricator.miraheze.org/T1176 downstream to provide a dblist including the custom domain names, otherwise it's not possible to automate this and we'll have to keep manually adding wikis with custom domains

Change 328860 had a related patch set uploaded (by Dzahn):
fix for updating miraheze wikis with custom URLs

https://gerrit.wikimedia.org/r/328860

Change 328860 merged by Dzahn:
fix for updating miraheze wikis with custom URLs

https://gerrit.wikimedia.org/r/328860

http://wikistats.wmflabs.org/display.php?t=mh now with allthetropes.org and other wikis with custom URLs