Page MenuHomePhabricator

labtestweb2001 should either be fully productionized or removed completely from production configuration
Closed, ResolvedPublic

Description

labtestweb2001 database cannot be connected from terbium. However, terbium cron jobs are scheduled for it as it is part of the production configuration. Either add it fully to production, or remove it entirely. This creates annoying daily cron-spam messages.

Example backtrace:

[a82b86761a33f17cfd586b11] [no req]   Wikimedia\Rdbms\DBConnectionError from line 999 of /srv/mediawiki/php-1.31.0-wmf.29/includes/libs/rdbms/database/Database.php: Cannot access the database: Host '10.64.32.13' is not allowed to connect to this MariaDB server (208.80.153.14)
Backtrace:
#0 /srv/mediawiki/php-1.31.0-wmf.29/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1133): Wikimedia\Rdbms\Database->reportConnectionError(string)
#1 /srv/mediawiki/php-1.31.0-wmf.29/includes/libs/rdbms/loadbalancer/LoadBalancer.php(748): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#2 /srv/mediawiki/php-1.31.0-wmf.29/includes/SiteStatsInit.php(200): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, string)
#3 /srv/mediawiki/php-1.31.0-wmf.29/includes/SiteStatsInit.php(51): SiteStatsInit::getDB(integer, string)
#4 /srv/mediawiki/php-1.31.0-wmf.29/maintenance/initSiteStats.php(44): SiteStatsInit->__construct(boolean)
#5 /srv/mediawiki/php-1.31.0-wmf.29/maintenance/doMaintenance.php(94): InitSiteStats->execute()
#6 /srv/mediawiki/php-1.31.0-wmf.29/maintenance/initSiteStats.php(82): require_once(string)
#7 /srv/mediawiki/multiversion/MWScript.php(100): require_once(string)
#8 {main}

High because on scripts not ready to capture that exception and continue gracefully, it will abort the execution for other legitimate wikis, technically causing a cron outage.

When this is fixed, all cronspam that references 208.80.153.14 will be gone.

Event Timeline

jcrespo triaged this task as Medium priority.Apr 17 2018, 8:15 AM
jcrespo created this task.
jcrespo raised the priority of this task from Medium to High.Apr 17 2018, 8:18 AM
jcrespo updated the task description. (Show Details)
jcrespo updated the task description. (Show Details)

Change 427257 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labtestweb: Allow access to db from maintenance hosts

https://gerrit.wikimedia.org/r/427257

Change 427257 merged by Andrew Bogott:
[operations/puppet@production] labtestweb: Allow access to db from maintenance hosts

https://gerrit.wikimedia.org/r/427257

jcrespo renamed this task from labtestweb2001 should either productionized of removed completely from production configuration to labtestweb2001 should either be fully productionized or removed completely from production configuration.Apr 17 2018, 9:31 PM

Change 427688 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labtestwikitech: add grants for labtestwiki, now on m5

https://gerrit.wikimedia.org/r/427688

Change 427688 merged by Andrew Bogott:
[operations/puppet@production] labtestwikitech: add grants for labtestwiki, now on m5

https://gerrit.wikimedia.org/r/427688

Change 427720 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] m5: allow labtestweb2001 mysql access

https://gerrit.wikimedia.org/r/427720

Change 427720 merged by Andrew Bogott:
[operations/puppet@production] m5: allow labtestweb2001 mysql access

https://gerrit.wikimedia.org/r/427720

Andrew claimed this task.

I moved the labtestwiki db from labtestweb2001 to m5.

Change 429841 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/mediawiki-config@master] labtestwikitech: use eqiad db host (db1073) even from codfw

https://gerrit.wikimedia.org/r/429841

Change 429841 abandoned by Andrew Bogott:
labtestwikitech: use eqiad db host (db1073) even from codfw

Reason:
now we have clouddb2001-dev for this

https://gerrit.wikimedia.org/r/429841