jcrespo (Jaime Crespo)
Sr Database Administrator

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
May 11 2015, 8:31 AM (175 w, 5 d)
Availability
Available
IRC Nick
jynus
LDAP User
Jcrespo
MediaWiki User
JCrespo (WMF) [ Global Accounts ]

Recent Activity

Thu, Sep 20

jcrespo added a comment to T203969: Create Icinga alerts on backup generation failure.

I have a first quick and dirty script to check for backup freshness:

Thu, Sep 20, 3:31 PM · Patch-For-Review, DBA
jcrespo added a comment to T204346: PHP-timed out requests also emit LoadBalancer::destruct error "you can't run this command now: COMMIT".

Not that it changes a lot, but indeed this doesn't happen only on COMMIT, also on ROLLBACK: Wikimedia\Rdbms\Database::close 10.192.32.7 2014 Commands out of sync; you can't run this command now (10.192.32.7) ROLLBACK

Thu, Sep 20, 1:30 PM · Performance-Team, Wikimedia-production-error, MediaWiki-Database
jcrespo added a comment to T204110: Add favicon to icinga and tendril.

Icinga is now showing me an icon, but etherpad isn't. Not sure how much of this is valid or it is my browser cache misbehaving.

Thu, Sep 20, 1:22 PM · Operations
jcrespo renamed T204928: Issues with mgmt interface on es2001 host from Several issues with mgmt interfaces on es200X hosts to Issues with mgmt interface on es2001 host.
Thu, Sep 20, 11:20 AM · Operations, DBA, ops-codfw
jcrespo updated the task description for T204928: Issues with mgmt interface on es2001 host.
Thu, Sep 20, 11:17 AM · Operations, DBA, ops-codfw
jcrespo updated the task description for T204928: Issues with mgmt interface on es2001 host.
Thu, Sep 20, 9:49 AM · Operations, DBA, ops-codfw
jcrespo moved T204928: Issues with mgmt interface on es2001 host from Triage to Backlog on the DBA board.
Thu, Sep 20, 9:41 AM · Operations, DBA, ops-codfw
jcrespo triaged T204928: Issues with mgmt interface on es2001 host as Lowest priority.
Thu, Sep 20, 9:40 AM · Operations, DBA, ops-codfw
jcrespo created T204928: Issues with mgmt interface on es2001 host.
Thu, Sep 20, 9:40 AM · Operations, DBA, ops-codfw
jcrespo moved T201392: Finish eqiad metadata database backup setup (s1-s8, x1) from In progress to Blocked external/Not db team on the DBA board.
Thu, Sep 20, 8:58 AM · Patch-For-Review, DBA
jcrespo assigned T183983: Re-institute query killer for the analytics WikiReplica to Banyek.
Thu, Sep 20, 8:58 AM · User-Banyek, Patch-For-Review, Data-Services, DBA
jcrespo assigned T204006: Execute the schema change for Partial Blocks to Marostegui.
Thu, Sep 20, 8:57 AM · Patch-For-Review, DBA, Blocked-on-schema-change, Anti-Harassment
jcrespo added a comment to T204593: dbstore2002 s2 crashed.

Looking good so far: https://grafana.wikimedia.org/dashboard/db/prometheus-machine-stats?panelId=18&fullscreen&orgId=1&var-server=dbstore2002&var-datasource=codfw%20prometheus%2Fops&from=now-3h&to=now-1m

Thu, Sep 20, 8:56 AM · Patch-For-Review, DBA
jcrespo moved T204593: dbstore2002 s2 crashed from Next to In progress on the DBA board.
Thu, Sep 20, 8:56 AM · Patch-For-Review, DBA
jcrespo added a comment to T202596: Write our anticipated "phase two" schemas and submit for review.

@Bawolff That new query you propose makes no sense to me- it just selects the first 100 revisions every single time(e.g. revision id 1 to 100, if they all existed). But sure, if they want to select that (ids and a bunch of NULLs), I don't see any problem with doing that one (although I would batch by rev id without using a limit for more deterministic speed), I am just not sure that is what they *really* want.

Thu, Sep 20, 7:42 AM · Patch-For-Review, DBA, Scoring-platform-team (Current), User-Joe, Operations, JADE
jcrespo added a comment to T164666: Long running query MessageCache::loadFromDB(en)-small on WMF "special" slaves.

I've personally don't remember this query on the top worring list anymore (unlike a bunch of others), I would personally close it as resolved.

Thu, Sep 20, 7:03 AM · Wikimedia-production-error, MediaWiki-Cache, MW-1.30-release-notes, MediaWiki-Database
jcrespo updated subscribers of T202596: Write our anticipated "phase two" schemas and submit for review.

That looks really bad performance. Not only that scans the revision table from top to bottom (>200GB of data) making it slow, it is nondeterministically slow- it will be faster or slower depending on the parameters and existing data.

Thu, Sep 20, 6:59 AM · Patch-For-Review, DBA, Scoring-platform-team (Current), User-Joe, Operations, JADE

Wed, Sep 19

jcrespo added a comment to T198176: Mediawiki page deletions should happen in batches of revisions.

If deleteBatch.php will be able to use to delete large pages in revision batches when the patch is merged, yes, it will fulfill fully my needs. I couldn't use it before because it batches pages, but not revisions within a page.

Wed, Sep 19, 3:58 PM · Wikimedia-production-error, Patch-For-Review, Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MediaWiki-Page-deletion
jcrespo moved T204838: Make Wikibase wb_terms term_search_key index the same as wb_term_search_key in WMF production from Triage to Blocked external/Not db team on the DBA board.

^I am assuming this doesn't have yet actionables for us (yet) just on the radar.

Wed, Sep 19, 3:53 PM · User-Marostegui, Wikidata-Campsite, MediaWiki-extensions-WikibaseRepository, Wikidata
jcrespo placed T174802: Archive and drop education program (ep_*) tables on all wikis up for grabs.

So public dumps are ready:

Wed, Sep 19, 3:07 PM · Datasets-General-or-Unknown, Data-Services, DBA
Marostegui awarded T204593: dbstore2002 s2 crashed a Mountain of Wealth token.
Wed, Sep 19, 2:39 PM · Patch-For-Review, DBA
jcrespo created P7568 https://gerrit.wikimedia.org/r/461406.
Wed, Sep 19, 2:10 PM
jcrespo triaged T172489: Monitor read_only on all databases, make it page on masters as Normal priority.
Wed, Sep 19, 1:08 PM · Patch-For-Review, Wikimedia-Incident, monitoring, DBA
jcrespo changed the status of T172489: Monitor read_only on all databases, make it page on masters from Open to Stalled.

The initial scope is not fulfilled:

Wed, Sep 19, 1:08 PM · Patch-For-Review, Wikimedia-Incident, monitoring, DBA
jcrespo changed the status of T172489: Monitor read_only on all databases, make it page on masters, a subtask of T171928: Wikidata and dewiki databases locked, from Open to Stalled.
Wed, Sep 19, 1:08 PM · Patch-For-Review, User-notice, Wikimedia-Incident, Operations, Wikidata
jcrespo added a comment to T195293: 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) .

to be scheduled independently.

Wed, Sep 19, 8:47 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Schema-change, Language-2018-July-September, User-Nikerabbit, MediaWiki-extensions-Translate, Wikimedia-Incident, Wikimedia-production-error, Operations
jcrespo added a comment to T204594: db1118 mysql process crashed (mysql 8.0 test host).

There is also the fact that I executed an ALTER table

It would fit in my theory because it would be (maybe) the first event with GTID that the host received in a long time.

Wed, Sep 19, 7:27 AM · User-Banyek, DBA
jcrespo added a comment to T204594: db1118 mysql process crashed (mysql 8.0 test host).

My suggestion for root cause is that mysql works well as long as gtid is not enabled on its master (it was enabled around some days of the crash). Replicating from a non-gtid enabled master worked well for a long time. I have not proof of this, it should be checked.

Wed, Sep 19, 6:49 AM · User-Banyek, DBA

Tue, Sep 18

jcrespo added a comment to T174802: Archive and drop education program (ep_*) tables on all wikis.

The tables cannot be dropped from other wikis just yet, as the extension is still enabled (but mostly disabled) on other wikis.

Tue, Sep 18, 2:18 PM · Datasets-General-or-Unknown, Data-Services, DBA
jcrespo moved T204593: dbstore2002 s2 crashed from In progress to Next on the DBA board.
Tue, Sep 18, 2:12 PM · Patch-For-Review, DBA
jcrespo placed T204593: dbstore2002 s2 crashed up for grabs.

dbstore2002:s2 is up and running (althought still catching up replication), enough for the backups to run in codfw unaltered.

Tue, Sep 18, 2:12 PM · Patch-For-Review, DBA
jcrespo added a comment to T174802: Archive and drop education program (ep_*) tables on all wikis.

List of tables to archive:

Tue, Sep 18, 11:20 AM · Datasets-General-or-Unknown, Data-Services, DBA
jcrespo claimed T174802: Archive and drop education program (ep_*) tables on all wikis.
Tue, Sep 18, 11:09 AM · Datasets-General-or-Unknown, Data-Services, DBA
jcrespo placed T85756: Make (redacted) log_search table available on ToolLabs up for grabs.
Tue, Sep 18, 11:09 AM · MediaWiki-General-or-Unknown, Security, Data-Services, DBA
jcrespo moved T85756: Make (redacted) log_search table available on ToolLabs from Backlog to In progress on the DBA board.
Tue, Sep 18, 11:07 AM · MediaWiki-General-or-Unknown, Security, Data-Services, DBA
jcrespo changed the status of T201392: Finish eqiad metadata database backup setup (s1-s8, x1) from Open to Stalled.

Blocked on new backup hardware setup.

Tue, Sep 18, 11:07 AM · Patch-For-Review, DBA
jcrespo changed the status of T201392: Finish eqiad metadata database backup setup (s1-s8, x1), a subtask of T138562: Improve regular production database backups handling, from Open to Stalled.
Tue, Sep 18, 11:07 AM · Epic, Wikimedia-Incident, DBA
jcrespo placed T201392: Finish eqiad metadata database backup setup (s1-s8, x1) up for grabs.
Tue, Sep 18, 11:06 AM · Patch-For-Review, DBA
jcrespo moved T193226: Test MySQL 8.0 with production data and evaluate its fit for WMF databases from In progress to Blocked external/Not db team on the DBA board.
Tue, Sep 18, 11:04 AM · Patch-For-Review, DBA
jcrespo placed T193226: Test MySQL 8.0 with production data and evaluate its fit for WMF databases up for grabs.
Tue, Sep 18, 11:03 AM · Patch-For-Review, DBA
jcrespo placed T202051: db2042 RAID battery failed up for grabs.
Tue, Sep 18, 11:03 AM · Operations, ops-codfw, DBA
jcrespo added a comment to T204593: dbstore2002 s2 crashed.

Cloning is ongoing from db2041 to dbstore2002 /srv/sqldata, will rename later to /srv/sqldata.s2 and restart replication there quickly. After the backups run, some compression process will have to be rerun, which may take quite a lot on the dbstore host with so much load.

Tue, Sep 18, 10:45 AM · Patch-For-Review, DBA
jcrespo renamed T204594: db1118 mysql process crashed (mysql 8.0 test host) from db1118 crashed (mysql 8.0 test host) to db1118 mysql process crashed (mysql 8.0 test host).
Tue, Sep 18, 9:18 AM · User-Banyek, DBA
jcrespo moved T203565: Reclone db1114 (s1 api) from another API host from Next to In progress on the DBA board.
Tue, Sep 18, 9:14 AM · User-Banyek, Patch-For-Review, DBA
jcrespo added a project to T204531: Wikidata dumps creating large amounts of log spam: Datacenter-Switchover-2018.

CC @Joe @akosiaris not to solve the specific issue, but to note it as a potential missed step/puppet code change related to switch datacenter.

Tue, Sep 18, 9:03 AM · Datacenter-Switchover-2018, MediaWiki-Logging, Wikidata, Dumps-Generation, Wikimedia-production-error
jcrespo added a comment to T100501: mysql user and group should be a system user/group.

So only pending labsdb1004,labsdb1005, dbstore1002 and the parsercache

Tue, Sep 18, 7:34 AM · Patch-For-Review, Operations, DBA
jcrespo claimed T204593: dbstore2002 s2 crashed.

I will do that, I'll see from where.

Tue, Sep 18, 7:17 AM · Patch-For-Review, DBA
jcrespo triaged T204594: db1118 mysql process crashed (mysql 8.0 test host) as Low priority.
Tue, Sep 18, 7:15 AM · User-Banyek, DBA

Mon, Sep 17

jcrespo updated the task description for T204594: db1118 mysql process crashed (mysql 8.0 test host).
Mon, Sep 17, 8:10 PM · User-Banyek, DBA
jcrespo created T204594: db1118 mysql process crashed (mysql 8.0 test host).
Mon, Sep 17, 8:08 PM · User-Banyek, DBA
jcrespo updated the task description for T204593: dbstore2002 s2 crashed.
Mon, Sep 17, 7:50 PM · Patch-For-Review, DBA
jcrespo created T204593: dbstore2002 s2 crashed.
Mon, Sep 17, 7:46 PM · Patch-For-Review, DBA
Marostegui awarded T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1 a Cookie token.
Mon, Sep 17, 6:12 PM · Patch-For-Review, Operations, DBA
jcrespo closed T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1 as Resolved.
Mon, Sep 17, 6:05 PM · Patch-For-Review, Operations, DBA
jcrespo closed T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1, a subtask of T189107: DB meta task for next DC failover issues, as Resolved.
Mon, Sep 17, 6:05 PM · Patch-For-Review, Epic, Operations, DBA
jcrespo updated the task description for T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1.
Mon, Sep 17, 6:05 PM · Patch-For-Review, Operations, DBA
jcrespo added a comment to T174802: Archive and drop education program (ep_*) tables on all wikis.

Should just be a case of making sure they're named well, gzip (or similar)

Mon, Sep 17, 4:37 PM · Datasets-General-or-Unknown, Data-Services, DBA
jcrespo awarded T204493: db1061 management interface busy (no sessions allowed) a Mountain of Wealth token.
Mon, Sep 17, 4:23 PM · ops-eqiad, Operations
jcrespo added a comment to T174802: Archive and drop education program (ep_*) tables on all wikis.

I may have time to work on this these week (the dumping) not sure if they will be available shortly after, as not sure what that entails for custom dumps.

Mon, Sep 17, 4:22 PM · Datasets-General-or-Unknown, Data-Services, DBA
jcrespo moved T174802: Archive and drop education program (ep_*) tables on all wikis from Backlog to Next on the DBA board.
Mon, Sep 17, 4:21 PM · Datasets-General-or-Unknown, Data-Services, DBA
jcrespo added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

The plan is for us dbas to test setting up a single API with the same structure than eqiad and do all assuming that fixies it, and later we will have to evaluate what is the right long-term status, given some unknowns and related tasks such as T202167:

One note: This index might be already existing but under name of tmp_1 or tmp_2 or tmp_3. I don't know how complex renaming an index is but that'd be useful.

Mon, Sep 17, 4:16 PM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata
jcrespo moved T202764: Wikidata produces a lot of failed requests for recentchanges API from Triage to In progress on the DBA board.
Mon, Sep 17, 4:11 PM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata
jcrespo added a project to T202764: Wikidata produces a lot of failed requests for recentchanges API: Datacenter-Switchover-2018.
Mon, Sep 17, 4:07 PM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata
jcrespo added a comment to T204493: db1061 management interface busy (no sessions allowed).

mgmt interface works for me again, waiting for you to be finished to continue with my maintenance.

Mon, Sep 17, 3:48 PM · ops-eqiad, Operations
jcrespo updated the task description for T204531: Wikidata dumps creating large amounts of log spam.
Mon, Sep 17, 3:38 PM · Datacenter-Switchover-2018, MediaWiki-Logging, Wikidata, Dumps-Generation, Wikimedia-production-error
jcrespo added a comment to T204493: db1061 management interface busy (no sessions allowed).

db1061 should be now fully down and ready for you! @Cmjohnson

Mon, Sep 17, 3:26 PM · ops-eqiad, Operations
jcrespo added a comment to T204493: db1061 management interface busy (no sessions allowed).

Shutting down service...

Mon, Sep 17, 3:18 PM · ops-eqiad, Operations
jcrespo added a comment to T203039: Storage of data for recommendation API.

Is 250K rows inserted per second fast enough? https://dbahire.com/testing-again-load-data-on-mysql-5-6-5-7-8-0-non-ga-and-mariadb-10-0-10-1-and-10-2-non-ga/ (I got those numbers on my 5-year old pc) or will you need more tuning? :-D You can download the sample code here: http://dbahire.com/testing-the-fastest-way-to-import-a-table-into-mysql-and-some-interesting-5-7-performance-results/ and of course we help people precisely with this kind of work. If batch import speed is important, rather read throughput, we can go in a classic master-replica async replication topology with automatic failover (as we have now with misc services) rather than a galera-like cluster.

Mon, Sep 17, 2:21 PM · Operations, DBA, Services (designing), Research
jcrespo created T204531: Wikidata dumps creating large amounts of log spam.
Mon, Sep 17, 2:04 PM · Datacenter-Switchover-2018, MediaWiki-Logging, Wikidata, Dumps-Generation, Wikimedia-production-error
jcrespo updated the task description for T204493: db1061 management interface busy (no sessions allowed).
Mon, Sep 17, 1:38 PM · ops-eqiad, Operations
jcrespo updated the task description for T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1.
Mon, Sep 17, 1:15 PM · Patch-For-Review, Operations, DBA
jcrespo updated the task description for T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1.
Mon, Sep 17, 11:53 AM · Patch-For-Review, Operations, DBA
jcrespo triaged T204493: db1061 management interface busy (no sessions allowed) as High priority.
Mon, Sep 17, 11:52 AM · ops-eqiad, Operations
jcrespo updated the task description for T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1.
Mon, Sep 17, 11:19 AM · Patch-For-Review, Operations, DBA
jcrespo edited projects for T204476: Lock wait timeout exceeded from WikiPage::insertOn (Unable to import "Template:IPA_Symbol"), added: MediaWiki-Export-or-Import, Wikimedia-production-error; removed DBA.
Mon, Sep 17, 8:47 AM · Wikimedia-production-error, MediaWiki-Export-or-Import
jcrespo added a subtask for T30499: 1205: Lock wait timeout exceeded; try restarting transaction (tracking): T204476: Lock wait timeout exceeded from WikiPage::insertOn (Unable to import "Template:IPA_Symbol").
Mon, Sep 17, 8:46 AM · Technical-Debt, Tracking, MediaWiki-Database
jcrespo added a parent task for T204476: Lock wait timeout exceeded from WikiPage::insertOn (Unable to import "Template:IPA_Symbol"): T30499: 1205: Lock wait timeout exceeded; try restarting transaction (tracking).
Mon, Sep 17, 8:46 AM · Wikimedia-production-error, MediaWiki-Export-or-Import

Fri, Sep 14

jcrespo added a comment to T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1.

db1090 took a long time to recover replication lag after db1062 initial maintenance, even more than dbstore1002- need to check why next week.

Fri, Sep 14, 4:02 PM · Patch-For-Review, Operations, DBA
jcrespo updated the task description for T204346: PHP-timed out requests also emit LoadBalancer::destruct error "you can't run this command now: COMMIT".
Fri, Sep 14, 3:20 PM · Performance-Team, Wikimedia-production-error, MediaWiki-Database
jcrespo updated the task description for T204346: PHP-timed out requests also emit LoadBalancer::destruct error "you can't run this command now: COMMIT".
Fri, Sep 14, 3:19 PM · Performance-Team, Wikimedia-production-error, MediaWiki-Database
jcrespo updated subscribers of T204346: PHP-timed out requests also emit LoadBalancer::destruct error "you can't run this command now: COMMIT".

Thanks, I will update the summary to remove my guess and point to this better evaluation. CCing @tstarling beacause it could be an intended consequence of the time limit and maybe those things can be captured/handled better (not sure)?

Fri, Sep 14, 3:17 PM · Performance-Team, Wikimedia-production-error, MediaWiki-Database
jcrespo added a comment to T204302: db1062 management interface busy (no sessions allowed).

I will shutdown this server fully at 14:11 UTC, this is a self reminder.

Fri, Sep 14, 1:37 PM · ops-eqiad, Operations
jcrespo removed a subtask for T189107: DB meta task for next DC failover issues: T204302: db1062 management interface busy (no sessions allowed).
Fri, Sep 14, 1:32 PM · Patch-For-Review, Epic, Operations, DBA
jcrespo edited parent tasks for T204302: db1062 management interface busy (no sessions allowed), added: T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1; removed: T189107: DB meta task for next DC failover issues.
Fri, Sep 14, 1:32 PM · ops-eqiad, Operations
jcrespo added a subtask for T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1: T204302: db1062 management interface busy (no sessions allowed).
Fri, Sep 14, 1:32 PM · Patch-For-Review, Operations, DBA
jcrespo claimed T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1.
Fri, Sep 14, 1:03 PM · Patch-For-Review, Operations, DBA
jcrespo moved T134809: Apache <=> mariadb SSL/TLS for cross-datacenter writes from Triage to Meta/Epic on the DBA board.
Fri, Sep 14, 1:02 PM · Availability (MediaWiki-MultiDC), DBA, Operations, Performance-Team
jcrespo moved T203039: Storage of data for recommendation API from Triage to Meta/Epic on the DBA board.
Fri, Sep 14, 1:02 PM · Operations, DBA, Services (designing), Research
jcrespo created T204346: PHP-timed out requests also emit LoadBalancer::destruct error "you can't run this command now: COMMIT".
Fri, Sep 14, 12:42 PM · Performance-Team, Wikimedia-production-error, MediaWiki-Database
jcrespo added a comment to P7548 what I did on db1070 after 10.1 upgrade.

Also https://gerrit.wikimedia.org/r/460506

Fri, Sep 14, 11:06 AM
jcrespo updated the task description for T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1.
Fri, Sep 14, 10:59 AM · Patch-For-Review, Operations, DBA
jcrespo added a comment to P7548 what I did on db1070 after 10.1 upgrade.

second puppet run get's stuck on master reimage because it expects mariadb being up and running.

Fri, Sep 14, 10:58 AM
jcrespo created P7548 what I did on db1070 after 10.1 upgrade.
Fri, Sep 14, 10:57 AM
jcrespo triaged T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1 as Normal priority.
Fri, Sep 14, 10:03 AM · Patch-For-Review, Operations, DBA
jcrespo added a comment to T204302: db1062 management interface busy (no sessions allowed).

@Cmjohnson I have restarted the service on this host, I will put it back down when you ping us you are available, as it cannot be down for long periods of time CC @Marostegui

Fri, Sep 14, 8:07 AM · ops-eqiad, Operations
jcrespo triaged T204302: db1062 management interface busy (no sessions allowed) as High priority.
Fri, Sep 14, 8:02 AM · ops-eqiad, Operations
jcrespo reassigned T204127: Reclone db2054 and db2068 from jcrespo to Marostegui.
Fri, Sep 14, 6:50 AM · Patch-For-Review, Datacenter-Switchover-2018, DBA

Thu, Sep 13

jcrespo claimed T204127: Reclone db2054 and db2068.

db2068 has been recloned, but needs time to catch up replication and then be slowly repooled with the above patch.

Thu, Sep 13, 4:28 PM · Patch-For-Review, Datacenter-Switchover-2018, DBA
jcrespo updated the task description for T204127: Reclone db2054 and db2068.
Thu, Sep 13, 4:27 PM · Patch-For-Review, Datacenter-Switchover-2018, DBA
jcrespo added a comment to T181205: Let quarry use the mariadb module.

profile 'profile::quarry::database' includes non-profile class mariadb::packages

Thu, Sep 13, 4:14 PM · Patch-For-Review, cloud-services-team (Kanban), Operations, Quarry