Marostegui (Manuel Aróstegui)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Sep 1 2016, 6:48 AM (54 w, 6 d)
Availability
Available
IRC Nick
marostegui
LDAP User
Marostegui
MediaWiki User
MArostegui (WMF)

Recent Activity

Thu, Sep 7

Marostegui triaged T175264: Decommission db1049 as Normal priority.
Thu, Sep 7, 1:23 PM · ops-eqiad, Operations, DBA
Marostegui moved T175264: Decommission db1049 from Triage to Backlog on the DBA board.
Thu, Sep 7, 1:23 PM · ops-eqiad, Operations, DBA
Marostegui created T175264: Decommission db1049.
Thu, Sep 7, 1:22 PM · ops-eqiad, Operations, DBA
Marostegui updated the task description for T170662: Productionize 22 new codfw database servers.
Thu, Sep 7, 11:28 AM · Patch-For-Review, DBA
Marostegui moved T175228: Degraded RAID on db2010 from In progress to Blocked external/Not db team on the DBA board.
Thu, Sep 7, 9:19 AM · DBA, Operations, ops-codfw
Marostegui added a comment to T161294: run pt-tablechecksum on s5.

db1100 has been cloned from db1049 and migrated to file per table.
It is now catching up, once it has caught up I will create the decommission task for db1049 but I won't act on it as I will be gone for two weeks, just create it as a reminder that it is ready to be decommissioned.

Thu, Sep 7, 9:18 AM · DBA
Marostegui updated the task description for T172679: Productionize 11 new eqiad database servers.
Thu, Sep 7, 9:16 AM · Patch-For-Review, DBA
Marostegui assigned T175228: Degraded RAID on db2010 to Papaul.
Thu, Sep 7, 8:39 AM · DBA, Operations, ops-codfw
Marostegui added a comment to T175228: Degraded RAID on db2010.

@Marostegui, are we sure we want this done, and not get rid of the host directly?- it is a very old host and we have its replacements setup. I would ask how many spare disks we have left, if any, and only change it if we have plently, start cloning it otherwise.

Thu, Sep 7, 8:31 AM · DBA, Operations, ops-codfw
Marostegui moved T89737: Make several mediawiki table fields unsigned ints on wmf databases from Triage to Backlog on the DBA board.
Thu, Sep 7, 5:26 AM · Blocked-on-schema-change, DBA
Marostegui added a comment to T174860: Define naming scheme for connecting to new wiki replica cluster.

@jcrespo & @Marostegui: do either of you have a reasoned objection to us using the <wiki>.(web|analytics).db.svc.eqiad.wmflabs naming scheme for Cloud Services user access to the new servers?

Thu, Sep 7, 5:24 AM · Patch-For-Review, cloud-services-team (Kanban), User-bd808, Data-Services
Marostegui closed T150306: truncate l10n_cache table on WMF wikis, a subtask of T54921: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking), as Resolved.
Thu, Sep 7, 5:21 AM · Epic, DBA, Tracking
Marostegui closed T150306: truncate l10n_cache table on WMF wikis as Resolved.

This is now all done

Thu, Sep 7, 5:21 AM · DBA
Marostegui moved T175228: Degraded RAID on db2010 from Triage to In progress on the DBA board.
Thu, Sep 7, 5:20 AM · DBA, Operations, ops-codfw
Marostegui added a project to T175228: Degraded RAID on db2010: DBA.

@Papaul please change the disk whenever you can.
Thanks!

Thu, Sep 7, 5:19 AM · DBA, Operations, ops-codfw

Wed, Sep 6

Marostegui added a comment to T175160: Identify the source of WHOIS data, the retrieval method, and update frequency.

@Marostegui I am removing the Schema Change tag.

Wed, Sep 6, 3:53 PM · Security-Reviews
Marostegui added a comment to T175160: Identify the source of WHOIS data, the retrieval method, and update frequency.

Is this meant to be a database schema change? https://wikitech.wikimedia.org/wiki/Schema_changes
If it is: can you please follow this template for the task, so DBAs can better understand it and thus try to get it resolved faster and without any issues?: https://wikitech.wikimedia.org/wiki/Schema_changes#Workflow_of_a_schema_change

Wed, Sep 6, 3:45 PM · Security-Reviews
Marostegui added a comment to T150306: truncate l10n_cache table on WMF wikis.

Truncation still happening on s3. I have throttled it quite a bit, because db1095 (sanitarium) was struggling to be able to replicate without any delay.

Wed, Sep 6, 3:36 PM · DBA
Marostegui moved T174569: Schema change for refactored comment storage from Triage to Backlog on the DBA board.
Wed, Sep 6, 1:37 PM · Dumps-Generation, Data-Services, Blocked-on-schema-change, DBA, MediaWiki-Platform-Team
Marostegui claimed T150306: truncate l10n_cache table on WMF wikis.
Wed, Sep 6, 9:24 AM · DBA
Marostegui added a comment to T150306: truncate l10n_cache table on WMF wikis.

Dropped from all the shards except from s3, which is being done now slowly

Wed, Sep 6, 9:15 AM · DBA
Marostegui added a comment to T161294: run pt-tablechecksum on s5.

I will migrate db1100 to file per table once the copy is done. db1049 was not using it.

Wed, Sep 6, 8:28 AM · DBA
Marostegui created P5962 (An Untitled Masterwork).
Wed, Sep 6, 7:18 AM
Marostegui triaged T150306: truncate l10n_cache table on WMF wikis as Normal priority.
Wed, Sep 6, 6:38 AM · DBA
Marostegui closed T174857: Degraded RAID on db1059 as Resolved.

This is all good now, thanks a lot Chris!

Wed, Sep 6, 5:29 AM · DBA, ops-eqiad, Operations

Tue, Sep 5

Marostegui added a comment to T175051: Drop "trackbacks" table on all wikis that have it.

Thanks for the initial list!
This is the complete list of wikis where it exists:

Tue, Sep 5, 6:28 PM · DBA
Marostegui moved T175051: Drop "trackbacks" table on all wikis that have it from Triage to Backlog on the DBA board.
Tue, Sep 5, 6:19 PM · DBA
Marostegui added a comment to T150306: truncate l10n_cache table on WMF wikis.

This table exists on:
s1:
enwiki

Tue, Sep 5, 6:17 PM · DBA
Marostegui added a comment to T150306: truncate l10n_cache table on WMF wikis.

You think we need to backup them (in case they are not empty?)

Tue, Sep 5, 5:54 PM · DBA
Marostegui moved T150306: truncate l10n_cache table on WMF wikis from Triage to Backlog on the DBA board.

Truncating these tables tables should be an easy win, and makes applying T146591: Add a primary key to l10n_cache across the cluster easier too

Tue, Sep 5, 5:53 PM · DBA
Marostegui added a comment to T146591: Add a primary key to l10n_cache.

I have tested https://gerrit.wikimedia.org/r/318553 on enwiki (db1089), dewiki and wikidata(db1092) and commons (db1091) and worked fine. I would say we can go ahead and get it merged
Thanks @Paladox!

Where are we with deploying this? Not that we actually use it in production.. It needs truncating, see T150306 and then the schema change should be very very cheap

https://gerrit.wikimedia.org/r/#/c/318553/21 is ready to go I think now.

Tue, Sep 5, 5:52 PM · Patch-For-Review, MediaWiki-Database
Marostegui added a comment to T174370: Create elections committee private wiki.

Going to remove the DBA tag from this task as our part is done, but I will remain subscribed as once the wiki is created, we'd need to double check the labs hosts to make sure nothing is actually being replicated.

That is exactly what DBA (done) column was created for (DBA work is done, but ticket cannot be closed yet) :-)

Tue, Sep 5, 5:41 PM · DBA, Patch-For-Review, Operations, Wiki-Setup (Create)
Marostegui updated the task description for T172679: Productionize 11 new eqiad database servers.
Tue, Sep 5, 5:39 PM · Patch-For-Review, DBA
Marostegui added a comment to T161294: run pt-tablechecksum on s5.

Well, of the new hosts, we will decide later which one goes to s8 and which ones go to s5. Let's just mark a new host as *old* master (because it is a copy of db1049), and later, before the database drop we decide if and where to duplicate. If we decide now, things are probably going to change by then, based on experience, so just clone db1049 so we get rid of it, and we will duplicate or whatever is needed later. Maybe, let0s say, db110X ends up on s5 or maybe it is on s8, we do not know yet.

Tue, Sep 5, 4:40 PM · DBA
Marostegui added a comment to T161294: run pt-tablechecksum on s5.

We can reclone db1071 from db1049 for instance then :)

No no, I meant a brand new one from this list- T172679. db1071 will probably be kept, we will have an exact copy of db1049 (which will be decommed) and we will have a host that eventually be part of s8. No need to delete s5 hosts (which we will need almost double the current number for the future s8). If we delete db1071 or db1092, we work twice as hard to get the same results.

Tue, Sep 5, 4:32 PM · DBA
Marostegui added a comment to T161294: run pt-tablechecksum on s5.

Cool with me, but I would actually keep db1092 untouched, and clone it to a new hosts, as I think the easiest way to setup s8 would be to clone s5 and later delete databases, so we can overprovision s5 for a while while not working twice.

Tue, Sep 5, 4:23 PM · DBA
Marostegui added a comment to T161294: run pt-tablechecksum on s5.

I was thinking that maybe we can reclone db1092 (for example) using db1049's data, (old master), so we can just decommission it...

Tue, Sep 5, 3:42 PM · DBA
Marostegui added a comment to T174509: Drop now redundant indexes from pagelinks and templatelinks.

And pagelinks:

root@db1083:/srv/sqldata/enwiki# ls -lh pagelinks.ibd
-rw-rw---- 1 mysql mysql 146G Sep  5 15:29 pagelinks.ibd
Tue, Sep 5, 3:29 PM · Patch-For-Review, DBA, Blocked-on-schema-change
Marostegui added a comment to T118764: Drop old devwikiinternal and rel13testwiki from s3.

devwikiinternal and rel13testwiki and last write happened on 2015.
I have taken a mysqldump from those two databases and placed them at:

root@dbstore1001:/srv/tmp/T118764# pwd
/srv/tmp/T118764
root@dbstore1001:/srv/tmp/T118764# ls -lh
total 420K
-rw-r--r-- 1 root root 208K Sep  5 15:07 devwikiinternal.sql
-rw-r--r-- 1 root root 212K Sep  5 15:07 rel13testwiki.sql
Tue, Sep 5, 3:09 PM · DBA
Marostegui closed T174782: Drop pr_index from wikis where ProofreadPage isn't enabled, a subtask of T54921: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking), as Resolved.
Tue, Sep 5, 2:57 PM · Epic, DBA, Tracking
Marostegui closed T174782: Drop pr_index from wikis where ProofreadPage isn't enabled as Resolved.

All dropped.

Tue, Sep 5, 2:57 PM · DBA
Marostegui added a comment to T174509: Drop now redundant indexes from pagelinks and templatelinks.

Actually, we could also optimize the table without depooling as it is an INPLACE operation (and I actually remember optimizing a big table not long ago without any lag): https://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-overview.html
We can double check anyways.

Tue, Sep 5, 2:11 PM · Patch-For-Review, DBA, Blocked-on-schema-change
Marostegui closed T174586: Remove ReaderFeedback tables from wikis as Resolved.

Tables dropped from: s5,s6 and s7.

Tue, Sep 5, 12:28 PM · DBA
Marostegui closed T174586: Remove ReaderFeedback tables from wikis, a subtask of T54921: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking), as Resolved.
Tue, Sep 5, 12:28 PM · Epic, DBA, Tracking
Marostegui added a comment to T174586: Remove ReaderFeedback tables from wikis.

Tables dropped from s3

Tue, Sep 5, 12:27 PM · DBA
Marostegui added a comment to T174586: Remove ReaderFeedback tables from wikis.

Tables dropped from s2

Tue, Sep 5, 11:57 AM · DBA
Marostegui added a comment to T174509: Drop now redundant indexes from pagelinks and templatelinks.

Sounds good to me. I will finish db1083 for as the defgramentation is already started on pagelinks.

Tue, Sep 5, 11:44 AM · Patch-For-Review, DBA, Blocked-on-schema-change
Marostegui added a comment to T174509: Drop now redundant indexes from pagelinks and templatelinks.

Maybe we should do a quick "index deletion" online and then track here the defragmention?

Tue, Sep 5, 11:38 AM · Patch-For-Review, DBA, Blocked-on-schema-change
Marostegui added a comment to T174509: Drop now redundant indexes from pagelinks and templatelinks.

This is template links after the defragmentation:

root@db1083:/srv/sqldata/enwiki# ls -lh templatelinks.ibd
-rw-rw---- 1 mysql mysql 85G Sep  5 11:33 templatelinks.ibd
Tue, Sep 5, 11:33 AM · Patch-For-Review, DBA, Blocked-on-schema-change
Marostegui added a comment to T174509: Drop now redundant indexes from pagelinks and templatelinks.

The ALTER itself takes seconds to run (less than 10 seconds on a SSD host).

root@PRODUCTION s1 slave[enwiki]> show create table pagelinks\G show create table templatelinks\G
*************************** 1. row ***************************
       Table: pagelinks
Create Table: CREATE TABLE `pagelinks` (
  `pl_from` int(8) unsigned NOT NULL DEFAULT '0',
  `pl_namespace` int(11) NOT NULL DEFAULT '0',
  `pl_title` varbinary(255) NOT NULL DEFAULT '',
  `pl_from_namespace` int(11) NOT NULL DEFAULT '0',
  PRIMARY KEY (`pl_from`,`pl_namespace`,`pl_title`),
  KEY `pl_namespace` (`pl_namespace`,`pl_title`,`pl_from`),
  KEY `pl_backlinks_namespace` (`pl_from_namespace`,`pl_namespace`,`pl_title`,`pl_from`)
) ENGINE=InnoDB DEFAULT CHARSET=binary
1 row in set (0.00 sec)
Tue, Sep 5, 6:40 AM · Patch-For-Review, DBA, Blocked-on-schema-change
Marostegui added a comment to T174509: Drop now redundant indexes from pagelinks and templatelinks.

I just realised that db1089 (enwiki) does not have those indexes.
db1083 does.

Tue, Sep 5, 6:24 AM · Patch-For-Review, DBA, Blocked-on-schema-change

Mon, Sep 4

Marostegui added a comment to T143366: Removal of {{#coordinates:}} leaves DB entries behind.

Thanks for the heads up. I assume this will be fixed at some point, right?

Mon, Sep 4, 1:44 PM · DBA, Maps-Sprint, Discovery, GeoData
Marostegui closed T166695: Substantive HTTP and mediawiki/database traffic coming from a single ip as Resolved.

Closing this as this has not happened again in months.
If it happens again, let's reopen and follow up

Mon, Sep 4, 1:36 PM · DBA, Operations, Traffic
Marostegui moved T174076: Decommission db1033 and db1028 from In progress to Done on the DBA board.
Mon, Sep 4, 1:29 PM · ops-eqiad, Operations, DBA
Marostegui closed T32824: Integrate Facebooks "Online Schema Change for MySQL" into MediaWiki as Resolved.

If no one disagrees I am going to close this as we are already using pt-online-schema-change and when available we use INPLACE operations supported on MariaDB.
As Jaime pointed out, the most popular tool out there now is gh-ost (along with pt-online-schema-change).
We could give gh-ost a go, but so far we don't have much bandwidth for it, and we can track it in an specific task for all the related tests.

Mon, Sep 4, 12:34 PM · DBA, MediaWiki-Database
Marostegui assigned T174902: Decommission db1037 to Cmjohnson.

MySQL is stopped.
This host is now ready for the remaining DC Ops steps to be completed

Mon, Sep 4, 12:24 PM · Patch-For-Review, Operations, ops-eqiad, DBA
Marostegui updated the task description for T174902: Decommission db1037.
Mon, Sep 4, 12:17 PM · Patch-For-Review, Operations, ops-eqiad, DBA
Marostegui updated the task description for T174902: Decommission db1037.
Mon, Sep 4, 12:08 PM · Patch-For-Review, Operations, ops-eqiad, DBA
Marostegui updated the task description for T174902: Decommission db1037.
Mon, Sep 4, 11:40 AM · Patch-For-Review, Operations, ops-eqiad, DBA
Marostegui moved T174902: Decommission db1037 from Triage to In progress on the DBA board.
Mon, Sep 4, 8:55 AM · Patch-For-Review, Operations, ops-eqiad, DBA
Marostegui triaged T174902: Decommission db1037 as Normal priority.
Mon, Sep 4, 8:55 AM · Patch-For-Review, Operations, ops-eqiad, DBA
Marostegui created T174902: Decommission db1037.
Mon, Sep 4, 8:54 AM · Patch-For-Review, Operations, ops-eqiad, DBA
Marostegui assigned T174806: Decommission db1045 to Cmjohnson.

This server is ready to be totally decommissioned and only pending the DC Ops steps, so assigning it to @Cmjohnson for the pending steps

Mon, Sep 4, 8:50 AM · Patch-For-Review, hardware-requests, ops-eqiad, DBA, Operations
Marostegui updated the task description for T174806: Decommission db1045.
Mon, Sep 4, 8:47 AM · Patch-For-Review, hardware-requests, ops-eqiad, DBA, Operations
Marostegui updated the task description for T174806: Decommission db1045.
Mon, Sep 4, 8:28 AM · Patch-For-Review, hardware-requests, ops-eqiad, DBA, Operations
Marostegui moved T168661: Apply schema change to add 3D filetype for STL files from In progress to Blocked external/Not db team on the DBA board.
Mon, Sep 4, 8:23 AM · Patch-For-Review, Blocked-on-schema-change, DBA
Marostegui updated the task description for T168661: Apply schema change to add 3D filetype for STL files.
Mon, Sep 4, 8:23 AM · Patch-For-Review, Blocked-on-schema-change, DBA
Marostegui updated the task description for T168661: Apply schema change to add 3D filetype for STL files.
Mon, Sep 4, 8:22 AM · Patch-For-Review, Blocked-on-schema-change, DBA
Marostegui reassigned T168661: Apply schema change to add 3D filetype for STL files from Marostegui to MarkTraceur.

@MarkTraceur I am going to re-assign this task back to you to see if you can arrange a read-only time for commonswiki.
I will be on holidays from 7th to 25th, so anytime after 25th of Sept (and probably best to avoid it being a Friday) would be good for me.

Mon, Sep 4, 8:21 AM · Patch-For-Review, Blocked-on-schema-change, DBA
Marostegui added a comment to T168661: Apply schema change to add 3D filetype for STL files.

The new 3D enum value was missing on the new created wiki kbpwiki on s3, as probably the patch was merged after it was created.
The rest of wikis look good across all the shards.

Mon, Sep 4, 8:17 AM · Patch-For-Review, Blocked-on-schema-change, DBA
Marostegui added a comment to T168661: Apply schema change to add 3D filetype for STL files.

Just for the record, I am checking again that the alter has been done across all the host and all the wikis and the only pending one would be commonswiki on s4 master - db1068

Mon, Sep 4, 5:26 AM · Patch-For-Review, Blocked-on-schema-change, DBA
Marostegui updated the task description for T168661: Apply schema change to add 3D filetype for STL files.
Mon, Sep 4, 5:04 AM · Patch-For-Review, Blocked-on-schema-change, DBA
Marostegui added a comment to T168661: Apply schema change to add 3D filetype for STL files.

I have done the alter table on enwiki master (db1052) and it has gone fine. I pre-warmed the big tables (image and filearchive) before and it went thru relatively quickly.
Before running the alter I checked the tables last modifications and indeed they were having low traffic at this time, so while running the alter I didn't see any connection waiting or errors happening so looks like it didn't impact anyone.
Image table took around 1 minute to be altered, and filearchive around 3 minutes.

Mon, Sep 4, 5:04 AM · Patch-For-Review, Blocked-on-schema-change, DBA

Sat, Sep 2

Marostegui moved T174857: Degraded RAID on db1059 from Triage to In progress on the DBA board.

@Cmjohnson please go ahead and replace the disk when you can

Sat, Sep 2, 7:47 PM · DBA, ops-eqiad, Operations

Fri, Sep 1

Marostegui awarded T162593: Run pt-table-checksum on s4 (commonswiki) a Love token.
Fri, Sep 1, 4:41 PM · DBA
Marostegui triaged T174806: Decommission db1045 as Normal priority.
Fri, Sep 1, 4:31 PM · Patch-For-Review, hardware-requests, ops-eqiad, DBA, Operations
Marostegui created T174806: Decommission db1045.
Fri, Sep 1, 4:30 PM · Patch-For-Review, hardware-requests, ops-eqiad, DBA, Operations
Marostegui moved T174801: Drop flaggedrevs tables on wikis where it is not enabled from Triage to Backlog on the DBA board.
Fri, Sep 1, 4:20 PM · DBA
Marostegui moved T174804: Truncate `updatelog` on all wikis from Triage to Backlog on the DBA board.
Fri, Sep 1, 4:13 PM · DBA
Marostegui updated the task description for T172679: Productionize 11 new eqiad database servers.
Fri, Sep 1, 3:39 PM · Patch-For-Review, DBA
Marostegui moved T174802: Drop education program (ep_*) tables on wikis where it is not enabled from Triage to Backlog on the DBA board.
Fri, Sep 1, 3:32 PM · DBA
Marostegui updated the task description for T172679: Productionize 11 new eqiad database servers.
Fri, Sep 1, 3:29 PM · Patch-For-Review, DBA
Marostegui moved T113842: `pr_index`to be replicated to Labs public databases from Next to In progress on the DBA board.
Fri, Sep 1, 1:45 PM · Patch-For-Review, DBA, Data-Services
Marostegui added a comment to T174782: Drop pr_index from wikis where ProofreadPage isn't enabled.

I have renamed the table on enwiki, on db1089 just to make sure nothing breaks

root@db1089[enwiki]> show tables like 'T17%';
+-------------------------+
| Tables_in_enwiki (T17%) |
+-------------------------+
| T174782_pr_index        |
+-------------------------+
1 row in set (0.00 sec)
Fri, Sep 1, 1:44 PM · DBA
Marostegui added a comment to T174782: Drop pr_index from wikis where ProofreadPage isn't enabled.

These empty tables current exist on the following wikis, and should be dropped from there:

Fri, Sep 1, 1:39 PM · DBA
Marostegui assigned T174763: Decommission db1026 to Cmjohnson.

db1026 is now ready to be decommissioned and all the pending steps are DC Ops ones, so I am handing this over to @Cmjohnson

Fri, Sep 1, 1:13 PM · Patch-For-Review, ops-eqiad, Operations, DBA
Marostegui updated the task description for T174763: Decommission db1026.
Fri, Sep 1, 1:10 PM · Patch-For-Review, ops-eqiad, Operations, DBA
Marostegui updated the task description for T174763: Decommission db1026.
Fri, Sep 1, 12:23 PM · Patch-For-Review, ops-eqiad, Operations, DBA
Marostegui updated the task description for T174763: Decommission db1026.
Fri, Sep 1, 12:22 PM · Patch-For-Review, ops-eqiad, Operations, DBA
Marostegui updated the task description for T174763: Decommission db1026.
Fri, Sep 1, 12:07 PM · Patch-For-Review, ops-eqiad, Operations, DBA
Marostegui moved T113842: `pr_index`to be replicated to Labs public databases from Triage to Next on the DBA board.
Fri, Sep 1, 12:04 PM · Patch-For-Review, DBA, Data-Services
Marostegui moved T174782: Drop pr_index from wikis where ProofreadPage isn't enabled from Triage to Next on the DBA board.
Fri, Sep 1, 11:59 AM · DBA
Marostegui added a comment to T113842: `pr_index`to be replicated to Labs public databases.

Wow... It's on enwiki

https://github.com/wikimedia/mediawiki-extensions-ProofreadPage/blob/master/sql/ProofreadIndex.sql

I'd propose dropping it from anywhere that isn't in this list...

'wmgUseProofreadPage' => [
	'default' => false,
	'wikisource' => true,
	'sourceswiki' => true, // FIXME: should be a part of wikisource (see T91534)
	'test2wiki' => true,
	'frrwiki' => true, // T40023
],
Fri, Sep 1, 11:50 AM · Patch-For-Review, DBA, Data-Services
Marostegui added a comment to T168661: Apply schema change to add 3D filetype for STL files.

All s4 hosts have been upgraded to 10.0.32.
Obviously not the master (see below)

Fri, Sep 1, 11:41 AM · Patch-For-Review, Blocked-on-schema-change, DBA
Marostegui triaged T174764: db2044 HW RAID failure as Normal priority.
Fri, Sep 1, 10:19 AM · Operations, ops-codfw, DBA
Marostegui closed T174764: db2044 HW RAID failure as Resolved.

After rebooting the server again, everything looks good again and I see no more HW errors.
I have started mysql and replication and everything is looking ok.

Fri, Sep 1, 10:18 AM · Operations, ops-codfw, DBA
Marostegui added a comment to T113842: `pr_index`to be replicated to Labs public databases.

Yeah - I thought about creating it on enwikisource to unblock this ticket first and then slowly create it on the big wikis and so forth

Fri, Sep 1, 9:41 AM · Patch-For-Review, DBA, Data-Services
Marostegui added a comment to T174764: db2044 HW RAID failure.

I have rebooted the server because it was basically unresponsive and it has came back fine apparently. I will do some more checks before starting MySQL

Fri, Sep 1, 7:10 AM · Operations, ops-codfw, DBA
Marostegui added a comment to T174764: db2044 HW RAID failure.

From the logs:

/system1/log1/record16
  Targets
  Properties
    number=16
    severity=Critical
    date=09/01/2017
    time=06:18
    description=Drive Array Controller Failure (Slot 0)
Fri, Sep 1, 7:02 AM · Operations, ops-codfw, DBA
Marostegui created T174764: db2044 HW RAID failure.
Fri, Sep 1, 6:59 AM · ops-codfw, Operations, DBA