Springle (Sean Pringle)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Oct 7 2014, 2:34 AM (210 w, 4 d)
Availability
Available
LDAP User
Springle
MediaWiki User
Unknown

Recent Activity

Aug 8 2018

jcrespo awarded T59186: Drop blob_tracking and blob_orphans everywhere a Love token.
Aug 8 2018, 6:58 AM · Patch-For-Review, DBA, MediaWiki-Database

Jun 11 2018

Gerrit Code Review <gerrit@wikimedia.org> committed rEBOPc12aac1b20c0: Update patch set 1 (authored by Springle).
Update patch set 1
Jun 11 2018, 5:28 PM

Jun 9 2018

Gerrit Code Review <gerrit@wikimedia.org> committed rECOG7e8351a0a666: Update patch set 1 (authored by Springle).
Update patch set 1
Jun 9 2018, 12:03 AM
Gerrit Code Review <gerrit@wikimedia.org> committed rECOG42f7d1a1dea4: Update patch set 1 (authored by Springle).
Update patch set 1
Jun 9 2018, 12:02 AM

Jun 8 2018

Gerrit Code Review <gerrit@wikimedia.org> committed rADESc0390bf5caca: Update patch set 1 (authored by Springle).
Update patch set 1
Jun 8 2018, 5:39 PM

May 21 2018

Springle added a comment to T179059: Consider skipping or modifying recombine step for page content dumps for wikidata.

Messing with bzip2, pbzip2, lbzip2 on snapshot001, looking at timing only (not archive size or non-default config).

May 21 2018, 3:00 AM · Patch-For-Review, Dumps-Generation

Apr 13 2018

Dzahn awarded T191478: Requesting access to shell (snapshot, dumpsdata) for springle a Like token.
Apr 13 2018, 11:57 PM · Patch-For-Review, Operations, SRE-Access-Requests

Apr 10 2018

Springle added a comment to T191478: Requesting access to shell (snapshot, dumpsdata) for springle.
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCZwhGWhhv+9QdjhhShbLdSZSV349oFxPH73CfvI0jRsQFXsQIlPQaSeKcFqw+kjhUoxvfgCw3YWoExHTT6jxHUxrOswI6ZVPeicHNBQ4kiRRY4uKE0xpqbdnkbLRSNWyru8zG1aB/uxpkhsQhwnUZ9fpGtDkXzX1In8NZ7X9jMQB6yrHFxqK/549WELGnpscL79lX7uKM2Ri/+v61th7kuDyn6VjsIMSLdt46dKoW9WgQ2UgkjEh67HOZd1FYt4V+OaQcNr2JtHj7nSI6YsXx9TQnBrQVqWQXk63AFNxw4uD7xFVByc4FIqefIYjHqHANRWpRmaNOcj6LaBTqXZUBSmtYRiLkXUhqhr1Tf1NiE75UjGKhknucpywXTYI02HaTdEcdxfN4C9guI+ojxwUKrIMEk9Wz3qcYzyN0QZmCL/6EcRxjEUzYDpEt0tMBRsRqE5Qp0TLPuDsK5trY1rtdzy/HckqmSik9N1p2WQ941SWs2EEiFji1jiCM4N8gwy1r6mf9xo5LWRVY/LtNYbCf/2EfW3mjreP9MaOGI+vedcS8I4sd6O3VP8WPpXZtoBU1+EKLhEHvfp/E9qYYr6iWIltCFySi67fWlv83cUNezJ6uMrDR++g8ANkFJKEWSHJzdVyrtf2fiwNNyIPrkEawHAcKHsZsVGdzkP9Xr8eBb7Q== sean@laptop
Apr 10 2018, 12:50 PM · Patch-For-Review, Operations, SRE-Access-Requests

Apr 5 2018

Springle added a comment to T191478: Requesting access to shell (snapshot, dumpsdata) for springle.

Updated to a non-wmf email in phabricator profile settings.

Apr 5 2018, 2:16 AM · Patch-For-Review, Operations, SRE-Access-Requests

Apr 4 2018

Springle created T191478: Requesting access to shell (snapshot, dumpsdata) for springle.
Apr 4 2018, 11:21 PM · Patch-For-Review, Operations, SRE-Access-Requests

Nov 2 2016

jcrespo awarded T97641: WebVideoTranscodeJob is keeping database connections open for several minutes on s4 master a Cookie token.
Nov 2 2016, 5:23 PM · TimedMediaHandler-Transcode, MediaWiki-Database

Aug 28 2015

Springle updated subscribers of T62539: [Task] Convert wb_terms term_row_id from INT to BIGINT on wikidatawiki.
Aug 28 2015, 4:32 AM · Blocked-on-schema-change, DBA, Wikidata, Schema-change, Wikidata.org
Springle updated subscribers of T79922: Set up backup strategy for es clusters.
Aug 28 2015, 4:30 AM · Operations, DBA
Springle updated subscribers of T86482: puppet stopped mysqld using orphan pid file from puppet agent.
Aug 28 2015, 4:28 AM · Operations, DBA
Springle updated subscribers of T88770: sleeper database connection surges during outage.
Aug 28 2015, 4:28 AM · Wikimedia-Incident, Operations, DBA, Incident-20150205-SiteOutage
Springle updated subscribers of T89986: Update GeoData schema.
Aug 28 2015, 4:27 AM · DBA, GeoData, Schema-change
Springle closed T96383: install/setup/deploy db2043-db2070 as Resolved.
Aug 28 2015, 4:27 AM · Operations, DBA
Springle updated subscribers of T96499: dbtree loads third party resources (from jquery.com and google.com).
Aug 28 2015, 4:26 AM · Privacy, Traffic, HTTPS, Operations, Patch-For-Review, DBA, WMF-Legal
Springle updated subscribers of T92841: Database replicas: replicate user.user_touched.
Aug 28 2015, 4:25 AM · DBA, Cloud-Services
Springle closed T105879: db1022 duplicate key errors as Resolved.
Aug 28 2015, 4:25 AM · Operations, Patch-For-Review, DBA
Springle updated subscribers of T70942: Database upgrade MariaDB 10: Engine / Option mismatch on table `user_properties`.
Aug 28 2015, 4:23 AM · DBA, Cloud-Services, Cloud-VPS
Springle closed T71110: Database upgrade MariaDB 10: 600 seconds timeout as Resolved.
Aug 28 2015, 4:23 AM · Cloud-Services, Cloud-VPS
Springle updated subscribers of T71127: Discrepancies with logging table on different wikis.
Aug 28 2015, 4:22 AM · Data-Services, DBA
Springle updated subscribers of T71182: Database upgrade MariaDB 10: Metadata access in INFORMATION_SCHEMA causes complete blocks.
Aug 28 2015, 4:21 AM · DBA, Cloud-Services, Cloud-VPS
Springle updated subscribers of T85266: Look into Maria 10 parallel-replication.
Aug 28 2015, 4:21 AM · DBA, Availability
Springle updated subscribers of T71463: Create a table in labs with replication lag data.
Aug 28 2015, 4:20 AM · Patch-For-Review, DBA, Analytics-General-or-Unknown

Aug 24 2015

Springle added a comment to T108850: Set up auto-purging after 90 days {tick}.

Regarding implementing this on replicas: nothing special is needed. Change on the master will propagate (but rate-limit everything, to avoid replag).

Aug 24 2015, 10:01 AM · User-Elukey, Analytics, Patch-For-Review, DBA
Springle added a comment to T108850: Set up auto-purging after 90 days {tick}.

A white list is fine. No real difference from DBA perspective, so +1.

Aug 24 2015, 9:59 AM · User-Elukey, Analytics, Patch-For-Review, DBA
Springle added a comment to T108856: Set up bucketization of editCount fields {tick}.

@mforns, yes, feel free.

Aug 24 2015, 9:57 AM · Analytics, DBA

Aug 13 2015

Jcornelius awarded T69223: Schema change for page content language a Like token.
Aug 13 2015, 3:33 PM · Community-Wishlist-Survey-2016, TechCom-RFC (TechCom-Approved), Blocked-on-schema-change, Wikimedia-Site-requests, DBA, Wikisource

Aug 7 2015

Springle added a comment to T108255: Enable MariaDB/MySQL's Strict Mode.

It's correct in that Mediawiki still supports MySQL 5.0.2 [1] with a default SQL_MODE setting [2]. Previous discussions about enforcing strict mode havn't gone far, mainly due to questions about MW extensions.

Aug 7 2015, 2:36 AM · Epic, WorkType-NewFunctionality, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure, DBA, MediaWiki-Database

Jul 30 2015

Springle committed rOPUP56af628418ec: reduce tendril memory footprint due to OOM, and switch to /srv (authored by Springle).
reduce tendril memory footprint due to OOM, and switch to /srv
Jul 30 2015, 10:16 PM
Springle added a comment to T107282: Reduce memory commitment on database hosts with many objects, specially s3, dbstore/research and labs.

Just to clarify, I don't think we've seen actual OOM killer on s[1-7], right? The only front-line production concern is swapping on s3 db1035?

Jul 30 2015, 2:15 AM · Patch-For-Review, Operations, DBA
Springle added a comment to T107265: MIMEsearchPage::reallyDoQuery failing on the logs due to taking too long to query.

Also, all bot driven, eg: tide543.microsoft.com

Jul 30 2015, 1:34 AM · MW-1.28-release (WMF-deploy-2016-10-11_(1.28.0-wmf.22)), MW-1.28-release-notes, MW-1.27-release (WMF-deploy-2015-12-15_(1.27.0-wmf.9)), MW-1.27-release-notes, Operations, WMF-deploy-2015-08-04_(1.26wmf17), WMF-deploy-2015-07-28_(1.26wmf16), MW-1.26-release, Developer-notice, User-notice, Patch-For-Review, Commons, MediaWiki-Special-pages
Springle added a comment to T107265: MIMEsearchPage::reallyDoQuery failing on the logs due to taking too long to query.

Continuing on db1042 right now and queries regularly hitting 5min limit. The LIMIT 10405000,501 is most of the problem here; shouldn't be possible to generate such a search.

Jul 30 2015, 1:33 AM · MW-1.28-release (WMF-deploy-2016-10-11_(1.28.0-wmf.22)), MW-1.28-release-notes, MW-1.27-release (WMF-deploy-2015-12-15_(1.27.0-wmf.9)), MW-1.27-release-notes, Operations, WMF-deploy-2015-08-04_(1.26wmf17), WMF-deploy-2015-07-28_(1.26wmf16), MW-1.26-release, Developer-notice, User-notice, Patch-For-Review, Commons, MediaWiki-Special-pages

Jul 28 2015

Springle added a comment to T104476: Provision a labsdb useraccount that can be used to run replica-addusers.pl.

At Yuvi's request on IRC I added 'labsdbadmin'@'10.64.37.7' with the same permissions/password as @jcrespo added for 'labsdbadmin'@'10.64.37.6', since the catastrophic failure of the latter.

Jul 28 2015, 2:21 AM · DBA, Cloud-Services

Jul 27 2015

Springle added a comment to T105843: new external storage cluster(s).

+1 to the provisioning.

Jul 27 2015, 1:14 AM · Operations, hardware-requests, DBA

Jul 24 2015

Springle added a comment to T106647: mariadb multi-source replication glitch with site_identifiers.

@jcrespo, no, I did not out-of-band change or use skip counter. I found the machine exactly as you described on IRC, and only did research by dumping logs to get the query examples shown in ticket description.

Jul 24 2015, 2:04 AM · Operations, DBA

Jul 23 2015

Springle committed rOMWC3f4a188e1e86: repool db1070 (authored by Springle).
repool db1070
Jul 23 2015, 6:41 AM
Springle created T106647: mariadb multi-source replication glitch with site_identifiers.
Jul 23 2015, 6:06 AM · Operations, DBA
Springle committed rOMWC0bde4f747fb2: depool db1070 (authored by Springle).
depool db1070
Jul 23 2015, 1:59 AM

Jul 22 2015

Springle committed rOMWC58f77c7dd2a2: repool db1071 (authored by Springle).
repool db1071
Jul 22 2015, 4:24 AM
Springle committed rOMWC1f43379113a2: depool db1071 (authored by Springle).
depool db1071
Jul 22 2015, 2:52 AM

Jul 20 2015

Springle updated the task description for T106312: m1-master switch from db1001 to db1016.
Jul 20 2015, 2:49 AM · Patch-For-Review, Operations, DBA
Springle created T106312: m1-master switch from db1001 to db1016.
Jul 20 2015, 2:49 AM · Patch-For-Review, Operations, DBA
Springle added a comment to T105135: Implement mariadb 10.0 masters.

All s1-7 slaves are now 10.0.

Jul 20 2015, 2:38 AM · Patch-For-Review, Operations, DBA
Springle added a project to T105135: Implement mariadb 10.0 masters: acl*sre-team.
Jul 20 2015, 2:30 AM · Patch-For-Review, Operations, DBA
Springle added a comment to T105879: db1022 duplicate key errors.

@jcrespo, ouch, so maybe something is broken in our physical backup process (was it xtrabackup?) that required the SQL_SLAVE_SKIP_COUNTER after reinstall on 2015-06-29? Scary...

Jul 20 2015, 2:23 AM · Operations, Patch-For-Review, DBA

Jul 17 2015

Springle committed rOMWC5b05a618a642: repool db1030 (authored by Springle).
repool db1030
Jul 17 2015, 3:31 AM

Jul 15 2015

Springle committed rOMWCefeceb5749fd: depool db1022 T105879 (authored by Springle).
depool db1022 T105879
Jul 15 2015, 1:11 PM
Springle added a comment to T105879: db1022 duplicate key errors.

For now, I will depool db1022.

Jul 15 2015, 1:06 PM · Operations, Patch-For-Review, DBA
Springle created T105879: db1022 duplicate key errors.
Jul 15 2015, 1:06 PM · Operations, Patch-For-Review, DBA
Springle committed rOPUP13a912f919a8: move db1030 to correct node definition (authored by Springle).
move db1030 to correct node definition
Jul 15 2015, 4:22 AM
Springle committed rOPUPb7d9b8d8cbd9: upgrade db1030 trusty (authored by Springle).
upgrade db1030 trusty
Jul 15 2015, 3:57 AM
Springle added a comment to T105713: missing database on replica server.

So, I need to get some more information from Jaime about what occurred on the weekend (he is unwell and on leave since then), but looking over the logs I see:

Jul 15 2015, 12:33 AM · Toolforge, Cloud-Services
Springle created T105843: new external storage cluster(s).
Jul 15 2015, 12:19 AM · Operations, hardware-requests, DBA

Jul 14 2015

Springle committed rMREL03f18d257866: abuse_filter_log table only uses tinyint not int for namespace ID (authored by Se4598).
abuse_filter_log table only uses tinyint not int for namespace ID
Jul 14 2015, 6:11 PM
Springle committed rOMWCd06eda3d11d4: repool db1037; depool db1030 (authored by Springle).
repool db1037; depool db1030
Jul 14 2015, 1:20 AM

Jul 13 2015

Springle added a comment to T103110: db1050 raid degraded.

Also, yes, time to plan another batch.

Jul 13 2015, 12:43 AM · Operations, ops-eqiad
Springle added a comment to T103110: db1050 raid degraded.

Sounds good.

Jul 13 2015, 12:42 AM · Operations, ops-eqiad

Jul 10 2015

Springle created T105440: include primary key field as secondary index suffix.
Jul 10 2015, 5:01 AM · DBA
Springle added a reverting change for rOMWC8cf12daa7188: repool db1037 as s6 logpager; depool db1030: rOMWC954e7cb2dfb4: Revert "repool db1037 as s6 logpager; depool db1030"..
Jul 10 2015, 4:35 AM
Springle committed rOMWC954e7cb2dfb4: Revert "repool db1037 as s6 logpager; depool db1030". (authored by Springle).
Revert "repool db1037 as s6 logpager; depool db1030".
Jul 10 2015, 4:35 AM
Springle committed rOMWC8cf12daa7188: repool db1037 as s6 logpager; depool db1030 (authored by Springle).
repool db1037 as s6 logpager; depool db1030
Jul 10 2015, 4:27 AM

Jul 9 2015

Springle committed rOMWC286b289af000: depool db1037 (authored by Springle).
depool db1037
Jul 9 2015, 1:36 AM

Jul 8 2015

Springle created T105135: Implement mariadb 10.0 masters.
Jul 8 2015, 10:49 AM · Patch-For-Review, Operations, DBA
Springle committed rOMWCbbba7493d1a8: repool db1041 (authored by Springle).
repool db1041
Jul 8 2015, 5:45 AM

Jul 7 2015

Springle committed rOPUP9cb94333b04a: upgrade db1041 to trusty + mariadb 10 (authored by Springle).
upgrade db1041 to trusty + mariadb 10
Jul 7 2015, 1:35 AM

Jul 6 2015

Springle committed rOMWCfaaa1feee513: repool db1034; depool db1041 (authored by Springle).
repool db1034; depool db1041
Jul 6 2015, 4:53 AM
Springle lowered the priority of T59176: ApiQueryExtLinksUsage::run query has crazy limit from High to Normal.
Jul 6 2015, 4:51 AM · MW-1.32-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), MW-1.29-release-notes, Patch-For-Review, Schema-change, DBA, MediaWiki-API, Performance, MediaWiki-Database
Springle added a comment to T104573: codfw frontends cannot connect to mysql at db2029.

A bunch of "unauthenticated user" in processlist still makes me suspect the thread pool, since that symptom has been seen on prod slaves with thread_pool_size=16 (but not the immediate all-connections-fail, which is indeed odd).

Jul 6 2015, 4:50 AM · Operations, DBA
Springle added a comment to T104699: Firewall configurations for database hosts.

Wasn't sure if you wanted to change that process :)

Jul 6 2015, 4:45 AM · DBA, Operations, Patch-For-Review

Jul 4 2015

Springle added a comment to T104699: Firewall configurations for database hosts.

Doing this to most production DBs seems straight forward. Pain points, due just to complexity, will be on M[1-4].

Jul 4 2015, 1:25 AM · DBA, Operations, Patch-For-Review
Springle created T104748: db1046 innodb signal 6 abort and restart.
Jul 4 2015, 1:11 AM · Analytics-EventLogging, DBA

Jul 3 2015

Springle updated subscribers of T104670: recentchanges deadlocks for dewiki (db1058).
Jul 3 2015, 1:17 AM · WMF-deploy-2015-07-14_(1.26wmf14), MW-1.26-release, Patch-For-Review, WMF-JobQueue, MediaWiki-JobQueue, DBA
Springle created T104670: recentchanges deadlocks for dewiki (db1058).
Jul 3 2015, 1:15 AM · WMF-deploy-2015-07-14_(1.26wmf14), MW-1.26-release, Patch-For-Review, WMF-JobQueue, MediaWiki-JobQueue, DBA
Springle added a comment to T104573: codfw frontends cannot connect to mysql at db2029.

(4) == EINTR on connect. Presumably the max_connections you observed, which in turn possibly something to do with:

Jul 3 2015, 12:32 AM · Operations, DBA

Jul 2 2015

Springle added a comment to T104471: Tables created on the s7 master have not been replicated to dbstore2001 and dbstore2002. Replication issue?.

Tried a restart of dbstore2002, but s7 replication behavior was unchanged: Yes/Yes for replication threads, master exec position advancing, yet no changes appearing.

Jul 2 2015, 12:13 AM · Patch-For-Review, DBA

Jul 1 2015

Springle committed rOPUP4a09514865c5: upgrade db1034 (authored by Springle).
upgrade db1034
Jul 1 2015, 2:43 AM

Jun 30 2015

Springle committed rOMWC5861bda01586: depool db1034 (authored by Springle).
depool db1034
Jun 30 2015, 5:16 AM

Jun 24 2015

Springle committed rOMWC40b4a90b5855: repool db1045 (authored by Springle).
repool db1045
Jun 24 2015, 5:10 AM

Jun 19 2015

Springle created T103110: db1050 raid degraded.
Jun 19 2015, 3:35 PM · Operations, ops-eqiad

Jun 18 2015

Springle closed T95927: centralauth database on dbstore1002 is out of date, replication stuck? as Resolved.

Oops, sorry, this was fixed a while back.

Jun 18 2015, 5:30 AM · Operations, SUL-Finalization
Springle closed T102871: Create flow_workflow_update_timestamp index as Resolved.

Schema change done on x1-master flowdb.

Jun 18 2015, 5:26 AM · WMF-deploy-2015-06-23_(1.26wmf11), Patch-For-Review, Collaboration-Team-Triage, StructuredDiscussions, Schema-change
Springle closed T102871: Create flow_workflow_update_timestamp index, a subtask of T51188: [DO NOT USE] Schema changes for Wikimedia wikis (tracking) [superseded by #Blocked-on-schema-change], as Resolved.
Jun 18 2015, 5:26 AM · DBA, Tracking, Schema-change
Springle claimed T102871: Create flow_workflow_update_timestamp index.
Jun 18 2015, 5:25 AM · WMF-deploy-2015-06-23_(1.26wmf11), Patch-For-Review, Collaboration-Team-Triage, StructuredDiscussions, Schema-change

Jun 17 2015

Springle triaged T102740: db1023 raid degraded as High priority.
Jun 17 2015, 5:43 AM · Operations, ops-eqiad
Springle created T102740: db1023 raid degraded.
Jun 17 2015, 5:36 AM · Operations, ops-eqiad
Springle committed rOMWC56d39699baac: depool db1045 (authored by Springle).
depool db1045
Jun 17 2015, 2:42 AM

Jun 16 2015

Springle committed rOMWCa431de183db8: repool db1073 (authored by Springle).
repool db1073
Jun 16 2015, 11:18 AM
Springle committed rOMWCa634098d3ef5: depool db1073 (authored by Springle).
depool db1073
Jun 16 2015, 3:05 AM
Springle committed rOMWCec7f09127d1b: repool db1057 (authored by Springle).
repool db1057
Jun 16 2015, 12:47 AM

Jun 15 2015

Springle created T102532: revision page_user_timestamp index problematic on large wikis.
Jun 15 2015, 6:26 PM · DBA, Performance, MediaWiki-Database
Springle committed rOMWC95ff57e1b9dc: depool db1057, take #2 (authored by Springle).
depool db1057, take #2
Jun 15 2015, 3:31 AM

Jun 11 2015

Springle added a comment to T101937: setup logging for #wikimedia-databases.

We'll use something other than !log, to maintain -operations supremacy. This is more about quickly posting notes on database hosts -- and tendril is just an obvious place, for us -- since db maintenance tasks tend to be slow, have multiple stages, and likely to be handed over to the next shift.

Jun 11 2015, 3:05 AM · Patch-For-Review, DBA
Springle committed rOMWC8514259383d5: repool db1057 (authored by Springle).
repool db1057
Jun 11 2015, 2:54 AM

Jun 10 2015

Springle added a comment to T101937: setup logging for #wikimedia-databases.

So, to be clear, this isn't intended to:

Jun 10 2015, 8:03 AM · Patch-For-Review, DBA
Springle created T101937: setup logging for #wikimedia-databases.
Jun 10 2015, 7:05 AM · Patch-For-Review, DBA
Springle committed rOPUP48ff9a45b043: m4 replication rule for analytics-store (authored by Springle).
m4 replication rule for analytics-store
Jun 10 2015, 2:35 AM
Springle committed rOMWCcda98f800a02: depool db1057 (authored by Springle).
depool db1057
Jun 10 2015, 1:33 AM