jcrespo (Jaime Crespo)
Sr Database Administrator

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
May 11 2015, 8:31 AM (153 w, 5 d)
Availability
Available
IRC Nick
jynus
LDAP User
Jcrespo
MediaWiki User
JCrespo (WMF)

Recent Activity

Today

jcrespo renamed T192710: [Feedback] Personal observations about the page previews feature usability from [Feedback] Personal observations about the page previews feature usuability to [Feedback] Personal observations about the page previews feature usability.
Sat, Apr 21, 11:14 AM · Page-Previews
jcrespo created T192710: [Feedback] Personal observations about the page previews feature usability.
Sat, Apr 21, 11:07 AM · Page-Previews

Yesterday

jcrespo added a comment to T189542: Update updatequerypages::cronjob and refreshlinks::cronjob now that silver no longer has a database.

Documented the "new" section at https://wikitech.wikimedia.org/wiki/Add_a_wiki#MediaWiki_configuration

Fri, Apr 20, 2:26 PM · Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org
jcrespo added a comment to T189542: Update updatequerypages::cronjob and refreshlinks::cronjob now that silver no longer has a database.

As a note, the list and the section name doesn't have to match, so there is nothing technically incorrect, but it is confusing (using the same name to refer to the same thing is something I recommend).

Fri, Apr 20, 2:20 PM · Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org
jcrespo updated the task description for T189542: Update updatequerypages::cronjob and refreshlinks::cronjob now that silver no longer has a database.
Fri, Apr 20, 2:17 PM · Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org
jcrespo added a comment to T189542: Update updatequerypages::cronjob and refreshlinks::cronjob now that silver no longer has a database.

Sorry, Andrew, insisting because I may be misunderstood. You have done step2. Step 1, for clarity, would be to edit db-eqiad.php and db-codfw.php structure (which I wouldn't recommend doing on a Friday and add us as reviewers so we are aware of it, as we edit that file heavily).

Fri, Apr 20, 2:17 PM · Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org
jcrespo added a comment to T189542: Update updatequerypages::cronjob and refreshlinks::cronjob now that silver no longer has a database.

section defined or named

Fri, Apr 20, 7:17 AM · Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org

Thu, Apr 19

jcrespo added a comment to T192551: atop on stretch overloading a host.

If I have to guess, I would say it is the combination of the stretch version + high load (if it is network, cpu or io, I cannot say)- I think enwiki API are hosts with logs of ongoing connections/traffic. We should ask Traffic if they have any large-traffic server with stretch.

Thu, Apr 19, 2:03 PM · monitoring, Operations

Wed, Apr 18

jcrespo added a comment to T191193: Move masters away from codfw C6.

For example, as a procedure, could activity be checked on the port before being disabled to check the host is down/moved away?

Wed, Apr 18, 7:22 AM · Patch-For-Review, ops-codfw, Operations, DBA
jcrespo closed T191193: Move masters away from codfw C6 as Resolved.

Okey, I feel we should check what went wrong (was it the clarity of the communication, was it a one-time mistake that will unlikely happen again, was it the extended downtime on icinga that made the issue not beeing immediately apparent)?

Wed, Apr 18, 7:02 AM · Patch-For-Review, ops-codfw, Operations, DBA
jcrespo added a comment to T191193: Move masters away from codfw C6.

Were the right interfaces disabled after the revert?

Wed, Apr 18, 6:58 AM · Patch-For-Review, ops-codfw, Operations, DBA
jcrespo added a comment to T191767: Important critical Etherpad release – 1.6.4.

Dzahn the reason this is should not be set to high (and almost could be public) is that there is already a workaround in place/the other vulnerability does not affect us. So according to multiple people (Moritz, Alex, this should not be a priority).

Wed, Apr 18, 6:57 AM · Wikimedia-Etherpad, Operations, Security

Tue, Apr 17

jcrespo reopened T191193: Move masters away from codfw C6 as "Open".
Tue, Apr 17, 9:39 PM · Patch-For-Review, ops-codfw, Operations, DBA
jcrespo renamed T192339: labtestweb2001 should either be fully productionized or removed completely from production configuration from labtestweb2001 should either productionized of removed completely from production configuration to labtestweb2001 should either be fully productionized or removed completely from production configuration.
Tue, Apr 17, 9:31 PM · Patch-For-Review, cloud-services-team
jcrespo added a comment to P7003 (An Untitled Masterwork).

root@labsdb1011.eqiad.wmnet[enwiki_p]> SELECT count(*) FROM recentchanges WHERE rc_timestamp > '201804151148';
+----------+

count(*)

+----------+

438676

+----------+
1 row in set (10.98 sec)

Tue, Apr 17, 4:14 PM
jcrespo created P7003 (An Untitled Masterwork).
Tue, Apr 17, 4:12 PM
jcrespo updated the task description for T156462: Framework to transfer files over the LAN.
Tue, Apr 17, 12:57 PM · DBA
jcrespo updated subscribers of T156462: Framework to transfer files over the LAN.

@Rduran Do you think you can take care of this? There is a prototype at https://gerrit.wikimedia.org/r/280947 but all the other Remote Calling methods should be dropped and use instead cumin ( https://wikitech.wikimedia.org/wiki/Cumin ). Sadly, Cumin is python2 only for now.

Tue, Apr 17, 12:56 PM · DBA
jcrespo moved T192358: Setup database logical backups on eqiad from Triage to In progress on the DBA board.

I am going to setup s1 on dbstore1001.

Tue, Apr 17, 12:32 PM · Patch-For-Review, DBA
jcrespo triaged T192358: Setup database logical backups on eqiad as Normal priority.
Tue, Apr 17, 12:32 PM · Patch-For-Review, DBA
jcrespo added a subtask for T138562: Improve regular production database backups handling: T156462: Framework to transfer files over the LAN.
Tue, Apr 17, 12:29 PM · Wikimedia-Incident, DBA
jcrespo added a parent task for T156462: Framework to transfer files over the LAN: T138562: Improve regular production database backups handling.
Tue, Apr 17, 12:29 PM · DBA
jcrespo added a comment to T187521: Optimize recentchanges and wbc_entity_usage table across wikis.

Done as T192349

Tue, Apr 17, 10:54 AM · Wikidata, DBA
jcrespo created T192349: deadlocks on INSERT IGNORE INTO wbc_entity_usage.
Tue, Apr 17, 10:53 AM · MW-1.32-release-notes (WMF-deploy-2018-04-24 (1.32.0-wmf.1)), Patch-For-Review, Wikidata-Ministry-Of-Magic, User-Ladsgroup, Wikidata-Ministry-Of-Magic-Tech-Debt, Wikimedia-log-errors, Wikidata
jcrespo added a comment to P7001 mysql.py.
while read host port; do ./mysql.py -h $host:$port enwiki -e "SHOW CREATE TABLE revision\G"; done < s1.hosts
Tue, Apr 17, 10:46 AM
jcrespo added a comment to P7001 mysql.py.
root@neodymium:~$ ./mysql.py -h db1052 enwiki
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 2447007520
Server version: 10.0.28-MariaDB MariaDB Server
Tue, Apr 17, 10:15 AM
jcrespo created P7001 mysql.py.
Tue, Apr 17, 10:13 AM
jcrespo updated subscribers of T189542: Update updatequerypages::cronjob and refreshlinks::cronjob now that silver no longer has a database.

Things to do:

Tue, Apr 17, 8:43 AM · Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org
jcrespo triaged T192340: Cron <www-data@terbium> /usr/local/bin/mwscriptwikiset extensions/FlaggedRevs/maintenance/updateStats.php flaggedrevs.dblist > /dev/null creates cron spam every 2 hours as Normal priority.
Tue, Apr 17, 8:33 AM · MediaWiki-extensions-FlaggedRevs
jcrespo updated the task description for T192339: labtestweb2001 should either be fully productionized or removed completely from production configuration.
Tue, Apr 17, 8:26 AM · Patch-For-Review, cloud-services-team
jcrespo updated the task description for T192339: labtestweb2001 should either be fully productionized or removed completely from production configuration.
Tue, Apr 17, 8:25 AM · Patch-For-Review, cloud-services-team
jcrespo raised the priority of T192339: labtestweb2001 should either be fully productionized or removed completely from production configuration from Normal to High.
Tue, Apr 17, 8:18 AM · Patch-For-Review, cloud-services-team
jcrespo triaged T192339: labtestweb2001 should either be fully productionized or removed completely from production configuration as Normal priority.
Tue, Apr 17, 8:15 AM · Patch-For-Review, cloud-services-team

Mon, Apr 16

jcrespo added a comment to T187521: Optimize recentchanges and wbc_entity_usage table across wikis.

commonswiki errors due to deadlocks on INSERT IGNORE INTO wbc_entity_usage seem to be common (not too worrying, but on of the most comon database errors), could the code be optimized to avoid those? I am guessing that the same row is written many times (once per change on the same item), and maybe that could be simplified somehow. INSERT IGNORE is a bit of a bad trick here, and we may be writing multiple times the same data without need. Given the changes are done by the job queue and arrive in any order, maybe transaction serialization can be relaxed?

Mon, Apr 16, 1:40 PM · Wikidata, DBA

Fri, Apr 13

jcrespo created P6989 (An Untitled Masterwork).
Fri, Apr 13, 3:32 PM
jcrespo edited projects for T191996: db1114 connection issues, added: netops; removed Patch-For-Review.

Adding the tag to reflect work done at network layer.

Fri, Apr 13, 2:28 PM · ops-eqiad, Patch-For-Review, netops, Operations, DBA
Gerrit Code Review <gerrit@wikimedia.org> committed rOSMDd35404ec26d5: Merge "Make WMFMariaDB.py flake8 compliant" (authored by jcrespo).
Merge "Make WMFMariaDB.py flake8 compliant"
Fri, Apr 13, 8:05 AM

Thu, Apr 12

jcrespo updated subscribers of T188913: "Obama" page on Beta Cluster often responds with 503.

Giuseppe mentioned some test stretch patches on beta, it may be unrelated, but so he is aware of ongoing issues.

Thu, Apr 12, 5:18 PM · Operations, Beta-Cluster-Infrastructure
jcrespo added a project to T191977: remote ipmi doesn't work for es2013: ops-codfw.
Thu, Apr 12, 3:20 PM · ops-codfw, DC-Ops, Patch-For-Review, DBA, Operations
jcrespo added a comment to T191391: Apply schema changes to an isolated database and examine the results.

I found T86530, which may be outdated, but may help with giving more options.

Thu, Apr 12, 3:18 PM · Wikidata-Ministry-Of-Magic, DBA, MediaWiki-extensions-WikibaseRepository, User-Ladsgroup, Wikidata
jcrespo added a comment to T191977: remote ipmi doesn't work for es2013.

The reset a previous ticket suggested was T191977#4123270 (racadm reset)

Thu, Apr 12, 3:07 PM · ops-codfw, DC-Ops, Patch-For-Review, DBA, Operations
jcrespo raised the priority of T191977: remote ipmi doesn't work for es2013 from Low to Normal.
Thu, Apr 12, 3:06 PM · ops-codfw, DC-Ops, Patch-For-Review, DBA, Operations
jcrespo reassigned T191977: remote ipmi doesn't work for es2013 from jcrespo to Papaul.

@Papaul you are now free to handle the server- it is up, but with all the service down and depooled. I would try the reset I proposed earlier first, and if that doesn't work, checking bios/admin config, maybe?

Thu, Apr 12, 3:06 PM · ops-codfw, DC-Ops, Patch-For-Review, DBA, Operations
jcrespo added a comment to T191977: remote ipmi doesn't work for es2013.

Now that I have a way to test it, we can proceed, depooling:

Thu, Apr 12, 2:55 PM · ops-codfw, DC-Ops, Patch-For-Review, DBA, Operations
jcrespo claimed T191977: remote ipmi doesn't work for es2013.

@Papaul @Marostegui Please don't do anything until it is clear what is the issue.

Thu, Apr 12, 2:46 PM · ops-codfw, DC-Ops, Patch-For-Review, DBA, Operations
jcrespo added a comment to T191977: remote ipmi doesn't work for es2013.

Not now, I will have to depool it. Give me 5 minutes.

Thu, Apr 12, 2:34 PM · ops-codfw, DC-Ops, Patch-For-Review, DBA, Operations
jcrespo added a comment to T190607: cp3048 hardware issues.

It probably crashed today at 2018-04-12 13:31:20, hardware logs should be checked.

Thu, Apr 12, 1:39 PM · Traffic, Operations, ops-esams
jcrespo added a comment to T191996: db1114 connection issues.

That would explain the disconnections- too many connections leads to heartbeat check fails, which leads to disconnections.

Thu, Apr 12, 9:55 AM · ops-eqiad, Patch-For-Review, netops, Operations, DBA
jcrespo added a comment to T191996: db1114 connection issues.

MMm, so api queries timing out get killed? That could be. But aren't those connection errors? Needs more research.

Thu, Apr 12, 9:46 AM · ops-eqiad, Patch-For-Review, netops, Operations, DBA
jcrespo added a comment to T191996: db1114 connection issues.

The errors would be consistent with the 10-interval in which the connections happen (bursts of high activity). But not as large as thinking it is a hardware error.

Thu, Apr 12, 9:06 AM · ops-eqiad, Patch-For-Review, netops, Operations, DBA
jcrespo awarded T190780: Schema changes to site_stats a Pterodactyl token.
Thu, Apr 12, 7:35 AM · Patch-For-Review, Blocked-on-schema-change, DBA
jcrespo added a comment to T190780: Schema changes to site_stats.

No issue or locking or strangeness of any kind on any server?

Thu, Apr 12, 7:35 AM · Patch-For-Review, Blocked-on-schema-change, DBA
jcrespo added a comment to T190425: GlobalPreferences deploy caused a significant increase in reads on s3.

We are fans of caching! :-)

Thu, Apr 12, 7:32 AM · MW-1.32-release-notes (WMF-deploy-2018-04-24 (1.32.0-wmf.1)), MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), Patch-For-Review, Community-Tech-Sprint, MediaWiki-extensions-GlobalPreferences
jcrespo added a comment to T181650: Change views for the new columns of the refactored comment storage.

My suggestion in the future, for new code/new views (not for regular things like dropping views or adding new wikis) would be to test extensively on a depooled host, to avoid bugs and security issues (maybe you did that already, I didn't follow all details).

Thu, Apr 12, 7:28 AM · cloud-services-team (Kanban), Data-Services
jcrespo added a comment to T191892: Reduce locking contention on deletion of pages.

@Anomie, you are the best!

Thu, Apr 12, 7:20 AM · MW-1.31-release-notes (WMF-deploy-2018-04-17 (1.31.0-wmf.30)), Wikimedia-Incident, Patch-For-Review, DBA, Operations, MediaWiki-Page-deletion
jcrespo awarded T191892: Reduce locking contention on deletion of pages a Love token.
Thu, Apr 12, 7:20 AM · MW-1.31-release-notes (WMF-deploy-2018-04-17 (1.31.0-wmf.30)), Wikimedia-Incident, Patch-For-Review, DBA, Operations, MediaWiki-Page-deletion

Wed, Apr 11

jcrespo added a comment to T191996: db1114 connection issues.

We know it is mediawiki, I discovered through application logs on logstash.

Wed, Apr 11, 4:16 PM · ops-eqiad, Patch-For-Review, netops, Operations, DBA
jcrespo updated the task description for T191996: db1114 connection issues.
Wed, Apr 11, 4:07 PM · ops-eqiad, Patch-For-Review, netops, Operations, DBA
jcrespo created T191996: db1114 connection issues.
Wed, Apr 11, 4:05 PM · ops-eqiad, Patch-For-Review, netops, Operations, DBA
jcrespo added a comment to T191977: remote ipmi doesn't work for es2013.

T150160 suggests racadm reset may fix it.

Wed, Apr 11, 2:30 PM · ops-codfw, DC-Ops, Patch-For-Review, DBA, Operations
jcrespo created T191977: remote ipmi doesn't work for es2013.
Wed, Apr 11, 2:26 PM · ops-codfw, DC-Ops, Patch-For-Review, DBA, Operations
jcrespo updated the task description for T191972: Scap sync-file failing for deploy1001.eqiad.wmnet.
Wed, Apr 11, 1:05 PM · Patch-For-Review, Deployments, Release-Engineering-Team, Operations
jcrespo renamed T191972: Scap sync-file failing for deploy1001.eqiad.wmnet from Scap sync-file failing for 9 hosts to Scap sync-file failing for deploy1001.eqiad.wmnet.
Wed, Apr 11, 12:47 PM · Patch-For-Review, Deployments, Release-Engineering-Team, Operations
jcrespo added a comment to T190780: Schema changes to site_stats.

@EddieGP Not queries will be lost, but if they pileup blocking the wiki's activity, it will be a worse issue (actual outage or edit outage).

Wed, Apr 11, 11:15 AM · Patch-For-Review, Blocked-on-schema-change, DBA
jcrespo added a comment to T188434: Perform code steward process for #geodata extension.

@EBjune The largest issue right now, from the reporter point of view, that would threaten the stability of the site is some database-related work. #DBAs want to take care of that, but may need some code maintenance. Is that something that your team could help with? It should be a 1-time thing, as far as the database bugs are concerned.

Wed, Apr 11, 7:18 AM · Discovery-Search, GeoData, Code-Stewardship-Reviews

Tue, Apr 10

jcrespo closed T191875: Deletion not working on English Wikipedia as Resolved.

I am going to close this ticket as the initial report, "Deletion not working", was resolved as soon as the maintenance finished. We hope that with T191892 that would mitigate the issues in the future, but only after that change is deployed we could test that is true. We will monitor and reopen if we gather more information/the mitigation doesn't work. Feel free to use the Incident talk page for more questions and comments rather than this ticket.

Tue, Apr 10, 3:38 PM · DBA, Wikimedia-Incident, Operations, MediaWiki-Page-deletion
jcrespo added a comment to T191892: Reduce locking contention on deletion of pages.

I believe this have been happening for some time now, but this incident only made it more real (happening not only for large deletes, but for small ones, too): https://logstash.wikimedia.org/goto/9facbbd99d63704f215285470b16d6f5

Tue, Apr 10, 1:30 PM · MW-1.31-release-notes (WMF-deploy-2018-04-17 (1.31.0-wmf.30)), Wikimedia-Incident, Patch-For-Review, DBA, Operations, MediaWiki-Page-deletion
jcrespo added a comment to T191875: Deletion not working on English Wikipedia.

I agree with everything you said, my comment was a quick sketch of what I wanted, and what you proposed was what I really wanted, creating T191892 to handle that there.

Tue, Apr 10, 1:28 PM · DBA, Wikimedia-Incident, Operations, MediaWiki-Page-deletion
jcrespo triaged T191892: Reduce locking contention on deletion of pages as Normal priority.
Tue, Apr 10, 1:26 PM · MW-1.31-release-notes (WMF-deploy-2018-04-17 (1.31.0-wmf.30)), Wikimedia-Incident, Patch-For-Review, DBA, Operations, MediaWiki-Page-deletion
jcrespo updated subscribers of T191875: Deletion not working on English Wikipedia.

CC @Anomie this is not directly related- maintenance was the direct cause, but I believe the new comment model may be creating worse locking patterns on deletion, with queries like:

SELECT  rev_id,rev_page,rev_text_id,rev_timestamp,rev_minor_edit,rev_deleted,rev_len,rev_parent_id,rev_sha1,COALESCE( comment_rev_comment.comment_text, rev_comment ) AS `rev_comment_text`,comment_rev_comment.comment_data AS `rev_comment_data`,comment_rev_comment.comment_id AS `rev_comment_cid`,rev_user,rev_user_text,NULL AS `rev_actor`,rev_content_format,rev_content_model  FROM `revision` LEFT JOIN `revision_comment_temp` `temp_rev_comment` ON ((temp_rev_comment.revcomment_rev = rev_id)) LEFT JOIN `comment` `comment_rev_comment` ON ((comment_rev_comment.comment_id = temp_rev_comment.revcomment_comment_id))   WHERE rev_page = 'X'   FOR UPDATE

Could the SELECT ... FOR UPDATE be restricted to the revision table and select the comment on a second query, without locking? My thesis is the extra locking could affect deletions as all will try to block exclusively the same "deletion reason comment", which creates higher contention. So something like:

Tue, Apr 10, 10:48 AM · DBA, Wikimedia-Incident, Operations, MediaWiki-Page-deletion
jcrespo moved T191875: Deletion not working on English Wikipedia from Triage to In progress on the DBA board.
Tue, Apr 10, 10:31 AM · DBA, Wikimedia-Incident, Operations, MediaWiki-Page-deletion
jcrespo added a project to T191875: Deletion not working on English Wikipedia: DBA.
Tue, Apr 10, 10:31 AM · DBA, Wikimedia-Incident, Operations, MediaWiki-Page-deletion
jcrespo added a comment to T191875: Deletion not working on English Wikipedia.

This were the queries ongoing at that time:
{P6973}

Tue, Apr 10, 10:00 AM · DBA, Wikimedia-Incident, Operations, MediaWiki-Page-deletion
jcrespo added a comment to T191875: Deletion not working on English Wikipedia.

What I saw was INSERTs into alter being blocked due to metadata locking, but that would not make sense except at the start of the command, or the command would fail in 30 seconds. Maybe it requires a second metadata lock under certain conditions?

Tue, Apr 10, 9:39 AM · DBA, Wikimedia-Incident, Operations, MediaWiki-Page-deletion
jcrespo lowered the priority of T191875: Deletion not working on English Wikipedia from Unbreak Now! to Normal.

Normal as the incident should be solved, we now have to research what actually happened.

Tue, Apr 10, 9:27 AM · DBA, Wikimedia-Incident, Operations, MediaWiki-Page-deletion
jcrespo added a subtask for T187962: Rack/cable/configure asw2-c-eqiad switch stack: T191792: Rack and setup db1116 - db1123.
Tue, Apr 10, 7:43 AM · Operations, ops-eqiad, netops
jcrespo added a parent task for T191792: Rack and setup db1116 - db1123: T187962: Rack/cable/configure asw2-c-eqiad switch stack.
Tue, Apr 10, 7:43 AM · Patch-For-Review, ops-eqiad, Operations, DBA
jcrespo added a comment to T187962: Rack/cable/configure asw2-c-eqiad switch stack.

I would do the second.

Tue, Apr 10, 7:43 AM · Operations, ops-eqiad, netops
jcrespo added a comment to T187962: Rack/cable/configure asw2-c-eqiad switch stack.

I would honestly move x1 replica (or the master directy), probably in a logical way, somewhere else- we don't want to serve the whole service from the same row, and x1 is like s4 and s8- it is not really that easy to put in read only because cross-wiki dependencies. x1 hosts will have to be moved anyway, but we can serve it for some time with a single host.

Tue, Apr 10, 7:37 AM · Operations, ops-eqiad, netops
jcrespo added a comment to T190780: Schema changes to site_stats.

Yes, we could do this on the masters even with a table reconstruction- but we should check if a table reconstruction is needed only for a definition change.

Tue, Apr 10, 7:22 AM · Patch-For-Review, Blocked-on-schema-change, DBA
jcrespo updated the task description for T188434: Perform code steward process for #geodata extension.
Tue, Apr 10, 7:18 AM · Discovery-Search, GeoData, Code-Stewardship-Reviews
jcrespo updated the task description for T188434: Perform code steward process for #geodata extension.
Tue, Apr 10, 6:39 AM · Discovery-Search, GeoData, Code-Stewardship-Reviews

Mon, Apr 9

jcrespo moved T190704: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy from Next to In progress on the DBA board.
Mon, Apr 9, 4:25 PM · Patch-For-Review, Operations, Goal, DBA
Liuxinyu970226 awarded T184280: Linter multiple database issues a Baby Tequila token.
Mon, Apr 9, 1:35 PM · User-notice, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Performance-Team (Radar), Security, Patch-For-Review, Wikimedia-log-errors, MediaWiki-extensions-Linter
jcrespo added projects to T191767: Important critical Etherpad release – 1.6.4: Operations, Wikimedia-Etherpad.
Mon, Apr 9, 8:45 AM · Wikimedia-Etherpad, Operations, Security
jcrespo created T191767: Important critical Etherpad release – 1.6.4.
Mon, Apr 9, 8:44 AM · Wikimedia-Etherpad, Operations, Security
jcrespo added a comment to T136687: Database error when filtering page log.

@TTO I think explain output is likely to change when data changes, and this is probably caused by a user with logs of change tags/logs, which may not show up on your local installation. It could also change based on the DBMS version used. Normally here I copy the results from production, so they are quite accurate- if not, we would have not received an error report in the first place.

Mon, Apr 9, 8:02 AM · TestMe, MediaWiki-Change-tagging, MediaWiki-Database
jcrespo awarded T184280: Linter multiple database issues a Like token.
Mon, Apr 9, 7:57 AM · User-notice, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Performance-Team (Radar), Security, Patch-For-Review, Wikimedia-log-errors, MediaWiki-extensions-Linter
jcrespo added a comment to T184280: Linter multiple database issues.

I am not seeing linter issues lately, I will open a new one if they come back.

Mon, Apr 9, 7:57 AM · User-notice, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Performance-Team (Radar), Security, Patch-For-Review, Wikimedia-log-errors, MediaWiki-extensions-Linter
jcrespo added a comment to T188434: Perform code steward process for #geodata extension.

I've changed the title to better reflect that I don't want to remove this, in fact, what I want is there is better support for it, which right now is affecting me.

Mon, Apr 9, 7:53 AM · Discovery-Search, GeoData, Code-Stewardship-Reviews
jcrespo renamed T188434: Perform code steward process for #geodata extension from Sunset #geodata extension to Perform code steward process for #geodata extension.
Mon, Apr 9, 7:52 AM · Discovery-Search, GeoData, Code-Stewardship-Reviews
jcrespo added a comment to T107610: Setup separate logical External Store for Flow in production.

That last suggestion looks like a blocker to me, at least to check it before doing anything.

Mon, Apr 9, 7:46 AM · DBA, Operations, WorkType-Maintenance, Collaboration-Team-Triage, StructuredDiscussions

Fri, Apr 6

jcrespo removed a project from T191626: Remove term_entity_type from wb_terms: DBA.

Wikidata team will test it on the test host and create a task for production deployment with all requested changes/full strategy. Until then, there is nothing for us to do here. If you need help, re-add us with a specific request.

Fri, Apr 6, 12:46 PM · MediaWiki-extensions-WikibaseRepository, Wikidata
jcrespo committed rOSMD15c0429b90fd: Make WMFMariaDB.py and recover_section.py flake8 compliant (authored by Rduran).
Make WMFMariaDB.py and recover_section.py flake8 compliant
Fri, Apr 6, 11:13 AM
jcrespo moved T123557: Database query error (internal_api_error_DBQueryError) while getting list=allrevisions from Backlog to Done on the DBA board.

Yes resolved from on our side.

Fri, Apr 6, 11:05 AM · DBA, MediaWiki-API
jcrespo committed rOSMD7a611d14fa9a: dump_section.py: Rename dump_sections to singular, update to HEAD (authored by jcrespo).
dump_section.py: Rename dump_sections to singular, update to HEAD
Fri, Apr 6, 11:04 AM
jcrespo added a comment to T189542: Update updatequerypages::cronjob and refreshlinks::cronjob now that silver no longer has a database.

On another side, cronjobs are still referred as silver on production, shouldn't that change too? Can you comment why this was closed as invalid?

Fri, Apr 6, 10:08 AM · Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org
jcrespo updated subscribers of P6953 RFC: Stop using puppet for mariadb dynamic configuration.
Fri, Apr 6, 9:09 AM · Operations-Software-Development, DBA, Puppet
jcrespo created P6953 RFC: Stop using puppet for mariadb dynamic configuration.
Fri, Apr 6, 9:09 AM · Operations-Software-Development, DBA, Puppet
jcrespo moved T107610: Setup separate logical External Store for Flow in production from Backlog to Next on the DBA board.
Fri, Apr 6, 5:07 AM · DBA, Operations, WorkType-Maintenance, Collaboration-Team-Triage, StructuredDiscussions