Page MenuHomePhabricator

jcrespo (Jaime Crespo)
Sr Database Administrator

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
May 11 2015, 8:31 AM (310 w, 1 d)
Availability
Available
IRC Nick
jynus
LDAP User
Jcrespo
MediaWiki User
JCrespo (WMF) [ Global Accounts ]

Recent Activity

Today

jcrespo added a comment to T280492: Upgrade all sanitarium masters to 10.4 and Buster.

I finished setting up db2139 with and s3 instance on buster- as soon as I merge the above patch (https://gerrit.wikimedia.org/r/681439) db2139:s3 will be the canonical location for s3 backups on codfw. I will do it now, unless any of you happen to be around still for a review.

Tue, Apr 20, 4:56 PM · Data-Persistence-Backup, Patch-For-Review, DBA
jcrespo added a comment to T280492: Upgrade all sanitarium masters to 10.4 and Buster.

So the current plan is to setup s3 on buster at db2139, move backups to dbprov2003, and then drop the db2098 s3 section. It will take me a bit to backup and transfer all the data. Will update when done, but think at least a few hours.

Tue, Apr 20, 11:04 AM · Data-Persistence-Backup, Patch-For-Review, DBA
jcrespo closed T271821: export_smart_data_dump.service failed on dbprov2001 because of a timeout in the raid facter as Declined.

I am going to close this for inactivity, because I haven't seen it happen again since last reported a few months ago, but we can reopen it if it happens again.

Tue, Apr 20, 10:26 AM · Data-Persistence-Backup
jcrespo moved T264274: Define a methodology to track WMF services backup requirements from In Progress to Blocked on the Data-Persistence-Backup board.
Tue, Apr 20, 10:24 AM · Goal, SRE, Data-Persistence-Backup
jcrespo moved T264275: Track all directly-owned SRE datasets into the new inventory system from In Progress to Blocked on the Data-Persistence-Backup board.
Tue, Apr 20, 10:24 AM · Goal, SRE, Data-Persistence-Backup
jcrespo moved T280492: Upgrade all sanitarium masters to 10.4 and Buster from Triage to In Progress on the Data-Persistence-Backup board.
Tue, Apr 20, 10:24 AM · Data-Persistence-Backup, Patch-For-Review, DBA
jcrespo added a project to T280492: Upgrade all sanitarium masters to 10.4 and Buster: Data-Persistence-Backup.

Adding a tag or I won't be able to find this task later.

Tue, Apr 20, 10:23 AM · Data-Persistence-Backup, Patch-For-Review, DBA
jcrespo added a comment to T280492: Upgrade all sanitarium masters to 10.4 and Buster.

time, @jcrespo, would you have time to reimage db2098 to Buster today or tomorrow?

Tue, Apr 20, 10:22 AM · Data-Persistence-Backup, Patch-For-Review, DBA

Fri, Apr 16

jcrespo edited P15390 sretest1002 backup configuration.
Fri, Apr 16, 1:36 PM
jcrespo created P15390 sretest1002 backup configuration.
Fri, Apr 16, 1:32 PM
jcrespo added a comment to T271573: upgrade conf2* servers to stretch.

Do you think, with the work done, we could drop support of jessie bacula backups (only etcd cluster was pending with jessie)?

Fri, Apr 16, 9:12 AM · Patch-For-Review, SRE, serviceops

Thu, Apr 15

jcrespo updated the task description for T280232: Uncached wiki requests partially unavailable due to excessive request rates from a bot.
Thu, Apr 15, 9:17 AM · SRE, Wikimedia-Incident
jcrespo updated the task description for T280232: Uncached wiki requests partially unavailable due to excessive request rates from a bot.
Thu, Apr 15, 9:12 AM · SRE, Wikimedia-Incident
jcrespo created T280232: Uncached wiki requests partially unavailable due to excessive request rates from a bot.
Thu, Apr 15, 9:10 AM · SRE, Wikimedia-Incident

Wed, Apr 14

jcrespo added a comment to T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.

Thanks to everybody that applied. I counted 10 successful final applications that I think were all related to this idea (or at least contained "databases" or "backups"), including everybody that had expressed interest here! Thank you for reaching to the end of the application process. I think this was a very successful idea thanks to you.

Wed, Apr 14, 4:06 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo awarded T138915: OTRS database is "too large" a Like token.
Wed, Apr 14, 3:41 PM · DBA, OTRS
jcrespo added a comment to T276220: Internal APT repository backup.

Feel free -if you find the time- to make some recovery tests, even on paper- it should be easy and doesn't hurt recovering on e.g. /var/tmp/new-dir and check you would be able to recover everything from that + puppet (which is also backed up). It will take a few minutes of your time and will avoid running into blockers issues in the future. :-)

Wed, Apr 14, 3:40 PM · Data-Persistence-Backup
jcrespo added a comment to T276448: Failover m1 master: db1080 -> db1159 Wed 14th April at 10 AM UTC.

Backup metadata looking good:

Wed, Apr 14, 1:25 PM · DBA
jcrespo added a comment to T274463: Backups for GitLab.

I just read there is an option for "Skipping tar creation", maybe that could be used to generate a consistent export of files that are incrementally backed up more efficiently? I don't know, it requires testing- specially of the restoring process.

Wed, Apr 14, 9:49 AM · Data-Persistence-Backup, Patch-For-Review, User-brennen, GitLab (Initialization)
jcrespo created P15315 m2 recommendationapi grants for deleted/nonexistent user.
Wed, Apr 14, 9:11 AM
jcrespo added a comment to T274463: Backups for GitLab.

What components are currently included in the hourly Gerrit backups? What is the retention policy for build artifacts, build data (logs etc)? Any information would be valuable here. Thank you.

Wed, Apr 14, 8:45 AM · Data-Persistence-Backup, Patch-For-Review, User-brennen, GitLab (Initialization)

Tue, Apr 13

jcrespo added a comment to T274463: Backups for GitLab.

Long term, the amount of data we currently have in Gerrit is our baseline.

Tue, Apr 13, 6:32 PM · Data-Persistence-Backup, Patch-For-Review, User-brennen, GitLab (Initialization)
jcrespo added a project to T274463: Backups for GitLab: Data-Persistence-Backup.
Tue, Apr 13, 4:29 PM · Data-Persistence-Backup, Patch-For-Review, User-brennen, GitLab (Initialization)
jcrespo added a comment to T274463: Backups for GitLab.

I asked a few questions related to backups needs at https://gerrit.wikimedia.org/r/c/operations/puppet/+/677970

Tue, Apr 13, 4:28 PM · Data-Persistence-Backup, Patch-For-Review, User-brennen, GitLab (Initialization)
jcrespo assigned T276220: Internal APT repository backup to LSobanski.

Thanks for the information, maybe I understood wrongly the task to do here. Assigning to @LSobanski.

Tue, Apr 13, 3:46 PM · Data-Persistence-Backup
jcrespo updated subscribers of T276220: Internal APT repository backup.

Hey, @jbond @MoritzMuehlenhoff sorry to ping you, but this is something that you may know how to do properly- as you are involved with operational security (please correct me if wrong, and add the right people instead).

Tue, Apr 13, 11:20 AM · Data-Persistence-Backup
jcrespo changed the status of T200035: DB backup restore skip empty databases from Open to Stalled.

Marking as stalled, as this is blocked on getting a patch from upstream.

Tue, Apr 13, 11:11 AM · Data-Persistence-Backup, Upstream
jcrespo moved T200035: DB backup restore skip empty databases from Triage to Blocked on the Data-Persistence-Backup board.
Tue, Apr 13, 11:10 AM · Data-Persistence-Backup, Upstream
jcrespo moved T200035: DB backup restore skip empty databases from Backlog to Reported Upstream on the Upstream board.
Tue, Apr 13, 11:09 AM · Data-Persistence-Backup, Upstream
jcrespo edited projects for T200035: DB backup restore skip empty databases, added: Data-Persistence-Backup; removed DBA.
Tue, Apr 13, 11:09 AM · Data-Persistence-Backup, Upstream
jcrespo added a comment to T224589: Migrate dbmonitor hosts to Buster.

Probably known (sorry) but the other alert I saw recently was: "CRITICAL: the following (6) node(s) change every puppet run: dbmonitor1001.wikimedia.org,...". Probably related to this?

Tue, Apr 13, 11:05 AM · Patch-For-Review, SRE
jcrespo added a comment to T276448: Failover m1 master: db1080 -> db1159 Wed 14th April at 10 AM UTC.

What about 10UTC? Would that work for backups? I will ping other owners if this works for you

Tue, Apr 13, 8:55 AM · DBA
jcrespo added a comment to T276448: Failover m1 master: db1080 -> db1159 Wed 14th April at 10 AM UTC.

As long as it is not too early in the morning, 14 will be ok. We may want to do it late in the morning so etherpad and other owners are around? So it should be ok as long we we merge the patch I prepared after switchover.

Tue, Apr 13, 8:40 AM · DBA
jcrespo added a comment to T276448: Failover m1 master: db1080 -> db1159 Wed 14th April at 10 AM UTC.

@jcrespo I would like to do this Wednesday 14th April - is this a good day or will it mess up with the backups? I have no problems in scheduling it any other day

Tue, Apr 13, 8:36 AM · DBA
jcrespo added a comment to T279552: Develop a web based monitoring dashboard to improve and monitor existing database backup inventory processes and improve long term maintainability of existing code.

No need to include a copy here, we will be able to see through Google's website after the deadline finishes. The problem was we would be unable to see it beforehand and provide feedback, but we will be able to see the final submission after the deadline.

Tue, Apr 13, 8:21 AM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a comment to T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.

A reminder that there are just a few hours left for the official and final submissions of GSOC proposals (less than 10 hours from the moment of this comment). So make sure your proposals have been submitted to the GSOC website https://summerofcode.withgoogle.com/ and are set to final before deadline. It is ok to do some final adjustments if necesary, but they have to be submitted to the Google's website to be accounted for.

Tue, Apr 13, 8:15 AM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)

Fri, Apr 9

jcrespo added a comment to T279552: Develop a web based monitoring dashboard to improve and monitor existing database backup inventory processes and improve long term maintainability of existing code.
  • You should register a nick on Freenode if selected, but that is not important for the proposal.
Fri, Apr 9, 2:42 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo assigned T279552: Develop a web based monitoring dashboard to improve and monitor existing database backup inventory processes and improve long term maintainability of existing code to h.krishna.
Fri, Apr 9, 1:54 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a subtask for T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects: T279552: Develop a web based monitoring dashboard to improve and monitor existing database backup inventory processes and improve long term maintainability of existing code.
Fri, Apr 9, 1:53 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a parent task for T279552: Develop a web based monitoring dashboard to improve and monitor existing database backup inventory processes and improve long term maintainability of existing code: T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.
Fri, Apr 9, 1:53 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo triaged T279552: Develop a web based monitoring dashboard to improve and monitor existing database backup inventory processes and improve long term maintainability of existing code as Medium priority.
Fri, Apr 9, 1:53 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a comment to T279552: Develop a web based monitoring dashboard to improve and monitor existing database backup inventory processes and improve long term maintainability of existing code.

apologies for the formatting

Fri, Apr 9, 1:52 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a comment to T253959: Check we are preparing (xtrabackup --prepare) with the same package version as the server version of which the backup was taken.

Unless I am not mistaken (we should test it), it would be the "server_version" from the generated file, to the "xtrabackup --version" standard output from the place where "--prepare" runs.

Fri, Apr 9, 1:46 PM · Patch-For-Review, manuel-and-jaime-gsoc2021, good first task
jcrespo added a comment to T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.

I think I have given at least some initial comments to every proposal so far. Thanks for taking the time, and of course, all are Work in Progress until the deadline, so you will have time to improve them if you consider it necessary (it was the whole point of having initial drafts).

Fri, Apr 9, 1:31 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a comment to T253959: Check we are preparing (xtrabackup --prepare) with the same package version as the server version of which the backup was taken.

I am building it on 20.04 Ubuntu focal fossa

Fri, Apr 9, 1:23 PM · Patch-For-Review, manuel-and-jaime-gsoc2021, good first task
jcrespo added a comment to T253959: Check we are preparing (xtrabackup --prepare) with the same package version as the server version of which the backup was taken.

I researched and found out that the xtrabackup_info itself has the server version. I think we can get the xtrabackup version from there

Fri, Apr 9, 1:21 PM · Patch-For-Review, manuel-and-jaime-gsoc2021, good first task
jcrespo triaged T279546: API to inventory all MySQL account metadata as Medium priority.

Small suggestion- given that contacting me by email was "against Wikimedia guidelines" (don't worry, other people did the same mistake and I direct them to Phabricator/Zulip, and it won't be taken into account- but we highly discourage it), -you will probably want to omit such a mistake from your proposal.

Fri, Apr 9, 1:06 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo triaged T279488: Develop Dashboard with Flask For Database backup inventory improvements as Medium priority.
Fri, Apr 9, 12:54 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo triaged T278744: Develop a web dashboard to help inventory zarcillo database as Medium priority.
Fri, Apr 9, 12:52 PM · Google-Summer-of-Code (2021), manuel-and-jaime-gsoc2021
jcrespo added a subtask for T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects: T279442: Develop a web dashboard to help MYSQL account metadata inventory.
Fri, Apr 9, 12:51 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a parent task for T279442: Develop a web dashboard to help MYSQL account metadata inventory: T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.
Fri, Apr 9, 12:51 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo removed a parent task for T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects: T279442: Develop a web dashboard to help MYSQL account metadata inventory.
Fri, Apr 9, 12:51 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo removed a subtask for T279442: Develop a web dashboard to help MYSQL account metadata inventory: T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.
Fri, Apr 9, 12:50 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo triaged T279442: Develop a web dashboard to help MYSQL account metadata inventory as Medium priority.
Fri, Apr 9, 12:50 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo assigned T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects to Marostegui.
Fri, Apr 9, 12:49 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a comment to T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.

I am assigning this task to @Marostegui because we cannot assign it to multiple people at the same time (both mentors) :-(, but that way it won't be confused with a student's proposal.

Fri, Apr 9, 12:49 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a comment to T279488: Develop Dashboard with Flask For Database backup inventory improvements.

One thing that probably is improvable is the deliverables section. While you have the challenge of probably not knowing the scope of the work before a deeper analysis, the deliverables are a bit vague. For example,

Fri, Apr 9, 12:42 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a comment to T279488: Develop Dashboard with Flask For Database backup inventory improvements.

I can devote 40 hours/week for GSoC project

Fri, Apr 9, 12:36 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a parent task for T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects: T279442: Develop a web dashboard to help MYSQL account metadata inventory.
Fri, Apr 9, 12:23 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a subtask for T279442: Develop a web dashboard to help MYSQL account metadata inventory: T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.
Fri, Apr 9, 12:23 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo removed a parent task for T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects: T279442: Develop a web dashboard to help MYSQL account metadata inventory.
Fri, Apr 9, 12:23 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo removed a subtask for T279442: Develop a web dashboard to help MYSQL account metadata inventory: T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.
Fri, Apr 9, 12:23 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a comment to T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.

For everybody- on the draft proposals here on Phabricator, and on the final proposal you should write for the Google Summer of Code website, please remember to highlight any past contribution to open source projects, be it by working on the suggested microtasks, helping others, or e.g. sending pull requests to other projects. While we will be evaluating your entire proposal, demonstrating initiative and/or past experience (even if very small) communicationg with an open source project is the best way you can make sure we know you will be able to complete the proposed work. Deep technical knowledge is something that is not that important for GSoC.

Fri, Apr 9, 12:18 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a comment to T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.

@Ashitabattu There are 4 microtask suggestions at the end of the body text of this ticket. None have been closed yet, so you can check if you can send a patch to any of them- I will try to review all, although be aware some have been started being worked by other students.

Fri, Apr 9, 12:06 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a comment to T279442: Develop a web dashboard to help MYSQL account metadata inventory.

The proposal seems solid, however I am worried about your courses commencing in August. Will you have enough time outside of class? You mention "having the most important functionality" written by then, however, when I see the timeline, I think you want to break down the backend first and then the frontend, and this last part can take a lot of time.

Fri, Apr 9, 12:02 PM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)

Tue, Apr 6

jcrespo edited projects for T279411: Determine why service responses are slow and what we can do about it, added: Data-Persistence (Consultation); removed DBA.
Tue, Apr 6, 8:01 AM · Growth-Team (Current Sprint), serviceops, Data-Persistence (Consultation), Add-Link
jcrespo added a comment to T279411: Determine why service responses are slow and what we can do about it.

Run:

EXPLAIN SELECT value FROM lr_cswiki_anchors WHERE lookup = {foo} LIMIT 1
Tue, Apr 6, 8:00 AM · Growth-Team (Current Sprint), serviceops, Data-Persistence (Consultation), Add-Link
jcrespo merged T278969: Problems generating thumbnails into T266155: Frequent "Error: 429, Too Many Requests" errors on pages with many (>50) thumbnails.
Tue, Apr 6, 7:34 AM · StructuredDataOnCommons, Patch-For-Review, SRE, MediaWiki-File-management, Thumbor, Commons
jcrespo merged task T278969: Problems generating thumbnails into T266155: Frequent "Error: 429, Too Many Requests" errors on pages with many (>50) thumbnails.
Tue, Apr 6, 7:34 AM · SRE-swift-storage, SRE
jcrespo added a comment to T278969: Problems generating thumbnails.

If what we have is 429s (rate limiting errors), I would merge this there. Feel free to reopen if you had a different experience.

Tue, Apr 6, 7:33 AM · SRE-swift-storage, SRE
jcrespo added a comment to T274234: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002.

Are the backup long TCP sessions or many small ones?

Tue, Apr 6, 7:26 AM · netops, SRE, Data-Persistence-Backup
jcrespo added a comment to T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.

Both mentors took most of last week away from the computer.

Tue, Apr 6, 7:13 AM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a comment to T278513: Create new tag #manuel-and-jaime-gsoc.

Thank you, Andre, and sorry for problems caused, they were unintended.

Tue, Apr 6, 6:59 AM · Project-Admins
jcrespo added a project to T278744: Develop a web dashboard to help inventory zarcillo database: manuel-and-jaime-gsoc2021.
Tue, Apr 6, 6:59 AM · Google-Summer-of-Code (2021), manuel-and-jaime-gsoc2021
jcrespo added a comment to T253959: Check we are preparing (xtrabackup --prepare) with the same package version as the server version of which the backup was taken.

@Palak199 Nobody else expressed interest on it so far, so I guess it is ok.

Tue, Apr 6, 6:58 AM · Patch-For-Review, manuel-and-jaime-gsoc2021, good first task
jcrespo added a project to T253959: Check we are preparing (xtrabackup --prepare) with the same package version as the server version of which the backup was taken: manuel-and-jaime-gsoc2021.
Tue, Apr 6, 6:55 AM · Patch-For-Review, manuel-and-jaime-gsoc2021, good first task
jcrespo added a project to T268258: transfer.py argument parsing exception: manuel-and-jaime-gsoc2021.
Tue, Apr 6, 6:54 AM · Google-Summer-of-Code (2021), manuel-and-jaime-gsoc2021, Patch-For-Review, good first task
jcrespo added a project to T277160: Make recover-dump show the time taken: manuel-and-jaime-gsoc2021.
Tue, Apr 6, 6:54 AM · Google-Summer-of-Code (2021), manuel-and-jaime-gsoc2021, Patch-For-Review, good first task
jcrespo added a project to T277162: recover-mariadb should use logging (logger) to indicate actions taken: manuel-and-jaime-gsoc2021.
Tue, Apr 6, 6:53 AM · Google-Summer-of-Code (2021), manuel-and-jaime-gsoc2021, Patch-For-Review, good first task
jcrespo added a project to T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects: manuel-and-jaime-gsoc2021.

can you please explain how to link them in backups table

Tue, Apr 6, 6:53 AM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a member for manuel-and-jaime-gsoc2021: Marostegui.
Tue, Apr 6, 6:42 AM
jcrespo added a member for manuel-and-jaime-gsoc2021: jcrespo.
Tue, Apr 6, 6:41 AM

Wed, Mar 31

jcrespo added a comment to T278969: Problems generating thumbnails.

@Wilfredor is this still happening to you? I saw some contention at the time of your report, due to high upload rate, but not at the moment.
It could be datacenter-dependent, though.

Wed, Mar 31, 4:46 PM · SRE-swift-storage, SRE
jcrespo added a comment to T278969: Problems generating thumbnails.

User Askeuhd is currently uploading files at a speed of ~>130 files per minute. I would guess it could be related to that.

Wed, Mar 31, 2:16 PM · SRE-swift-storage, SRE
jcrespo added a comment to T278719: load-datasets.py: Lock wait timeout exceeded; try restarting transaction.

That is doable, I think. I am thinking maybe some of the disconnections without commits may try to rollback, rather than commit the changes, and that leads to locks. It needs more debugging.

Wed, Mar 31, 12:16 PM · Patch-For-Review, Data-Persistence (Consultation), Growth-Team (Current Sprint), GrowthExperiments, Add-Link
jcrespo added a comment to T278719: load-datasets.py: Lock wait timeout exceeded; try restarting transaction.

FWIW all the tables are basically cache. Data is read from the stats server, processed, and then inserted as key-value data into the tables read by the link recommendation service. On the next import, all data is discarded and replaced by new data extracted from the stats server. All reads are simple key lookups. So consistency or complexity of schema changes is not much of a concern.

Wed, Mar 31, 11:30 AM · Patch-For-Review, Data-Persistence (Consultation), Growth-Team (Current Sprint), GrowthExperiments, Add-Link
jcrespo added a comment to T278719: load-datasets.py: Lock wait timeout exceeded; try restarting transaction.

To clarify, there is just one point of entry, and it varies on which datacenter is active, yes.

Wed, Mar 31, 10:55 AM · Patch-For-Review, Data-Persistence (Consultation), Growth-Team (Current Sprint), GrowthExperiments, Add-Link
jcrespo added a comment to T278744: Develop a web dashboard to help inventory zarcillo database.

Hi @jcrespo, I have left
Add any other relevant information such as UI mockups, references to related projects, a link to your proof of concept code, etc
this as WIP.
Please suggest if I should prepare some mockups for the web app? or add something else to this question.

Wed, Mar 31, 10:34 AM · Google-Summer-of-Code (2021), manuel-and-jaime-gsoc2021
jcrespo added a comment to T278744: Develop a web dashboard to help inventory zarcillo database.

I will be giving a few ideas on several comments. As a reminder, this is just feedback, your the person that will be finally deciding on your own proposal.

Wed, Mar 31, 10:31 AM · Google-Summer-of-Code (2021), manuel-and-jaime-gsoc2021
jcrespo renamed T278929: Extend ldap access to Superset for amy-wmde from Extend access Superset to Extend ldap access to Superset for amy-wmde.
Wed, Mar 31, 10:09 AM · LDAP-Access-Requests, SRE
jcrespo assigned T278929: Extend ldap access to Superset for amy-wmde to jijiki.

Assigning to the right person (feel free to manage that in the best way for you). :-)

Wed, Mar 31, 10:06 AM · LDAP-Access-Requests, SRE
jcrespo added a comment to T271725: Allow WMDE intern Amrutha to access Superset.

Sorry about that :-(. I used the outdated topic.

Wed, Mar 31, 10:02 AM · LDAP-Access-Requests, SRE, WMF-NDA-Requests
jcrespo closed T271725: Allow WMDE intern Amrutha to access Superset as Resolved.

Hey, @amy_rc Let's create a separate ticket for that, so you don't need to go over the trouble of a regular access request- just the extension, which should be super easy in comparison. Just having your manager confirm the extension should be enough.

Wed, Mar 31, 9:51 AM · LDAP-Access-Requests, SRE, WMF-NDA-Requests
jcrespo added a comment to T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.

1)Is the database host("db1115.eqiad.wmnet") mentioned in script directly accessible from the public network or should I be using some VPN to connect to it?

Wed, Mar 31, 9:34 AM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a comment to T278719: load-datasets.py: Lock wait timeout exceeded; try restarting transaction.

Let's focus on the immediate issues first, later on better architecture.

Wed, Mar 31, 9:18 AM · Patch-For-Review, Data-Persistence (Consultation), Growth-Team (Current Sprint), GrowthExperiments, Add-Link

Tue, Mar 30

jcrespo added a comment to T278350: "Lock wait timeout exceeded" moving a page back with ~18800 watchers on en.wp.

This is the full trace- easy to locate thanks to the user reporting the request Id https://logstash.wikimedia.org/goto/d1e2ffabc0987ddb3395cf2ec4cdbda4:

Tue, Mar 30, 2:47 PM · User-Urbanecm, Editing-team, MediaWiki-Watchlist, Growth-Team, WMF-JobQueue, Wikimedia-production-error, User-notice, Wikimedia-database-error, MediaWiki-Page-rename
jcrespo edited projects for T278719: load-datasets.py: Lock wait timeout exceeded; try restarting transaction, added: Data-Persistence (Consultation); removed DBA.

"Lock wait timeout exceeded; try restarting transaction" means 2 writes are happening at the same time exclusively, blocking the same rows .

Tue, Mar 30, 1:06 PM · Patch-For-Review, Data-Persistence (Consultation), Growth-Team (Current Sprint), GrowthExperiments, Add-Link
jcrespo added a comment to T278744: Develop a web dashboard to help inventory zarcillo database.

Thank you, I will find some time to read it and try to provide useful feedback. As a tip- don't spend much time on "styling" in phabricator- as long as it is readable it is ok. This is just supposed to be a draft until you submit the definitive one to Google.

Tue, Mar 30, 10:50 AM · Google-Summer-of-Code (2021), manuel-and-jaime-gsoc2021
jcrespo added a subtask for T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects: T278744: Develop a web dashboard to help inventory zarcillo database.
Tue, Mar 30, 10:48 AM · manuel-and-jaime-gsoc2021, Google-Summer-of-Code (2021)
jcrespo added a parent task for T278744: Develop a web dashboard to help inventory zarcillo database: T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.
Tue, Mar 30, 10:48 AM · Google-Summer-of-Code (2021), manuel-and-jaime-gsoc2021