Page MenuHomePhabricator
Feed Advanced Search

Mon, Apr 22

Liz added a comment to T363077: High replication lag for enwiki (db1154 s1 replication crashed).

Well, my "complaints" are actually a request for information on when this problem will be fixed. Where else can I ask? I was directed to come here.

Mon, Apr 22, 3:49 AM · DBA, Data-Services
Liz added a comment to T363077: High replication lag for enwiki (db1154 s1 replication crashed).

Now a 7 hour system lag.

Mon, Apr 22, 12:59 AM · DBA, Data-Services
Liz added a comment to T352010: Gradually drop old pagelinks columns.

Oh, no, this task isn't done yet? System lag again? I hope this one is shorter than the last one.

It's not related to this ticket per se. you should check T363077: High replication lag for enwiki (db1154 s1 replication crashed)

Mon, Apr 22, 12:56 AM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data

Sun, Apr 21

Liz added a comment to T352010: Gradually drop old pagelinks columns.

Oh, no, this task isn't done yet? System lag again? I hope this one is shorter than the last one.

Sun, Apr 21, 7:21 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data

Thu, Apr 18

Liz added a comment to T362732: enwiki_p database replica has stopped updating.

Yeah! Life returns to normal.

Thu, Apr 18, 2:05 AM · Data-Services
Liz added a comment to T352010: Gradually drop old pagelinks columns.

Looks like everything is back to normal. Thanks for all of the work you do for the project.

Thu, Apr 18, 2:03 AM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data

Wed, Apr 17

Liz added a comment to T352010: Gradually drop old pagelinks columns.

Looks like the lag is decreasing now.

Wed, Apr 17, 8:25 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Liz added a comment to T352010: Gradually drop old pagelinks columns.

Do you have any idea when this task will be done, the system will catch up to the 10 hour time lag and everything goes back to normal?

It'll take a while:

grafik.png (306×1 px, 59 KB)

My rough estimate is around 6-8 more hours

Wed, Apr 17, 4:32 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Liz added a comment to T362732: enwiki_p database replica has stopped updating.

I updated the description. Any tools that rely on database replicas, including all toolforge tools that rely on data not available through the API, are affected by this.

Wed, Apr 17, 4:26 PM · Data-Services
Liz added a comment to T352010: Gradually drop old pagelinks columns.

Do you have any idea when this task will be done, the system will catch up to the 10 hour time lag and everything goes back to normal?

Wed, Apr 17, 3:27 AM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Liz added a comment to T362732: enwiki_p database replica has stopped updating.

I can't run any of my Quarry queries and all of the bots I use for editing purposes aren't updating as well. When will this system update be complete and the system lag catch up? Tonight? Tomorrow morning? Next week?

Wed, Apr 17, 1:21 AM · Data-Services

Jun 4 2023

Liz added a comment to T338006: [jobs-api,jobs-cli,infra] Transient cronjob scheduling failures on Toolforge k8s.

And both reports were skipped again tonight for the 4th day in a row. We work with a 7 day lag between when the G13 soon report is issued and when action is required and we're now down to a three day supply of reports until we are really in trouble.

Jun 4 2023, 12:14 AM · Toolforge, Kubernetes

Jun 1 2023

Liz added a comment to T337446: Rebuild sanitarium hosts.

I'm not sure what's causing it (regarding s1), but I'm finding some bots are not returning up-to-date reports. With s1 down for 5 days, there should be a backlog of lengthy reports but I'm seeing short reports or none at all. Did every new edit since May 25th get restored and integrated? Sorry that I don't know the correct terminology.

Jun 1 2023, 12:55 AM · User-notice-archive, TaxonBot, cloud-services-team, Data-Engineering, Data-Services, DBA

May 30 2023

Liz added a comment to T337446: Rebuild sanitarium hosts.

Looks like it's just s1 to be restored (or whatever the correct term is).

May 30 2023, 5:34 PM · User-notice-archive, TaxonBot, cloud-services-team, Data-Engineering, Data-Services, DBA

May 26 2023

Liz added a comment to T337446: Rebuild sanitarium hosts.

Any status updates on this issue? Replag is at 19 hours now.

May 26 2023, 1:18 AM · User-notice-archive, TaxonBot, cloud-services-team, Data-Engineering, Data-Services, DBA

May 22 2023

Liz added a comment to T169452: Replace Quarry with an installation of Superset.

My only concern is that I'm not a programmer, just a regular editor, and I run several queries evert day that were written for me by more tech-minded editors. I hope if we move from Quarry to Superset, it is at least as user-friendly and easy to use as Quarry is (and hopefully, even more so!).

May 22 2023, 10:57 PM · cloud-services-team (FY2023/2024-Q3-Q4), superset.wmcloud.org, Quarry

Aug 23 2022

Liz added a comment to T307314: Lag in updating Special Pages?.

I have been told, probably repeatedly, that Special Pages are generated automatically every 3 days but the ones I regularly use https://en.wikipedia.org/wiki/Special:UnusedCategories and https://en.wikipedia.org/wiki/Special:WantedCategories, were supposed to update some time on August 22nd and didn't.

Aug 23 2022, 7:38 PM · DBA, serviceops, WMF-General-or-Unknown

Jun 13 2022

Liz added a comment to T310431: Unable to view all Wikimedia projects.

This happened again in the past 15 minutes and lasted about 4 or 5 minutes.

Jun 13 2022, 4:16 AM · Wikimedia-Incident, SRE, Traffic
Liz added a comment to T310325: s1 (enwiki) wiki replicas replication has stopped.

Well, I noticed that some bots, like AnomieBOT III, is back to issuing reports, while others, like SDZeroBot, are still affected. I guess it takes a while for everything to return to normal.

Jun 13 2022, 12:55 AM · cloud-services-team (Kanban), DBA, Data-Services

Jun 12 2022

Liz added a comment to T310325: s1 (enwiki) wiki replicas replication has stopped.

I don't understand the difference between what is "production" and what is not (and what difference that makes to the developers) but, again, I'm with Jonesey95. All information that is communicated is valuable to us volunteer editors, including what you have just shared here today. Most of us have our daily editing routines and handling problems that show up on bot or database reports is part of our daily work. When they go down, we understandably have questions.

Jun 12 2022, 12:36 AM · cloud-services-team (Kanban), DBA, Data-Services

Jun 11 2022

Liz added a comment to T310325: s1 (enwiki) wiki replicas replication has stopped.

Monday is not "soon", at least not to me on this Friday night. But I appreciate any specific estimate.

Jun 11 2022, 5:54 AM · cloud-services-team (Kanban), DBA, Data-Services

Jun 10 2022

Liz added a comment to T310325: s1 (enwiki) wiki replicas replication has stopped.

And now there is a 34 hour lag! How can a system catch up with a lag longer than a day?

Jun 10 2022, 8:20 PM · cloud-services-team (Kanban), DBA, Data-Services
Liz added a comment to T310325: s1 (enwiki) wiki replicas replication has stopped.

And now it's 18 hours behind.

Jun 10 2022, 4:08 AM · cloud-services-team (Kanban), DBA, Data-Services

Jun 1 2022

Liz added a comment to T307314: Lag in updating Special Pages?.

Just a reminder that today is June 1st and the Special Pages, at least the ones we utilize on En.Wiki, typically get generated on the 1st of the month...and then the 4th, 7th, 10th, 13th, 16th, 19th, 22nd, 25th, 28th and sometimes the 31st. If they could get back to being generated on their old, regular schedule, that would be great!

Jun 1 2022, 7:22 PM · DBA, serviceops, WMF-General-or-Unknown

May 31 2022

Liz added a comment to T307314: Lag in updating Special Pages?.

This happened again today. The special pages involving categories eventually updated at around 00:30 UTC on May 29th but so far, no updates today on May 31st and they usually also update on the 1st of every month so the reports should have been issued both today and also tomorrow....so we'll see.

May 31 2022, 11:00 PM · DBA, serviceops, WMF-General-or-Unknown
Liz added a comment to T309570: quarry is unable to access enwiki_p.page table.

Looks like everything is fixed now. Thank you for getting right on this bug report!

May 31 2022, 4:13 AM · Quarry, Wikimedia-production-error
Liz added a comment to T309569: HTTP 500 - View 'enwiki_p.page' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them.

I'm also experience problems with Quarry, 2 bots on en.wiki which haven't issued their regularly scheduled reports and receiving "overflow" messages when I try to look at page histories earlier today. Are these connected?

May 31 2022, 12:47 AM · XTools
Liz added a comment to T309570: quarry is unable to access enwiki_p.page table.

I'm also experience problems with 2 bots on en.wiki which haven't issued their regularly scheduled reports and receiving "overflow" messages when I try to look at page histories earlier today. Are these connected?

May 31 2022, 12:43 AM · Quarry, Wikimedia-production-error
Liz added a comment to T157670: Periodically run refreshLinks.php on production sites..

I'm also experience problems with two bots on en.wiki which haven't issued their regularly scheduled reports and receiving "overflow" messages when I try to look at page histories earlier today. Are these connected?

May 31 2022, 12:43 AM · Platform Engineering Roadmap Decision Making, MediaWiki-Parser, MediaWiki-Page-editing, Parsing-Team--ARCHIVED

May 28 2022

Liz reopened T307314: Lag in updating Special Pages? as "Open".

Looks like this special pages posting lag is happening again, on En.Wiki, special pages involving categories typically, for years, updated around 13:00-15:00 UTC time every 3 days but on May 25th, they updated at 19:00 UTC and today, they haven't updated at all. I'd prefer that they "catch up" and post sometime today rather than skipping a date and just updating again on June 1st as they did at the beginning of May when this lag was first reported. A late update is better than omitting an update!

May 28 2022, 8:11 PM · DBA, serviceops, WMF-General-or-Unknown

May 1 2022

Liz created T307314: Lag in updating Special Pages?.
May 1 2022, 9:51 PM · DBA, serviceops, WMF-General-or-Unknown

Apr 27 2021

Liz created T281281: Error message when trying to get edit count/summary.
Apr 27 2021, 3:43 PM · XTools

Feb 23 2021

Liz created T275466: Split wikistats metrics out by namespace.
Feb 23 2021, 3:43 AM · Data-Engineering, Data-Engineering-Wikistats