Page MenuHomePhabricator

Ladsgroup (Amir Sarabadani)
Shah of Bugs, Emir of database architecture, World-renowned rubber duckAdministrator

Projects (35)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 6 2014, 9:53 PM (489 w, 1 d)
Roles
Administrator
Availability
Available
IRC Nick
Amir1
LDAP User
Ladsgroup
MediaWiki User
Ladsgroup [ Global Accounts ]

Staff Database Architect in SRE data persistence team in WMF. Used to be Wikidata software engineer in WMDE

I'm also open source enthusiast, mediawiki volunteer developer, and long-term Wikipedian.

All edits on tickets about databases are in my work capacity and anything else is in my volunteer capacity unless mentioned otherwise.

Babel: fa-N, en-4, de-2, tr-1, hu-1

Recent Activity

Yesterday

Ladsgroup claimed T358011: Set up mailing list for zh.wikipedia.

Waiting to check some stuff.

Tue, Feb 20, 11:58 PM · SRE, Wikimedia-Mailing-lists
Ladsgroup closed T357996: Captcha on betacluster Special:CreateAccount does not change, a subtask of T270771: Keep selenium-daily-beta(commons)-MediaWiki Jenkins jobs green, as Resolved.
Tue, Feb 20, 11:47 PM · Browser-Tests, User-zeljkofilipin
Ladsgroup closed T357996: Captcha on betacluster Special:CreateAccount does not change as Resolved.

I ran it again with my own wordlist which is in my home directory. It's not great. I know but such is life.

Tue, Feb 20, 11:47 PM · ConfirmEdit (CAPTCHA extension), User-zeljkofilipin, Beta-Cluster-reproducible, Beta-Cluster-Infrastructure
Ladsgroup added a comment to T357995: `Database error` error page when creating account on the beta cluster.

It should switch to virtual domains I think.

Tue, Feb 20, 4:19 PM · MediaWiki-Platform-Team, AntiSpoof, MediaWiki-extensions-CentralAuth, User-zeljkofilipin, Beta-Cluster-reproducible

Mon, Feb 19

Ladsgroup committed rSCHCHae442606dbbb: Merge branch 'phab/T357189' into 'main' (authored by Ladsgroup).
Merge branch 'phab/T357189' into 'main'
Mon, Feb 19, 11:04 AM
Ladsgroup committed rSCHCHeb8f9ebf06ac: Phab/T357189 (authored by ABran-WMF).
Phab/T357189
Mon, Feb 19, 11:04 AM
Ladsgroup added a comment to T357824: Ignore one lagging replica in waitForReplication.

the proposal is to just stop complaining about it in logs

OK, you can change the log messages, I don't care about those.

Mon, Feb 19, 10:14 AM · DBA, MediaWiki-libs-Rdbms
Ladsgroup closed T350431: Run maintenance scripts on Serbian projects as Resolved.

@Ladsgroup Can you run it again on Monday? srwiki is having wmf.18 as I can see.

Mon, Feb 19, 9:08 AM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Wikimedia-Site-requests, Serbian-Sites, Wikimedia-maintenance-script-run
Ladsgroup closed T350431: Run maintenance scripts on Serbian projects, a subtask of T185421: Allow the use of Latin and Ijekavian magic words and special page aliases on the Serbian Wikipedia, as Resolved.
Mon, Feb 19, 9:07 AM · User-Kizule, Language-Technical Support, Serbian-Sites, I18n, MediaWiki-Internationalization
Ladsgroup added a comment to T352010: Gradually drop old pagelinks columns.

pagelinks in azwiki in db2156 is corrupted. I can't change its PK there due to duplicate entry while that's fine in other replicas. I'd say I need to clone it from another host.

Mon, Feb 19, 8:27 AM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data

Sun, Feb 18

Ladsgroup added a comment to T314020: LoadMonitor connection weighting reimagined.

Oh and another thing I remembered: Due to T327852: MediaWiki replication lags are inflated with average of half a second 80 to 90% of the replag being measured is just randomness (assuming the lag is usually between 50-100ms) and that means LM has been just moving weights around mostly at random in the past couple of years.

Sun, Feb 18, 4:56 PM · MW-1.41-notes (1.41.0-wmf.27; 2023-09-19), Patch-For-Review, Sustainability (Incident Followup), Performance-Team, MediaWiki-libs-Rdbms
Ladsgroup added a comment to T357824: Ignore one lagging replica in waitForReplication.

I actually realized something: waitForReplication() doesn't take into account the secondary datacenter and WMCS and other replicas which has led in many cases for maint scripts to cause significant lag there when they ran for long period of time and as such if a maint script is running for long-enough period they must also have --sleep option between each batch (I introduced several one of them myself) too which means the idea of "The goal of waitForReplication() is [...] to reduce the write rate to match the speed of the slowest replica." is moot because it hasn't been doing that for years (including replicas that serve production traffic but just in the secondary datacenter).

Sun, Feb 18, 4:52 PM · DBA, MediaWiki-libs-Rdbms

Fri, Feb 16

Ladsgroup added a comment to T357824: Ignore one lagging replica in waitForReplication.

I think my proposal didn't come through, probably I didn't write it better.

Fri, Feb 16, 11:48 PM · DBA, MediaWiki-libs-Rdbms
Ladsgroup added a comment to T314020: LoadMonitor connection weighting reimagined.

I'm tasked to see this through by end of this quarter. I read through the code, the proposal and the WIP patches. I have some comments, specially from DBA/operations point of view. Sorry it's quite long:

  • The biggest problem with status quo is that LM uses replication lag as the only metric for the load on a replica. That doesn't make any sense to me. Almost all replag we have in production is due to large write or lots of writes and in no way indicator of read pressure. It can mean some read pressure when the replica basically gives up and is down which defies the point. Again, in almost all cases, LM currently treats a replica that is going through a large write the same as a replica that's having too many read connections. While it treats a a db that's getting a lot of reads is and on the verge of going down as fully functioning "everything is fine" replica. No wonder this never helped in any outages so far.
    • Maybe it made sense in HDD times. Regardless, It's not the case anymore.
  • That means I fully support the idea of using connections or connection rates as a proxy metric for load. My worry is that building a WRstats system for this is quite costly specially if you put it in APCu it'll be much noisier with mw-on-k8s.
    • Another issue with APCu is that if a replica is overloaded because of extra load from the API cluster, The web cluster appservers will happily continue to send more connections to that replica because they are not seeing any extra load there.
    • I think there might be a simpler solution here, just do a count of processlist on the replica instead of getting replag. I don't know if grants need to be changed or if we can find some other way to gather metric on number of connections but I generally think that mysql should provide you with a usable metric rather than mw trying to guess based on some internal counters. There are lots of options, I'm going to research them and come back to you with some ideas. Maybe it's not possible, then we can revisit.
  • We should look at failure scenarios. I have a couple in mind.
    • One replica is overloaded with connections: It should stop sending requests there. It currently doesn't
    • One replica is inaccessible: It logs an errors (It shouldn't T357824: Ignore one lagging replica in waitForReplication) and doesn't send requests there (good)
    • One replica has a cold cache and is slow to respond: It should give it less connection but slowly ramp it up. It currently doesn't and forcing DBAs to repool a host in four steps over 45 minutes.
    • A whole section is overloaded (a common case actually): It happily sends connection to the replicas which leads to the whole appservers running out CPUs. This has happened so many times I lost count. LM needs to become a circuit breaker for whole sections too and this is not too hard to implement in the class which makes me happy. Even 40% of user requests failing is much better than 100% of them failing.
  • As part of T343098: [epic] Data Persistence Hypothesis WE 3.2.1 this class needs a lot of simplification and clean up. I don't understand the point of ::isStateRefreshDue(), you can add jitter, no need for this. And more stuff like this.
  • Instead of moving average, it could take the previous value and basically turn it into a PID controller. They have been heavily used in designing many large-scale load balancers. Biggest complication is figuring out the coefficients, maybe it's not needed, I need to check this in more depth. Ideas are welcome.
Fri, Feb 16, 9:55 PM · MW-1.41-notes (1.41.0-wmf.27; 2023-09-19), Patch-For-Review, Sustainability (Incident Followup), Performance-Team, MediaWiki-libs-Rdbms
Ladsgroup created T357824: Ignore one lagging replica in waitForReplication.
Fri, Feb 16, 9:16 PM · DBA, MediaWiki-libs-Rdbms
Ladsgroup added a comment to T352010: Gradually drop old pagelinks columns.

Awesome. I gently recommend unsubscribing from this ticket :D, we are not even 30% done.

Fri, Feb 16, 8:41 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Ladsgroup added a comment to T357778: Require logged-out users to complete a CAPTCHA on temporary account creations in certain circumstances.

Fair, in adding links, the user clicks on Publish and then sees a captcha, maybe if we show captcha before that it would reduce the barrier but yeah. Very non-scientific gut feeling. Nothing concrete.

Fri, Feb 16, 6:34 PM · Temporary accounts
Ladsgroup added a comment to T352010: Gradually drop old pagelinks columns.

Can you try again?

Fri, Feb 16, 6:32 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Ladsgroup added a comment to T352010: Gradually drop old pagelinks columns.

It probably needs a maintain view, let me try it.

Fri, Feb 16, 6:32 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Ladsgroup added a comment to T357778: Require logged-out users to complete a CAPTCHA on temporary account creations in certain circumstances.

I'd go even further given how easy it is to change IPs to just require it on every case. That simplifies the logic.

Fri, Feb 16, 6:20 PM · Temporary accounts
Ladsgroup awarded T357778: Require logged-out users to complete a CAPTCHA on temporary account creations in certain circumstances a Love token.
Fri, Feb 16, 6:19 PM · Temporary accounts
Ladsgroup added a comment to T342880: Decide what the rate limit should be for temporary account creations.

My 2c that you can easily ignore (I think you mentioned it): I really like this idea: First edit that would trigger a new temp account should require a captcha, the subsequent edits by the same temp account shouldn't (unless the usual case of adding external links which is automatically enforced by ConfirmEdit extension). It would put up a decent-ish barrier against large-scale abuse.

Fri, Feb 16, 6:04 PM · Temporary accounts
Ladsgroup added a comment to T166010: The Great Namespaceization Effort.

After deploy of next week, I will run lsc 😈

Fri, Feb 16, 5:58 PM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Patch-For-Review, MediaWiki-General, MW-1.41-notes (1.41.0-wmf.28; 2023-09-26), Platform Engineering Roadmap Decision Making, Platform Team Workboards (Initiatives), Wikimania-Hackathon-2019, TechCom-RFC (TechCom-RFC-Closed), Epic
Ladsgroup added a comment to T356984: Stop sending change notification email if edit is done by a bot.

@UOzurumba Hi, this was already announced in the last tech news, why announcing it again?

Fri, Feb 16, 11:32 AM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice

Thu, Feb 15

Dreamy_Jazz awarded T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients a Yellow Medal token.
Thu, Feb 15, 3:08 PM · Beta-Cluster-Infrastructure
kostajh awarded T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients a Love token.
Thu, Feb 15, 2:52 PM · Beta-Cluster-Infrastructure
Ladsgroup added a comment to T356108: Use a third party email service for beta cluster emails.

FWIW, I don't think this would be a good idea. Beside using propriety systems, it's not even really justified, fixing that bug was one day of work at most which is much less work and resources than setting up mailgun (like a dedicated TLD domain) and pay and such.

Thu, Feb 15, 2:06 PM · Beta-Cluster-Infrastructure
Ladsgroup closed T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients as Resolved.
Thu, Feb 15, 2:04 PM · Beta-Cluster-Infrastructure
Ladsgroup added a comment to T354099: FancyCaptcha isn't compatible with pillow 10.x.

We need add a comparison between draw points and swap them if needed. That's straightforward if anyone is willing to take a look

Thu, Feb 15, 1:05 PM · Patch-For-Review, MW-1.42-notes (1.42.0-wmf.16; 2024-01-30), MW-1.41-release, MW-1.40-release, MW-1.39-release, ConfirmEdit (CAPTCHA extension)

Wed, Feb 14

Ladsgroup added a comment to T357072: Echo: Drop droppable rows from user_properties.

Sigh for loginwiki it's this:

mysql:research@s3-analytics-replica.eqiad.wmnet [loginwiki]> select up_property, count(*) from user_properties group by up_property order by count(*) desc limit 50;
+------------------------------------------+----------+
| up_property                              | count(*) |
+------------------------------------------+----------+
| wlenhancedfilters-seen-tour              | 20382152 |
| rcenhancedfilters-seen-tour              | 20382130 |
Wed, Feb 14, 6:35 PM · Notifications, DBA, Growth-Team
Ladsgroup added a comment to T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients.

Cherry-picked this in beta cluster and it works.

Wed, Feb 14, 6:23 PM · Beta-Cluster-Infrastructure
Ladsgroup added a comment to T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients.

An update:

Wed, Feb 14, 4:16 PM · Beta-Cluster-Infrastructure
Ladsgroup closed T355499: Add "edituserjson" permission to ruwiki's "engineer" usergroup as Resolved.
Wed, Feb 14, 4:12 PM · Patch-For-Review, Wikimedia-Site-requests, Russian-Sites
Ladsgroup added a comment to T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients.

The whitespace change fixed the lookup and now it should be working. I got the mail as well.

Wed, Feb 14, 2:54 PM · Beta-Cluster-Infrastructure
Ladsgroup added a comment to T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients.

I love this, the relay allows internal IPs

hostlist relay_from_hosts = <; @[] ; 127.0.0.1 ; ::1 ; 172.16.0.0/12 ; 127.0.0.0/8 ; ::1/128
Wed, Feb 14, 2:28 PM · Beta-Cluster-Infrastructure
Ladsgroup added a comment to T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients.

Replaced the host with IP and now I'm getting this:

2024-02-14 14:21:08 1raG8J-0002l3-Tc ** ladsgroup+beta@gmail.com R=wiki_mail T=remote_smtp H=185.15.56.115 [185.15.56.115] I=[172.16.2.65]: SMTP error from remote mail server after RCPT TO:<ladsgroup+beta@gmail.com>: 550 Relay not permitted

Let me see what I can do about that.

Wed, Feb 14, 2:23 PM · Beta-Cluster-Infrastructure
Ladsgroup added a comment to T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients.

Prgoress?

2024-02-14 14:09:13 1raFwn-0005fc-Me <= wiki@wikimedia.beta.wmflabs.org U=www-data P=local S=2898 id=enwiki.65ccc989a56369.31715278@en.wikipedia.beta.wmflabs.org
2024-02-14 14:09:13 1raFwn-0005fc-Me no IP address found for host  deployment-mx03.deployment-prep.eqiad1.wikimedia.cloud
2024-02-14 14:09:13 1raFwn-0005fc-Me == ladsgroup+beta@gmail.com R=wiki_mail defer (-32): lookup of host "\302\240deployment-mx03.deployment-prep.eqiad1.wikimedia.cloud" failed in wiki_mail router
Wed, Feb 14, 2:11 PM · Beta-Cluster-Infrastructure
Ladsgroup added a comment to T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients.
Notice: /Stage[main]/Exim4/File[/etc/exim4/exim4.conf]/content: 
--- /etc/exim4/exim4.conf	2023-12-11 16:45:46.519337352 +0000
+++ /tmp/puppet-file20240214-10749-1yae50w	2024-02-14 14:03:14.872509648 +0000
@@ -45,6 +45,12 @@
 	allow_defer
 	forbid_file
Wed, Feb 14, 2:04 PM · Beta-Cluster-Infrastructure
Ladsgroup added a comment to T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients.

It's getting routed to WMCS's main mailservers:

2024-02-14 13:01:57 1raEth-0006mk-Lf <= wiki-enwiki-s9-s8uk79-se/dKyYlA6mLoB6r@beta.wmflabs.org U=www-data P=local S=1644 id=enwiki.65ccb9c59d2e20.08051219@en.wikipedia.beta.wmflabs.org
2024-02-14 13:01:57 1raEth-0006mk-Lf => ladsgroup@gmail.com R=smart_route T=remote_smtp S=1680 H=mx-out03.wmcloud.org [172.16.6.237] I=[172.16.2.65] K C="250- 1680 byte chunk, total 1680\\n250 OK id=1raEth-0005rF-Ml" DT=0s
Wed, Feb 14, 1:57 PM · Beta-Cluster-Infrastructure
Ladsgroup added a comment to T353225: Echo: Make use of conditional user defaults.

This works:

wikiadmin2023@10.192.48.205(fawiki)> select count(user_id) from user left join user_properties on up_user = user_id and up_property = 'echo-subscriptions-email-mention' where user_registration > '20130000000000' and (up_value is null or up_value = '0') and user_name in (select log_title from logging where log_type = 'newusers' and log_action = 'create');
Wed, Feb 14, 12:14 PM · Growth-Team (Sprint 8 (Growth Team)), User-notice, Patch-For-Review, Notifications
Ladsgroup awarded T357507: Move 50% of mediawiki external requests to mw on k8s a Love token.
Wed, Feb 14, 11:27 AM · Release-Engineering-Team, SRE, Traffic, serviceops, MW-on-K8s
Ladsgroup awarded T357508: Move 60% of mediawiki external requests to mw on k8s a Love token.
Wed, Feb 14, 11:27 AM · Release-Engineering-Team, SRE, Traffic, serviceops, MW-on-K8s
Ladsgroup updated the task description for T352010: Gradually drop old pagelinks columns.
Wed, Feb 14, 10:40 AM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data

Tue, Feb 13

Ladsgroup added a comment to T353225: Echo: Make use of conditional user defaults.

Unfortunately, detecting the autocreated status retrospectively is a significant challenge, especially from MediaWiki Core itself (it is not easy even from CentralAuth, but at least it can be determined with reasonable accuracy from centralauth.localuser.lu_attachment_method).

We can check the oldest newusers log entry for the user and see if it's an autocreation or not. It's a range query on the log_actor_type_time index with a limit of 1, should be very efficient. There are some scenarios in which no log entry is created (e.g. autocreation by a global admin) but only for autocreation.
If we need to turn it into a batch operation some time in the future, that will be problematic, but that's already true for the registration date.

Tue, Feb 13, 9:06 PM · Growth-Team (Sprint 8 (Growth Team)), User-notice, Patch-For-Review, Notifications
Ladsgroup added a comment to T355980: [vrts] Move attachment storage out of mysql.

Fair, if someone can take a look at what queues are the biggest, maybe we can trim non-permission ones.

Tue, Feb 13, 8:06 PM · Patch-For-Review, Znuny, collaboration-services
Ladsgroup added a comment to T355980: [vrts] Move attachment storage out of mysql.

Something to consider, definitely needs to talk to the community first: Is really attachments from ten years ago needed? Can we set up a threshold and delete attachments after certain time (to be determined by VRTS admins)?

Tue, Feb 13, 7:51 PM · Patch-For-Review, Znuny, collaboration-services
Ladsgroup added a comment to T355672: Investigate: How to make the GUC query performant.

I didn't elaborate on IP ranges, but doing that is pretty fast as-is, by simply querying ip_changes (relevant code). To be clear, I'm not claiming this to be the "best" way to do it, rather I can vouch that it's fast enough – and that's using the slower Toolforge replicas.

ip_changes won't be written to for temp users, since their IP addresses will only be stored in CheckUser tables. Would work on the same principle though.

[...]
To that end, would the new table be part of CentralAuth? I'm guessing for this project 3rd party support isn't a hard requirement for an MVP, but it would be cool to remove the dependency on CentralAuth, specifically, as that's not really recommended outside WMF. The dependency on CheckUser is of course fine and necessary in our case.

Having GUC in CheckUser makes sense to me too - I think the fewer places we store and look up IPs the better - mostly so the permissions are all handled in one place, but also to keep purging in sync, in case the time we're allowed to keep the data ever changes. For that reason, ideally the central table would be in CheckUser too, but I'm not sure if that's possible to do cleanly since CheckUser isn't aware of other wikis. (Could we have a config for whether to use the central table? Can you manage a single central table from an extension installed on many wikis?)

Tue, Feb 13, 7:15 PM · Epic, XTools, Tool-Global-user-contributions, Stewards-and-global-tools, Temporary accounts
Ladsgroup added a comment to T355672: Investigate: How to make the GUC query performant.

I didn't elaborate on IP ranges, but doing that is pretty fast as-is, by simply querying ip_changes (relevant code). To be clear, I'm not claiming this to be the "best" way to do it, rather I can vouch that it's fast enough – and that's using the slower Toolforge replicas.

That said, while a new table sounds great, you could just query cu_changes directly using the same per-slice strategy and run no more than in 8 queries (on our cluster) for a single IP/range/account.

Tue, Feb 13, 5:58 PM · Epic, XTools, Tool-Global-user-contributions, Stewards-and-global-tools, Temporary accounts
Ladsgroup added a comment to T356159: Performance review for PageNotice extension.

Exactly, that's what I'm saying :)

Tue, Feb 13, 5:44 PM · Wikimedia-Extension-setup, MediaWiki-extensions-PageNotice
Ladsgroup added a comment to T356159: Performance review for PageNotice extension.

One thing to consider is that this is page notice (header and footer) and not site notice. So it makes sense to be dependent on the page. Whether it's authority control data or the message's language.

Tue, Feb 13, 5:17 PM · Wikimedia-Extension-setup, MediaWiki-extensions-PageNotice
Ladsgroup added a comment to T355914: Change default image thumbnail size.

@RHo Thank you! Something to consider is also looking at thumbnail sizes in other areas. Mostly notably:

Tue, Feb 13, 4:55 PM · Web-Team-Backlog (Needs Prioritization (Tech)), Design, Wikimedia-Design, Thumbor, Traffic, SRE-swift-storage, Data-Persistence, SRE, Wikimedia-Site-requests
Krinkle awarded T354194: Redesign DBAccessObjectUtils a Orange Medal token.
Tue, Feb 13, 1:51 PM · MW-1.42-notes (1.42.0-wmf.19; 2024-02-20), Patch-For-Review, DBA, MediaWiki-libs-Rdbms
Ladsgroup added a comment to T356159: Performance review for PageNotice extension.

Can we take a couple of steps back?

This is injecting a system message into the page output, depending on the namespace.
We inject dozens of system messages in the user's language into the output on every request to build the skin.

Tue, Feb 13, 12:58 PM · Wikimedia-Extension-setup, MediaWiki-extensions-PageNotice
Ladsgroup updated the task description for T352010: Gradually drop old pagelinks columns.
Tue, Feb 13, 12:53 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Ladsgroup updated the task description for T352010: Gradually drop old pagelinks columns.
Tue, Feb 13, 12:53 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Ladsgroup added a comment to T352010: Gradually drop old pagelinks columns.

pagelinks is enwikinews is partitioned in ["db1212", "db2105", "db2109", "db2139:3313", "db2149", "db2156", "db2177", "db2190"] and not partitioned in ["db1140:3313", "db1150:3313", "db1166", "db1175", "db1189", "db1198", "db1223", "db1240:3313", "dbstore1007:3313"]

Tue, Feb 13, 11:59 AM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Ladsgroup added a comment to T299947: Normalize pagelinks table.

@Huji Hi, the data has been fully populated for fawiki. If you want check and switch your tools and queries, it should be fine to do so now.

Tue, Feb 13, 11:52 AM · Platform Engineering, MediaWiki-Page-derived-data

Mon, Feb 12

Ladsgroup updated subscribers of T355499: Add "edituserjson" permission to ruwiki's "engineer" usergroup.

From the looks of the discussion it seems it is indeed closed by a 'crat but I can't read Russian and only guess this based on Google Translate. Can one of you confirm this? @Iniquity or @stjn

Mon, Feb 12, 8:38 PM · Patch-For-Review, Wikimedia-Site-requests, Russian-Sites
Ladsgroup added a comment to T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients.

I haven't looked at the exim4 configs yet to see what needs to be done regarding routing (maybe it's already handled via ad-hoc cherry-picked puppet). exim4 configs are notoriously complicated, give me a bit.

Mon, Feb 12, 8:16 PM · Beta-Cluster-Infrastructure
Ladsgroup added a comment to T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients.

I forgot, I removed beta.wmflabs.org SPF record too.

Mon, Feb 12, 8:12 PM · Beta-Cluster-Infrastructure
Ladsgroup added a comment to T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients.

I think I set up the SPF record:

$dig wikimedia.beta.wmflabs.org txt
Mon, Feb 12, 8:12 PM · Beta-Cluster-Infrastructure
Ladsgroup added a comment to T343925: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients.

I assigned float IP 185.15.56.115 to deployment-mx03

Mon, Feb 12, 7:55 PM · Beta-Cluster-Infrastructure
Ladsgroup updated the task description for T352010: Gradually drop old pagelinks columns.
Mon, Feb 12, 6:10 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Ladsgroup added a comment to T352010: Gradually drop old pagelinks columns.

pagelinks is partitioned in enwikinews:

root@db1212.eqiad.wmnet[enwikinews]> show create table pagelinks;
+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->
| Table     | Create Table                                                                                                                                                                                        >
+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->
| pagelinks | CREATE TABLE `pagelinks` (
  `pl_from` int(8) unsigned NOT NULL DEFAULT 0,
  `pl_namespace` int(11) NOT NULL DEFAULT 0,
  `pl_title` varbinary(255) NOT NULL DEFAULT '',
  `pl_from_namespace` int(11) NOT NULL DEFAULT 0,
  `pl_target_id` bigint(20) unsigned DEFAULT NULL,
  PRIMARY KEY (`pl_from`,`pl_namespace`,`pl_title`),
  KEY `pl_namespace` (`pl_namespace`,`pl_title`,`pl_from`),
  KEY `pl_backlinks_namespace` (`pl_from_namespace`,`pl_namespace`,`pl_title`,`pl_from`),
  KEY `pl_target_id` (`pl_target_id`,`pl_from`),
  KEY `pl_backlinks_namespace_target_id` (`pl_from_namespace`,`pl_target_id`,`pl_from`)
) ENGINE=InnoDB DEFAULT CHARSET=binary
 PARTITION BY RANGE (`pl_namespace`)
(PARTITION `p_3` VALUES LESS THAN (4) ENGINE = InnoDB,
 PARTITION `p_4` VALUES LESS THAN (5) ENGINE = InnoDB,
 PARTITION `p_9` VALUES LESS THAN (10) ENGINE = InnoDB,
 PARTITION `p_10` VALUES LESS THAN (11) ENGINE = InnoDB,
 PARTITION `p_max` VALUES LESS THAN MAXVALUE ENGINE = InnoDB) |
+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->
1 row in set (0.001 sec)
Mon, Feb 12, 5:55 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Ladsgroup closed T357326: Email List request as Declined.

We don't create mailing lists for such cases. See https://meta.wikimedia.org/wiki/Mailing_lists

Mon, Feb 12, 5:08 PM · SRE, Wikimedia-Mailing-lists
Ladsgroup updated the task description for T352010: Gradually drop old pagelinks columns.
Mon, Feb 12, 1:55 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Ladsgroup added a comment to T355914: Change default image thumbnail size.

@WhatamIdoing Welcome to the mess of thumbnailing sizes in mediawiki. I wrote in more details in T211661#8377883. TLDR: That pregen sizes are the sizes you see under the image description page. e.g. (today's POTD in commons):
https://commons.wikimedia.org/wiki/File:Gilbert_Studios_photograph_of_Harriet_Jacobs.jpg

grafik.png (249×979 px, 106 KB)

Mon, Feb 12, 1:48 PM · Web-Team-Backlog (Needs Prioritization (Tech)), Design, Wikimedia-Design, Thumbor, Traffic, SRE-swift-storage, Data-Persistence, SRE, Wikimedia-Site-requests
Ladsgroup closed T356984: Stop sending change notification email if edit is done by a bot, a subtask of T354436: 1.42.0-wmf.18 deployment blockers, as Resolved.
Mon, Feb 12, 1:26 PM · User-brennen, Release-Engineering-Team (Priority Backlog 📥), Release, Train Deployments
Ladsgroup closed T356984: Stop sending change notification email if edit is done by a bot as Resolved.

I call this resolved. Let me know if you have any concerns. I continue to work on improvements.

Mon, Feb 12, 1:26 PM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice
Ladsgroup added a comment to T356159: Performance review for PageNotice extension.

The part of PageNotice extension that provides page-specific notices is not planned to be used in WMF prod, but only the namespace-level notices. They can use variables like {{PAGENAME}} so a cache strategy like that of core sitenotices (which are parsed once and reused across all pages) doesn't work. @daniel @Ladsgroup Can you confirm if parser caching is still workable? I am at a loss to understand how cache invalidation would work:

  • When a namespace notice is updated, we can't purge the PC of all pages in the namespace as that could be in the millions.

Yeah exactly, the idea is that we just don't and let it get regenerated with refreshlinks/edits/expiry. I don't think page notices are time sensitive (similar to templates). I could be wrong though.

Mon, Feb 12, 11:31 AM · Wikimedia-Extension-setup, MediaWiki-extensions-PageNotice
Ladsgroup added a comment to T356984: Stop sending change notification email if edit is done by a bot.

I don't think what you're describing will happen. Will try it after deploy and let us know.

Mon, Feb 12, 11:25 AM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice
Ladsgroup updated the task description for T352010: Gradually drop old pagelinks columns.
Mon, Feb 12, 11:06 AM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Ladsgroup added a comment to T356984: Stop sending change notification email if edit is done by a bot.

If that's data loss, we are having data loss since T29884

Mon, Feb 12, 9:11 AM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice
Ladsgroup added a comment to T356984: Stop sending change notification email if edit is done by a bot.

It doesn't lead to data loss...

Mon, Feb 12, 9:04 AM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice
Ladsgroup added a comment to T356984: Stop sending change notification email if edit is done by a bot.

If you disagree with a change, it doesn't mean it has to be reverted.

Mon, Feb 12, 8:57 AM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice
Ladsgroup removed a parent task for T356984: Stop sending change notification email if edit is done by a bot: T354436: 1.42.0-wmf.18 deployment blockers.
Mon, Feb 12, 8:57 AM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice
Ladsgroup removed a subtask for T354436: 1.42.0-wmf.18 deployment blockers: T356984: Stop sending change notification email if edit is done by a bot.
Mon, Feb 12, 8:57 AM · User-brennen, Release-Engineering-Team (Priority Backlog 📥), Release, Train Deployments
Ladsgroup lowered the priority of T356984: Stop sending change notification email if edit is done by a bot from Unbreak Now! to Medium.
Mon, Feb 12, 8:57 AM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice

Fri, Feb 9

Ladsgroup updated the task description for T357189: Drop iwl_prefix_from_title from iwlinks.
Fri, Feb 9, 11:39 PM · Schema-change-in-production, DBA
Ladsgroup added a project to T357189: Drop iwl_prefix_from_title from iwlinks: Schema-change-in-production.
Fri, Feb 9, 11:39 PM · Schema-change-in-production, DBA
Ladsgroup triaged T357189: Drop iwl_prefix_from_title from iwlinks as Medium priority.
Fri, Feb 9, 11:39 PM · Schema-change-in-production, DBA
Ladsgroup created T357189: Drop iwl_prefix_from_title from iwlinks.
Fri, Feb 9, 11:38 PM · Schema-change-in-production, DBA
Ladsgroup closed T356984: Stop sending change notification email if edit is done by a bot as Resolved.
Fri, Feb 9, 11:34 PM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice
Ladsgroup added a comment to T350181: Enable desktop diff page on mobile site.

Noted. Thanks!

Fri, Feb 9, 6:43 PM · Wikimedia-Site-requests, Web-Team-Backlog (Needs Prioritization (Tech)), User-Jdlrobson, MobileFrontend (MobileFrontend Special Pages), Technical-Debt (RW-Tech-Debt)
Ladsgroup added a comment to T356984: Stop sending change notification email if edit is done by a bot.

I responded there, I also added a link to this ticket in tech news.

Fri, Feb 9, 6:42 PM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice
Ladsgroup added a comment to T350181: Enable desktop diff page on mobile site.

The coloring looks off if you click on Special:MobileDiff. e.g. this: https://fa.wikipedia.org/wiki/Special:MobileDiff/38611902

grafik.png (430×984 px, 49 KB)

Fri, Feb 9, 4:43 PM · Wikimedia-Site-requests, Web-Team-Backlog (Needs Prioritization (Tech)), User-Jdlrobson, MobileFrontend (MobileFrontend Special Pages), Technical-Debt (RW-Tech-Debt)
Ladsgroup added a comment to T356159: Performance review for PageNotice extension.

This extension is going to be showing a notice over every page in the wiki (mw message per ns)

Fri, Feb 9, 2:04 PM · Wikimedia-Extension-setup, MediaWiki-extensions-PageNotice
Ladsgroup updated the task description for T352010: Gradually drop old pagelinks columns.
Fri, Feb 9, 1:23 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Ladsgroup updated the task description for T352010: Gradually drop old pagelinks columns.
Fri, Feb 9, 1:20 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Ladsgroup closed T356565: Clean up logs table as Resolved.

I'm seeing that it's deployed. We could do much more but at least for now it's not exploding.

Fri, Feb 9, 1:14 PM · LibUp
Ladsgroup renamed T355914: Change default image thumbnail size from Changing default image thumbnail size on English Wikipedia to Change default image thumbnail size.
Fri, Feb 9, 12:49 PM · Web-Team-Backlog (Needs Prioritization (Tech)), Design, Wikimedia-Design, Thumbor, Traffic, SRE-swift-storage, Data-Persistence, SRE, Wikimedia-Site-requests
Ladsgroup added a comment to T355914: Change default image thumbnail size.

As a random rant, perhaps it is my myopia speaking, but I would really love to have bigger default sizes across Wikipedias for the images. In fact it is in my todo list to propose increase for default Infobox sizes on Ukrainian Wikipedia, and come to think of it making default thumbnail sizes bigger would be a boon too.

Fri, Feb 9, 12:49 PM · Web-Team-Backlog (Needs Prioritization (Tech)), Design, Wikimedia-Design, Thumbor, Traffic, SRE-swift-storage, Data-Persistence, SRE, Wikimedia-Site-requests
Ladsgroup added a comment to T355914: Change default image thumbnail size.

At risk of scope creep... while we're here, it'd be nice to also re-examine gallery sizes. See this conversation on Wikitech-l :

Short version: 120px is outrageously small and not in keeping with how others do it. The default gallery size should be a lot larger - maybe something for a user study if we want details, but really any increase would be an improvement. If the same size as the "new" default thumbnail is picked, then there isn't even any need to do anything extra. Or alternatively, just match an existing size created by the thumbnailer.

I agree but that should be done after this to avoid melting thumbor :(

Fri, Feb 9, 12:41 PM · Web-Team-Backlog (Needs Prioritization (Tech)), Design, Wikimedia-Design, Thumbor, Traffic, SRE-swift-storage, Data-Persistence, SRE, Wikimedia-Site-requests
Ladsgroup added a comment to T356159: Performance review for PageNotice extension.

Yeah it should be page language. I don't think it's reasonable to make this be parsed on the fly on every page view :(

Fri, Feb 9, 12:40 PM · Wikimedia-Extension-setup, MediaWiki-extensions-PageNotice
Ladsgroup added a comment to T356984: Stop sending change notification email if edit is done by a bot.

Yup

Fri, Feb 9, 11:37 AM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice
Ladsgroup added a comment to T356984: Stop sending change notification email if edit is done by a bot.

It should be included ASAP. We are planning to deploy this really soon.

Fri, Feb 9, 1:44 AM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice

Thu, Feb 8

Ladsgroup triaged T357072: Echo: Drop droppable rows from user_properties as Medium priority.

I will get to it next week.

Thu, Feb 8, 11:02 PM · Notifications, DBA, Growth-Team
Ladsgroup added a comment to T356984: Stop sending change notification email if edit is done by a bot.

Hi, something along the lines of:

If you have "Email me when a page or a file on my watchlist is changed " option enabled, bot edits will no longer trigger notification emails. You can still see bot edits in [[Special:Watchlist]].

Thu, Feb 8, 10:55 PM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), Infrastructure-Foundations, Mail, User-notice
Ladsgroup added a comment to T706: Requests for addition to the #acl*Project-Admins group (in comments).

Hi, I am a manager for the Research team (I report to Leila). I would like to have Admin permissions (something like that) because I need to manage tags, projects and assignees

Thu, Feb 8, 9:18 PM · Tracking-Neverending, Project-Admins
Ladsgroup added a member for acl*Project-Admins: XiaoXiao-WMF.
Thu, Feb 8, 9:14 PM