Page MenuHomePhabricator

tstarling (Tim Starling)
UserAdministrator

Projects (23)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Oct 15 2014, 8:27 PM (340 w, 1 d)
Roles
Administrator
Availability
Available
LDAP User
Tim Starling
MediaWiki User
Tim Starling (WMF) [ Global Accounts ]

Recent Activity

Yesterday

tstarling added a comment to T280220: Error "Lock wait timeout exceeded" from User::loadFromDatabase (via API action=options).

The drop can also be seen in the ApiOptions p99 request service time, which dropped from ~25 seconds to ~5 seconds. The median service time is about 700ms, and for pretty much all of that time, the user row is locked, since User::getInstanceForUpdate() is the first thing the module does, and User::saveSettings() is the last thing it does before output.

Thu, Apr 22, 8:30 PM · Platform Team Workboards (Clinic Duty Team), MediaWiki-User-management, MediaWiki-API, Wikimedia-production-error

Tue, Apr 20

tstarling added a comment to T280220: Error "Lock wait timeout exceeded" from User::loadFromDatabase (via API action=options).

It's interesting to see that there are so many JS callers of mw.Api().saveOption(). It reminds me of T128602. I think preferences could be redesigned around this use case -- it was originally imagined as mostly being set in bulk via Special:Preferences, rather than being a generic user-linked data store.

Tue, Apr 20, 9:19 PM · Platform Team Workboards (Clinic Duty Team), MediaWiki-User-management, MediaWiki-API, Wikimedia-production-error
tstarling added a comment to T278917: Clean up obsolete ActorMigration usages for non-temp tables.

That concludes the code search for WMF deployed extensions. Any other extensions using the old ActorMigration fields will emit deprecation warnings after https://gerrit.wikimedia.org/r/c/mediawiki/core/+/676179 is merged.

Tue, Apr 20, 4:24 AM · MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), Platform Team Workboards (MW Expedition), MW-1.36-notes (1.36.0-wmf.38; 2021-04-06), Patch-For-Review

Mon, Apr 19

tstarling updated subscribers of T278917: Clean up obsolete ActorMigration usages for non-temp tables.

According to https://www.mediawiki.org/wiki/Extension:Translate , the Translate extension has a "master" b/c policy going back to 1.33. But its extension.json says it requires 1.34+. Can @Nikerabbit please confirm that it is safe to drop ActorMigration calls in the Translate extension that are there to support MW 1.33 and earlier?

Mon, Apr 19, 4:03 AM · MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), Platform Team Workboards (MW Expedition), MW-1.36-notes (1.36.0-wmf.38; 2021-04-06), Patch-For-Review

Sun, Apr 18

Krinkle awarded T277445: Hourly log rotation for large MW logs a Orange Medal token.
Sun, Apr 18, 12:09 AM · Developer Productivity, Platform Team Workboards (Clinic Duty Team), observability

Fri, Apr 16

tstarling added a comment to T279832: Some pages moves fail with "InvalidArgumentException: The Title object yields no ID. Perhaps the page doesn't exist?".

We could just remove the exception. Log a warning instead.

Fri, Apr 16, 6:47 AM · MediaWiki-extensions-Scribunto, Platform Team Workboards (Clinic Duty Team), MediaWiki-Page-rename, Wikimedia-production-error

Tue, Apr 13

tstarling updated the task description for T267532: Shellbox MediaWiki integration.
Tue, Apr 13, 9:11 PM · Shellbox, Platform Team Workboards (Purple), MW-on-K8s
tstarling created T279982: Add index on oi_timestamp.
Tue, Apr 13, 5:32 AM · Schema-change

Mon, Apr 12

tstarling added a comment to T278033: Upgrade testing-Wikimini from MediaWiki 1.28.0 to 1.35.

I would suggest temporarily removing ipblocks from $wgSharedTables during the upgrade. Re-add it once all wikis are upgraded.

Mon, Apr 12, 11:57 PM · WMCH-Wikimini, WMCH-Infrastructure
tstarling added a comment to T278033: Upgrade testing-Wikimini from MediaWiki 1.28.0 to 1.35.

I am seeing the same error migrating ipblocks, probably due to $wgSharedTables having ipblocks in it. I think the solution will be to unshare the tables.

Mon, Apr 12, 2:04 AM · WMCH-Wikimini, WMCH-Infrastructure

Fri, Apr 9

tstarling added a comment to T114098: Remote file thumb generation blocks serving a page.

The URLs were changed to point to Commons, but transformVia404 has no effect since it is implemented in the parent File::transform() which is not called. Also, there is a bug in the cache expiry code, meaning the thumbnail is downloaded every time instead of once per month. So the full thumbnail is downloaded and stored every time a page containing a commons image is rendered.

Fri, Apr 9, 2:10 AM · Commons, MW-1.27-release-notes, MW-1.27-release (WMF-deploy-2015-11-10_(1.27.0-wmf.6)), Patch-For-Review, Performance Issue, MediaWiki-File-management, Multimedia

Thu, Apr 8

tstarling closed Restricted Task, a subtask of T262331: Make Reply Tool available as opt-out preference at all Wikipedias, as Invalid.
Thu, Apr 8, 10:19 PM · DiscussionTools, Editing-team, OWC2020
tstarling closed Restricted Task, a subtask of T271964: Make New Discussion Tool available as opt-out feature at partner wikis, as Invalid.
Thu, Apr 8, 10:19 PM · Editing-team (FY2020-21 Kanban Board), DiscussionTools
tstarling closed Restricted Task, a subtask of T275256: Make New Discussion Tool available as opt-out feature at all wikis, as Invalid.
Thu, Apr 8, 10:19 PM · DiscussionTools, Editing-team, OWC2020

Wed, Apr 7

tstarling added a comment to T279496: Revisit mangleFlashPolicy().

I grepped the Ruffle source and found no implementation of cross-domain policy. I found a couple of complaints in the Ruffle bug tracker about it not supporting crossdomain.xml. The response from the devs is "use CORS".

Wed, Apr 7, 4:59 AM · Technical-Debt, MediaWiki-General, Security
tstarling added a comment to T279496: Revisit mangleFlashPolicy().

Ruffle is also provided as a native browser extension, so I suppose that should be reviewed for security.

Wed, Apr 7, 4:47 AM · Technical-Debt, MediaWiki-General, Security
tstarling added a comment to T279496: Revisit mangleFlashPolicy().

But I suppose someone might have a copy predating that "EOL will come" time bomb.

Wed, Apr 7, 4:33 AM · Technical-Debt, MediaWiki-General, Security
tstarling created T279496: Revisit mangleFlashPolicy().
Wed, Apr 7, 2:06 AM · Technical-Debt, MediaWiki-General, Security
tstarling reopened T278579: Special:Export broken: always generates an empty file as "Open".

The problem is that the ob_end_clean() in wfResetOutputBuffers() results in a call to OutputHandler::callback(), so Content-Length: 0 is sent. Previously the buggy code was only activated if the request was HTTP/1.0, but Aaron's patch exposed it for other kinds of requests.

Wed, Apr 7, 12:26 AM · MW-1.37-notes (1.37.0-wmf.1; 2021-04-13), MW-1.36-notes (1.36.0-wmf.38; 2021-04-06), MW-1.35-notes, Patch-For-Review, MW-1.35-release, Beta-Cluster-reproducible, MediaWiki-Core-Snapshots
tstarling removed a subtask for T278343: 1.36.0-wmf.37 deployment blockers: T278579: Special:Export broken: always generates an empty file.
Wed, Apr 7, 12:18 AM · Patch-For-Review, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release, Train Deployments
tstarling removed a parent task for T278579: Special:Export broken: always generates an empty file: T278343: 1.36.0-wmf.37 deployment blockers.
Wed, Apr 7, 12:18 AM · MW-1.37-notes (1.37.0-wmf.1; 2021-04-13), MW-1.36-notes (1.36.0-wmf.38; 2021-04-06), MW-1.35-notes, Patch-For-Review, MW-1.35-release, Beta-Cluster-reproducible, MediaWiki-Core-Snapshots

Tue, Apr 6

tstarling added a comment to T278579: Special:Export broken: always generates an empty file.

A wholesale revert is not really a resolution. Maybe we can reopen with lower priority and without the train blocker parent task?

Tue, Apr 6, 11:40 PM · MW-1.37-notes (1.37.0-wmf.1; 2021-04-13), MW-1.36-notes (1.36.0-wmf.38; 2021-04-06), MW-1.35-notes, Patch-For-Review, MW-1.35-release, Beta-Cluster-reproducible, MediaWiki-Core-Snapshots

Wed, Mar 31

tstarling updated the task description for T278917: Clean up obsolete ActorMigration usages for non-temp tables.
Wed, Mar 31, 5:46 AM · MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), Platform Team Workboards (MW Expedition), MW-1.36-notes (1.36.0-wmf.38; 2021-04-06), Patch-For-Review
tstarling added a comment to T275246: Populate rev_actor and rev_comment_id.

I'm splitting out the cleanup task to T278917.

Wed, Mar 31, 5:40 AM · MW-1.37-release, Patch-For-Review, Platform Engineering Roadmap, Code-Health-Objective, Platform Team Initiatives (Revision Storage Schema Improvements), Technical-Debt
tstarling added a subtask for T161671: Compacting the revision table: T278917: Clean up obsolete ActorMigration usages for non-temp tables.
Wed, Mar 31, 5:39 AM · Platform Team Workboards (Epics), Platform Team Initiatives (Revision Storage Schema Improvements), MediaWiki-Revision-backend, Multi-Content-Revisions, Epic, Patch-For-Review, Schema-change
tstarling added a parent task for T278917: Clean up obsolete ActorMigration usages for non-temp tables: T161671: Compacting the revision table.
Wed, Mar 31, 5:39 AM · MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), Platform Team Workboards (MW Expedition), MW-1.36-notes (1.36.0-wmf.38; 2021-04-06), Patch-For-Review
tstarling created T278917: Clean up obsolete ActorMigration usages for non-temp tables.
Wed, Mar 31, 5:39 AM · MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), Platform Team Workboards (MW Expedition), MW-1.36-notes (1.36.0-wmf.38; 2021-04-06), Patch-For-Review

Tue, Mar 30

tstarling added a comment to T252853: updateSearchIndex.php sql error not all tables locked.

I'm trying to understand why it is locking tables at all. Apparently I added it in June 2004 72652c4bc72e7043f3bf12b71abc959510132b5d. I'm not sure why, does anyone have any idea? I tried reading the relevant chapter in the MySQL 3.23 manual, but there's no suggestion there to lock tables.

Tue, Mar 30, 8:58 PM · MW-1.36-notes, MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), Patch-For-Review, Platform Team Workboards (Clinic Duty Team), Discovery-Search (Current work), MW-1.35-release, MediaWiki-User-management, Wikimedia-Rdbms, MediaWiki-Search, MediaWiki-Maintenance-system

Sun, Mar 28

tstarling added a comment to T278621: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007.

The change was made to allow MediaWiki to be installable with the UTF-8 character set selected. The maximum key length in MyISAM is 1000 bytes: https://dev.mysql.com/doc/refman/8.0/en/myisam-storage-engine.html . With the index (job_cmd, job_namespace, job_title), job_namespace is 4 bytes, job_title is 255*3=765 bytes, leaving 231 bytes for job_cmd. Before the referenced patch, job_cmd would require 765 bytes, causing an error on install, so I reduced that to 60 bytes, believing that that would be long enough for such an identifier.

Sun, Mar 28, 7:04 AM · DBA

Thu, Mar 25

tstarling added a comment to T274041: Investigate performance impact of HookContainer loading 500+ interfaces.

Ideas from technical planning meeting:

  • Make a separate hook runner for use within load.php?
  • Preload? T240775 Should be easy after k8s deployment.
  • Optimise PSR-4 code
  • Concatenate files
  • Have abstract classes representing hook modules, multiple hooks per class
Thu, Mar 25, 10:04 PM · Platform Engineering Roadmap Decision Making, Performance-Team (Radar), Patch-For-Review, Performance Issue, MediaWiki-Core-Hooks
tstarling moved T274041: Investigate performance impact of HookContainer loading 500+ interfaces from Untriaged to Tech Debt Jam on the Platform Engineering Roadmap Decision Making board.
Thu, Mar 25, 9:26 PM · Platform Engineering Roadmap Decision Making, Performance-Team (Radar), Patch-For-Review, Performance Issue, MediaWiki-Core-Hooks
tstarling edited projects for T274041: Investigate performance impact of HookContainer loading 500+ interfaces, added: Platform Engineering Roadmap Decision Making; removed Platform Engineering.
Thu, Mar 25, 9:26 PM · Platform Engineering Roadmap Decision Making, Performance-Team (Radar), Patch-For-Review, Performance Issue, MediaWiki-Core-Hooks
tstarling moved T267928: RFC: Provide mechanism for defining and utilizing configuration sets for local development and browser / API-testing tests from Tech Planning Review to Tracking/Watching on the Platform Engineering board.
Thu, Mar 25, 9:14 PM · Platform Engineering Roadmap Decision Making, tech-decision-forum, User-Soham, User-brennen, User-zeljkofilipin, MediaWiki-Configuration, Platform Engineering, Testing-Roadblocks, TechCom-RFC
tstarling added a comment to T275246: Populate rev_actor and rev_comment_id.

There's a lot of cleanup of deprecated code which needs to be done first.

Thu, Mar 25, 5:20 AM · MW-1.37-release, Patch-For-Review, Platform Engineering Roadmap, Code-Health-Objective, Platform Team Initiatives (Revision Storage Schema Improvements), Technical-Debt

Wed, Mar 24

tstarling claimed T275246: Populate rev_actor and rev_comment_id.
Wed, Mar 24, 11:04 PM · MW-1.37-release, Patch-For-Review, Platform Engineering Roadmap, Code-Health-Objective, Platform Team Initiatives (Revision Storage Schema Improvements), Technical-Debt
tstarling updated the task description for T215466: Remove revision_comment_temp and revision_actor_temp.
Wed, Mar 24, 10:55 PM · Data-Persistence (Consultation), Platform Engineering Roadmap Decision Making, Analytics-Radar, MW-1.35-notes (1.35.0-wmf.32; 2020-05-12), Platform Team Initiatives (Revision Storage Schema Improvements), Technical-Debt, Epic
tstarling updated the task description for T215466: Remove revision_comment_temp and revision_actor_temp.
Wed, Mar 24, 10:55 PM · Data-Persistence (Consultation), Platform Engineering Roadmap Decision Making, Analytics-Radar, MW-1.35-notes (1.35.0-wmf.32; 2020-05-12), Platform Team Initiatives (Revision Storage Schema Improvements), Technical-Debt, Epic

Mar 22 2021

tstarling created P14958 vardump.php.
Mar 22 2021, 6:29 AM

Mar 21 2021

tstarling added a comment to T270225: Finish session storage to actually meet multi-DC requirements.

The semantics as implemented were based on the idea that anything that wrote state was the result of an HTTP request with a corresponding verb (POST, PUT, DELETE, ...), whereas anything that read state was the result of a verb corresponding to reads (GET, HEAD, ...). I don't know if this was meant to explain how things actually were at the time, or if it was what we were working toward.

Mar 21 2021, 11:19 PM · MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), Platform Engineering Roadmap, MediaWiki-Authentication-and-authorization

Mar 19 2021

tstarling created T277834: Two-level session storage and the consistency problem with serialized blob stores.
Mar 19 2021, 4:45 AM · Performance-Team (Radar), MediaWiki-Authentication-and-authorization
tstarling added a comment to T267270: Determine multi-dc strategy for CentralAuth.

I am vaguely aware of the various different slabs and different storage areas Memcached allocates, but I've not come away from it thinking that it is "very likely" for single-use tokens to safely persist for N minutes in the Memc main cluster. Or did you envison a tiny separate memcached cluster for this?

Mar 19 2021, 4:03 AM · MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Platform Team Workboards (Clinic Duty Team), Performance-Team (Radar), serviceops, Sustainability (MediaWiki-MultiDC), Code-Health-Objective, Platform Team Initiatives (Session Management Service (CDP2))
tstarling added a comment to T270225: Finish session storage to actually meet multi-DC requirements.
  1. Request flow A
Mar 19 2021, 3:38 AM · MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), Platform Engineering Roadmap, MediaWiki-Authentication-and-authorization
tstarling added a comment to T270225: Finish session storage to actually meet multi-DC requirements.

For the login request [...]

Mar 19 2021, 2:55 AM · MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), Platform Engineering Roadmap, MediaWiki-Authentication-and-authorization
tstarling updated subscribers of T270225: Finish session storage to actually meet multi-DC requirements.

We instead rely on "sticky DC" cookies to pin a user the few seconds around session replication

Mar 19 2021, 1:13 AM · MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), Platform Engineering Roadmap, MediaWiki-Authentication-and-authorization

Mar 18 2021

Ferdi2005 awarded T6845: CAPTCHA doesn't work for people with visual impairments a Heartbreak token.
Mar 18 2021, 1:48 PM · Security, ConfirmEdit (CAPTCHA extension), Accessibility, Design, WCAG-Level-A

Mar 17 2021

tstarling added a comment to T276476: Creating a page with {{safesubst:self}} throws RevisionAccessException: No such slot: main.
In T276476#6920215, @IN wrote:

I am not sure how this problem is triggered

Mar 17 2021, 11:46 PM · MW-1.37-notes (1.37.0-wmf.1; 2021-04-13), MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Patch-For-Review, Platform Team Workboards (Clinic Duty Team), MediaWiki-Revision-backend, Wikimedia-production-error
tstarling added a comment to T277618: var_dump() on various objects writes gigabytes of data and takes minutes to run.

Trait which reads @noDebugInfo doc comment annotations: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/672845

Mar 17 2021, 5:06 AM · Platform Team Workboards (Clinic Duty Team), Upstream
tstarling added a comment to T277618: var_dump() on various objects writes gigabytes of data and takes minutes to run.

If it's worth doing, it's worth overdoing, am I right?

Mar 17 2021, 4:57 AM · Platform Team Workboards (Clinic Duty Team), Upstream
tstarling created T277618: var_dump() on various objects writes gigabytes of data and takes minutes to run.
Mar 17 2021, 3:35 AM · Platform Team Workboards (Clinic Duty Team), Upstream
tstarling added a comment to T277246: Omnipay mini test development and assessment .

I would suggest high priority for that, per https://www.mediawiki.org/wiki/Phabricator/Project_management#Priority_levels . I am regularly reviewing all UBN tasks.

Mar 17 2021, 3:09 AM · Patch-For-Review, Fundraising Sprint File Systems Stage Show, Fundraising-Backlog
tstarling added a comment to T277246: Omnipay mini test development and assessment .

Hi, I see you triaged this task as Unbreak Now. Do you need help?

Mar 17 2021, 2:59 AM · Patch-For-Review, Fundraising Sprint File Systems Stage Show, Fundraising-Backlog
tstarling lowered the priority of T271655: A2 release from Unbreak Now! to Medium.

As Thiemo says, please do not misuse UBN priority.

Mar 17 2021, 2:56 AM · Humaniki
tstarling added a comment to T276476: Creating a page with {{safesubst:self}} throws RevisionAccessException: No such slot: main.

DerivedPageDataUpdater says

Mar 17 2021, 1:11 AM · MW-1.37-notes (1.37.0-wmf.1; 2021-04-13), MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Patch-For-Review, Platform Team Workboards (Clinic Duty Team), MediaWiki-Revision-backend, Wikimedia-production-error

Mar 16 2021

tstarling renamed T276476: Creating a page with {{safesubst:self}} throws RevisionAccessException: No such slot: main from RevisionAccessException: No such slot: main to Creating a page with {{safesubst:self}} throws RevisionAccessException: No such slot: main.
Mar 16 2021, 10:52 PM · MW-1.37-notes (1.37.0-wmf.1; 2021-04-13), MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Patch-For-Review, Platform Team Workboards (Clinic Duty Team), MediaWiki-Revision-backend, Wikimedia-production-error
tstarling added a comment to T276476: Creating a page with {{safesubst:self}} throws RevisionAccessException: No such slot: main.

I will make a patch that improves reporting, so we get the ID of the offending revision.

Mar 16 2021, 10:50 PM · MW-1.37-notes (1.37.0-wmf.1; 2021-04-13), MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Patch-For-Review, Platform Team Workboards (Clinic Duty Team), MediaWiki-Revision-backend, Wikimedia-production-error
tstarling moved T276739: Get Wikimedia added to disconnect.me entity list from Ready (WIP:5) to Later on the Platform Team Workboards (Clinic Duty Team) board.

Let's leave this until we have some sort of confirmation that it would actually help us. I reviewed the Disconnect Firefox extension source code -- entities.json doesn't actually seem to be used there. But it looks like everything in entities.json has a corresponding entry in services.json, and everything in services.json is categorized as some sort of tracker. So the reason we're not in entities.json is because we're not considered to be a tracker. Probably adding Wikimedia would give people the option to block Wikimedia in the extension configuration. There are a number of open or closed bug reports against the list from companies asking to be removed, and from users asking for trackers to be added, but nobody is asking for non-trackers to be added to the list. So, maybe it is harmful.

Mar 16 2021, 5:53 AM · Platform Team Workboards (Clinic Duty Team), User-notice, Browser-Support-Firefox, MediaWiki-extensions-CentralAuth

Mar 15 2021

tstarling added a comment to T277445: Hourly log rotation for large MW logs.

If you propose a puppet change, I can review it and verify that it's working after deployment.

Mar 15 2021, 10:27 PM · Developer Productivity, Platform Team Workboards (Clinic Duty Team), observability
tstarling created T277445: Hourly log rotation for large MW logs.
Mar 15 2021, 9:44 AM · Developer Productivity, Platform Team Workboards (Clinic Duty Team), observability

Mar 12 2021

tstarling added a comment to T267270: Determine multi-dc strategy for CentralAuth.

Writing that patch forced me to properly review all current usages of session storage. Foreign API tokens also need DC pinning or some other special solution.

Mar 12 2021, 7:03 AM · MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Platform Team Workboards (Clinic Duty Team), Performance-Team (Radar), serviceops, Sustainability (MediaWiki-MultiDC), Code-Health-Objective, Platform Team Initiatives (Session Management Service (CDP2))
tstarling added a comment to T267270: Determine multi-dc strategy for CentralAuth.

There are some details here to be figured out. Probably most callers of setCentralSession() will need to send such a cookie.

Mar 12 2021, 5:08 AM · MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Platform Team Workboards (Clinic Duty Team), Performance-Team (Radar), serviceops, Sustainability (MediaWiki-MultiDC), Code-Health-Objective, Platform Team Initiatives (Session Management Service (CDP2))

Mar 11 2021

tstarling added a comment to T277124: Create project tag for Slow-DB-Query.

Is it necessary for tags to be non-overlapping? A query can be slow without being an error, and there can be errors without slowness.

Mar 11 2021, 6:27 AM · Project-Admins
tstarling added a comment to T267270: Determine multi-dc strategy for CentralAuth.

As an optimisation, special routing for those path prefixes could be skipped if there is no session or token cookie in the request. Anonymous auto-login is always going to fail and will account for most requests. The code would be the same as the existing special cases for session and token cookies in the Varnish and ATS configuration.

Mar 11 2021, 6:14 AM · MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Platform Team Workboards (Clinic Duty Team), Performance-Team (Radar), serviceops, Sustainability (MediaWiki-MultiDC), Code-Health-Objective, Platform Team Initiatives (Session Management Service (CDP2))
tstarling added a comment to T267270: Determine multi-dc strategy for CentralAuth.
  • All URIs involved in auto-login (what are they?) being routed to the primary DC
Mar 11 2021, 6:04 AM · MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Platform Team Workboards (Clinic Duty Team), Performance-Team (Radar), serviceops, Sustainability (MediaWiki-MultiDC), Code-Health-Objective, Platform Team Initiatives (Session Management Service (CDP2))
tstarling created T277124: Create project tag for Slow-DB-Query.
Mar 11 2021, 4:52 AM · Project-Admins
tstarling lowered the priority of T275591: PHP Warning: Memcached::getMulti(): Memcached constructor was not called from High to Low.

@aaron and I reviewed the memcached PECL code, but we did not find anything which would cause this.

Mar 11 2021, 2:24 AM · Platform Team Workboards (Clinic Duty Team), MediaWiki-Cache, Wikimedia-production-error

Mar 10 2021

tstarling closed T276017: Error: 503, Backend fetch failed while attempting to delete "c:File:Tholden28 test, 2020 US presidential election results.svg" (file history lists 778 versions of this file) as Resolved.

Complete after 10 minutes.

Mar 10 2021, 4:27 AM · MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Platform Team Workboards (Clinic Duty Team), MediaWiki-Page-deletion, Commons
tstarling added a comment to T276017: Error: 503, Backend fetch failed while attempting to delete "c:File:Tholden28 test, 2020 US presidential election results.svg" (file history lists 778 versions of this file).

When I looked at the file description page, I missed the fact that the file history is pageable. There are 778 versions of this file. I'll just delete it with deleteBatch.php. Please advise Tholden28 to not do that in future.

Mar 10 2021, 4:07 AM · MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Platform Team Workboards (Clinic Duty Team), MediaWiki-Page-deletion, Commons
tstarling moved T276017: Error: 503, Backend fetch failed while attempting to delete "c:File:Tholden28 test, 2020 US presidential election results.svg" (file history lists 778 versions of this file) from Blocked Externally to Doing(WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.

Actually, grepping for that request ID in FileOperation.log shows evidence of MediaWiki doing many, many requests to Swift in those 200 seconds. So maybe I was too quick to blame it on Swift. There are about 3000 lines like

Mar 10 2021, 3:28 AM · MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Platform Team Workboards (Clinic Duty Team), MediaWiki-Page-deletion, Commons
tstarling moved T276017: Error: 503, Backend fetch failed while attempting to delete "c:File:Tholden28 test, 2020 US presidential election results.svg" (file history lists 778 versions of this file) from Ready (WIP:5) to Blocked Externally on the Platform Team Workboards (Clinic Duty Team) board.

This is a reproducible timeout in swift. Following is the backtrace from when I tried it.

Mar 10 2021, 2:16 AM · MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Platform Team Workboards (Clinic Duty Team), MediaWiki-Page-deletion, Commons

Mar 9 2021

tstarling added a comment to T275960: Partially failed user suppression left revisions visible, with rev_deleted=0.

The relevant user (user_id=9299073) only has the one revision, there are no suppressed revisions for that user. I'm not sure how the transaction got split up, since neither the current code nor the code in MW 1.15 seems to have explicit transactions, so you would think there would be one transaction covering both the ipblocks insertion and the revision update. It wasn't a race between a new edit and a suppression: the edit occurred at 04:56 and the suppression was at 10:46.

Mar 9 2021, 10:49 PM · Platform Team Workboards (Clinic Duty Team), MediaWiki-extensions-CentralAuth, MediaWiki-Logevents, MediaWiki-User-management
tstarling renamed T275960: Partially failed user suppression left revisions visible, with rev_deleted=0 from Partially failed user suppression left some revisions visible, with rev_deleted=0 to Partially failed user suppression left revisions visible, with rev_deleted=0.
Mar 9 2021, 10:36 PM · Platform Team Workboards (Clinic Duty Team), MediaWiki-extensions-CentralAuth, MediaWiki-Logevents, MediaWiki-User-management
tstarling renamed T275960: Partially failed user suppression left revisions visible, with rev_deleted=0 from User contributions with a non-existent user to Partially failed user suppression left some revisions visible, with rev_deleted=0.
Mar 9 2021, 10:35 PM · Platform Team Workboards (Clinic Duty Team), MediaWiki-extensions-CentralAuth, MediaWiki-Logevents, MediaWiki-User-management
tstarling added a comment to T275960: Partially failed user suppression left revisions visible, with rev_deleted=0.

User suppression is supposed to set the DELETED_RESTRICTED bit in rev_deleted, but rev_deleted is 0 in this case. But are we really talking about a suppression from 2009? The logs have been purged, I can't investigate a possible failed suppression from 2009. If there's not a lot of affected revisions, you can just suppress them manually.

Mar 9 2021, 10:02 PM · Platform Team Workboards (Clinic Duty Team), MediaWiki-extensions-CentralAuth, MediaWiki-Logevents, MediaWiki-User-management
tstarling triaged T276017: Error: 503, Backend fetch failed while attempting to delete "c:File:Tholden28 test, 2020 US presidential election results.svg" (file history lists 778 versions of this file) as High priority.

I tried deleting it and confirmed that it's slow.

Mar 9 2021, 9:33 PM · MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Platform Team Workboards (Clinic Duty Team), MediaWiki-Page-deletion, Commons
tstarling added a project to T276682: Featured article is broken for bs.wiki: MediaWiki-extensions-FeaturedFeeds.

Tagging FeaturedFeeds since that's what it looks like from the message names in the task description.

Mar 9 2021, 9:22 PM · Wikifeeds, MediaWiki-extensions-FeaturedFeeds, Wikipedia-iOS-App-Backlog, Product-Infrastructure-Team-Backlog, Wikipedia-Android-App-Backlog
tstarling added a comment to T267270: Determine multi-dc strategy for CentralAuth.
  • How do our local and global sessions work currently? What makes it so that if I come back after two days to a wiki I haven't been on in a few days that I'm still (properly) logged in (e.g. not just triggering auto-login).
Mar 9 2021, 6:52 AM · MW-1.36-notes (1.36.0-wmf.36; 2021-03-23), Platform Team Workboards (Clinic Duty Team), Performance-Team (Radar), serviceops, Sustainability (MediaWiki-MultiDC), Code-Health-Objective, Platform Team Initiatives (Session Management Service (CDP2))

Mar 8 2021

tstarling removed a project from T276739: Get Wikimedia added to disconnect.me entity list: Browser-Support-Apple-Safari.
Mar 8 2021, 4:51 AM · Platform Team Workboards (Clinic Duty Team), User-notice, Browser-Support-Firefox, MediaWiki-extensions-CentralAuth
tstarling added a comment to T276739: Get Wikimedia added to disconnect.me entity list.

I think the criteria should be:

Mar 8 2021, 4:18 AM · Platform Team Workboards (Clinic Duty Team), User-notice, Browser-Support-Firefox, MediaWiki-extensions-CentralAuth
tstarling created T276739: Get Wikimedia added to disconnect.me entity list.
Mar 8 2021, 3:32 AM · Platform Team Workboards (Clinic Duty Team), User-notice, Browser-Support-Firefox, MediaWiki-extensions-CentralAuth

Mar 5 2021

tstarling lowered the priority of T226797: Login state not propagated across domains in Safari and in Firefox with strict mode from High to Medium.
Mar 5 2021, 4:37 AM · Platform Engineering Roadmap Decision Making, User-notice, Browser-Support-Firefox, Browser-Support-Apple-Safari, MediaWiki-extensions-CentralAuth
tstarling added a comment to T226797: Login state not propagated across domains in Safari and in Firefox with strict mode.

Indeed, in a default installation it still appears to work. I was able to reproduce the problem in a new Firefox profile by enabling "strict" enhanced tracking protection. And I was able to permit cross-site login by disabling tracking protection for en.wikipedia.org via the shield icon.

Mar 5 2021, 2:51 AM · Platform Engineering Roadmap Decision Making, User-notice, Browser-Support-Firefox, Browser-Support-Apple-Safari, MediaWiki-extensions-CentralAuth

Mar 4 2021

tstarling moved T248686: Allow subclasses of ApiBase to be used in phpunit tests without MediaWiki being initialized from External Code Review In Progress to External Code Review Completed on the Platform Team Workboards (External Code Reviews) board.
Mar 4 2021, 10:58 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), Platform Team Workboards (External Code Reviews), Testing-Roadblocks, MediaWiki-Core-Testing, MediaWiki-API
tstarling added a comment to T226797: Login state not propagated across domains in Safari and in Firefox with strict mode.

I see this is getting some heat. I think we need a power-user workaround, involving editing about:config.

Mar 4 2021, 6:46 AM · Platform Engineering Roadmap Decision Making, User-notice, Browser-Support-Firefox, Browser-Support-Apple-Safari, MediaWiki-extensions-CentralAuth
tstarling closed T276287: Excimer segfaults when compiled in ZTS mode as Resolved.
Mar 4 2021, 4:33 AM · Excimer
tstarling added a comment to T206591: Run LuaSandbox tests against PHP with ZTS.

I did that manual testing, it seems to work. Sorry for the noise on a mostly irrelevant task.

Mar 4 2021, 4:31 AM · Continuous-Integration-Config, LuaSandbox
tstarling added a comment to T206591: Run LuaSandbox tests against PHP with ZTS.

We need to do some manual testing before Excimer is released, like a high-concurrency multi-threaded stress test, since Remi is questioning whether it will work even after https://gerrit.wikimedia.org/r/c/mediawiki/php/excimer/+/667970 . Remi is saying we should just make it throw an error during configure "excimer does not support ZTS".

Mar 4 2021, 1:11 AM · Continuous-Integration-Config, LuaSandbox

Mar 3 2021

tstarling added a comment to T226797: Login state not propagated across domains in Safari and in Firefox with strict mode.

On the upstream bug I asked for architecture advice, since I don't see a way to make our SUL system work with state partitioning. I'm also asking for temporary whitelisting since a full rearchitecture of sign-on (we might call it SUL3) is going to take months. Tim Huang, a relevant developer and co-author of the "Introducing State Partitioning" article on the Mozilla blog, is the "triage owner" of my bug, but it's unclear whether that means he has been notified. The next step for progressing the upstream bug is to make sure Tim Huang is aware of it.

Mar 3 2021, 11:24 PM · Platform Engineering Roadmap Decision Making, User-notice, Browser-Support-Firefox, Browser-Support-Apple-Safari, MediaWiki-extensions-CentralAuth
tstarling added a comment to T226797: Login state not propagated across domains in Safari and in Firefox with strict mode.

I filed the upstream bug https://bugzilla.mozilla.org/show_bug.cgi?id=1696095

Mar 3 2021, 4:41 AM · Platform Engineering Roadmap Decision Making, User-notice, Browser-Support-Firefox, Browser-Support-Apple-Safari, MediaWiki-extensions-CentralAuth
tstarling added a comment to T275591: PHP Warning: Memcached::getMulti(): Memcached constructor was not called.

I tried to reproduce this locally with P14574, which throws an exception once per millisecond, and if an exception occurs during client destruction, it reuses the client. But it didn't turn up anything. Maybe in production?

Mar 3 2021, 3:01 AM · Platform Team Workboards (Clinic Duty Team), MediaWiki-Cache, Wikimedia-production-error
tstarling edited P14574 memcached-excimer.php.
Mar 3 2021, 2:58 AM
tstarling created P14574 memcached-excimer.php.
Mar 3 2021, 2:56 AM
tstarling added a comment to T276287: Excimer segfaults when compiled in ZTS mode.

config.m4 was forcing the compiler option -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1, but that option is broken for shared extensions due to some kind of dynamic linking issue. The assembly generated by gcc -S is correct, but _tsrm_ls_cache ends up at the wrong address.

Mar 3 2021, 12:29 AM · Excimer

Mar 2 2021

tstarling added a comment to T276287: Excimer segfaults when compiled in ZTS mode.

It's a bit comical given the amount of code devoted to supposedly supporting ZTS mode. Evidently I never got around to testing it.

Mar 2 2021, 11:03 PM · Excimer
tstarling created T276287: Excimer segfaults when compiled in ZTS mode.
Mar 2 2021, 10:36 PM · Excimer
tstarling added a comment to T260321: Function calls for built-in functions (orchestrator-only).

Please do not use the Unbreak Now priority for things that are not actually urgent. I am reviewing all tasks with this priority every day.

Mar 2 2021, 5:37 AM · function-orchestrator, WikiLambda, Epic, Abstract Wikipedia team (Phase δ)

Mar 1 2021

tstarling updated subscribers of T275452: addWiki.php throws Wikimedia\Assert\PreconditionException from line 58 of /srv/mediawiki/php-1.36.0-wmf.31/includes/dao/WikiAwareEntityTrait.php: Expected User to belong to 'altwiki', but it belongs to the local wiki on writing main page.

I think the best way to test it is to actually run it for real in production.

I bet you know it, but I note it anyway: People should not run it unless they really know what they're doing.

Mar 1 2021, 9:49 PM · MW-1.36-notes (1.36.0-wmf.33; 2021-03-02), Patch-For-Review, Platform Team Workboards (MW Expedition), Wiki-Setup, MediaWiki-extensions-WikimediaMaintenance
tstarling lowered the priority of T275453: addWiki.php warns Deprecated: Premature access to HookContainer, ObjectFactory and ServiceContainer from High to Medium.

I'm lowering the priority since it's hard to imagine some deprecation warnings preventing you from creating wikis. The actual blocker is presumably T275452.

Mar 1 2021, 6:37 AM · Platform Team Workboards (Clinic Duty Team), Wiki-Setup, MediaWiki-extensions-WikimediaMaintenance
tstarling added a comment to T275452: addWiki.php throws Wikimedia\Assert\PreconditionException from line 58 of /srv/mediawiki/php-1.36.0-wmf.31/includes/dao/WikiAwareEntityTrait.php: Expected User to belong to 'altwiki', but it belongs to the local wiki on writing main page.

I think the best way to test it is to actually run it for real in production. In principle, the script should be re-runnable if it fails halfway through. But don't use altwiki since that wiki seems to be fully set up and working fine. Which wikis need to be created?

Mar 1 2021, 6:20 AM · MW-1.36-notes (1.36.0-wmf.33; 2021-03-02), Patch-For-Review, Platform Team Workboards (MW Expedition), Wiki-Setup, MediaWiki-extensions-WikimediaMaintenance
tstarling renamed T275453: addWiki.php warns Deprecated: Premature access to HookContainer, ObjectFactory and ServiceContainer from addWiki.php throws PreconditionException from WikiAwareEntityTrait to addWiki.php warns Deprecated: Premature access to HookContainer, ObjectFactory and ServiceContainer.
Mar 1 2021, 6:07 AM · Platform Team Workboards (Clinic Duty Team), Wiki-Setup, MediaWiki-extensions-WikimediaMaintenance
tstarling added a comment to T275453: addWiki.php warns Deprecated: Premature access to HookContainer, ObjectFactory and ServiceContainer.

Oh right, there is a separate task for the exception, which was fixed already.

Mar 1 2021, 6:06 AM · Platform Team Workboards (Clinic Duty Team), Wiki-Setup, MediaWiki-extensions-WikimediaMaintenance