Page MenuHomePhabricator

aaron (Aaron Schulz)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Oct 20 2014, 5:25 PM (235 w, 1 d)
Availability
Available
IRC Nick
AaronSchulz
LDAP User
Aaron Schulz
MediaWiki User
Aaron Schulz [ Global Accounts ]

Recent Activity

Fri, Apr 19

aaron added a comment to T212550: Implement support for ChronologyProtection in RDF export.

@aaron Is there any docs how to use the client ID on the client side? I see there's support for cpPosIndex cookie which is $index@$time#$clientId but if I have only client ID that is not going to help me. So I am not sure how would one use it - I can't find any trace of ChronologyClientId HTTP header.

Fri, Apr 19, 9:40 PM · MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), Performance-Team, Patch-For-Review, User-Smalyshev, Wikidata, Wikidata-Query-Service
aaron added a comment to T212550: Implement support for ChronologyProtection in RDF export.

Another question for @aaron as well - If we start storing this chronologyprotector field in kafka etc. that means it can survive longer and no longer has an expiry like we do for cookies. I vaguely recall an issue in the past where clients that kept the value too long caused subsequent requests to stall for many seconds as MW mistook the unknown value for a value that hasn't appeared yet.

It may've been mitigated already, but if not, we'll need to sort that out before this goes live.

Fri, Apr 19, 9:38 PM · MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), Performance-Team, Patch-For-Review, User-Smalyshev, Wikidata, Wikidata-Query-Service
aaron committed rECOGccf2f475022e: Fix Phan errors with return values and Database type hints (authored by aaron).
Fix Phan errors with return values and Database type hints
Fri, Apr 19, 9:21 PM
aaron committed rECOGbd14541219fc: Fix Phan errors with Database type hints (authored by aaron).
Fix Phan errors with Database type hints
Fri, Apr 19, 8:54 PM

Thu, Apr 18

aaron committed rECOG0ef22f81e73f: Make CacheUpdateJob super call explicitly pass the empty parameters (authored by aaron).
Make CacheUpdateJob super call explicitly pass the empty parameters
Thu, Apr 18, 11:45 PM
aaron committed rECOSb11cf0d907a6: Explicitly pass empty parameters to SMWUpdateJob constructor (authored by aaron).
Explicitly pass empty parameters to SMWUpdateJob constructor
Thu, Apr 18, 11:12 PM

Tue, Apr 16

aaron added a comment to T216287: CentralNotice must not query master on GET request (page views).

If it's not actually part of a job, it should be DB_REPLICA. Also, even when it should use DB_MASTER it should not be using getCachedGroupDefinitions() since it should skip the cache.

Tue, Apr 16, 10:00 PM · Fundraising-Backlog, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Wikimedia-production-error, MediaWiki-extensions-CentralNotice, Beta-Cluster-reproducible
aaron added a comment to T212550: Implement support for ChronologyProtection in RDF export.

I think you can just add a method, similar to LBFactory::getChronologyProtectorTouched, that exposes the client ID, maybe call it LBFactory::getChronologyProtectorClientId();

Tue, Apr 16, 4:56 AM · MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), Performance-Team, Patch-For-Review, User-Smalyshev, Wikidata, Wikidata-Query-Service
aaron added a comment to T189821: Use stashedit in 2017 wikitext editor (wikitext mode within the visual editor).

Can someone please prioritize this? This basically means that very few people are using edit stashing on enwiki, which has this editor as the default. That seems like a pretty bad regression from the regular editor being the default.

Tue, Apr 16, 2:15 AM · VisualEditor, Performance-Team (Radar), MediaWiki-Page-editing, VisualEditor-MediaWiki-2017WikitextEditor

Mon, Apr 15

aaron moved T220470: Investigate backend save timing regression starting at 2019-04-08 19:15:00 from Inbox to Doing on the Performance-Team board.
Mon, Apr 15, 8:02 PM · MediaWiki-Page-editing, Performance-Team

Fri, Apr 12

aaron added a comment to T216664: MWException when viewing or comparing certain pages with Preprocessor_DOM (PHP7 beta feature).

Given all the PHP memory/speed enhancements, some benchmarking is probably in order. Maybe BenchmarkParse could use a preprocess-only option. I'm curious how PHP7 compares to HHVM for large pages.

Fri, Apr 12, 1:55 AM · Core Platform Team, Patch-For-Review, PHP 7.2 support, MediaWiki-History-and-Diffs, MediaWiki-Parser, Wikimedia-production-error
aaron placed T204606: Some requests time out after 60 seconds in MobileFrontend transforms. What to do? up for grabs.
Fri, Apr 12, 12:50 AM · Performance-Team (Radar), Wikimedia-production-error, MobileFrontend, Readers-Web-Backlog
aaron added a comment to T204606: Some requests time out after 60 seconds in MobileFrontend transforms. What to do?.

So, I don't really think loading the parsed HTML into a DOM object like this is really acceptable performance-wise, and I don't see any trick to make it fast.

Fri, Apr 12, 12:46 AM · Performance-Team (Radar), Wikimedia-production-error, MobileFrontend, Readers-Web-Backlog

Thu, Apr 11

aaron added a comment to T204606: Some requests time out after 60 seconds in MobileFrontend transforms. What to do?.

I made a version of the page that extracts less djvu pages (but still many) and tried to profile again:

Thu, Apr 11, 6:13 PM · Performance-Team (Radar), Wikimedia-production-error, MobileFrontend, Readers-Web-Backlog
aaron added a comment to T211488: Audit and sync INI settings as needed between HHVM and PHP 7 .
  • Session

    We don't use the default session storage in PHP. But, we do use some of its functions and legacy variables like $_SESSION. I don't know whether some of those might vary their behaviour through these.

I'll be honest, I had no idea either, at least for some of these settings. But from a read of the docs, I don't see how the security ones might have an impact.

What does have an impact is

session.upload_progress.enabled = 1

which will indeed interact with $_SESSION by adding a variable named

session.upload_progress.name = PHP_SESSION_UPLOAD_PROGRESS

in the session to track file uploads progress.

Do we want this to be disabled? I'm not sure, maybe it's potentially useful?

Thu, Apr 11, 5:38 PM · PHP 7.2 support, Patch-For-Review, Performance-Team (Radar), Operations

Tue, Apr 9

aaron updated the task description for T220470: Investigate backend save timing regression starting at 2019-04-08 19:15:00.
Tue, Apr 9, 5:57 AM · MediaWiki-Page-editing, Performance-Team
aaron created T220470: Investigate backend save timing regression starting at 2019-04-08 19:15:00.
Tue, Apr 9, 5:34 AM · MediaWiki-Page-editing, Performance-Team

Mon, Apr 8

aaron added a comment to T220312: categories are broken on sqlite as a result of calling beginMasterChanges() prior to the job insert being comitted.

I can't reproduce this on cf8f1fe9b0b5647d1ed3955f6ea6bce7d76fed44.

Mon, Apr 8, 7:16 PM · SQLite, MediaWiki-Database
aaron added a comment to T218207: Use disk-based LCStore by default in MediaWiki 1.34.

Have you compared LCStore with sqlite (as defined by the current installer) vs cdb?

Mon, Apr 8, 6:09 PM · Performance-Team, Core Platform Team Backlog (Later), Core Platform Team (Security, stability, performance and scalability (TEC1)), Language-Team, MediaWiki-Internationalization, MW-1.34-release

Sun, Apr 7

aaron committed rELGN3b22f0648216: Use IDatabase for type hints rather than Database (authored by aaron).
Use IDatabase for type hints rather than Database
Sun, Apr 7, 1:17 AM
aaron committed rELGNa61b9d958078: Use IDatabase for type hints rather than Database (authored by aaron).
Use IDatabase for type hints rather than Database
Sun, Apr 7, 12:52 AM
aaron committed rELGNb6e4fa17f557: Use IDatabase for type hints rather than Database (authored by aaron).
Use IDatabase for type hints rather than Database
Sun, Apr 7, 12:52 AM

Fri, Apr 5

Mill <mill@mail.com> committed rEWCS72831de6c663: gaaaaaaaaaaaaa (authored by aaron).
gaaaaaaaaaaaaa
Fri, Apr 5, 10:42 PM
aaron added a comment to T210725: Replace parsercache keys to something more meaningful on db-XXXX.php.

I would like to elaborate more on my idea on how to deploy the change to eqiad for the first key (the one we already changed in codfw):

Someday at 05:00AM UTC

  • Merge: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/499737/
  • Run scap pull on the canaries: ` `
  • Create a manual lock on deploy1001 by doing: touch /var/lock/scap.operations_mediawiki-config.lock
  • Check the parsercache graphs, errors, browse the sites using mwdebug1001 and mwdebug1002, keep checking for errors.
  • After 3-4 hours, if all goes good and nothing weird is seeing on graphs and logs, remove the lock.
  • Do a normal scap sync-file to deploy the key everywhere.
Fri, Apr 5, 6:12 PM · Patch-For-Review, MediaWiki-Cache, Performance-Team (Radar), DBA, User-Marostegui
aaron added a comment to T137900: Deal with poor edit stash hit rate due to Lua modules using {{REVISIONID}}.

What about {{REVISIONTIMESTAMP}}?

Fri, Apr 5, 7:13 AM · MediaWiki-Parser, Performance-Team, User-notice, Parsoid

Mon, Apr 1

aaron closed T179760: File metadata gets refreshed when accessing the file description page as Resolved.
Mon, Apr 1, 7:54 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Performance-Team, Patch-For-Review, Multimedia, Commons, MediaWiki-File-management
aaron closed T179760: File metadata gets refreshed when accessing the file description page, a subtask of T92357: Fix problematic database master queries performed on HTTP GET/HEAD, as Resolved.
Mon, Apr 1, 7:54 PM · Availability (MediaWiki-MultiDC), Patch-For-Review, MediaWiki-General-or-Unknown

Fri, Mar 29

aaron committed rELGN75773a929b2c: Clean up $params argument to Job subclasses to type hint an array (authored by aaron).
Clean up $params argument to Job subclasses to type hint an array
Fri, Mar 29, 6:47 PM
aaron closed T138619: TranslateMetadata::get does full table scans unnecessarily as Resolved.

Some cases still tale scan, but those are pages or API modules that aggregate everything, so it's not avoidable there.

Fri, Mar 29, 6:24 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Performance-Team (Radar), Patch-For-Review, MediaWiki-extensions-Translate
aaron renamed T219592: Frequent Echo DB_MASTER write queries on HTTP GET from Frequent Echo queries on HTTP GET to Frequent Echo DB_MASTER write queries on HTTP GET.
Fri, Mar 29, 5:32 PM · Growth-Team, Notifications, Services (watching), Core Platform Team Backlog (Watching / External), Core Platform Team (Multi-DC (TEC1)), Performance-Team (Radar), Availability (MediaWiki-MultiDC)
aaron created T219592: Frequent Echo DB_MASTER write queries on HTTP GET.
Fri, Mar 29, 8:53 AM · Growth-Team, Notifications, Services (watching), Core Platform Team Backlog (Watching / External), Core Platform Team (Multi-DC (TEC1)), Performance-Team (Radar), Availability (MediaWiki-MultiDC)

Thu, Mar 28

aaron committed rEWCS21d297ce4403: [DNM] Jenkins RDBMs test commit (authored by aaron).
[DNM] Jenkins RDBMs test commit
Thu, Mar 28, 11:45 PM
aaron closed T218388: quibble-vendor-mysql-hhvm-docker fails on EchoDiscussionParserTest as Resolved.
Thu, Mar 28, 7:17 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Patch-For-Review, Growth-Team (Current Sprint), Wikimedia-production-error (Shared Build Failure), Notifications
aaron closed T194632: mediawiki-extensions-hhvm-jessie failures on EchoDiscussionParserTest as Resolved.
Thu, Mar 28, 7:16 PM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.21; 2019-03-12), Growth-Team (Current Sprint), Wikimedia-production-error (Shared Build Failure), Notifications
aaron closed T37854: LCStore_DB cache is not refreshed on action=purge as Resolved.

LCStore_DB has gone through numerous changes since this report. For example, with mysql/postgres a separate auto-commit mode connection is used to avoid staleness and contention. The sqlite installer make a dedicated LCStore_DB database file to avoid contention and failed writes.

Thu, Mar 28, 3:17 AM · MediaWiki-Internationalization

Wed, Mar 27

aaron closed T42589: Database "Error: 1213 Deadlock" while running rebuildLocalisationCache.php as Resolved.

Table not used anymore.

Wed, Mar 27, 9:48 PM · MediaWiki-Maintenance-scripts
aaron closed T42589: Database "Error: 1213 Deadlock" while running rebuildLocalisationCache.php, a subtask of T30499: 1205: Lock wait timeout exceeded; try restarting transaction (tracking), as Resolved.
Wed, Mar 27, 9:48 PM · MediaWiki-General-or-Unknown, Technical-Debt, Tracking-Neverending
aaron closed T42589: Database "Error: 1213 Deadlock" while running rebuildLocalisationCache.php, a subtask of T41480: Issues affecting translatewiki.net, as Resolved.
Wed, Mar 27, 9:48 PM · Tracking-Neverending, MediaWiki-General-or-Unknown
aaron added a comment to T100585: "Uncommitted DB writes" errors are getting creepy.

Does this still occur?

Wed, Mar 27, 9:46 PM · Fundraising-Backlog, Recurring-Donations, MediaWiki-extensions-DonationInterface, Fundraising Tech Backlog
aaron added a comment to T210725: Replace parsercache keys to something more meaningful on db-XXXX.php.

FYI, I was playing around with my PHP interpreter last night using:

Wed, Mar 27, 5:09 PM · Patch-For-Review, MediaWiki-Cache, Performance-Team (Radar), DBA, User-Marostegui
aaron added a comment to T210725: Replace parsercache keys to something more meaningful on db-XXXX.php.

They don't seem related nor are actually thrown exceptions (just warnings caught by MWExceptionHandler and logged using Exception objects).

Wed, Mar 27, 7:57 AM · Patch-For-Review, MediaWiki-Cache, Performance-Team (Radar), DBA, User-Marostegui

Tue, Mar 26

aaron closed T218087: Performance review of Extension:WikimediaEditorTasks as Resolved.
Tue, Mar 26, 5:48 PM · MW-1.33-notes (1.33.0-wmf.23; 2019-03-26), DBA, Patch-For-Review, Performance-Team, Reading-Infrastructure-Team-Backlog, WikimediaEditorTasks
aaron closed T218087: Performance review of Extension:WikimediaEditorTasks, a subtask of T212795: Build infrastructure to track counts of qualifying edits for the Suggested Edits feature, as Resolved.
Tue, Mar 26, 5:48 PM · WikimediaEditorTasks, Wikipedia-Android-App-Backlog, Reading-Infrastructure-Team-Backlog (Kanban)
aaron closed T218473: Simultaneous configuring of pending changes and page protection fails to apply page protection as Resolved.

T217469 had a patch deployed... Is the issue on this task fixed, also, maybe?

Tue, Mar 26, 5:17 AM · Performance-Team, MediaWiki-Page-protection, MediaWiki-extensions-FlaggedRevs
aaron added a comment to T218087: Performance review of Extension:WikimediaEditorTasks.

@aaron How are we doing on this? Do you expect for this to be complete in time to enable in production on Thursday of this week? Seems like you've already gone over all of the codebase and made tweaks, but I'm not sure if there's more you had planned.

Tue, Mar 26, 4:58 AM · MW-1.33-notes (1.33.0-wmf.23; 2019-03-26), DBA, Patch-For-Review, Performance-Team, Reading-Infrastructure-Team-Backlog, WikimediaEditorTasks

Mar 24 2019

aaron added a comment to T215850: LoadBalancer.php uses $wgDBname even though $wgDBservers is defined.

You can even just omit the dbname from $wgDBservers. That is also a cleaner config for wiki farms. The servers (masters + X replicas) themselves don't really have a concept of "the table prefix", or even "the database" if it's mysql (rather than sqlite/postgres). Each wiki uses some db/prefix (might be the same DB or per-wiki or grouped or whatever). Calls to LoadBalancer::getConnection() or wfGetDB( DB_REPLICA, [], $wikiDbDomain ) can just use the db/prefix contained in $wikiDBDomain, which is $wgDBname and $wgDBprefix if it's the current wiki.

Mar 24 2019, 7:33 AM · MW-1.33-notes (1.33.0-wmf.23; 2019-03-26), Patch-For-Review, MediaWiki-Database
aaron reopened T193832: JobQueue not working with certain config ($wgLocalDatabases set to the default value of an empty array()) as "Open".

Code should always handle the current wiki (e.g. wfWikiId/$wgDBname) case and not require it to be set in $wgLocalDatabases. The job queue should be fine now (at least in master, 32, and 31). If anything somewhere still has that problem, then it's a bug.

Mar 24 2019, 7:15 AM · MediaWiki-JobQueue

Mar 22 2019

aaron added a comment to T218511: After opening a diff, entry on Special:Watchlist sometimes stays unread (bold).

Thanks, when might that change go live?

Mar 22 2019, 6:02 PM · MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), Patch-For-Review, Growth-Team, MediaWiki-Watchlist
aaron added a comment to T215850: LoadBalancer.php uses $wgDBname even though $wgDBservers is defined.

From T193832:

I don't see a way to avoid having to set $wgDBname. It was already used to determine things like wfWikiId(), which is used by some things to get DB connections. Having them both set and mismatched has long since been a use case that is not well supported. The correspondence needs better documentation though.

Maybe I could detect this in MWLBFactory and error out with an informative message. Mysql does not require specifying a DB in order to get a connection (unlike postgres) and 'dbname' is only used for when the $domain to wfGetDB() is '' (e.g. unspecified). That means that it's pretty much pointless to have it in $wgDBservers now, so if it exists and does not match $wgDBname, an error could be thrown. The 'schema' is always NULL for mysql. I suppose 'tablePrefix' could be checked against $wgDBprefix too.

Mar 22 2019, 7:30 AM · MW-1.33-notes (1.33.0-wmf.23; 2019-03-26), Patch-For-Review, MediaWiki-Database
aaron merged T193832: JobQueue not working with certain config ($wgLocalDatabases set to the default value of an empty array()) into T215850: LoadBalancer.php uses $wgDBname even though $wgDBservers is defined.
Mar 22 2019, 7:23 AM · MW-1.33-notes (1.33.0-wmf.23; 2019-03-26), Patch-For-Review, MediaWiki-Database
aaron merged task T193832: JobQueue not working with certain config ($wgLocalDatabases set to the default value of an empty array()) into T215850: LoadBalancer.php uses $wgDBname even though $wgDBservers is defined.
Mar 22 2019, 7:23 AM · MediaWiki-JobQueue
aaron added a comment to T193832: JobQueue not working with certain config ($wgLocalDatabases set to the default value of an empty array()).

I don't see a way to avoid having to set $wgDBname. It was already used to determine things like wfWikiId(), which is used by some things to get DB connections. Having them both set and mismatched has long since been a use case that is not well supported. The correspondence needs better documentation though.

Mar 22 2019, 5:37 AM · MediaWiki-JobQueue

Mar 21 2019

aaron added a comment to T124196: Fatal "cannot perform this operation with arrays" from CirrusSearch/ElasticaWrite (using JobQueueDB).

Why does the job itself have all of the transformed text rather than just a revision/page ID and use them to derive the transformed text? I get that some metadata is not stored elsewhere and would have to go in the job.

Mar 21 2019, 11:28 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Discovery-Search (Current work), Patch-For-Review, Wikimedia-production-error, MediaWiki-JobQueue, User-aude, Wikidata, CirrusSearch, Discovery
aaron committed rEWET022e63944114: Use the right DB handles for the deferred updates (authored by aaron).
Use the right DB handles for the deferred updates
Mar 21 2019, 9:38 PM
aaron added a comment to T218087: Performance review of Extension:WikimediaEditorTasks.

66 millon rows is a small table, the size itself does not worry me, how writes when updating the table works does, specially with wikidata, where normally multiple writes of the same item happens at the same time.

Mar 21 2019, 7:47 PM · MW-1.33-notes (1.33.0-wmf.23; 2019-03-26), DBA, Patch-For-Review, Performance-Team, Reading-Infrastructure-Team-Backlog, WikimediaEditorTasks
aaron closed T33534: SQL errors with corrupted table name aliases Database::select() generated queries as Declined.
Mar 21 2019, 6:41 AM · MediaWiki-Database
Mill <mill@mail.com> committed rEWETd4d9a6d550da: ynaaaaaaaaaaaa (authored by aaron).
ynaaaaaaaaaaaa
Mar 21 2019, 12:27 AM
aaron committed rEWET82fe1c8d0bd0: Fix various bugs and make some optimizations (authored by aaron).
Fix various bugs and make some optimizations
Mar 21 2019, 12:27 AM
aaron committed rEWETb11751e9d7b1: Fix various bugs and make some optimizations (authored by aaron).
Fix various bugs and make some optimizations
Mar 21 2019, 12:27 AM
aaron committed rEWET9ae285d53cca: Small fixes and tweaks to PopulateEntityDescriptionExistsTable (authored by aaron).
Small fixes and tweaks to PopulateEntityDescriptionExistsTable
Mar 21 2019, 12:27 AM
aaron committed rEWCS5e1a672b3572: [DNM] Jenkins RDBMs test commit (authored by aaron).
[DNM] Jenkins RDBMs test commit
Mar 21 2019, 12:25 AM
aaron committed rEWCSfad93f5b8796: [DNM] Jenkins RDBMs test commit (authored by aaron).
[DNM] Jenkins RDBMs test commit
Mar 21 2019, 12:25 AM
aaron committed rEWCS5bb456a64b18: [DO NOT MERGE] Test for broken Echo tests (authored by Catrope).
[DO NOT MERGE] Test for broken Echo tests
Mar 21 2019, 12:25 AM
aaron committed rEWCS392046a57a97: [DO NOT MERGE] Test for broken Echo tests (authored by Catrope).
[DO NOT MERGE] Test for broken Echo tests
Mar 21 2019, 12:25 AM
aaron committed rEWCSff01cb60f2fe: [DNM] Jenkins RDBMs test commit (authored by aaron).
[DNM] Jenkins RDBMs test commit
Mar 21 2019, 12:25 AM
aaron committed rEWCSa231b456703d: [DNM] Jenkins RDBMs test commit (authored by aaron).
[DNM] Jenkins RDBMs test commit
Mar 21 2019, 12:25 AM
aaron committed rEWCSb8dc19d34ede: [DNM] Jenkins RDBMs test commit (authored by aaron).
[DNM] Jenkins RDBMs test commit
Mar 21 2019, 12:25 AM

Mar 20 2019

aaron updated subscribers of T218087: Performance review of Extension:WikimediaEditorTasks.

The wetede_rand index seems like it will need to be changed as I mentioned, given the size. Otherwise, there are two many cases that could involve massing scanning.

Mar 20 2019, 7:01 PM · MW-1.33-notes (1.33.0-wmf.23; 2019-03-26), DBA, Patch-For-Review, Performance-Team, Reading-Infrastructure-Team-Backlog, WikimediaEditorTasks
aaron added a comment to T193832: JobQueue not working with certain config ($wgLocalDatabases set to the default value of an empty array()).

The $wiki !== wfWikiID() check was meant for the local wiki case (as wgLocalDatabases is either fully set for some farms and often empty for smaller sites).

Mar 20 2019, 7:30 AM · MediaWiki-JobQueue

Mar 19 2019

aaron updated subscribers of T203913: User.php: Cannot create a user with no name, no ID, and no actor ID (Unable to delete certain pages on Chinese Wikisource).

@Tgr might know better.

Mar 19 2019, 5:41 PM · Growth-Team, MediaWiki-Page-deletion, MediaWiki-User-management, Wikimedia-production-error, Chinese-Sites
aaron added a comment to T218388: quibble-vendor-mysql-hhvm-docker fails on EchoDiscussionParserTest.

Maybe the User::saveSettings() call has its clearSharedCache() call deferred in the old code, due to hasOrMadeRecentMasterChanges(), but is immediate in the new code. If the tests relied on the old cached value being there, then they could fail.

Mar 19 2019, 4:56 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Patch-For-Review, Growth-Team (Current Sprint), Wikimedia-production-error (Shared Build Failure), Notifications
aaron added a comment to T218388: quibble-vendor-mysql-hhvm-docker fails on EchoDiscussionParserTest.

Actually, given the nature of tests all using DB_MASTER, I don't yet see what would be significantly affected here. It can't be hasMasterChanges() since that wouldn't have changed, doneWrites() is not used, lastDoneWrites() is only used by lastMasterChangeTimestamp(). That in turn is used by hasOrMadeRecentMasterChanges() and waitForReplication().

Mar 19 2019, 4:36 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Patch-For-Review, Growth-Team (Current Sprint), Wikimedia-production-error (Shared Build Failure), Notifications
aaron added a comment to T218388: quibble-vendor-mysql-hhvm-docker fails on EchoDiscussionParserTest.

I'd imagine --use-normal tables would pass and not having that would fail. In production, there is no reason to treat temporary table operations as writes (e.g. for triggering commitMasterChanges as so on). They would just be used for complex reads and accumulated results. For tests, without --use-normal-tables, everything is temporary (though it it's meant to represent permanent stuff). Even still, this rarely matters, though things like User do check hasOrMadeRecentMasterChanges(), which would hit the edge case of everything being temporary (no longer returning true).

Mar 19 2019, 4:25 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Patch-For-Review, Growth-Team (Current Sprint), Wikimedia-production-error (Shared Build Failure), Notifications

Mar 18 2019

aaron added a project to T218473: Simultaneous configuring of pending changes and page protection fails to apply page protection: Performance-Team.
Mar 18 2019, 10:28 PM · Performance-Team, MediaWiki-Page-protection, MediaWiki-extensions-FlaggedRevs
aaron closed T218226: Improve Database connection loss handling as Resolved.
Mar 18 2019, 10:27 PM · MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), Performance-Team, Patch-For-Review, MediaWiki-Database

Mar 17 2019

aaron added a comment to T202149: Exception thrown for failure to save settings appears ~ 1000 times/day.

Seems to rarely occur (usually either 0 times per 30 minutes or 1 time per 30 minutes).

Mar 17 2019, 5:33 AM · MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), Patch-For-Review, BetaFeatures, Wikimedia-production-error, MediaWiki-Authentication-and-authorization

Mar 16 2019

aaron added a comment to T218473: Simultaneous configuring of pending changes and page protection fails to apply page protection.

Seems like it might be T217649 (which should be re-titled itself).

Mar 16 2019, 5:54 PM · Performance-Team, MediaWiki-Page-protection, MediaWiki-extensions-FlaggedRevs

Mar 15 2019

aaron closed T188801: Migrate wl_notificationtimestamp updates to the job queue as Resolved.

Report on WP:VP/T:

Today is Thursday. And now when I visit a link from my watchlist, and then go back to the watchlist, the green bullet stays green instead of turning blue. This problem occurs even for deleted redlinked pages.

Might be related to this change ?

Mar 15 2019, 5:11 PM · MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), Growth-Team, MediaWiki-Watchlist, Patch-For-Review, Availability (MediaWiki-MultiDC)
aaron closed T188801: Migrate wl_notificationtimestamp updates to the job queue, a subtask of T92357: Fix problematic database master queries performed on HTTP GET/HEAD, as Resolved.
Mar 15 2019, 5:11 PM · Availability (MediaWiki-MultiDC), Patch-For-Review, MediaWiki-General-or-Unknown
aaron claimed T218226: Improve Database connection loss handling.
Mar 15 2019, 8:25 AM · MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), Performance-Team, Patch-For-Review, MediaWiki-Database

Mar 13 2019

aaron added a comment to T215107: Global rename of The_Photographer → Wilfredor: supervision needed.

So that's an impressive cascade of failures:

  • Updates for the image table are not batched so the query times out. (Why doesn't the DB error have a stack trace? I thoght we are logging that.)
Mar 13 2019, 4:03 PM · MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), Patch-For-Review, User-MarcoAurelio, DBA, Wikimedia-Site-requests
aaron created T218226: Improve Database connection loss handling.
Mar 13 2019, 4:00 PM · MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), Performance-Team, Patch-For-Review, MediaWiki-Database
aaron added a comment to T218197: CacheAwarePropertyInfoStore & CachingPropertyInfoLookup should use WANObjectCache instead of BagOStuff.

Yes, and CacheAwarePropertyInfoStore should use delete() or such for purges rather than set(). Using set() would only effect one DC.

Mar 13 2019, 2:53 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Patch-For-Review, User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata
aaron added a comment to T217731: Consider raising Memcached MWObject cache memory size limit.

Can the (extra) space be dedicated more so towards the larger slabs, were we have more problems AFAIK?

Mar 13 2019, 2:50 PM · Patch-For-Review, Performance-Team (Radar), User-Elukey, MediaWiki-Cache, Operations
aaron committed rEWETb9e8d19f0dd5: [WIP] Fix various bugs and make some optimizations (authored by aaron).
[WIP] Fix various bugs and make some optimizations
Mar 13 2019, 5:44 AM
aaron added a comment to T218087: Performance review of Extension:WikimediaEditorTasks.

How will wikimedia_editor_tasks_entity_description_exists be populated and about how large will it be? Can wetede_language actually go up to 255 length? If the table can be huge, then I also wonder if the wetede_rand index should be on (wetede_language,wetede_description_exists,wetede_rand) instead.

Mar 13 2019, 4:38 AM · MW-1.33-notes (1.33.0-wmf.23; 2019-03-26), DBA, Patch-For-Review, Performance-Team, Reading-Infrastructure-Team-Backlog, WikimediaEditorTasks

Mar 12 2019

aaron added a comment to T138619: TranslateMetadata::get does full table scans unnecessarily.

Is this the same as T204026?

Mar 12 2019, 10:54 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Performance-Team (Radar), Patch-For-Review, MediaWiki-extensions-Translate
aaron added a comment to T204742: Unable to store parser output in StashEdit (Memcached error: ITEM TOO BIG).

It's also useful as a tunable option for spreading I/O accross cache nodes rather than only being used to get around hard limits.

Mar 12 2019, 5:41 AM · MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), Performance-Team, MediaWiki-Page-editing, Patch-For-Review, User-Elukey, Regression, Wikimedia-production-error
aaron added a comment to T204742: Unable to store parser output in StashEdit (Memcached error: ITEM TOO BIG).

Which SQL layer? We have SQLBagOStuff using pcxxxx servers, which I suppose could be co-opted for this. As for ES, I wouldn't want to spam a bunch of one-off blobs in there since it is meant to be append-only. Depending on how much code it is I don't mind a bit of complexity. I'm still experimenting around with different ways to do it, but I don't think it has to be that complex.

Mar 12 2019, 5:41 AM · MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), Performance-Team, MediaWiki-Page-editing, Patch-For-Review, User-Elukey, Regression, Wikimedia-production-error

Mar 9 2019

aaron added a comment to T217942: Argument 2 passed to WatchedItemStore::__construct() must be an instance of JobQueueGroup, instance of HashBagOStuff given, called in /srv/mediawiki/php-master/includes/ServiceWiring.php on line 610.

Does that still happen? I can't see how that would happen in master without files from two versions.

Mar 9 2019, 3:39 AM · Scap, Beta-Cluster-Infrastructure

Mar 8 2019

aaron committed rEPFM38d72ba153b8: Avoid use of deprecated ParserOptions::legacyOptions when possible (authored by aaron).
Avoid use of deprecated ParserOptions::legacyOptions when possible
Mar 8 2019, 10:43 PM
aaron added a comment to T204742: Unable to store parser output in StashEdit (Memcached error: ITEM TOO BIG).

Two pieces, one of them still huge. I'm working on a generic segmentation wrapper for BagOStuff atm, but that will take longer to do (lots of ways to arrange the classes, hard to choice from).

Mar 8 2019, 6:39 PM · MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), Performance-Team, MediaWiki-Page-editing, Patch-For-Review, User-Elukey, Regression, Wikimedia-production-error

Mar 7 2019

aaron added a comment to T211721: Establish an SLA for session storage.

The SET metric for redis is very slow, so wouldn't use 10x that figure.

Mar 7 2019, 10:42 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Patch-For-Review, Core Platform Team Backlog (Later), Performance-Team (Radar), TechCom, Services (next), Operations, User-Clarakosi, Core Platform Team (Session Management Service (CDP2)), User-Eevans
Ladsgroup awarded T188801: Migrate wl_notificationtimestamp updates to the job queue a Love token.
Mar 7 2019, 3:39 PM · MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), Growth-Team, MediaWiki-Watchlist, Patch-For-Review, Availability (MediaWiki-MultiDC)
aaron closed T98288: Stop relying on Redis LUA scripting for jobrunner as Resolved.

Job queue has long since been migrated to changeprop

Mar 7 2019, 6:35 AM · Availability
aaron created T217813: Grant root on MediaWiki maintenance hosts to perf-roots.
Mar 7 2019, 1:07 AM · Patch-For-Review, Operations, SRE-Access-Requests

Mar 6 2019

aaron created T217754: Add FileBackend statsd metrics and a dashboard.
Mar 6 2019, 11:44 AM · monitoring, media-storage, Performance-Team

Mar 5 2019

aaron added a comment to T203786: Mcrouter periodically reports soft TKOs for mc1029 (was mc1035, mc1022) leading to MW Memcached exceptions.

It reduces set()/setInterimKey() calls to volatile (recently invalidated) keys. Each of those calls translates into a GET/CAS since they use merge(). I don't think it will prevent the huge GET spikes we saw (since most did not correspond to CAS and were probably not merge() calls).

Mar 5 2019, 7:10 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, Performance-Team (Radar), Wikimedia-production-error, User-Elukey, MediaWiki-Cache, Operations

Mar 4 2019

aaron closed T204423: MW 1.31 install reports "InvalidArgumentException ... DatabaseDomain.php: Domain has too few or too many parts " as Resolved.
Mar 4 2019, 9:25 PM · MW-1.31-release-notes, MW-1.32-notes, Patch-For-Review, Performance-Team, MW-1.31-release, MediaWiki-Database, MediaWiki-Installer
aaron added a comment to T199044: Special:ActiveUsers queries result in fatal error (due to query/request timeout).

Here is a form of the query without filesorts:

Mar 4 2019, 4:52 AM · Core Platform Team Kanban (Done with CPT), MW-1.33-notes (1.33.0-wmf.17; 2019-02-12), Core Platform Team (Security, stability, performance and scalability (TEC1)), Patch-For-Review, Commons, User-Josve05a, MediaWiki-Special-pages, MediaWiki-Database, Wikimedia-production-error