tstarling (Tim Starling)Administrator
User

Projects (16)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Oct 15 2014, 8:27 PM (200 w, 3 d)
Roles
Administrator
Availability
Available
LDAP User
Tim Starling
MediaWiki User
Tim Starling (WMF) [ Global Accounts ]

Recent Activity

Yesterday

tstarling added a comment to T202107: Job queue should not overload the DB servers when there is replication lag.

I can stop replication or partially bring one server down and show it, but @tstarling won't let me for now.

Fri, Aug 17, 10:35 AM · Patch-For-Review, ChangeProp, Services, Availability
tstarling added a comment to T202107: Job queue should not overload the DB servers when there is replication lag.

I think I've found the correct configuration file now, at mediawiki/services/change-propagation/jobqueue-deploy/scap/vars.yaml . I couldn't tell if the concurrency limits are normally reached, and I couldn't figure out how they add up to a global connection count. Looking at current connection counts from scb* to jobrunner.svc with netstat I see counts of 113, 318, 52, 107. MediaWiki has 60 job types, is it correct to multiply that by 30, which is the top-level concurrency in vars.yaml, and then to adjust for the overridden queue types? 55 classes with 30 connections each plus the 5 overrides would make 1970 connections. Then I multiply this by 4 scb servers, for a total of 7880 maximum connections. Is this correct?

Fri, Aug 17, 5:50 AM · Patch-For-Review, ChangeProp, Services, Availability
tstarling changed the visibility for T202107: Job queue should not overload the DB servers when there is replication lag.
Fri, Aug 17, 1:56 AM · Patch-For-Review, ChangeProp, Services, Availability
tstarling triaged T202107: Job queue should not overload the DB servers when there is replication lag as Normal priority.
Fri, Aug 17, 1:18 AM · Patch-For-Review, ChangeProp, Services, Availability

Thu, Aug 16

tstarling added a comment to T183488: MCR schema migration stage 2: populate new fields.

Current status: everything is done except enwiki and the T202032 wikis. enwiki has about another 49 hours to run.

Thu, Aug 16, 10:05 PM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Multi-Content-Revisions (MCR Deployment), Patch-For-Review, Structured-Data-Commons, Wikidata
tstarling triaged T202032: Duplicate ar_rev_id values in several wikis as Normal priority.
Thu, Aug 16, 1:32 AM · Patch-For-Review, Wikidata, Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Multi-Content-Revisions (MCR Deployment), Structured-Data-Commons
tstarling created P7461 aawikibooks ar_rev_id conflicts.
Thu, Aug 16, 12:06 AM

Wed, Aug 15

tstarling added a comment to T183488: MCR schema migration stage 2: populate new fields.

So how do we end up trying to insert a row for revision 3003 twice?

Wed, Aug 15, 10:23 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Multi-Content-Revisions (MCR Deployment), Patch-For-Review, Structured-Data-Commons, Wikidata

Tue, Aug 14

tstarling added a comment to T183488: MCR schema migration stage 2: populate new fields.

You can see the full logs at mwmaint1001:/var/log/mediawiki/populateContentTables/ . On both aawikibooks and gotwikibooks, the error occurred on the second batch of the archive table, starting at ar_rev_id 2001. In both cases it was also the last batch, with the maximum ar_rev_id being 3275 and 3175 respectively.

Tue, Aug 14, 10:32 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Multi-Content-Revisions (MCR Deployment), Patch-For-Review, Structured-Data-Commons, Wikidata
tstarling closed T200881: Create ParserFactory service as Resolved.
Tue, Aug 14, 6:56 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, MediaWiki-Parser, Core-Platform-Team (CPT-Q1-Jul-Sep-2018)
tstarling closed T200246: Introduce ContentLanguage service to replace $wgContLang as Resolved.
Tue, Aug 14, 6:54 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MediaWiki-General-or-Unknown, Technical-Debt
tstarling closed T200246: Introduce ContentLanguage service to replace $wgContLang, a subtask of T160815: Deprecate $wgContLang, as Resolved.
Tue, Aug 14, 6:54 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, Technical-Debt (Deprecation), MediaWiki-General-or-Unknown
tstarling closed T110209: Maintenance scripts should fail on unknown parameters as Resolved.
Tue, Aug 14, 6:52 AM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Patch-For-Review, Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Wikimedia-Incident, Incident-20150825-Redis, MediaWiki-Maintenance-scripts
tstarling moved T200864: Pingback on non-MySQL databases fails to save to updatelog, generates a high rate of unique pings from In Progress to Done on the Core-Platform-Team (CPT-Q1-Jul-Sep-2018) board.
Tue, Aug 14, 6:50 AM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), MediaWiki-General-or-Unknown, MW-1.31-release, Patch-For-Review, Core-Platform-Team (CPT-Q1-Jul-Sep-2018)
tstarling moved T200861: Web upgrade of SQLite does not work, just skips to install from In Progress to Done on the Core-Platform-Team (CPT-Q1-Jul-Sep-2018) board.
Tue, Aug 14, 6:50 AM · MW-1.31-release-notes, MW-1.30-release-notes, MW-1.29-release-notes, MW-1.31-release, MW-1.30-release, MW-1.29-release, MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Patch-For-Review, MediaWiki-Database, SQLite, Core-Platform-Team (CPT-Q1-Jul-Sep-2018)
tstarling added a comment to T193565: Foreign query for metawiki fails with "Table 'centralauth.page' doesn't exist" (DBConnRef mixup?).

I tried importing a file into testwiki with curl, forcing a centralauth DB connection in the same request by first deleting the global:centralauth-user:... cache key, still could not reproduce.

Tue, Aug 14, 6:45 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Wikimedia-log-errors, MediaWiki-Database
tstarling added a comment to T183488: MCR schema migration stage 2: populate new fields.

Log summary:

Tue, Aug 14, 5:32 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Multi-Content-Revisions (MCR Deployment), Patch-For-Review, Structured-Data-Commons, Wikidata
tstarling added a comment to T183488: MCR schema migration stage 2: populate new fields.

@tstarling Please stop writes going to *s2* unless they have already finished

Tue, Aug 14, 5:12 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Multi-Content-Revisions (MCR Deployment), Patch-For-Review, Structured-Data-Commons, Wikidata

Mon, Aug 13

tstarling updated subscribers of T193565: Foreign query for metawiki fails with "Table 'centralauth.page' doesn't exist" (DBConnRef mixup?).

It doesn't have to be a LoadBalancer bug, it could just be some other extension calling reuseConnection() inappropriately. It's hard to debug without a reproduction procedure. I see in the logs that there was a series of these on 2018-08-06 with URL https://sat.wikipedia.org/w/index.php?title=%E1%B1%9F%E1%B1%A5%E1%B1%9A%E1%B1%A0%E1%B1%9F%E1%B1%AD:Import&action=submit , and the failed query indicates that the user was @MF-Warburg , who did have successful file upload imports at that time in the logs: https://sat.wikipedia.org/wiki/%E1%B1%9F%E1%B1%A5%E1%B1%9A%E1%B1%A0%E1%B1%9F%E1%B1%AD:Log/import

Mon, Aug 13, 6:54 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Wikimedia-log-errors, MediaWiki-Database
tstarling added a comment to T201799: Should ParserFactory call firstCallInit()?.

It's important to avoid running it on requests that don't need it. In particular, requests that only call $wgParser->setHook() but not Parser::parse() should not call firstCallInit(). Maybe the risk of that is fading but my understanding is that it's not quite gone yet.

Mon, Aug 13, 4:15 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MediaWiki-Parser

Sat, Aug 11

tstarling added a comment to T194697: Multi-blocks — Allow for multiple, simultaneously blocks with different expiration dates..
  1. Is there a reason why bt_auto exists in block_target instead of block_entry. It feels to me that autoblocks can be just another entry
Sat, Aug 11, 11:43 PM · MediaWiki-User-management, Anti-Harassment

Thu, Aug 9

tstarling added a comment to T134976: SpecialRecentChangesLinked::doMainQuery blocking database infrastructure.

The initial report showed a query which didn't even use ORES, so it seems unfair to assign it to them.

Thu, Aug 9, 4:38 AM · Growth-Team, MediaWiki-Recent-changes, Core-Platform-Team, Wikimedia-log-errors
tstarling added a comment to T194697: Multi-blocks — Allow for multiple, simultaneously blocks with different expiration dates..

Here's my proposal.

Thu, Aug 9, 1:01 AM · MediaWiki-User-management, Anti-Harassment

Wed, Aug 8

jcrespo awarded T201482: LinksUpdate fails, spams exception logs, whenever replication lag on any server rises above 10s a Love token.
Wed, Aug 8, 6:49 AM · Patch-For-Review, Performance-Team (Radar), Core-Platform-Team, MediaWiki-Database
tstarling renamed T59186: Drop blob_tracking and blob_orphans everywhere from blob_tracking indexes apparently unused to Drop blob_tracking and blob_orphans everywhere.
Wed, Aug 8, 6:28 AM · Patch-For-Review, DBA, MediaWiki-Database
tstarling added a comment to T201240: Transaction timeout for LinksUpdate::updateLinksTimestamp (SET page_links_updated) .

We have debug logs for this request. On mwlog1001 do zgrep W2XVZApAAC4AAEKMbQAAAAAV /srv/mw-log/archive/test2wiki.log-20180805.gz

Wed, Aug 8, 4:35 AM · Performance-Team, Core-Platform-Team, Regression, Wikimedia-log-errors, MediaWiki-Page-editing
tstarling closed T198049: Investigate possible outage on wikidata on 25th June - 04:13AM UTC - 05:27AM UTC as Resolved.
Wed, Aug 8, 2:37 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Wikidata, Operations
tstarling created T201482: LinksUpdate fails, spams exception logs, whenever replication lag on any server rises above 10s.
Wed, Aug 8, 2:34 AM · Patch-For-Review, Performance-Team (Radar), Core-Platform-Team, MediaWiki-Database
tstarling created T201481: API maxlag stats.
Wed, Aug 8, 2:07 AM · Core-Platform-Team, MediaWiki-API
tstarling added a comment to T198049: Investigate possible outage on wikidata on 25th June - 04:13AM UTC - 05:27AM UTC.

db1071, the master, had no writes

Wed, Aug 8, 1:48 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Wikidata, Operations

Tue, Aug 7

tstarling added a comment to T198049: Investigate possible outage on wikidata on 25th June - 04:13AM UTC - 05:27AM UTC.

The drop may have been caused by the API maxlag parameter. Wikidata:Bots recommends using a maxlag parameter, and some client libraries set maxlag=5 by default. The point of this feature is to make bots pause during replication lag, to prioritise human users and avoid worsening the situation.

Tue, Aug 7, 11:57 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Wikidata, Operations
tstarling added a comment to T198049: Investigate possible outage on wikidata on 25th June - 04:13AM UTC - 05:27AM UTC.

51,715 exceptions with:

[{exception_id}] {exception_url} Wikimedia\Rdbms\DBReplicationWaitError from line 426 of /srv/mediawiki/php-1.32.0-wmf.8/includes/libs/rdbms/lbfactory/LBFactory.php: Could not wait for replica DBs to catch up to db1071
Tue, Aug 7, 6:14 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Wikidata, Operations
tstarling assigned T182748: $wgExternalDiffEngine should have shell restrictions to BPirkle.
Tue, Aug 7, 2:11 AM · Core-Platform-Team, MediaWiki-History-or-Diffs, MediaWiki-Shell
tstarling assigned T179901: Create a tmp directory just for MediaWiki to BPirkle.
Tue, Aug 7, 2:11 AM · Core-Platform-Team, Security-Core, MediaWiki-General-or-Unknown
tstarling assigned T198176: Mediawiki page deletions should happen in batches of revisions to BPirkle.
Tue, Aug 7, 2:00 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MediaWiki-Page-deletion

Mon, Aug 6

tstarling added a comment to T183488: MCR schema migration stage 2: populate new fields.

@greg The WN31 things are done now, only 1081 seconds for mediawikiwiki and 9252 seconds for metawiki. For metawiki the rate was about the same as anomie got for testwiki, 2000 rows per second for the revision table and 600 rows per second for the archive table. At that rate, we can expect wikidatawiki to take about 91 hours and commonswiki to take about 48 hours. We can run them concurrently since they are on different DB clusters, and that way maybe get them done by the end of the week.

Mon, Aug 6, 10:16 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Multi-Content-Revisions (MCR Deployment), Patch-For-Review, Structured-Data-Commons, Wikidata
tstarling added a comment to T200960: Logstash packet loss.

Back up to ~60% loss now, due to a slow drop in capacity on logstash1008 and logstash1009. And there was a similar event on August 4, which was fixed when @fgiunchedi restarted logstash. Can we have a daily restart cron job now?

Mon, Aug 6, 6:56 AM · Operations, Patch-For-Review, Wikimedia-Logstash
tstarling closed T197816: Enable MCR migration stage "write both, read old" on live systems as Resolved.
Mon, Aug 6, 4:06 AM · Patch-For-Review, Multi-Content-Revisions (MCR Deployment), Structured-Data-Commons, Wikidata
tstarling closed T197816: Enable MCR migration stage "write both, read old" on live systems, a subtask of T183488: MCR schema migration stage 2: populate new fields, as Resolved.
Mon, Aug 6, 4:06 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Multi-Content-Revisions (MCR Deployment), Patch-For-Review, Structured-Data-Commons, Wikidata
tstarling closed T197816: Enable MCR migration stage "write both, read old" on live systems, a subtask of T194750: Deploy Structured Data on Commons baseline , as Resolved.
Mon, Aug 6, 4:06 AM · Multi-Content-Revisions (MCR-SDC File Caption Support - phase 2), Epic, Multimedia-Team-Working-Board, Wikidata, Multimedia, Structured-Data-Commons
tstarling added a comment to T200827: "PHP Notice: Undefined index: N" from RemexHtml/Serializer.

The core patch is enough to kill the warning, at the expense of extra memory usage. Merging the RemexHtml patch, releasing and updating composer will reduce the memory impact.

Mon, Aug 6, 2:40 AM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Patch-For-Review, Parsing-Team, MediaWiki-Parser, RemexHtml, Wikimedia-log-errors

Sat, Aug 4

Liuxinyu970226 awarded T89331: Replace HTML4 Tidy in MW parser with an equivalent HTML5 based tool a Party Time token.
Sat, Aug 4, 2:22 PM · TechCom-RFC (TechCom-Approved), Services (watching), User-notice, Tidy, Proposal, Parsing-Team, Parsoid

Fri, Aug 3

mmodell awarded T200960: Logstash packet loss a Orange Medal token.
Fri, Aug 3, 3:20 PM · Operations, Patch-For-Review, Wikimedia-Logstash
tstarling added a comment to T200827: "PHP Notice: Undefined index: N" from RemexHtml/Serializer.

Short version: RemexCompatMunger case B/b calls endTag() on a p-wrapper which still has children in the stack. There is in fact an effect on the output.

Fri, Aug 3, 11:18 AM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Patch-For-Review, Parsing-Team, MediaWiki-Parser, RemexHtml, Wikimedia-log-errors

Thu, Aug 2

tstarling added a comment to T200827: "PHP Notice: Undefined index: N" from RemexHtml/Serializer.

Reduced test case:

Thu, Aug 2, 11:58 PM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Patch-For-Review, Parsing-Team, MediaWiki-Parser, RemexHtml, Wikimedia-log-errors
tstarling added a comment to T200960: Logstash packet loss.

The logspam patches removed the extra demand caused by the 1.32.0-wmf.10 deployment. Packet loss is now down to ~20%, i.e. still bad but comparable to the long term average of available Prometheus data, which starts in May. There was a drop in successfully consumed packets with this demand drop, which should leave logstash with some capacity headroom. So the next most obvious theory is stalling, and yet the 4MB receive buffer didn't help reduce the packet loss, which is amazing since the bitrate is low, 4MB should correspond to a stall of about 2.7 seconds. It seems to be slow or stalling for even longer periods of time than that.

Thu, Aug 2, 10:56 AM · Operations, Patch-For-Review, Wikimedia-Logstash
tstarling added a comment to T200960: Logstash packet loss.

I tried restarting logstash on logstash1007 with no other change, to avoid confounding the test, then I quadrupled the default receive buffer size (net.core.rmem_default) and restarted it again. The restart alone increased throughput by a factor of 4.7, nice but hardly enough to put a dent in the packet loss graph. The receive buffer change had no effect or a negative effect.

Thu, Aug 2, 7:18 AM · Operations, Patch-For-Review, Wikimedia-Logstash
tstarling added a comment to T183488: MCR schema migration stage 2: populate new fields.

test2wiki and testwikidatawiki are complete.

Thu, Aug 2, 5:44 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Multi-Content-Revisions (MCR Deployment), Patch-For-Review, Structured-Data-Commons, Wikidata
tstarling added a comment to T183488: MCR schema migration stage 2: populate new fields.

Daniel proposed the following schedule:

Thu, Aug 2, 5:29 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Multi-Content-Revisions (MCR Deployment), Patch-For-Review, Structured-Data-Commons, Wikidata
tstarling added a comment to T200960: Logstash packet loss.

There is T200362 for exporting logstash metrics.

Thu, Aug 2, 5:18 AM · Operations, Patch-For-Review, Wikimedia-Logstash
tstarling closed T200963: Cannot save edits at Wikitech-wiki - exception - Wikimedia\Rdbms\DBQueryError as Resolved.

I ran the schema change on labswiki and labtestwiki. I confirmed that no other wiki in all.dblist is missing this schema change. Edits work now. Stashbot died (quit all channels), I restarted it.

Thu, Aug 2, 5:16 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), wikitech.wikimedia.org
Krinkle awarded T200960: Logstash packet loss a Orange Medal token.
Thu, Aug 2, 4:19 AM · Operations, Patch-For-Review, Wikimedia-Logstash
tstarling updated subscribers of T200960: Logstash packet loss.
Thu, Aug 2, 3:37 AM · Operations, Patch-For-Review, Wikimedia-Logstash
tstarling renamed T200960: Logstash packet loss from Logstash has high packet loss since June 29 to Logstash has ~90% packet loss since June 29.
Thu, Aug 2, 3:34 AM · Operations, Patch-For-Review, Wikimedia-Logstash
tstarling created T200960: Logstash packet loss.
Thu, Aug 2, 2:14 AM · Operations, Patch-For-Review, Wikimedia-Logstash

Wed, Aug 1

tstarling added a comment to T199511: Security review for fluidsynth.

I don't know if @tstarling ever did any kind of review of lilypond or timidity.

Wed, Aug 1, 5:07 AM · MediaWiki-extensions-Score, Security-Team, Security-Reviews
tstarling moved T200861: Web upgrade of SQLite does not work, just skips to install from Backlog to In Progress on the Core-Platform-Team (CPT-Q1-Jul-Sep-2018) board.
Wed, Aug 1, 1:35 AM · MW-1.31-release-notes, MW-1.30-release-notes, MW-1.29-release-notes, MW-1.31-release, MW-1.30-release, MW-1.29-release, MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Patch-For-Review, MediaWiki-Database, SQLite, Core-Platform-Team (CPT-Q1-Jul-Sep-2018)
tstarling moved T200864: Pingback on non-MySQL databases fails to save to updatelog, generates a high rate of unique pings from Backlog to In Progress on the Core-Platform-Team (CPT-Q1-Jul-Sep-2018) board.
Wed, Aug 1, 1:34 AM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), MediaWiki-General-or-Unknown, MW-1.31-release, Patch-For-Review, Core-Platform-Team (CPT-Q1-Jul-Sep-2018)
tstarling created T200864: Pingback on non-MySQL databases fails to save to updatelog, generates a high rate of unique pings.
Wed, Aug 1, 1:28 AM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), MediaWiki-General-or-Unknown, MW-1.31-release, Patch-For-Review, Core-Platform-Team (CPT-Q1-Jul-Sep-2018)
tstarling created T200861: Web upgrade of SQLite does not work, just skips to install.
Wed, Aug 1, 12:17 AM · MW-1.31-release-notes, MW-1.30-release-notes, MW-1.29-release-notes, MW-1.31-release, MW-1.30-release, MW-1.29-release, MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Patch-For-Review, MediaWiki-Database, SQLite, Core-Platform-Team (CPT-Q1-Jul-Sep-2018)

Tue, Jul 31

tstarling added a comment to T200759: Security Issue Access Request for CCicalese_WMF.

I gave her WMF-NDA access just now for the same reason.

Tue, Jul 31, 4:39 AM · Security
tstarling created T200759: Security Issue Access Request for CCicalese_WMF.
Tue, Jul 31, 4:38 AM · Security
tstarling added a member for WMF-NDA: CCicalese_WMF.
Tue, Jul 31, 4:33 AM
tstarling moved T180192: Audit $wgExtensionFunction and extension.json callback usage by Wikimedia-deployed extensions from Backlog to In Progress on the Core-Platform-Team (CPT-Q1-Jul-Sep-2018) board.
Tue, Jul 31, 4:28 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MW-1.32-release-notes (WMF-deploy-2018-07-31 (1.32.0-wmf.15)), Patch-For-Review, Performance-Team (Radar), Wikimedia-General-or-Unknown
tstarling moved T180192: Audit $wgExtensionFunction and extension.json callback usage by Wikimedia-deployed extensions from Backlog to CPT-Q1-Jul-Sep-2018 on the Core-Platform-Team board.
Tue, Jul 31, 4:28 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MW-1.32-release-notes (WMF-deploy-2018-07-31 (1.32.0-wmf.15)), Patch-For-Review, Performance-Team (Radar), Wikimedia-General-or-Unknown

Mon, Jul 30

tstarling moved T110209: Maintenance scripts should fail on unknown parameters from Backlog to In Progress on the Core-Platform-Team (CPT-Q1-Jul-Sep-2018) board.
Mon, Jul 30, 11:25 PM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Patch-For-Review, Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Wikimedia-Incident, Incident-20150825-Redis, MediaWiki-Maintenance-scripts
tstarling moved T200247: Introduce MagicWordFactory service from Backlog to In Progress on the Core-Platform-Team (CPT-Q1-Jul-Sep-2018) board.
Mon, Jul 30, 11:25 PM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Patch-For-Review, MediaWiki-General-or-Unknown, Core-Platform-Team (CPT-Q1-Jul-Sep-2018)
tstarling moved T200246: Introduce ContentLanguage service to replace $wgContLang from Backlog to In Progress on the Core-Platform-Team (CPT-Q1-Jul-Sep-2018) board.
Mon, Jul 30, 11:25 PM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MediaWiki-General-or-Unknown, Technical-Debt

Fri, Jul 27

tstarling added a comment to T200247: Introduce MagicWordFactory service.

You can talk to @Legoktm about it on IRC, he should be around, 9:30pm for you would be 11:30am for him, and he has a similar outlook to me on these questions.

Fri, Jul 27, 6:47 AM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Patch-For-Review, MediaWiki-General-or-Unknown, Core-Platform-Team (CPT-Q1-Jul-Sep-2018)

Thu, Jul 26

tstarling added a comment to T200247: Introduce MagicWordFactory service.

@tstarling, what do you want me to do about things like InfoAction or ApiQuerySiteinfo? They use static MagicWord methods, but it doesn't seem like it would be easy to give them a MagicWordFactory member that the caller creates. Should they use MediaWikiServices (which seems to defeat the point of the exercise), or create their own MagicWordFactory, or what?

If we have nothing in core calling MediaWikiServices::getInstance()->getMagicWordFactory(), it seems like we could eliminate MagicWordFactory::clearCache() entirely, which would be nice.

Thu, Jul 26, 12:47 PM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Patch-For-Review, MediaWiki-General-or-Unknown, Core-Platform-Team (CPT-Q1-Jul-Sep-2018)
tstarling added a comment to T200246: Introduce ContentLanguage service to replace $wgContLang.

@tstarling Do we need the initContLang() method? I don't see anything that overrides it, or any explanation of why it was added. 43b2fb56 added it as a branch merge without mention in the commit message. It looks like the commit in the branch that made the actual change also doesn't explain why it was added:

https://phabricator.wikimedia.org/rSVN15753#change-lx4YK1Qml5dI

It would simplify overriding the ContLang service if it was just any old Language object without any initialization method having to be called on it.

Thu, Jul 26, 12:31 PM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MediaWiki-General-or-Unknown, Technical-Debt

Wed, Jul 25

tstarling moved T200297: Introduce a new namespace for collaborative judgments about wiki entities from Inbox to Under discussion on the TechCom-RFC board.
Wed, Jul 25, 8:31 PM · Scoring-platform-team (Current), DBA, Operations, JADE, TechCom-RFC
tstarling moved T20493: Unify various deletion systems from Request IRC meeting to Under discussion on the TechCom-RFC board.
Wed, Jul 25, 8:24 PM · TechCom-RFC, Stewards-and-global-tools, MediaWiki-Page-deletion

Tue, Jul 24

tstarling created T200247: Introduce MagicWordFactory service.
Tue, Jul 24, 6:45 AM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Patch-For-Review, MediaWiki-General-or-Unknown, Core-Platform-Team (CPT-Q1-Jul-Sep-2018)
tstarling created T200246: Introduce ContentLanguage service to replace $wgContLang.
Tue, Jul 24, 6:21 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MediaWiki-General-or-Unknown, Technical-Debt
tstarling moved T199594: Exception "Job queue is read-only" from Inbox to Watching on the Core-Platform-Team board.
Tue, Jul 24, 1:47 AM · Services (done), MW-1.32-release-notes (WMF-deploy-2018-07-24 (1.32.0-wmf.14)), User-Joe, Operations, Wikimedia-log-errors, Core-Platform-Team, WMF-JobQueue
tstarling moved T178517: Fix tests of PoolCounter extension from Inbox to Backlog on the Core-Platform-Team board.
Tue, Jul 24, 1:46 AM · Core-Platform-Team, Release-Engineering-Team (Watching / External), Patch-For-Review, PoolCounter
tstarling closed T187980: Memcached error "A TIMEOUT OCCURRED" for keys as Resolved.

Seems fixed now. Please reopen if necessary.

Tue, Jul 24, 1:44 AM · Core-Platform-Team, Performance-Team (Radar), Wikimedia-log-errors, MediaWiki-Cache
tstarling moved T198176: Mediawiki page deletions should happen in batches of revisions from Inbox to Backlog on the Core-Platform-Team board.
Tue, Jul 24, 1:42 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MediaWiki-Page-deletion
tstarling moved T199008: sql enwik gives a poor error message when db doesn't exist from Inbox to Watching on the Core-Platform-Team board.
Tue, Jul 24, 1:38 AM · Core-Platform-Team, MediaWiki-Maintenance-scripts, Operations
tstarling moved T199762: WikiPage::updateCategoryCounts causing Lock wait timeout exceeded from Inbox to Watching on the Core-Platform-Team board.
Tue, Jul 24, 1:37 AM · Wikimedia-log-errors, Performance-Team, Core-Platform-Team, MediaWiki-Database
tstarling moved T200055: Some pages have revisions by invalid usernames (with leading or trailing space) which break the "Info" and "History" views from Inbox to Backlog on the Core-Platform-Team board.
Tue, Jul 24, 1:36 AM · Wikimedia-log-errors, Core-Platform-Team, MediaWiki-General-or-Unknown
tstarling added a project to T198176: Mediawiki page deletions should happen in batches of revisions: Core-Platform-Team.
Tue, Jul 24, 12:56 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MediaWiki-Page-deletion
tstarling added a comment to T152776: API action=parsoid-batch not available on Graphite.

This should be fixed with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/447041

Tue, Jul 24, 12:52 AM · Release-Engineering-Team (Watching / External), Services (watching), Performance, Parsoid, MediaWiki-API, monitoring

Fri, Jul 20

tstarling added a comment to T198176: Mediawiki page deletions should happen in batches of revisions.

I'm imagining that we would do it without a progress bar, just a message like you say. It's a UX improvement compared to an exception message. But I wonder whether we should still require the bigdelete right before launching a job? Currently $wgDeleteRevisionsLimit is 5000, and above that number of revisions, the bigdelete right is required. Is 5000 also an appropriate threshold for queueing a job?

Fri, Jul 20, 2:38 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MediaWiki-Page-deletion

Jul 19 2018

tstarling closed T199316: "sql wikishared" doesn't work on mwmaint1001 as Resolved.
Jul 19 2018, 6:28 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Patch-For-Review, Scap
tstarling updated subscribers of T199594: Exception "Job queue is read-only".

To serve read traffic correctly, $wgReadOnly needs to be false. $wgReadOnly is mostly a UI-layer concept which shows some informative message to the user, not just on POST, but also on confirmation pages. So it's not really necessary to fix this to implement active-active support, we can just set $wgReadOnly to false. So I don't think this is high priority, it is just logspam.

Jul 19 2018, 6:25 AM · Services (done), MW-1.32-release-notes (WMF-deploy-2018-07-24 (1.32.0-wmf.14)), User-Joe, Operations, Wikimedia-log-errors, Core-Platform-Team, WMF-JobQueue
tstarling added a comment to T199008: sql enwik gives a poor error message when db doesn't exist.

Poor but specific. The old script was just guessing when it said "Error looking up DB", it would have said that if the PHP binary was missing. In the new script I put in different messages for different error cases. Feel free to submit a patch if you want to reword it, then I can +2. It's on line 91 of maintenance/mysql.php.

Jul 19 2018, 5:57 AM · Core-Platform-Team, MediaWiki-Maintenance-scripts, Operations
tstarling added a comment to T199152: mysql.php reacts to signals intended for mysql.

In production, mwscript/sudo is causing its own set of problems with ctrl-Z even when running eval.php. To test it locally I skipped my own wrappers and ran php directly, since I figured the (evident) problems with the wrappers were out of scope.

Jul 19 2018, 5:10 AM · MW-1.32-release-notes (WMF-deploy-2018-07-24 (1.32.0-wmf.14)), Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Patch-For-Review, MediaWiki-Maintenance-scripts
tstarling added a comment to T20493: Unify various deletion systems.

Not allowing undeletion of deleted revisions (an archived page) would just add extra steps.

  1. Delete the undeleted revisions (the existing page)
  2. Undelete everything
Jul 19 2018, 12:52 AM · TechCom-RFC, Stewards-and-global-tools, MediaWiki-Page-deletion
tstarling added a comment to T198176: Mediawiki page deletions should happen in batches of revisions.

It can't be done in the request because request threads can die at any time. But it could be done in the job queue, which has the means to retry jobs which do not complete successfully, so it could be reasonably reliable.

Jul 19 2018, 12:43 AM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MediaWiki-Page-deletion
tstarling added a comment to T20493: Unify various deletion systems.

Meeting minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2018/wikimedia-office.2018-07-18-21.00.html

Jul 19 2018, 12:15 AM · TechCom-RFC, Stewards-and-global-tools, MediaWiki-Page-deletion

Jul 18 2018

tstarling moved T40010: Re-evaluate librsvg as SVG renderer on Wikimedia wikis from Inbox to Backlog on the TechCom-RFC board.
Jul 18 2018, 8:54 PM · TechCom-RFC, MediaWiki-File-management, Commons, Multimedia, Wikimedia-SVG-rendering
tstarling moved T199004: RFC: Add a language agnostic build step to skins/extensions to our deploy process from Inbox to Backlog on the TechCom-RFC board.
Jul 18 2018, 8:46 PM · TechCom-RFC, Proposal, User-Jdlrobson
tstarling moved T199917: RFC: Block users by page/namespace from Inbox to Under discussion on the TechCom-RFC board.
Jul 18 2018, 8:39 PM · Anti-Harassment, TechCom-RFC
tstarling moved T175745: Do not overwrite edits when conflicting with self from Inbox to Backlog on the TechCom board.
Jul 18 2018, 8:37 PM · Core-Platform-Team, TechCom, Patch-For-Review, User-Daniel, MediaWiki-Page-editing, Editing-team
tstarling added a comment to T199152: mysql.php reacts to signals intended for mysql.

A lesser issue is that Ctrl+Z properly suspends both mysql and the wrapper, but executing fg does not seem to send SIGCONT through to mysql.

Jul 18 2018, 6:40 AM · MW-1.32-release-notes (WMF-deploy-2018-07-24 (1.32.0-wmf.14)), Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Patch-For-Review, MediaWiki-Maintenance-scripts
tstarling closed T32452: gracefully handle a poolcounterd outage as Resolved.

As detailed in T105378, this was fixed by reducing the connect timeout.

Jul 18 2018, 6:31 AM · Availability, Core-Platform-Team, PoolCounter
tstarling triaged T199383: WaitConditionLoop callers need to log on timeout as Low priority.
Jul 18 2018, 6:28 AM · MW-1.32-release-notes (WMF-deploy-2018-07-24 (1.32.0-wmf.14)), Patch-For-Review, MediaWiki-General-or-Unknown, Performance-Team (Radar), Wikimedia-Incident, Core-Platform-Team

Jul 17 2018

tstarling updated subscribers of T199842: replace rsvg with resvg.

@JoKalliauer: I do not know why you subscribed 28 people to this task?

Jul 17 2018, 9:52 PM · MediaWiki-File-management, Commons, Multimedia, Wikimedia-SVG-rendering