Page MenuHomePhabricator

Start reading from change_tag_def in production
Closed, ResolvedPublic

Description

On some date (to be defined), set $wgChangeTagsSchemaMigrationStage to MIGRATION_NEW in order to start reading from the new change_tag_def table.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 8 2018, 2:24 PM
Vvjjkkii renamed this task from Start reading from change_tag_def in production to vddaaaaaaa.Jul 1 2018, 1:11 AM
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from vddaaaaaaa to Start reading from change_tag_def in production.Jul 1 2018, 3:31 PM
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added a subscriber: Aklapper.

Change 466271 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/mediawiki-config@master] Set some small wikis to read new for change tag backend

https://gerrit.wikimedia.org/r/466271

Restricted Application added a project: Wikidata. · View Herald TranscriptOct 11 2018, 10:00 AM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptOct 11 2018, 10:00 AM

Change 466271 merged by jenkins-bot:
[operations/mediawiki-config@master] Set some small wikis to read new for change tag backend

https://gerrit.wikimedia.org/r/466271

Mentioned in SAL (#wikimedia-operations) [2018-10-11T11:30:41Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:466271|Set some small wikis to read new for change tag backend (T194164)]] (duration: 00m 50s)

Change 467307 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/mediawiki-config@master] Enable reading from ct_tag_id in s7

https://gerrit.wikimedia.org/r/467307

Change 467307 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable reading from ct_tag_id in s7

https://gerrit.wikimedia.org/r/467307

Mentioned in SAL (#wikimedia-operations) [2018-10-15T11:09:32Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:467307|Enable reading from ct_tag_id in s7 (T194164)]] (duration: 00m 49s)

Change 467315 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/mediawiki-config@master] Enable reading from new backend of change_tag in s7

https://gerrit.wikimedia.org/r/467315

Addshore triaged this task as Normal priority.Oct 15 2018, 12:28 PM

Change 467315 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable reading from new backend of change_tag in s7

https://gerrit.wikimedia.org/r/467315

Mentioned in SAL (#wikimedia-operations) [2018-10-15T18:59:48Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:467315|Enable reading from new backend of change_tag in s7 (T194164)]] (duration: 00m 49s)

Legoktm added a subscriber: Legoktm.EditedOct 16 2018, 6:27 AM

Going to revert because of:

2018-10-16 06:26:29 [W8WElQpAMFcAAKy-ox4AAAIY] mw1252 hewiki 1.32.0-wmf.24 exception ERROR: [W8WElQpAMFcAAKy-ox4AAAIY] /w/index.php?namespace=lWCq&invert=1%27%3BSELECT%20%28CASE%20WHEN%20%285897%3D5897%29%20THEN%205897%20ELSE%201%2
F%28SELECT%200%29%20END%29--&associated=1&tagfilter=Uvhq&target=%D7%A2%D7%9E%D7%95%D7%93%20%D7%A8%D7%90%D7%A9%D7%99&showlinkedto=1&title=%D7%9E%D7%99%D7%95%D7%97%D7%93%3A%D7%A9%D7%99%D7%A0%D7%95%D7%99%D7%99%D7%9D%20%D7%91%D7%93%D7%
A4%D7%99%D7%9D%20%D7%94%D7%9E%D7%A7%D7%95%D7%A9%D7%A8%D7%99%D7%9D   InvalidArgumentException from line 2139 of /srv/mediawiki/php-1.32.0-wmf.24/includes/libs/rdbms/database/Database.php: Wikimedia\Rdbms\Database::makeList: empty in
put for field ct_tag_id {"exception_id":"W8WElQpAMFcAAKy-ox4AAAIY","exception_url":"/w/index.php?namespace=lWCq&invert=1%27%3BSELECT%20%28CASE%20WHEN%20%285897%3D5897%29%20THEN%205897%20E
LSE%201%2F%28SELECT%200%29%20END%29--&associated=1&tagfilter=Uvhq&target=%D7%A2%D7%9E%D7%95%D7%93%20%D7%A8%D7%90%D7%A9%D7%99&showlinkedto=1&title=%D7%9E%D7%99%D7%95%D7%97%D7%93%3A%D7%A9%D7%99%D7%A0%D7%95%D7%99%D7%99%D7%9D%20%D7%91%D7%93%D7%A4%D7%99%D7%9D%20%D7%94%D7%9E%D7%A7%D7%95%D7%A9%D7%A8%D7%99%D7%9D","caught_by":"mwe_handler"} 
[Exception InvalidArgumentException] (/srv/mediawiki/php-1.32.0-wmf.24/includes/libs/rdbms/database/Database.php:2139) Wikimedia\Rdbms\Database::makeList: empty input for field ct_tag_id
  #0 /srv/mediawiki/php-1.32.0-wmf.24/includes/libs/rdbms/database/Database.php(1732): Wikimedia\Rdbms\Database->makeList(array, integer)
  #1 /srv/mediawiki/php-1.32.0-wmf.24/includes/specials/SpecialRecentchangeslinked.php(211): Wikimedia\Rdbms\Database->selectSQLText(array, array, array, string, array, array)
  #2 /srv/mediawiki/php-1.32.0-wmf.24/includes/specialpage/ChangesListSpecialPage.php(934): SpecialRecentChangesLinked->doMainQuery(array, array, array, array, array, FormOptions)
  #3 /srv/mediawiki/php-1.32.0-wmf.24/includes/specialpage/ChangesListSpecialPage.php(629): ChangesListSpecialPage->getRows()
  #4 /srv/mediawiki/php-1.32.0-wmf.24/includes/specials/SpecialRecentchanges.php(167): ChangesListSpecialPage->execute(NULL)
  #5 /srv/mediawiki/php-1.32.0-wmf.24/includes/specialpage/SpecialPage.php(569): SpecialRecentChanges->execute(NULL)
  #6 /srv/mediawiki/php-1.32.0-wmf.24/includes/specialpage/SpecialPageFactory.php(568): SpecialPage->run(NULL)
  #7 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
  #8 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(868): MediaWiki->performRequest()
  #9 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(525): MediaWiki->main()
  #10 /srv/mediawiki/php-1.32.0-wmf.24/index.php(42): MediaWiki->run()
  #11 /srv/mediawiki/w/index.php(3): include(string)
  #12 {main}

Happening about 200/minute per Joe.

Mentioned in SAL (#wikimedia-operations) [2018-10-16T06:32:09Z] <legoktm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Revert "Enable reading from new backend of change_tag in s7" (T194164) (duration: 00m 50s)

Tgr added a subscriber: Tgr.Oct 16 2018, 7:11 AM

Seems to have started an hour ago, though, not when the patch was deployed: https://logstash.wikimedia.org/goto/ed4d3af593dd6d272ee6b5ef0bed5d67

Tgr added a comment.Oct 16 2018, 7:13 AM

And all the URLs are crazy like that. Is that some kind of SQL injection scanning?

And all the URLs are crazy like that. Is that some kind of SQL injection scanning?

Yes, they are trying to scan for SQL injections and that's why it started fatal 18 hours after deploying the change (exactly when I was asleep). We should ban them IMO but still it should not trigger fatals. I'm on it.

Change 467666 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/core@master] Avoid fatals when the filter tags is empty

https://gerrit.wikimedia.org/r/467666

Change 467670 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/core@wmf/1.32.0-wmf.24] Avoid fatals when the filter tags is empty

https://gerrit.wikimedia.org/r/467670

Change 467666 merged by jenkins-bot:
[mediawiki/core@master] Avoid fatals when the filter tags is empty

https://gerrit.wikimedia.org/r/467666

Change 467670 merged by Ladsgroup:
[mediawiki/core@wmf/1.32.0-wmf.24] Avoid fatals when the filter tags is empty

https://gerrit.wikimedia.org/r/467670

Mentioned in SAL (#wikimedia-operations) [2018-10-16T12:03:08Z] <ladsgroup@deploy1001> Synchronized php-1.32.0-wmf.24/includes/changetags/ChangeTags.php: SWAT: [[gerrit:467670|Avoid fatals when the filter tags is empty (T194164)]] (duration: 00m 50s)

Mentioned in SAL (#wikimedia-operations) [2018-10-16T12:08:33Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:467669|Enable reading from new backend of change_tag in s7 (T194164)]] (duration: 00m 50s)

Addshore moved this task from incoming to in progress on the Wikidata board.Oct 16 2018, 1:49 PM

Change 467946 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Enable reading from new backend of change tag everywhere

https://gerrit.wikimedia.org/r/467946

Change 467946 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable reading from new backend of change tag everywhere

https://gerrit.wikimedia.org/r/467946

Mentioned in SAL (#wikimedia-operations) [2018-10-17T11:40:01Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:467946|Enable reading from new backend of change tag everywhere (T194164)]] (duration: 00m 57s)

Addshore closed this task as Resolved.Nov 6 2018, 12:13 PM
Addshore added a subscriber: Addshore.

The "start" as described in this ticket is done, I'll file a followup for wikidata.org which was rolled back so it is still not done there yet.