Database field 'workflow.workflow_wiki' too short (not compatible with mediawiki's iw_wikiid)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Mar 21 2015, 1:38 AM

Description

The latest builds are failing due to the database names having increased slightly (as side effects of lots of other layers and dependencies) with the switch to MySQL.

https://integration.wikimedia.org/ci/job/mediawiki-extensions-zend/8069/consoleFull

00:06:19.836 1) Flow\Tests\Api\ApiFlowEditPostTest::testEditPost
00:06:19.836 Flow\Exception\CrossWikiException: Interwiki to 'jenkins_u2_mw-un' from 'jenkins_u2_mw-unittest_'  not implemented
00:06:19.836 
00:06:19.836 /mnt/jenkins-workspace/workspace/mediawiki-extensions-zend/src/extensions/Flow/includes/Model/Workflow.php:195
00:06:19.837 /mnt/jenkins-workspace/workspace/mediawiki-extensions-zend/src/extensions/Flow/includes/Model/Workflow.php:166
00:06:19.837 /mnt/jenkins-workspace/workspace/mediawiki-extensions-zend/src/extensions/Flow/includes/Model/Workflow.php:278
00:06:19.837 /mnt/jenkins-workspace/workspace/mediawiki-extensions-zend/src/extensions/Flow/includes/WorkflowLoaderFactory.php:121

Flow (unchanged since the first 2013 prototype):

CREATE TABLE /*_*/flow_workflow (
5		workflow_id binary(11) not null,
6		workflow_wiki varchar(16) binary not null,
…

MediaWiki:

CREATE TABLE /*_*/interwiki (
  iw_wikiid varchar(64) NOT NULL,

EDIT: Actually, I'm not sure how this previously worked. Previously, the sqlite wikis were named my_wiki. Which, with the phpunit suffix, would also exceed 16 characters.

> "my_wiki-unittest_".length
17

Details

Subject	Repo	Branch	Lines +/-
Temporarily remove Flow from mediawiki-extensions combo group	integration/config	master	+0 -1
Increase length of *_wiki fields to 64	mediawiki/extensions/Flow	wmf/1.25wmf22	+20 -6
Increase length of *_wiki fields to 64	mediawiki/extensions/Flow	master	+20 -6

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	hashar	T53908 Jenkins should flag usage of deprecated features
Resolved	hashar	T50002 Jenkins: Assert no PHP errors (notices, warnings) were raised or exceptions were thrown
Resolved	• Mattflaschen-WMF	T93463 Database field 'workflow.workflow_wiki' too short (not compatible with mediawiki's iw_wikiid)
Resolved	• Springle	T93844 Run Flow's patch-increase_width_wiki_fields in WMF production

Event Timeline

Krinkle created this task.Mar 21 2015, 1:38 AM

Krinkle raised the priority of this task from to Unbreak Now!.

Krinkle updated the task description. (Show Details)

Krinkle added projects: StructuredDiscussions, Continuous-Integration-Infrastructure.

Krinkle subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 21 2015, 1:38 AM

Krinkle unsubscribed.Mar 21 2015, 2:06 AM

Krinkle updated the task description. (Show Details)Mar 21 2015, 2:08 AM

Krinkle set Security to None.

Change 198456 had a related patch set uploaded (by Krinkle):
Temporarily remove Flow from mediawiki-extensions combo group

https://gerrit.wikimedia.org/r/198456

gerritbot added a project: Patch-For-Review.Mar 21 2015, 2:16 AM

Change 198456 merged by jenkins-bot:
Temporarily remove Flow from mediawiki-extensions combo group

https://gerrit.wikimedia.org/r/198456

Krinkle removed a project: Patch-For-Review.Mar 21 2015, 6:32 AM

Krinkle removed a subscriber: gerritbot.

@Krinkle SQLite does not limit the length of strings, so a varchar(n) is just a text.

Krinkle updated the task description. (Show Details)Mar 21 2015, 8:29 PM

Krinkle moved this task from Untriaged to Done on the Continuous-Integration-Infrastructure board.Mar 22 2015, 10:06 PM

Krinkle mentioned this in rCICFc4bb24a6ee94: Temporarily remove Flow from mediawiki-extensions combo group.Mar 23 2015, 3:56 PM

@hashar Ah, that explains why this worked in testing. That makes the issue even more important as this means things can actually impact production. A quick git-blame exposes that the Flow database schema has been incompatible with MediaWiki core in this matter from its first prototype in 2013.

• Mattflaschen-WMF subscribed.Mar 25 2015, 2:03 AM

It looks like these are now live and breaking all merges for Flow? 2 days notice is not nearly enough time to throw something on someones plate and then break everything.

• Mattflaschen-WMF claimed this task.Mar 25 2015, 2:57 AM

In T93463#1147477, @EBernhardson wrote:

It looks like these are now live and breaking all merges for Flow? 2 days notice is not nearly enough time to throw something on someones plate and then break everything.

Sorry but this was not a scheduled breakage and as such there was no notice. Not even two days. I created this task after all jobs in MediaWiki extensions and core were already broken. Hence I removed Flow from that test group as immediate response so that at least some repositories can still test their code.

The latest MediaWiki core master and SQLite on Ubuntu Trusty are not compatible. This is due to a bugfix in the newer sqlite that exposed an issue in how MediaWiki core does certain atomic queries. See T89180.

Since we migrated to Trusty two months ago this caused random jobs to fail over the place. Most affected were front-end projects that use a browser (which results in concurrent http requests to load.php and stress sqlite more). But even PHPUnit jobs were affected and randomly failed in 1/50 times or so.

This was unacceptable and made things very frustrating for everyone. As such we prioritised MySQL with @greg. This was a long term goal anyway, but was moved up to restore stability.

After a month of work this was finalised a few days ago. As this should only improve stability, no notice beforehand seemed needed. I could not have foreseen that Flow's database schema was incompatible under MySQL (it worked by accident up until now).

Reverting to SQLite is not an option. And our infrastructure currently does not provide a mechanism to easily change this for individual projects. We're overloaded and understaffed as it is. If you like I can help you disable unit tests using Database in Flow until this is resolved. Or alternatively, I could deploy a live hack in Jenkins to switch Flow jobs back to SQLite. Or we could make the job non-voting (and you'd inspect test results manually to ensure no regressions are introduced).

Note that this is a very real bug and is likely causing issues in production every day. It's not just a CI thing. The Interwiki map has contained entries with keys longer than 16 characters for about 2 years now. MediaWiki itself has supported it for at least 5 years.

Again my apologies for the inconvenience but I only had two bad choices. Neither of which I could feasibly know ahead of time would impact Flow.

I think the comparable (not the same, but we should be able to rely on it for length limit) is actually iw_wikiid (the wiki ID/DB name, only applicable for local interwikis).

workflow_wiki in flow_workflow is certainly the DB name. I'm also pretty confident from looking at interwiki.sql that iw_prefix is the code (e.g 'mw' for an interwiki link to MediaWiki.org). The documentation indicates iw_wikiid is the database name.

So it should really be 64.

In T93463#1147502, @Krinkle wrote:

Reverting to SQLite is not an option. And our infrastructure currently does not provide a mechanism to easily change this for individual projects. We're overloaded and understaffed as it is. If you like I can help you disable unit tests using Database in Flow until this is resolved.

I will have it fixed shortly. We're not asking for anything like that.

One thing I don't know is why the failures only started today (e.g. https://gerrit.wikimedia.org/r/#/c/199316/ passed Jenkins 6 hours ago, well after the 20th).

Note that this is a very real bug and is likely causing issues in production every day. It's not just a CI thing. The Interwiki map has contained entries with keys longer than 16 characters for about 2 years now. MediaWiki itself has supported it for at least 5 years.

Thankfully, it don't think it affects Flow in WMF production (can't speak for third-party wikis), since the longest database name we currently have is mediawikiwiki.

• Mattflaschen-WMF renamed this task from Database field 'workflow.workflow_wiki' too short (not compatible with mediawiki's iw_prefix) to Database field 'workflow.workflow_wiki' too short (not compatible with mediawiki's iw_wikiid).Mar 25 2015, 3:40 AM

• Mattflaschen-WMF updated the task description. (Show Details)

In T93463#1147531, @Mattflaschen wrote:

workflow_wiki in flow_workflow is certainly the DB name. I'm also pretty confident from looking at interwiki.sql that iw_prefix is the code (e.g 'mw' for an interwiki link to MediaWiki.org). The documentation indicates iw_wikiid is the database name.

So it should really be 64.

Good catch :)

In T93463#1147502, @Krinkle wrote:

Reverting to SQLite is not an option. And our infrastructure currently does not provide a mechanism to easily change this for individual projects. We're overloaded and understaffed as it is. If you like I can help you disable unit tests using Database in Flow until this is resolved.

I will have it fixed shortly. We're not asking for anything like that.

Okay. I'd be happy to help in that area though. I understand making a schema change can be non-trivial and would not be a fair demand without notice :D

One thing I don't know is why the failures only started today (e.g. https://gerrit.wikimedia.org/r/#/c/199316/ passed Jenkins 6 hours ago, well after the 20th).

Ah, that explains why Erik thought it had two days notice. I'll look into that.

Thankfully, it don't think it affects Flow in WMF production (can't speak for third-party wikis), since the longest database name we currently have is mediawikiwiki.

Perfect. That's comforting to know.

Change 199573 had a related patch set uploaded (by Mattflaschen):
Increase length of *_wiki fields to 64

https://gerrit.wikimedia.org/r/199573

gerritbot added a project: Patch-For-Review.Mar 25 2015, 4:20 AM

@DannyH, @EBernhardson, @matthiasmullie, until this is deployed we can't deploy to any wikis that are longer than 16 characters. Hopefully, we can get this deployed rapidly, and just not deploy to anything until then for simplicity.