Page MenuHomePhabricator

Beta cluster update database is broken because of Flow db patch db_patches/patch-ref_target_not_null.sql
Closed, ResolvedPublic

Description

The poor job https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/ is broken since Feb 16th 5:20am UTC.

Looking at https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/fatalmonitor that happens on multiple wikis while sourcing:

DatabaseBase::sourceFile( /mnt/srv/mediawiki-staging/php-master/extensions/Flow/db_patches/patch-ref_target_not_null.sql )
10.68.16.193	1170	BLOB/TEXT column 'ref_target' used in key specification without a key length (10.68.16.193)
                        ALTER TABLE `flow_ext_ref` CHANGE ref_target ref_target BLOB NOT NULL

Event Timeline

hashar created this task.Feb 16 2016, 9:33 AM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a subscriber: hashar.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 16 2016, 9:33 AM
hashar renamed this task from Beta cluster update database is broken to Beta cluster update database is broken because of Flow db patch db_patches/patch-ref_target_not_null.sql .Feb 16 2016, 9:41 AM
hashar updated the task description. (Show Details)
hashar added projects: DBA, StructuredDiscussions.
hashar set Security to None.
Restricted Application added a project: Collaboration-Team-Triage. · View Herald TranscriptFeb 16 2016, 9:41 AM

I guess patch-reference_wiki.sql didn't fully make it to beta cluster.

Any chance you could paste the current schema for flow_ext_ref & flow_wiki_ref? I want to make sure the index updates aren't the only thing missing there.

You should be able to reach the beta cluster databases via:

ssh deployment-tin.deployment-prep.eqiad.wmflabs
sql enwiki

enwiki has:

flow_ext_ref | CREATE TABLE `flow_ext_ref` (
  `ref_src_object_id` binary(11) NOT NULL,
  `ref_src_object_type` varbinary(32) NOT NULL,
  `ref_src_workflow_id` binary(11) NOT NULL,
  `ref_src_namespace` int(11) NOT NULL,
  `ref_src_title` varbinary(255) NOT NULL,
  `ref_target` blob NOT NULL,
  `ref_type` varbinary(16) NOT NULL,
  `ref_src_wiki` varchar(16) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
  `ref_id` binary(11) NOT NULL,
  PRIMARY KEY (`ref_id`),
  KEY `flow_ext_ref_idx_v2` (`ref_src_wiki`,`ref_src_namespace`,`ref_src_title`,`ref_type`,`ref_target`(255),`ref_src_object_type`,`ref_src_object_id`),
  KEY `flow_ext_ref_revision_v2` (`ref_src_wiki`,`ref_src_namespace`,`ref_src_title`,`ref_src_object_type`,`ref_src_object_id`,`ref_type`,`ref_target`(255))
) ENGINE=InnoDB DEFAULT CHARSET=latin1
flow_wiki_ref | CREATE TABLE `flow_wiki_ref` (
  `ref_src_object_id` binary(11) NOT NULL,
  `ref_src_object_type` varbinary(32) NOT NULL,
  `ref_src_workflow_id` binary(11) NOT NULL,
  `ref_src_namespace` int(11) NOT NULL,
  `ref_src_title` varbinary(255) NOT NULL,
  `ref_target_namespace` int(11) NOT NULL,
  `ref_target_title` varbinary(255) NOT NULL,
  `ref_type` varbinary(16) NOT NULL,
  `ref_src_wiki` varchar(16) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
  `ref_id` binary(11) NOT NULL,
  PRIMARY KEY (`ref_id`),
  KEY `flow_wiki_ref_idx_v2` (`ref_src_wiki`,`ref_src_namespace`,`ref_src_title`,`ref_type`,`ref_target_namespace`,`ref_target_title`,`ref_src_object_type`,`ref_src_object_id`),
  KEY `flow_wiki_ref_revision_v2` (`ref_src_wiki`,`ref_src_namespace`,`ref_src_title`,`ref_src_object_type`,`ref_src_object_id`,`ref_type`,`ref_target_namespace`,`ref_target_title`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

After a few turns of update.php the build of Feb 16th at 10:20 UTC passed. Maybe the patch is flagged as passed (which would be a bug) or another attempt make it work eventually.

Ha, deployment-tin! Dear god, been awhile since I ssh'ed into beta :)

But the schema looks fine, so the patch seems to have been applied eventually. Strange though.

There could be several issues here, some beta-related, some potentially real issues. Beta is using here latin1, if that is due to beta or flow update, I do not know. Different collation means potential different sized for things like indexes.

In adition, as far as I remember, beta uses MySQL 5.5. Different InnoDB configuration may result in incompatibilities for things like maximum index size. I will not investigate this further as it is resolved, but I would suggest flow devels to try the updates on several different mysql configurations if that is important for external users to try to detect potential incompatibilities.

Mattflaschen-WMF closed this task as Resolved.Feb 17 2016, 7:35 PM