Some transcoding fails because of "is not a valid storage path" in swift
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Yann
	Jan 13 2017, 2:06 PM

Description

Transcoding failed:
https://commons.wikimedia.org/wiki/File:%D0%9F%D1%80%D0%B5%D0%B7%D0%B8%D0%B4%D0%B5%D0%BD%D1%82_%D0%A0%D0%BE%D1%81%D1%81%D0%B8%D0%B8_%E2%80%94_2016-01-14_%E2%80%94_%D0%97%D0%B0%D1%81%D0%B5%D0%B4%D0%B0%D0%BD%D0%B8%D0%B5_%D0%BD%D0%B0%D0%B1%D0%BB%D1%8E%D0%B4%D0%B0%D1%82%D0%B5%D0%BB%D1%8C%D0%BD%D0%BE%D0%B3%D0%BE_%D1%81%D0%BE%D0%B2%D0%B5%D1%82%D0%B0_%D0%90%D0%B3%D0%B5%D0%BD%D1%82%D1%81%D1%82%D0%B2%D0%B0_%D1%81%D1%82%D1%80%D0%B0%D1%82%D0%B5%D0%B3%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%B8%D1%85_%D0%B8%D0%BD%D0%B8%D1%86%D0%B8%D0%B0%D1%82%D0%B8%D0%B2.webm
File:Президент России — 2016-01-14 — Заседание наблюдательного совета Агентства стратегических инициатив.webm

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Fix length calculation for shortening transcode filenames	mediawiki/extensions/TimedMediaHandler	wmf/1.29.0-wmf.7	+4 -3
	Fix length calculation for shortening transcode filenames	mediawiki/extensions/TimedMediaHandler	master	+4 -3

Customize query in gerrit

Related Objects

Mentioned Here: T57658: Transcode error for long cyrillic filename: ....ogv.360p.webm" is not a valid storage path.
T155168: Wikimedia videoscalers cannot transcode videos with frame rate lower than 1 fps to VP8

Event Timeline

Yann created this task.Jan 13 2017, 2:06 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 13 2017, 2:06 PM

In T155168#2936070, @zhuyifei1999 wrote:

* "mwstore&#58;//local-swift-eqiad/local-transcoded/b/b5/Президент_России_—_2016-01-14_—_Заседание_наблюдательного_совета_Агентства_стратегических_инициатив.webm/Президент_России_—_2016-01-14_—_Заседание_наблюдательного_совета_Агентства_стратегических_инициатив.webm.160p.ogv" is not a valid storage path.
* "mwstore&#58;//local-swift-codfw/local-transcoded/b/b5/Президент_России_—_2016-01-14_—_Заседание_наблюдательного_совета_Агентства_стратегических_инициатив.webm/Президент_России_—_2016-01-14_—_Заседание_наблюдательного_совета_Агентства_стратегических_инициатив.webm.160p.ogv" is not a valid storage path.

The filename does not seem long enough, at 194 bytes utf-8.

zhuyifei1999 renamed this task from Transcoding fails: name too long? to Some transcoding fails because of "is not a valid storage path" in swift.Jan 13 2017, 3:02 PM

zhuyifei1999 added a project: Video.Jan 13 2017, 3:07 PM

zhuyifei1999 added a project: SRE-swift-storage.Jan 13 2017, 3:11 PM

Looks related to old bug T57658 -- perhaps the calculation for the fix there is off.

Per the comment:

	 * Swift has problems where the url-encoded version of
	 * the path (ie 'filename.ogv/filename.ogv.720p.webm' )
	 * is greater that > 1024 bytes, so shorten in that case.

The calculation used in the actual comparison before shortening the filename is this:

		if ( strlen( urlencode( $name ) ) * 2 + 12 > 1024 ) {
			return 'transcode' . '.' . $suffix;
		} else {
			return $name . '.' . $suffix;
		}

which here gives us false:

php > echo strlen(urlencode('Президент_России_—_2016-01-14_—_Заседание_наблюдательного_совета_Агентства_стратегических_инициатив.webm')) * 2 + 12;
1022

So TMH thinks it *can* save the filename, hitting 1022 chars which is under the 1024-char limit.

Possibility: I suspect the "b/b5/" might count in that length too, which would push it to 1031 bytes.

Change 331985 had a related patch set uploaded (by Brion VIBBER):
Fix length calculation for shortening transcode filenames

https://gerrit.wikimedia.org/r/331985

gerritbot added a project: Patch-For-Review.Jan 13 2017, 3:45 PM

Paladox subscribed.Jan 13 2017, 4:36 PM

Change 331985 merged by jenkins-bot:
Fix length calculation for shortening transcode filenames

https://gerrit.wikimedia.org/r/331985

ReleaseTaggerBot added a project: MW-1.29-release (WMF-deploy-2017-01-17_(1.29.0-wmf.8)).Jan 13 2017, 6:00 PM

Change 332009 had a related patch set uploaded (by Paladox):
Fix length calculation for shortening transcode filenames

https://gerrit.wikimedia.org/r/332009

Change 332009 abandoned by Chad:
Fix length calculation for shortening transcode filenames

Reason:
Already included in wmf.8, which is live everywhere.