Page MenuHomePhabricator

[026f63a8-bebd-49dd-a536-746796d71575] /w/api.php Exception: Errors saving HLS playlist LL-Q8097_(tel)-V_Bhavya-క్రొ.wav.m3u8
Closed, DuplicatePublicPRODUCTION ERROR

Description

Error
labels.normalized_message
[026f63a8-bebd-49dd-a536-746796d71575] /w/api.php   Exception: Errors saving HLS playlist LL-Q8097_(tel)-V_Bhavya-క్రొ.wav.m3u8
error.stack_trace
from /srv/mediawiki/php-1.41.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(1251)
#0 /srv/mediawiki/php-1.41.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(1040): MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::updateStreamingManifests(LocalFile)
#1 /srv/mediawiki/php-1.41.0-wmf.30/extensions/TimedMediaHandler/includes/Hooks.php(331): MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::removeTranscodes(LocalFile)
#2 /srv/mediawiki/php-1.41.0-wmf.30/includes/HookContainer/HookContainer.php(161): MediaWiki\TimedMediaHandler\Hooks->onFileUpload(LocalFile, boolean, boolean)
#3 /srv/mediawiki/php-1.41.0-wmf.30/includes/HookContainer/HookRunner.php(1744): MediaWiki\HookContainer\HookContainer->run(string, array)
#4 /srv/mediawiki/php-1.41.0-wmf.30/includes/filerepo/file/LocalFile.php(2014): MediaWiki\HookContainer\HookRunner->onFileUpload(LocalFile, boolean, boolean)
#5 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/AutoCommitUpdate.php(45): LocalFile->{closure}(Wikimedia\Rdbms\DBConnRef, string)
#6 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/DeferredUpdates.php(463): AutoCommitUpdate->doUpdate()
#7 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/DeferredUpdates.php(192): DeferredUpdates::attemptUpdate(AutoCommitUpdate)
#8 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/DeferredUpdates.php(276): DeferredUpdates::run(AutoCommitUpdate)
#9 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(AutoCommitUpdate, integer)
#10 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#11 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/DeferredUpdates.php(295): DeferredUpdatesScope->processUpdates(integer, Closure)
#12 /srv/mediawiki/php-1.41.0-wmf.30/includes/MediaWiki.php(727): DeferredUpdates::doUpdates(integer)
#13 /srv/mediawiki/php-1.41.0-wmf.30/includes/api/ApiMain.php(934): MediaWiki::preOutputCommit(DerivativeContext)
#14 /srv/mediawiki/php-1.41.0-wmf.30/includes/api/ApiMain.php(879): ApiMain->executeActionWithErrorHandling()
#15 /srv/mediawiki/php-1.41.0-wmf.30/api.php(95): ApiMain->execute()
#16 /srv/mediawiki/php-1.41.0-wmf.30/api.php(48): wfApiMain()
#17 /srv/mediawiki/w/api.php(3): require(string)
#18 {main}
Impact

I guess the video is not transcoded? :)

Notes

There are several of them showing up in the logs, maybe due to malformed videos?

Event Timeline

hashar added a subscriber: brooke.

I thought it was due to some bad input, but the code throwing the error has been introduced as part of this week train: HTTP Live Streaming (HLS for iOS) and WebM transcode cleanup https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TimedMediaHandler/+/888279

Looking at the trace.id in Logstash there are a few more emitted:

FileOperation
StoreFileOp failed: {"src":"/tmp/LL-Q8097_(tel)-V_Bhavya-\u0c15\u0c4d\u0c30\u0c4a.wav.m3u84e303476a3b0.m3u8","dst":"mwstore://local-swift-eqiad/local-transcoded/0/03/LL-Q8097_(tel)-V_Bhavya-\u0c15\u0c4d\u0c30\u0c4a.wav/LL-Q8097_(tel)-V_Bhavya-\u0c15\u0c4d\u0c30\u0c4a.wav.m3u8","overwrite":true,"headers":[],"failedAction":"attempt"}
FileOperation
HTTP 401 (Unauthorized) in 'SwiftFileBackend::doStoreInternal' (given '{"async":false,"src":"/tmp/LL-Q8097_(tel)-V_Bhavya-\u0c15\u0c4d\u0c30\u0c4a.wav.m3u84e303476a3b0.m3u8","dst":"mwstore://local-swift-eqiad/local-transcoded/0/03/LL-Q8097_(tel)-V_Bhavya-\u0c15\u0c4d\u0c30\u0c4a.wav/LL-Q8097_(tel)-V_Bhavya-\u0c15\u0c4d\u0c30\u0c4a.wav.m3u8","overwrite":true,"headers":[]}')

So that looks like a permission issue with Swift?

Note there shouldn't be any streaming transcodes made for .wav files at this time, so something's amiss as well.

Oh I see it's saving empty playlists anyway heh. Well that's fine, it should work and be harmless...

I could make these non-fatal errors I guess, just log and move on. :(

Ah great, thank you @brion. I guess you can log a warning/info message instead of raising an exception.

Related to it are errors such as:

MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::updateStreamingManifests: Cannot flush pre-lock snapshot; writes from transaction Wikimedia\Rdbms\Database::beginIfImplied (MediaWiki\Block\DatabaseBlock::newLoad) are still pending (WikiPage::lockAndGetLatest, MediaWiki\CommentStore\CommentStore::createComment, MediaWiki\Revision\RevisionStore::insertRevisionRowOn, MediaWiki\Revision\RevisionStore::insertSlotRowOn, MediaWiki\Revision\RevisionStore::insertSlotRowOn, WikiPage::updateRevisionOn, MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::removeTranscodes, MediaWiki\Title\Title::invalidateCache, MediaWiki\Title\Title::invalidateCache)

trace.id: 00ba4a10-0c2e-4d71-a9e5-d03849a31f28

My rough understanding is the exception happens inside a database transaction which is not properly abandoned/discarded.

Change 965520 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@master] Don't try to lock to serialize m3u8 file writes

https://gerrit.wikimedia.org/r/965520

Change 965520 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] Don't try to lock to serialize m3u8 file writes

https://gerrit.wikimedia.org/r/965520

Change 965220 had a related patch set uploaded (by Jforrester; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@wmf/1.41.0-wmf.30] Don't try to lock to serialize m3u8 file writes

https://gerrit.wikimedia.org/r/965220

Change 965221 had a related patch set uploaded (by Jforrester; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@REL1_41] Don't try to lock to serialize m3u8 file writes

https://gerrit.wikimedia.org/r/965221

Change 965221 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@REL1_41] Don't try to lock to serialize m3u8 file writes

https://gerrit.wikimedia.org/r/965221

Change 965220 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@wmf/1.41.0-wmf.30] Don't try to lock to serialize m3u8 file writes

https://gerrit.wikimedia.org/r/965220

Mentioned in SAL (#wikimedia-operations) [2023-10-16T07:57:35Z] <hashar@deploy2002> jforrester and hashar: Backport for [[gerrit:965220|Don't try to lock to serialize m3u8 file writes (T348689 T348667 T348375 T348753)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-10-16T08:10:25Z] <hashar@deploy2002> Finished scap: Backport for [[gerrit:965220|Don't try to lock to serialize m3u8 file writes (T348689 T348667 T348375 T348753)]] (duration: 27m 04s)

hashar assigned this task to brooke.

I have deployed the backport an hour or so ago. Thank you @brion !