Page MenuHomePhabricator
Create Task
Maniphest T348753

[026f63a8-bebd-49dd-a536-746796d71575] /w/api.php Exception: Errors saving HLS playlist LL-Q8097_(tel)-V_Bhavya-క్రొ.wav.m3u8
Closed, DuplicatePublicPRODUCTION ERROR
Actions

Description

Error
labels.normalized_message
[026f63a8-bebd-49dd-a536-746796d71575] /w/api.php   Exception: Errors saving HLS playlist LL-Q8097_(tel)-V_Bhavya-క్రొ.wav.m3u8
error.stack_trace
from /srv/mediawiki/php-1.41.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(1251)
#0 /srv/mediawiki/php-1.41.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(1040): MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::updateStreamingManifests(LocalFile)
#1 /srv/mediawiki/php-1.41.0-wmf.30/extensions/TimedMediaHandler/includes/Hooks.php(331): MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::removeTranscodes(LocalFile)
#2 /srv/mediawiki/php-1.41.0-wmf.30/includes/HookContainer/HookContainer.php(161): MediaWiki\TimedMediaHandler\Hooks->onFileUpload(LocalFile, boolean, boolean)
#3 /srv/mediawiki/php-1.41.0-wmf.30/includes/HookContainer/HookRunner.php(1744): MediaWiki\HookContainer\HookContainer->run(string, array)
#4 /srv/mediawiki/php-1.41.0-wmf.30/includes/filerepo/file/LocalFile.php(2014): MediaWiki\HookContainer\HookRunner->onFileUpload(LocalFile, boolean, boolean)
#5 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/AutoCommitUpdate.php(45): LocalFile->{closure}(Wikimedia\Rdbms\DBConnRef, string)
#6 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/DeferredUpdates.php(463): AutoCommitUpdate->doUpdate()
#7 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/DeferredUpdates.php(192): DeferredUpdates::attemptUpdate(AutoCommitUpdate)
#8 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/DeferredUpdates.php(276): DeferredUpdates::run(AutoCommitUpdate)
#9 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(AutoCommitUpdate, integer)
#10 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#11 /srv/mediawiki/php-1.41.0-wmf.30/includes/deferred/DeferredUpdates.php(295): DeferredUpdatesScope->processUpdates(integer, Closure)
#12 /srv/mediawiki/php-1.41.0-wmf.30/includes/MediaWiki.php(727): DeferredUpdates::doUpdates(integer)
#13 /srv/mediawiki/php-1.41.0-wmf.30/includes/api/ApiMain.php(934): MediaWiki::preOutputCommit(DerivativeContext)
#14 /srv/mediawiki/php-1.41.0-wmf.30/includes/api/ApiMain.php(879): ApiMain->executeActionWithErrorHandling()
#15 /srv/mediawiki/php-1.41.0-wmf.30/api.php(95): ApiMain->execute()
#16 /srv/mediawiki/php-1.41.0-wmf.30/api.php(48): wfApiMain()
#17 /srv/mediawiki/w/api.php(3): require(string)
#18 {main}
Impact

I guess the video is not transcoded? :)

Notes

There are several of them showing up in the logs, maybe due to malformed videos?

Details

Request URL
https://commons.wikimedia.org/w/api.php
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Don't try to lock to serialize m3u8 file writesmediawiki/extensions/TimedMediaHandlerwmf/1.41.0-wmf.30+28 -22
Don't try to lock to serialize m3u8 file writesmediawiki/extensions/TimedMediaHandlerREL1_41+28 -22
Don't try to lock to serialize m3u8 file writesmediawiki/extensions/TimedMediaHandlermaster+28 -22
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Oct 12 2023, 1:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 12 2023, 1:56 PM
hashar added a project: SRE-swift-storage.Oct 12 2023, 2:02 PM
hashar added a subscriber: • brooke.

I thought it was due to some bad input, but the code throwing the error has been introduced as part of this week train: HTTP Live Streaming (HLS for iOS) and WebM transcode cleanup https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TimedMediaHandler/+/888279

Looking at the trace.id in Logstash there are a few more emitted:

FileOperation
StoreFileOp failed: {"src":"/tmp/LL-Q8097_(tel)-V_Bhavya-\u0c15\u0c4d\u0c30\u0c4a.wav.m3u84e303476a3b0.m3u8","dst":"mwstore://local-swift-eqiad/local-transcoded/0/03/LL-Q8097_(tel)-V_Bhavya-\u0c15\u0c4d\u0c30\u0c4a.wav/LL-Q8097_(tel)-V_Bhavya-\u0c15\u0c4d\u0c30\u0c4a.wav.m3u8","overwrite":true,"headers":[],"failedAction":"attempt"}
FileOperation
HTTP 401 (Unauthorized) in 'SwiftFileBackend::doStoreInternal' (given '{"async":false,"src":"/tmp/LL-Q8097_(tel)-V_Bhavya-\u0c15\u0c4d\u0c30\u0c4a.wav.m3u84e303476a3b0.m3u8","dst":"mwstore://local-swift-eqiad/local-transcoded/0/03/LL-Q8097_(tel)-V_Bhavya-\u0c15\u0c4d\u0c30\u0c4a.wav/LL-Q8097_(tel)-V_Bhavya-\u0c15\u0c4d\u0c30\u0c4a.wav.m3u8","overwrite":true,"headers":[]}')

So that looks like a permission issue with Swift?

Aklapper added a project: Regression.Oct 12 2023, 3:08 PM
• brooke added a comment.Oct 12 2023, 3:11 PM

Note there shouldn't be any streaming transcodes made for .wav files at this time, so something's amiss as well.

• brooke added a comment.Oct 12 2023, 3:13 PM

Oh I see it's saving empty playlists anyway heh. Well that's fine, it should work and be harmless...

• brooke added a comment.Oct 12 2023, 3:14 PM

I could make these non-fatal errors I guess, just log and move on. :(

hashar added a comment.Oct 12 2023, 3:26 PM

Ah great, thank you @brion. I guess you can log a warning/info message instead of raising an exception.

Related to it are errors such as:

MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::updateStreamingManifests: Cannot flush pre-lock snapshot; writes from transaction Wikimedia\Rdbms\Database::beginIfImplied (MediaWiki\Block\DatabaseBlock::newLoad) are still pending (WikiPage::lockAndGetLatest, MediaWiki\CommentStore\CommentStore::createComment, MediaWiki\Revision\RevisionStore::insertRevisionRowOn, MediaWiki\Revision\RevisionStore::insertSlotRowOn, MediaWiki\Revision\RevisionStore::insertSlotRowOn, WikiPage::updateRevisionOn, MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::removeTranscodes, MediaWiki\Title\Title::invalidateCache, MediaWiki\Title\Title::invalidateCache)

trace.id: 00ba4a10-0c2e-4d71-a9e5-d03849a31f28

My rough understanding is the exception happens inside a database transaction which is not properly abandoned/discarded.

gerritbot added a comment.Oct 12 2023, 3:28 PM

Change 965520 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@master] Don't try to lock to serialize m3u8 file writes

https://gerrit.wikimedia.org/r/965520

gerritbot added a project: Patch-For-Review.Oct 12 2023, 3:28 PM
hashar mentioned this in T347081: 1.41.0-wmf.30 deployment blockers.Oct 12 2023, 4:07 PM
gerritbot added a comment.Oct 12 2023, 4:46 PM

Change 965520 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] Don't try to lock to serialize m3u8 file writes

https://gerrit.wikimedia.org/r/965520

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.1; 2023-10-17).Oct 12 2023, 5:00 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 12 2023, 5:10 PM
gerritbot added a comment.Oct 12 2023, 6:50 PM

Change 965220 had a related patch set uploaded (by Jforrester; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@wmf/1.41.0-wmf.30] Don't try to lock to serialize m3u8 file writes

https://gerrit.wikimedia.org/r/965220

gerritbot added a project: Patch-For-Review.Oct 12 2023, 6:50 PM

Change 965221 had a related patch set uploaded (by Jforrester; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@REL1_41] Don't try to lock to serialize m3u8 file writes

https://gerrit.wikimedia.org/r/965221

hashar moved this task from Untriaged to October 2023 on the Wikimedia-production-error board.Oct 12 2023, 7:12 PM
gerritbot added a comment.Oct 12 2023, 7:43 PM

Change 965221 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@REL1_41] Don't try to lock to serialize m3u8 file writes

https://gerrit.wikimedia.org/r/965221

gerritbot added a comment.Oct 16 2023, 7:42 AM

Change 965220 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@wmf/1.41.0-wmf.30] Don't try to lock to serialize m3u8 file writes

https://gerrit.wikimedia.org/r/965220

Stashbot mentioned this in T348689: WebVideoTranscode::updateStreamingManifests: Cannot flush pre-lock snapshot.Oct 16 2023, 7:43 AM
Stashbot mentioned this in T348375: Commons removal of last remaining caption: Caught exception of type Wikimedia\Rdbms\DBUnexpectedError.

Mentioned in SAL (#wikimedia-operations) [2023-10-16T07:43:20Z] <hashar@deploy2002> Started scap: Backport for [[gerrit:965220|Don't try to lock to serialize m3u8 file writes (T348689 T348667 T348375 T348753)]]

Stashbot mentioned this in T348667: DBUnexpectedError while deleting or moving file.Oct 16 2023, 7:43 AM
Stashbot added a comment.Oct 16 2023, 7:57 AM

Mentioned in SAL (#wikimedia-operations) [2023-10-16T07:57:35Z] <hashar@deploy2002> jforrester and hashar: Backport for [[gerrit:965220|Don't try to lock to serialize m3u8 file writes (T348689 T348667 T348375 T348753)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.30; 2023-10-10).Oct 16 2023, 8:00 AM
Stashbot added a comment.Oct 16 2023, 8:10 AM

Mentioned in SAL (#wikimedia-operations) [2023-10-16T08:10:25Z] <hashar@deploy2002> Finished scap: Backport for [[gerrit:965220|Don't try to lock to serialize m3u8 file writes (T348689 T348667 T348375 T348753)]] (duration: 27m 04s)

Maintenance_bot removed a project: Patch-For-Review.Oct 16 2023, 8:10 AM
hashar closed this task as Resolved.Oct 16 2023, 9:05 AM
hashar assigned this task to • brooke.
hashar mentioned this in T348354: 1.42.0-wmf.1 deployment blockers.

I have deployed the backport an hour or so ago. Thank you @brion !

Jdforrester-WMF added a parent task: T347081: 1.41.0-wmf.30 deployment blockers.Oct 16 2023, 1:31 PM
Frostly closed this task as a duplicate of T348375: Commons removal of last remaining caption: Caught exception of type Wikimedia\Rdbms\DBUnexpectedError.Oct 17 2023, 2:05 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits