These two extensions shell out to pdfinfo, pdftext, tiffinfo, etc. to extract metadata out of uploads and will need to be converted to use Shellbox.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T289225 Convert all Wikimedia-deployed extensions to shell out via Shellbox | |||
| Open | None | T289228 Convert media handling code (PdfHandler, PagedTiffHandler) to use Shellbox | |||
| Open | • tstarling | T292322 Support large files in Shellbox | |||
| Resolved | Scott_French | T381830 Deploy Shellbox 4.1.1 server |
Change 717124 had a related patch set uploaded (by Legoktm; author: Legoktm):
[mediawiki/extensions/PdfHandler@master] Remove questionable PdfHandler::isEnabled() implementation
Change 717154 had a related patch set uploaded (by Legoktm; author: Legoktm):
[mediawiki/extensions/PdfHandler@master] [WIP] Port retrieveMetaData to BoxedCommand
Change 717124 merged by jenkins-bot:
[mediawiki/extensions/PdfHandler@master] Remove questionable PdfHandler::isEnabled() implementation
Change 719651 had a related patch set uploaded (by Legoktm; author: Legoktm):
[mediawiki/extensions/PagedTiffHandler@master] [WIP] Port retrieveMetaData to BoxedCommand
Change 720143 had a related patch set uploaded (by Legoktm; author: Legoktm):
[mediawiki/core@master] [WIP] media: Port DjVuImage::retrieveMetaData() to use BoxedCommand
Change 721603 had a related patch set uploaded (by Legoktm; author: Legoktm):
[mediawiki/libs/Shellbox@master] pipeline: Build image for media handling
Change 721603 merged by jenkins-bot:
[mediawiki/libs/Shellbox@master] pipeline: Build image for media handling
Change 721633 had a related patch set uploaded (by Legoktm; author: Legoktm):
[labs/private@master] Add k8s users/tokens for shellbox-media
Change 721633 merged by Legoktm:
[labs/private@master] Add k8s users/tokens for shellbox-media
Change 721634 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/puppet@production] Add tokens and users for shellbox-media service
Change 721635 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/deployment-charts@master] Add namespace for shellbox-media service
Change 721634 merged by Legoktm:
[operations/puppet@production] Add tokens and users for shellbox-media service
Change 721635 merged by jenkins-bot:
[operations/deployment-charts@master] Add namespace for shellbox-media service
Change 721637 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/deployment-charts@master] helmfile.d: Add shellbox-media
Change 721637 merged by jenkins-bot:
[operations/deployment-charts@master] helmfile.d: Add shellbox-media
Change 721904 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/puppet@production] Add LVS for new Shellboxes: media, syntaxhighlight & timeline
Change 721905 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/puppet@production] service: Switch new Shellboxes to lvs_setup
Change 721906 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/puppet@production] service: Switch new Shellboxes to monitoring_setup
Change 721908 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/dns@master] Add *.svc.{codfw,eqiad}.wmnet entries for new Shellboxes
Change 721907 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/puppet@production] service: Switch new Shellboxes to production
Change 721909 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/dns@master] Add new Shellboxes to discovery
Change 719651 merged by jenkins-bot:
[mediawiki/extensions/PagedTiffHandler@master] Port retrieveMetaData to BoxedCommand
Change 717154 merged by jenkins-bot:
[mediawiki/extensions/PdfHandler@master] Port retrieveMetaData to BoxedCommand
Change 721908 merged by Legoktm:
[operations/dns@master] Add *.svc.{codfw,eqiad}.wmnet entries for new Shellboxes
Change 721904 merged by Legoktm:
[operations/puppet@production] Add LVS for new Shellboxes: media, syntaxhighlight & timeline
Change 721905 merged by Legoktm:
[operations/puppet@production] service: Switch new Shellboxes to lvs_setup
Change 721906 merged by Legoktm:
[operations/puppet@production] service: Switch new Shellboxes to monitoring_setup
Change 721907 merged by Legoktm:
[operations/puppet@production] service: Switch new Shellboxes to production
Change 721909 merged by Legoktm:
[operations/dns@master] Add new Shellboxes to discovery
Change 722736 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/puppet@production] services_proxy: Add envoy proxies for new Shellboxes
Change 722737 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/mediawiki-config@master] ProductionServices: Add new Shellboxes
Change 722736 merged by Legoktm:
[operations/puppet@production] services_proxy: Add envoy proxies for new Shellboxes
Change 722737 merged by jenkins-bot:
[operations/mediawiki-config@master] ProductionServices: Add new Shellboxes
Change 723050 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/mediawiki-config@master] Have PdfHandler use Shellbox service on group0 wikis
Change 723052 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/mediawiki-config@master] Have PagedTiffHandler use Shellbox service on group0 wikis
Change 723052 merged by jenkins-bot:
[operations/mediawiki-config@master] Have PagedTiffHandler use Shellbox service on group0 wikis
Mentioned in SAL (#wikimedia-operations) [2021-09-27T22:13:35Z] <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Have PagedTiffHandler use Shellbox service on group0 wikis (T289228) (1/2) (duration: 00m 57s)
Mentioned in SAL (#wikimedia-operations) [2021-09-27T22:14:48Z] <legoktm@deploy1002> Synchronized wmf-config/CommonSettings.php: Have PagedTiffHandler use Shellbox service on group0 wikis (T289228) (2/2) (duration: 00m 58s)
Change 723050 merged by jenkins-bot:
[operations/mediawiki-config@master] Have PdfHandler use Shellbox service on group0 wikis
Mentioned in SAL (#wikimedia-operations) [2021-09-27T22:25:07Z] <legoktm@deploy1002> sync-file aborted: Have PdfHandler use Shellbox service on group0 wikis (T289228) (duration: 00m 00s)
Mentioned in SAL (#wikimedia-operations) [2021-09-27T22:26:08Z] <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Have PdfHandler use Shellbox service on group0 wikis (T289228) (1/2) (duration: 00m 57s)
Mentioned in SAL (#wikimedia-operations) [2021-09-27T22:27:25Z] <legoktm@deploy1002> Synchronized wmf-config/CommonSettings.php: Have PdfHandler use Shellbox service on group0 wikis (T289228) (2/2) (duration: 00m 56s)
So far we're not really getting any traffic from group0 wikis which is mostly expected. When we move to group1, we should exclude commons until the final step since that's actually going to be our biggest source of requests.
I forgot to tag https://gerrit.wikimedia.org/r/724572 with this bug, but PdfHandler+PagedTiffHandler now use Shellbox on all non-Commons wikis. Even that is no meaningful traffic, we'll have to do some % rollout on Commons.
Change 724576 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/mediawiki-config@master] Have PdfHandler use Shellbox on 10% of requests
Change 724577 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/mediawiki-config@master] Have PagedTiffHandler use Shellbox on 10% of requests
Change 725121 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/deployment-charts@master] Scale up shellbox-media
Change 725121 merged by jenkins-bot:
[operations/deployment-charts@master] Scale up shellbox-media
Change 724576 merged by jenkins-bot:
[operations/mediawiki-config@master] Have PdfHandler use Shellbox on Commons for 10% of requests
Mentioned in SAL (#wikimedia-operations) [2021-10-01T04:00:31Z] <legoktm@deploy1002> Synchronized wmf-config/CommonSettings.php: Have PdfHandler use Shellbox on Commons for 10% of requests (T289228) (duration: 00m 59s)
I've spot checked newly uploaded PDFs on Commons and they seem to be fine in getting metadata. I'll check again in my morning before bumping it up again, probalby 50%.
I forgot to add statsd metrics to PagedTiffHandler, so waiting for https://gerrit.wikimedia.org/r/725183 before enabling that.
Reviewing Special:NewFiles on Commons, I realized Shellbox probably won't be happy if we tried posting a 1GB+ tiff file to it, at the very least we need to increase max sizes that envoy, apache and php-fpm allow. Joe and I had previously discussed having the container download the file from swift for videoscalers, we should probably do something like that for large files too. I'll file a separate task for that.
Change 720143 merged by jenkins-bot:
[mediawiki/core@master] media: Port DjVuImage::retrieveMetaData() to use BoxedCommand
Change 738636 had a related patch set uploaded (by Ladsgroup; author: Legoktm):
[mediawiki/core@wmf/1.38.0-wmf.7] media: Port DjVuImage::retrieveMetaData() to use BoxedCommand
Change 738636 merged by jenkins-bot:
[mediawiki/core@wmf/1.38.0-wmf.7] media: Port DjVuImage::retrieveMetaData() to use BoxedCommand
Mentioned in SAL (#wikimedia-operations) [2021-11-15T09:59:14Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.7/includes/media/: Backport: [[gerrit:738636|media: Port DjVuImage::retrieveMetaData() to use BoxedCommand (T289228)]] (duration: 00m 56s)
Change 724577 abandoned by Legoktm:
[operations/mediawiki-config@master] Have PagedTiffHandler use Shellbox on Commons for 10% of requests
Reason:
Not yet
The DjVu part was reverted and needs to be reworked. Other than that we're still blocked on T292322: Support large files in Shellbox.
Change 979386 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):
[mediawiki/core@master] media: Port DjVuImage::retrieveMetaData() to use BoxedCommand
Change 979386 merged by jenkins-bot:
[mediawiki/core@master] media: Port DjVuImage::retrieveMetaData() to use BoxedCommand
We still need to investigate/fix the regression on CLI for DjVuImage, see https://gerrit.wikimedia.org/r/c/mediawiki/core/+/720143/comments/8f625c32_80c745cd