Page MenuHomePhabricator
Create Task
Maniphest T289227

Convert SyntaxHighlight to use Shellbox
Closed, ResolvedPublic
Actions

Description

SyntaxHighlight shells out to Pygments to highlight stuff. There are 3 primary things that need to be converted:

  • Getting the generated CSS
  • Getting the language list
  • Actually doing the syntax highlighting

Currently the first two are just hardcoded, and can probably remain that way if it's using the bundled pygments. Otherwise it should call shellbox to avoid version mismatches.

Related: T271751: SyntaxHighlight extension is slow (needs more active stewardship).

  • Review charts
  • shellbox-syntaxhighlight namespaces in k8s
  • shellbox-syntaxhighlight accounts in k8s.
  • shellbox-syntaxhighlight puppet private tokens.
  • Generate TLS certificates
  • Review helmfile.d files:
  • LVS setup
  • DNS for LVS records
  • Discovery DNS
  • Monitoring dashboard
  • Integration and Acceptance tests

Details

SubjectRepoBranchLines +/-
Throw more resources at shellbox-syntaxhighlightoperations/deployment-chartsmaster+8 -1
Bump shellbox-syntaxhighlight up to 6 replicasoperations/deployment-chartsmaster+1 -1
Have SyntaxHighlight use Shellbox on group1 wikis toooperations/mediawiki-configmaster+1 -0
Have SyntaxHighlight use Shellbox service on group0 wikisoperations/mediawiki-configmaster+11 -0
ProductionServices: Add new Shellboxesoperations/mediawiki-configmaster+6 -0
services_proxy: Add envoy proxies for new Shellboxesoperations/puppetproduction+18 -0
Add new Shellboxes to discoveryoperations/dnsmaster+6 -0
service: Switch new Shellboxes to productionoperations/puppetproduction+3 -3
service: Switch new Shellboxes to monitoring_setupoperations/puppetproduction+3 -3
service: Switch new Shellboxes to lvs_setupoperations/puppetproduction+3 -3
Add LVS for new Shellboxes: media, syntaxhighlight & timelineoperations/puppetproduction+120 -0
Add *.svc.{codfw,eqiad}.wmnet entries for new Shellboxesoperations/dnsmaster+15 -2
helmfile.d: Add shellbox-syntaxhighlightoperations/deployment-chartsmaster+65 -0
Port to BoxedCommandmediawiki/extensions/SyntaxHighlight_GeSHimaster+382 -81
pipeline: Build image for SyntaxHighlight with Pygmentsmediawiki/libs/Shellboxmaster+15 -0
admin: Add namespace for shellbox-syntaxhighlightoperations/deployment-chartsmaster+1 -0
Add k8s users/tokens for shellbox-{syntaxhighlight,timeline}operations/puppetproduction+20 -0
Add k8s users/tokens for shellbox-{syntaxhighlight,timeline}labs/privatemaster+24 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Legoktm created this task.Aug 19 2021, 1:55 AM
Legoktm added a subtask: T289367: blubber should make it convenient to install apt packages from a component.Aug 20 2021, 5:25 PM
Jdforrester-WMF subscribed.Sep 1 2021, 6:18 PM
Legoktm updated the task description. (Show Details)Sep 1 2021, 9:34 PM
gerritbot added a comment.Sep 1 2021, 9:59 PM

Change 716048 had a related patch set uploaded (by Legoktm; author: Legoktm):

[labs/private@master] Add k8s users/tokens for shellbox-{syntaxhighlight,timeline}

https://gerrit.wikimedia.org/r/716048

gerritbot added a project: Patch-For-Review.Sep 1 2021, 9:59 PM
gerritbot added a comment.Sep 1 2021, 10:03 PM

Change 716051 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/puppet@production] Add k8s tokens/users for shellbox-{syntaxhighlight,timeline}

https://gerrit.wikimedia.org/r/716051

gerritbot added a comment.Sep 1 2021, 10:16 PM

Change 716048 merged by Legoktm:

[labs/private@master] Add k8s users/tokens for shellbox-{syntaxhighlight,timeline}

https://gerrit.wikimedia.org/r/716048

gerritbot added a comment.Sep 1 2021, 10:16 PM

Change 716051 merged by Legoktm:

[operations/puppet@production] Add k8s users/tokens for shellbox-{syntaxhighlight,timeline}

https://gerrit.wikimedia.org/r/716051

gerritbot added a comment.Sep 1 2021, 10:20 PM

Change 716054 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/deployment-charts@master] admin: Add namespace for shellbox-syntaxhighlight

https://gerrit.wikimedia.org/r/716054

Legoktm mentioned this in rLPRI4123326114eb: Add k8s users/tokens for shellbox-{syntaxhighlight,timeline}.Sep 1 2021, 10:24 PM
gerritbot added a comment.Sep 1 2021, 10:27 PM

Change 716054 merged by jenkins-bot:

[operations/deployment-charts@master] admin: Add namespace for shellbox-syntaxhighlight

https://gerrit.wikimedia.org/r/716054

Legoktm updated the task description. (Show Details)Sep 1 2021, 11:07 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 1 2021, 11:10 PM
Legoktm added a comment.Sep 2 2021, 6:52 PM

My plan is to have SyntaxHighlight effectively split into two modes: using bundled and not bundled (this is similar to what Scribunto's LuaStandalone does).

In bundled mode:

  • The version of pygments will be hardcoded
  • The CSS is read from the bundled stylesheet, using a ResourceLoaderFileModule
  • The lexer list is read from the bundled PHP data file

In not-bundled mode (what Wikimedia will use):

  • The version of pygments will be fetched by parsing pygmentize -V, and cached for an hour in apcu
  • The CSS is read by shelling out, using a custom ResourceLoaderModule. Unclear if it'll need additional caching here on top of RL's caching
  • The lexer list is read by shelling out, cached for a day in apcu, keyed on the cached version.

The actual highlighting will use BoxedCommand, cached on version (new) in WAN for 1 month (pre-existing). To keep things relatively simple I'm not planning on adding abstraction for all of this, but it should be straightforward for someone to refactor in the future for adding a microservice backed highlighter.

gerritbot added a comment.Sep 3 2021, 2:30 AM

Change 716756 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/extensions/SyntaxHighlight_GeSHi@master] [WIP] Port to BoxedCommand

https://gerrit.wikimedia.org/r/716756

gerritbot added a project: Patch-For-Review.Sep 3 2021, 2:31 AM
gerritbot added a comment.Sep 3 2021, 6:42 AM

Change 717020 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/libs/Shellbox@master] pipeline: Build image for SyntaxHighlight with Pygments

https://gerrit.wikimedia.org/r/717020

• dduvall closed subtask T289367: blubber should make it convenient to install apt packages from a component as Resolved.Sep 3 2021, 5:43 PM
gerritbot added a comment.Sep 3 2021, 9:57 PM

Change 717020 merged by jenkins-bot:

[mediawiki/libs/Shellbox@master] pipeline: Build image for SyntaxHighlight with Pygments

https://gerrit.wikimedia.org/r/717020

gerritbot added a comment.Sep 10 2021, 7:07 PM

Change 720378 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/deployment-charts@master] helmfile.d: Add shellbox-syntaxhighlight

https://gerrit.wikimedia.org/r/720378

gerritbot added a comment.Sep 10 2021, 7:27 PM

Change 716756 merged by jenkins-bot:

[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Port to BoxedCommand

https://gerrit.wikimedia.org/r/716756

ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.23; 2021-09-13).Sep 10 2021, 8:00 PM
gerritbot added a comment.Sep 10 2021, 9:16 PM

Change 720378 merged by jenkins-bot:

[operations/deployment-charts@master] helmfile.d: Add shellbox-syntaxhighlight

https://gerrit.wikimedia.org/r/720378

Legoktm updated the task description. (Show Details)Sep 10 2021, 9:33 PM
Legoktm claimed this task.Sep 17 2021, 12:10 AM
gerritbot added a comment.Sep 17 2021, 10:03 PM

Change 721904 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/puppet@production] Add LVS for new Shellboxes: media, syntaxhighlight & timeline

https://gerrit.wikimedia.org/r/721904

gerritbot added a comment.Sep 17 2021, 10:03 PM

Change 721905 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/puppet@production] service: Switch new Shellboxes to lvs_setup

https://gerrit.wikimedia.org/r/721905

gerritbot added a comment.Sep 17 2021, 10:03 PM

Change 721906 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/puppet@production] service: Switch new Shellboxes to monitoring_setup

https://gerrit.wikimedia.org/r/721906

gerritbot added a comment.Sep 17 2021, 10:03 PM

Change 721908 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/dns@master] Add *.svc.{codfw,eqiad}.wmnet entries for new Shellboxes

https://gerrit.wikimedia.org/r/721908

gerritbot added a comment.Sep 17 2021, 10:03 PM

Change 721907 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/puppet@production] service: Switch new Shellboxes to production

https://gerrit.wikimedia.org/r/721907

gerritbot added a comment.Sep 17 2021, 10:03 PM

Change 721909 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/dns@master] Add new Shellboxes to discovery

https://gerrit.wikimedia.org/r/721909

gerritbot added a comment.Sep 22 2021, 4:45 AM

Change 721908 merged by Legoktm:

[operations/dns@master] Add *.svc.{codfw,eqiad}.wmnet entries for new Shellboxes

https://gerrit.wikimedia.org/r/721908

gerritbot added a comment.Sep 22 2021, 4:54 AM

Change 721904 merged by Legoktm:

[operations/puppet@production] Add LVS for new Shellboxes: media, syntaxhighlight & timeline

https://gerrit.wikimedia.org/r/721904

gerritbot added a comment.Sep 22 2021, 5:11 AM

Change 721905 merged by Legoktm:

[operations/puppet@production] service: Switch new Shellboxes to lvs_setup

https://gerrit.wikimedia.org/r/721905

gerritbot added a comment.Sep 22 2021, 5:33 AM

Change 721906 merged by Legoktm:

[operations/puppet@production] service: Switch new Shellboxes to monitoring_setup

https://gerrit.wikimedia.org/r/721906

gerritbot added a comment.Sep 22 2021, 5:41 AM

Change 721907 merged by Legoktm:

[operations/puppet@production] service: Switch new Shellboxes to production

https://gerrit.wikimedia.org/r/721907

gerritbot added a comment.Sep 22 2021, 5:45 AM

Change 721909 merged by Legoktm:

[operations/dns@master] Add new Shellboxes to discovery

https://gerrit.wikimedia.org/r/721909

gerritbot added a comment.Sep 22 2021, 5:56 AM

Change 722736 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/puppet@production] services_proxy: Add envoy proxies for new Shellboxes

https://gerrit.wikimedia.org/r/722736

gerritbot added a comment.Sep 22 2021, 5:58 AM

Change 722737 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/mediawiki-config@master] ProductionServices: Add new Shellboxes

https://gerrit.wikimedia.org/r/722737

gerritbot added a comment.Sep 22 2021, 6:02 AM

Change 722736 merged by Legoktm:

[operations/puppet@production] services_proxy: Add envoy proxies for new Shellboxes

https://gerrit.wikimedia.org/r/722736

gerritbot added a comment.Sep 22 2021, 6:05 PM

Change 722737 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices: Add new Shellboxes

https://gerrit.wikimedia.org/r/722737

gerritbot added a comment.Sep 23 2021, 7:39 AM

Change 723049 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/mediawiki-config@master] Have SyntaxHighlight use Shellbox service on group0 wikis

https://gerrit.wikimedia.org/r/723049

Legoktm updated the task description. (Show Details)Sep 23 2021, 7:41 AM
gerritbot added a comment.Sep 23 2021, 7:56 AM

Change 723049 merged by jenkins-bot:

[operations/mediawiki-config@master] Have SyntaxHighlight use Shellbox service on group0 wikis

https://gerrit.wikimedia.org/r/723049

Stashbot added a comment.Sep 23 2021, 8:02 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-23T08:02:54Z] <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Have SyntaxHighlight use Shellbox service on group0 wikis (1/2) (T289227) (duration: 01m 06s)

Stashbot added a comment.Sep 23 2021, 8:04 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-23T08:04:16Z] <legoktm@deploy1002> Synchronized wmf-config/CommonSettings.php: Have SyntaxHighlight use Shellbox service on group0 wikis (2/2) (T289227) (duration: 01m 05s)

Legoktm added a comment.Sep 23 2021, 8:12 AM

SyntaxHighlight on group0 is now running via the Shellbox service.

If it goes well over the next few days and weekend we can roll it out further next week.

Legoktm added a comment.Sep 23 2021, 8:23 AM

For some reason I see no hits for fetchGeneratedCSS() in statsd nor exec.log. It should've been called at least once and then stored in WAN cache...

Legoktm added a comment.Sep 23 2021, 8:47 AM
In T289227#7373646, @Legoktm wrote:

For some reason I see no hits for fetchGeneratedCSS() in statsd nor exec.log. It should've been called at least once and then stored in WAN cache...

I now see 1 entry (logstash id DXvSEXwB9aenX452ZvP6), maybe was waiting on some RL cache to expire? Things look fine (visually) though!

gerritbot added a comment.Sep 27 2021, 10:16 PM

Change 724198 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/mediawiki-config@master] Have SyntaxHighlight use Shellbox on group1 wikis too

https://gerrit.wikimedia.org/r/724198

gerritbot added a comment.Sep 27 2021, 10:32 PM

Change 724198 merged by jenkins-bot:

[operations/mediawiki-config@master] Have SyntaxHighlight use Shellbox on group1 wikis too

https://gerrit.wikimedia.org/r/724198

Stashbot added a comment.Sep 27 2021, 10:34 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-27T22:34:43Z] <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Have SyntaxHighlight use Shellbox on group1 wikis too (T289227) (duration: 00m 57s)

gerritbot added a comment.Sep 27 2021, 10:48 PM

Change 724203 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/deployment-charts@master] Bump shellbox-syntaxhighlight up to 6 replicas

https://gerrit.wikimedia.org/r/724203

gerritbot added a comment.Sep 27 2021, 10:55 PM

Change 724203 merged by jenkins-bot:

[operations/deployment-charts@master] Bump shellbox-syntaxhighlight up to 6 replicas

https://gerrit.wikimedia.org/r/724203

Legoktm closed this task as Resolved.Sep 29 2021, 1:55 AM

Things look relatively stable now \o/. I also wrote up https://www.mediawiki.org/wiki/Extension:SyntaxHighlight/Updating_pygments which explains the new process for upgrading pygments.

I think things can still be tuned a bit further, I'll keep watching it.

TheDJ subscribed.Sep 29 2021, 9:19 AM

Something shot up to 56 secs just there... https://grafana.wikimedia.org/d/VTCkm29Wz/envoy-telemetry?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-origin=appserver&var-origin_instance=All&var-destination=shellbox-syntaxhighlight&from=1632906198966&to=1632907078395

Legoktm reopened this task as Open.Sep 29 2021, 5:13 PM

It looked pretty bad on the Shellbox side too: https://grafana.wikimedia.org/d/RKogW1m7z/shellbox?orgId=1&var-dc=eqiad%20prometheus%2Fk8s&var-service=shellbox&var-namespace=shellbox-syntaxhighlight&var-release=main&from=1632906027990&to=1632909636707

Per https://grafana.wikimedia.org/d/vcOTDuSnk/syntaxhighlight?orgId=1&from=1632906048981&to=1632907839781 MediaWiki sent 1300 highlight requests in a minute. Zooming out over the past 24 hours I do see spikes reaching over 1200 at other times. Let me do a bit of napkin math and increase the replica count.

@TheDJ I'm curious if you were just looking at the dashboards and noticed that or if you or someone else experienced page loading/saving being slow.

taavi subscribed.Sep 29 2021, 5:16 PM
TheDJ added a comment.Sep 29 2021, 5:29 PM

I was going through my phab notifications. I was looking at https://phabricator.wikimedia.org/T271751 and this ticket around the same time. And opened https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/queries/examples and then some time later I looked at the graph link you posted in this ticket. They could be related I guess but I didn't experience any slowness or anything, so i didn't consider the option of them being related.

Legoktm added a comment.Sep 30 2021, 6:23 AM

That page is like the worst case performance example for SyntaxHighlight :)

A few hours ago we also had:
16:37:33 <+icinga-wm> PROBLEM - LVS shellbox-syntaxhighlight eqiad port 4014/tcp - Shellbox SyntaxHighlight- shellbox-syntaxhighlight.svc.eqiad.wmnet IPv4 on shellbox-syntaxhighlight.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems
16:39:33 <+icinga-wm> RECOVERY - LVS shellbox-syntaxhighlight eqiad port 4014/tcp - Shellbox SyntaxHighlight- shellbox-syntaxhighlight.svc.eqiad.wmnet IPv4 on shellbox-syntaxhighlight.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 358 bytes in 1.052 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems

https://grafana.wikimedia.org/d/vcOTDuSnk/syntaxhighlight?orgId=1&from=1632967782148&to=1632970799524 looks like it hit 1400 requests/min.

gerritbot added a comment.Sep 30 2021, 6:44 AM

Change 724869 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/deployment-charts@master] Throw more resources at shellbox-syntaxhighlight

https://gerrit.wikimedia.org/r/724869

gerritbot added a comment.Sep 30 2021, 6:55 AM

Change 724869 merged by jenkins-bot:

[operations/deployment-charts@master] Throw more resources at shellbox-syntaxhighlight

https://gerrit.wikimedia.org/r/724869

Legoktm closed this task as Resolved.Oct 4 2021, 4:19 PM

Things have looked good ever since the expansion.

TheDJ mentioned this in T292736: SyntaxHighlight always highlights syntax errors in red.Oct 8 2021, 1:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL