Page MenuHomePhabricator

[EasyTimeline] No text included / no font rendered / displayed at all in PNG graph output
Closed, ResolvedPublic

Description

Steps to reproduce:

Expected result:

  • Chart is updated as expected

Actual result:

  • The overlay text disappears

Also tried at https://meta.wikimedia.org/wiki/Template:StewardsChart to the same result (with preview only)

Before edit:

image.png (592×939 px, 161 KB)

After edit:

image.png (584×958 px, 27 KB)

Event Timeline

I don't see any recent changes in the extension itself, https://gerrit.wikimedia.org/r/q/EasyTimeline , the only recent change is some typos being fixed in comments, so adding a broader tag

Aklapper renamed this task from [EasyTimeline] overlay text disappears upon update to [EasyTimeline] No text included / displayed at all in PNG graph output.Feb 16 2021, 9:27 AM
Aklapper triaged this task as High priority.
Aklapper raised the priority of this task from High to Needs Triage.EditedFeb 16 2021, 9:35 AM

At first I thought of an issue in the font rendering stack (though I have no clue what EasyTimeline/Platypus use - which exact font, FontConfig, etc), but I could not confirming this when editing https://en.wikipedia.org/w/index.php?title=Template:Timeline_MediaWiki and looking at the preview.
I can confirm with the given type of graph however, at https://www.mediawiki.org/wiki/User:AKlapper_(WMF)/Sandbox#T274822

So it's unclear if other PNG images are affected, or if this depends on some types of graphs.

Removed Wikimedia-production-error tag as there's no error message logged.

Aklapper renamed this task from [EasyTimeline] No text included / displayed at all in PNG graph output to [EasyTimeline] No text included / no font rendered / displayed at all in PNG graph output.Feb 16 2021, 9:42 AM

bd808 and cdanis on IRC mentioned a potential relation to the appserver migration from Stretch to Debian Buster in T245757

Krinkle triaged this task as High priority.Feb 16 2021, 7:53 PM
Krinkle added a project: serviceops.

I havent touched to EasyTimeline / ploticus in age. The version up to Stretch uses gd for rendering and that finds the fonts via GDFONTPATH=/srv/mediawiki/fonts. The fonts are deployed via operations/mediawiki-config submodule ./fonts which points to operations/mediawiki-config/fonts.git), that got done in T147481.

The EasyTimeline code is in https://gerrit.wikimedia.org/r/mediawiki/extensions/timeline , it has a perl script to parse the user input and that wfShellExec() ploticus. So if there are any errors reported by ploticus, we should have them in whatever captures the php stdout/stderr.

Maybe we are missing some font files / they are not recognized / the default font got renamed.

This could also could be caused by the Shellbox deployment.

hashar raised the priority of this task from High to Unbreak Now!.Feb 16 2021, 8:30 PM

In logstash: environment var GDFONTPATH not found. That started today at roughly 13:00 UTC with way more at 14:30 UTC. They do not show up in our mediawiki-errors logstash dashboard cause the error are logged in channel:exec and we filter on channel being one of exception, error, jsonTruncated.

View without filters looking for GDFONTPATH https://logstash.wikimedia.org/goto/d82065d223dde19b45a672d7b434f728

/srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/UnboxedExecutor.php:407

Error running /bin/bash '/srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/usr/bin/perl'\'' '\''/srv/mediawiki/php-1.36.0-wmf.30/extensions/timeline/EasyTimeline.pl'\'' -i '\''/tmp/timeline_d272655b3f9e'\'' -m -P '\''/usr/bin/ploticus'\'' -T '\''/tmp'\'' -A '\''/wiki/$1'\'' -f '\''FreeSans'\''' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=50; SB_CGROUP='\''/sys/fs/cgroup/memory/mediawiki/job'\''; SB_MEM_LIMIT=1073741824; SB_FILE_SIZE_LIMIT=536870912; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes': pl proc page:  12358: warning: environment var GDFONTPATH not found. See ploticus fonts docs.
pl proc page:  12358: warning: environment var GDFONTPATH not found. See ploticus fonts docs. 
Could not find/open font (FreeSans)
...
Could not find/open font (FreeSans) (width calc)
...
from /srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/UnboxedExecutor.php(407)
#0 /srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/UnboxedCommand.php(29): Shellbox\Command\UnboxedExecutor->execute(MediaWiki\Shell\Command)
#1 /srv/mediawiki/php-1.36.0-wmf.30/includes/shell/Command.php(227): Shellbox\Command\UnboxedCommand->execute()
#2 /srv/mediawiki/php-1.36.0-wmf.30/includes/GlobalFunctions.php(2101): MediaWiki\Shell\Command->execute()
#3 /srv/mediawiki/php-1.36.0-wmf.30/extensions/timeline/includes/Timeline.php(113): wfShellExec(string, NULL)
#4 /srv/mediawiki/php-1.36.0-wmf.30/includes/parser/Parser.php(3929): Timeline::renderTimeline(string, array, Parser, PPFrame_Hash)
#5 /srv/mediawiki/php-1.36.0-wmf.30/includes/parser/PPFrame_Hash.php(341): Parser->extensionSubstitution(array, PPFrame_Hash)
#6 /srv/mediawiki/php-1.36.0-wmf.30/includes/parser/Parser.php(2874): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#7 /srv/mediawiki/php-1.36.0-wmf.30/includes/parser/Parser.php(1551): Parser->replaceVariables(string)
#8 /srv/mediawiki/php-1.36.0-wmf.30/includes/parser/Parser.php(644): Parser->internalParse(string)
#9 /srv/mediawiki/php-1.36.0-wmf.30/includes/content/WikitextContent.php(375): Parser->parse(string, Title, ParserOptions, boolean, boolean, NULL)
#10 /srv/mediawiki/php-1.36.0-wmf.30/includes/content/AbstractContent.php(591): WikitextContent->fillParserOutput(Title, NULL, ParserOptions, boolean, ParserOutput)
#11 /srv/mediawiki/php-1.36.0-wmf.30/includes/Revision/RenderedRevision.php(266): AbstractContent->getParserOutput(Title, NULL, ParserOptions, boolean)
#12 /srv/mediawiki/php-1.36.0-wmf.30/includes/Revision/RenderedRevision.php(235): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#13 /srv/mediawiki/php-1.36.0-wmf.30/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#14 /srv/mediawiki/php-1.36.0-wmf.30/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#15 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#16 /srv/mediawiki/php-1.36.0-wmf.30/includes/Revision/RenderedRevision.php(197): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#17 /srv/mediawiki/php-1.36.0-wmf.30/includes/Storage/DerivedPageDataUpdater.php(1337): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#18 [internal function]: MediaWiki\Storage\DerivedPageDataUpdater->getCanonicalParserOutput()
#19 /srv/mediawiki/php-1.36.0-wmf.30/includes/edit/PreparedEdit.php(104): call_user_func(array)
#20 /srv/mediawiki/php-1.36.0-wmf.30/includes/edit/PreparedEdit.php(119): MediaWiki\Edit\PreparedEdit->getOutput()
#21 /srv/mediawiki/php-1.36.0-wmf.30/includes/Storage/DerivedPageDataUpdater.php(1315): MediaWiki\Edit\PreparedEdit->__get(string)
#22 /srv/mediawiki/php-1.36.0-wmf.30/includes/page/WikiPage.php(2177): MediaWiki\Storage\DerivedPageDataUpdater->getPreparedEdit()
#23 /srv/mediawiki/php-1.36.0-wmf.30/includes/Storage/PageEditStash.php(137): WikiPage->prepareContentForEdit(WikitextContent, NULL, User, string, boolean)
#24 /srv/mediawiki/php-1.36.0-wmf.30/includes/api/ApiStashEdit.php(189): MediaWiki\Storage\PageEditStash->parseAndCache(WikiPage, WikitextContent, User, string)
#25 /srv/mediawiki/php-1.36.0-wmf.30/includes/api/ApiMain.php(1612): ApiStashEdit->execute()
#26 /srv/mediawiki/php-1.36.0-wmf.30/includes/api/ApiMain.php(592): ApiMain->executeAction()
#27 /srv/mediawiki/php-1.36.0-wmf.30/includes/api/ApiMain.php(563): ApiMain->executeActionWithErrorHandling()
#28 /srv/mediawiki/php-1.36.0-wmf.30/api.php(90): ApiMain->execute()
#29 /srv/mediawiki/php-1.36.0-wmf.30/api.php(45): wfApiMain()
#30 /srv/mediawiki/w/api.php(3): require(string)
#31 {main}

Note: beside the few traces above, I am not actively dealing with the issue, it is late here and I am heading to bed'

In logstash: environment var GDFONTPATH not found. That started today at roughly 13:00 UTC with way more at 14:30 UTC. They do not show up in our mediawiki-errors logstash dashboard cause the error are logged in channel:exec and we filter on channel being one of exception, error, jsonTruncated.

Nothing really happened around that time according to SAL. If I had to guess, that's when people tried to edit timelines.

View without filters looking for GDFONTPATH https://logstash.wikimedia.org/goto/d82065d223dde19b45a672d7b434f728

/srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/UnboxedExecutor.php:407

Error running /bin/bash '/srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/usr/bin/perl'\'' '\''/srv/mediawiki/php-1.36.0-wmf.30/extensions/timeline/EasyTimeline.pl'\'' -i '\''/tmp/timeline_d272655b3f9e'\'' -m -P '\''/usr/bin/ploticus'\'' -T '\''/tmp'\'' -A '\''/wiki/$1'\'' -f '\''FreeSans'\''' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=50; SB_CGROUP='\''/sys/fs/cgroup/memory/mediawiki/job'\''; SB_MEM_LIMIT=1073741824; SB_FILE_SIZE_LIMIT=536870912; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes': pl proc page:  12358: warning: environment var GDFONTPATH not found. See ploticus fonts docs.
pl proc page:  12358: warning: environment var GDFONTPATH not found. See ploticus fonts docs. 
Could not find/open font (FreeSans)
...
Could not find/open font (FreeSans) (width calc)
...

This is set in MediaWiki config: https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/1abd0649db08b03a6b3ee1a60a3be010610f24d1/wmf-config/CommonSettings.php#941

Ah right, this was caused by https://gerrit.wikimedia.org/r/c/mediawiki/libs/Shellbox/+/662134

Instead of having a global putenv command, I think the correct thing is for Timeline to have a $wgTimelineFontDirectory, which is passed as an explicit environment option as GDFONTPATH when shelling out.

Change 664668 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/extensions/timeline@master] Add $wgTimelineFontDirectory to be passed as GDFONTPATH

https://gerrit.wikimedia.org/r/664668

Change 664669 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/mediawiki-config@master] Set $wgTimelineFontDirectory

https://gerrit.wikimedia.org/r/664669

Change 664670 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/mediawiki-config@master] Remove putenv() for GDFONTPATH

https://gerrit.wikimedia.org/r/664670

Legoktm lowered the priority of this task from Unbreak Now! to High.Feb 16 2021, 9:09 PM

I'm downgrading this to just "high" because 1) timeline is an unmaintained feature 2) viewing of timelines still works until someone edits them. I've submitted patches and will backport them whenever someone can review them.

code reviewed. I just gave a +1 to the config changes but I voted +2 to the change in the extension.

Change 664668 merged by jenkins-bot:
[mediawiki/extensions/timeline@master] Add $wgTimelineFontDirectory to be passed as GDFONTPATH

https://gerrit.wikimedia.org/r/664668

Thanks for your help @Legoktm. This isnt the first time that the timeline images have broken and I wonder why is it an unmaintained feature? Is there a better way of presenting the data that the timelines on wiki contain? There are at least 8 tropical cyclone articles per year that currently rely on this feature to be working and I have always wondered if there was a better way of doing them.

Change 664649 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/extensions/timeline@wmf/1.36.0-wmf.31] Add $wgTimelineFontDirectory to be passed as GDFONTPATH

https://gerrit.wikimedia.org/r/664649

Change 664650 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/extensions/timeline@wmf/1.36.0-wmf.30] Add $wgTimelineFontDirectory to be passed as GDFONTPATH

https://gerrit.wikimedia.org/r/664650

Change 664669 merged by jenkins-bot:
[operations/mediawiki-config@master] Set $wgTimelineFontDirectory

https://gerrit.wikimedia.org/r/664669

Change 664649 merged by jenkins-bot:
[mediawiki/extensions/timeline@wmf/1.36.0-wmf.31] Add $wgTimelineFontDirectory to be passed as GDFONTPATH

https://gerrit.wikimedia.org/r/664649

Change 664650 merged by jenkins-bot:
[mediawiki/extensions/timeline@wmf/1.36.0-wmf.30] Add $wgTimelineFontDirectory to be passed as GDFONTPATH

https://gerrit.wikimedia.org/r/664650

Mentioned in SAL (#wikimedia-operations) [2021-02-17T00:13:32Z] <legoktm@deploy1001> Synchronized wmf-config/timeline.php: Set $wgTimelineFontDirectory (T274822) (duration: 01m 05s)

Mentioned in SAL (#wikimedia-operations) [2021-02-17T00:15:47Z] <legoktm@deploy1001> Synchronized php-1.36.0-wmf.31/extensions/timeline/: Add $wgTimelineFontDirectory to be passed as GDFONTPATH (T274822) (duration: 01m 02s)

Mentioned in SAL (#wikimedia-operations) [2021-02-17T00:17:19Z] <legoktm@deploy1001> Synchronized php-1.36.0-wmf.30/extensions/timeline/: Add $wgTimelineFontDirectory to be passed as GDFONTPATH (T274822) (duration: 01m 06s)

Thanks for your help @Legoktm. This isnt the first time that the timeline images have broken and I wonder why is it an unmaintained feature? Is there a better way of presenting the data that the timelines on wiki contain? There are at least 8 tropical cyclone articles per year that currently rely on this feature to be working and I have always wondered if there was a better way of doing them.

The technology behind EasyTimeline (A Perl wrapper around ploticus) is just really outdated and there doesn't seem to be anyone interested in working on it. The plan was to migrate usage to the Graph extension (see T137291: Transition all use of EasyTimeline to the Graph extension and decommission it from Wikimedia's servers), except the Graph extension is now also unmaintained.

Personally I'd recommend looking into something like https://en.wikipedia.org/wiki/Template:Horizontal_timeline and seeing if a friendly Lua/template author can help you implement what you specifically want.

Legoktm claimed this task.

This should be fixed now. I made https://ua.wikimedia.org/w/index.php?title=%D0%A8%D0%B0%D0%B1%D0%BB%D0%BE%D0%BD%3A%D0%93%D1%80%D0%B0%D1%84%D1%96%D0%BA_%D0%BA%D0%B5%D1%80%D1%96%D0%B2%D0%BD%D0%B8%D1%85_%D0%BE%D1%80%D0%B3%D0%B0%D0%BD%D1%96%D0%B2&type=revision&diff=71903&oldid=71874 to verify.

Note that for now the text passed to <timeline> must be different than anything that was rendered while this was broken - you can use a empty comment like I did in my edit.

+User-notice

Suggested entry: Trying to edit a timeline may have removed all the text from it, this is now fixed. You may need to edit the timeline again for it to display properly.

Getting rid of the environment variable in favor of a regular config setting ($wgTimelineFontDirectory) is definitely an improvement. Thank you @Legoktm.

Note we can invalidate all the timelines by bumping $wgTimelineEpochTimestamp.

Please edit the timeline of that article again, to display properly.

Thanks @Aklapper - Fix made by adding a temporary comment and then removing:
https://en.wikipedia.org/w/index.php?title=Metallica&type=revision&diff=1008495071&oldid=1005601846

That was before I read this:

  • you can use a empty comment like I did in my edit.

Perhaps something a bot could do because as far as I can tell, the timeline had not been edited in the last six months, so any EasyTimeline could have been broken by any change in the entire article that contains it, while the bug was active (?)

Maybe not the best place to discuss it, here, but just bringing it to your attention, if needed.
e.g. (Metallica) Feb 2nd revision was okay ... Feb 8th, no change to timeline but timeline was broken by an edit in an unrelated section.

Thanks, again.