Page MenuHomePhabricator

Make it possible to run pandoc in Toolforge's jobs framework
Closed, DeclinedPublic

Description

tools.wmcz involves job that does markdown conversion tasks (namely, [generic] HTML to wikitext) and makes use of pandoc. The code looks like:

import subprocess

p = subprocess.Popen(['pandoc', '-f', 'html', '-t', 'mediawiki'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
out, err = p.communicate(input=post.get('post_content').encode('utf-8'))
post_content_wikitext = out.decode('utf-8')

This code runs w/o problems at the grid engine (thanks to T279787), but it doesn't execute properly in the jobs framework, because pandoc is missing:

(venv) tools.wmcz@shell-1693074154 ~/wmczbot/generate-meta-reports
$ python3 generate_news.py 2023-01
Traceback (most recent call last):
  File "/data/project/wmcz/wmczbot/generate-meta-reports/generate_news.py", line 103, in <module>
    p = subprocess.Popen(['pandoc', '-f', 'html', '-t', 'mediawiki'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1024, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.11/subprocess.py", line 1901, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'pandoc'
CRITICAL: Exiting due to uncaught exception FileNotFoundError: [Errno 2] No such file or directory: 'pandoc'
(venv) tools.wmcz@shell-1693074154 ~/wmczbot/generate-meta-reports
$

Can we get pandoc installed? Or is there a different way in how I should do the conversion in k8s?

Event Timeline

This seems like a good use case for a custom image built with the build service.

Hi @Urbanecm! Were you able to get this working? Do you need more help on getting started with the build service? (I see your tool migrated already, so probably got it sorted out)

The grid engine has been shut down, so I'm closing any remaining migration tasks as Declined. If you're still planning to migrate this tool, please re-open this task and add one or more active project tags to it. (If you need a project tag for your tool, those can be created via the Toolforge admin console.)