Problem
- For security, we need better isolation of external binaries from MediaWiki.
- If we run MediaWiki itself under Kubernetes, the resulting container should be as small as possible, so it should ideally exclude unnecessary binaries.
- It's difficult to deploy bleeding-edge versions of external binaries when they necessarily share an OS with MediaWiki.
Proposal
Have a PHP microservice, accessible via HTTP, which takes POSTed inputs, writes them to the container's filesystem as temporary files, runs a shell command, and responds with gathered output files.
The client and server components, as well as abstractions and local execution, shall be in a new composer library called Shellbox.
The key new abstraction is called BoxedCommand. This is essentially a Command with files attached. Executing a BoxedCommand sets up a temporary directory, places input files inside that directory, executes the command, reads output files, and then cleans up the temporary directory. BoxedCommand can be instructed to either return an output file as a string, or to copy/move the file to another location.
A BoxedResult object is returned, which is similar to the existing Result object except that it also encapsulates output files.
A BoxedCommand can be executed remotely, via the client library, or locally. The execution code will be moved out to a separate Executor hierarchy.
class BoxedCommand { // ... public function setExecutor( BoxedExecutor $executor ) { $this->executor = $executor; } public function execute() { return $this->executor->execute( $this ); }
So for callers in MediaWiki, the interface will look like this:
$result = MediaWikiServices::getInstance() ->getCommandFactory() ->getBoxedCommand( ... params ... ) ->execute(); if ( $result->getExitCode() === 0 ) { $output = $result->getStdout(); }
The existing Command class is not quite suitable as a parent class of BoxedCommand, because BoxedCommand will be in a composer library whereas Command has b/c considerations and some MediaWiki dependencies. MediaWiki's Command can probably become a subclass of a loosely-coupled UnboxedCommand class within Shellbox.
I considered having the client post the data as multipart/form-data, and then the server can just use $_FILE. The problem with that is that for security, we want to have HMAC authentication of the POST contents. So my current plan is to post the data as multipart/mixed -- an efficient binary format used by email which is essentially identical to multipart/form-data -- and to parse it in PHP. That allows HMAC authentication to be done without having to reconstruct PHP's $_FILE logic.
I previously considered making the service be aware of Swift, but I don't think there is much benefit after T260504 is done.
multipart/mixed is a nice format for file transfer because it is binary-safe. JSON is officially UTF-8, using it to transfer binary data without base-64 encoding is quite dodgy.
Structured parameters within the multipart/mixed request are transferred as a JSON part.
File API
File names within the box are relative paths under a working directory. Running a command with files might look like this:
$result = $commandFactory->getBoxedCommand() ->inputFileFromString( 'input1.svg', $inputString ) ->inputFileFromFile( 'config.json', __DIR__ . '/config.json' ) ->outputFileToFile( 'out1.png', TempFSFile::factory( 'out', 'png' )->getPath() ) ->params( 'transform', "--input=input1.svg", "--conf=config.json", "--output=out1.png" ) ->execute(); $outputStr = $result->getFileContents( 'out1.png' );
Backwards compatibility
Shell::command() and wfShellExec() will continue to execute commands locally, without using the service, even in WMF production. MediaWiki's shell command handling will be moved to the Shellbox library, which will be required by MediaWiki core as a composer dependency. There will be some refactoring but "unboxed" commands will continue to work roughly as they always have.
The new BoxedCommand abstraction will be available via CommandFactory. BoxedCommand can either execute commands locally (the default) or it can execute commands remotely using the Shellbox service.
PHP RPC
It's trivial to add an endpoint to the server which runs arbitrary PHP code. This allows PHP execution to be done with time and memory limits, for example for T240884. There will be shared client library responsible for the protocol. Shell and RPC endpoints will wrap this client library.
Caller survey
- Media transformation
- Subclasses get a local copy of a file from swift and transform it, with the output being placed in a local directory for the framework to upload to swift
- Source download in File::getThumbnailSource() has a PoolCounter to avoid network saturation
- Thumbor theoretically replaces a lot of this, but it is loosely integrated so these code paths may end up getting hit anyway.
- BitmapHandler
- Scaling: runs ImageMagick with environment overrides
- Also standalone rotation without scaling
- Also “convert -version” with cache
- DjVuHandler
- Uses a pipeline ddjvu | pnmtojpeg
- SvgHandler
- rsvg with weird command template config
- JpegHandler
- Postprocess with exiftool
- Rotation with jpegtran
- PagedTiffHandler
- ImageMagick
- TimedMediaHandler
- ffmpeg
- fluidsynth
- PdfHandler
- gs | convert
- VipsScaler
- vips, exiv2
- Puts a bunch of vips commands into an array, runs them all
- The commands talk to each other via temporary files
- 3D
- xvfb-run 3d2png.js
- With environment
- Media metadata
- Framework provides a local source path, IIRC this is prior to Swift publication so the file is already local.
- Output goes into the DB
- DjVuImage
- Runs djvudump and djvutext on the source file
- JpegHandler
- exiftool
- PagedTiffHandler
- tiffinfo
- PdfHandler
- pdfinfo, pdftotext
- Video transcode
- TMH does a pcntl_fork(), then the pcntl_exec() is commented out and replaced by plain old wfShellExec()
- The parent monitors the output file size and provides progress updates
- Runs kill, ps directly with exec(), not via MW wrapper
- ffmpeg, fluidsynth
- @Joe notes: this case is complex enough to warrant its own separate service
- Special pages
- SecurePoll
- gpg for encryption
- CodeReview
- svn
- OpenStackManager
- ssh-keygen via proc_open
- SecurePoll
- Parser
- EasyTimeline
- Perl runs Ploticus
- SyntaxHighlight_GeSHi
- Poorly named extension runs pygmentize
- Score
- lilypond, gs, convert, fluidsynth
- EasyTimeline
- ResourceLoaderImage::rasterize()
- Runs rsvg with proc_open
- GlobalIdGenerator
- Runs ifconfig to get the MAC address, with local file cache
- GitInfo
- Uses git to inspect the local source tree. For Special:Version etc.
- Things that run MediaWiki maintenance scripts
- SiteConfiguration::getConfig()
- CirrusSearch/includes/Api/SuggestIndex.php
- FlaggedRevs Special:ValidationStatistics
- wfMerge()
- Runs diff3 for conflict merging on page save
- Uses popen() directly, no firejail
- Makes temporary input files, reads output from pipe
- Maintenance scripts
- Maintenance::getTermSize(): stty
- Maintenance::readlineEmulation(): readline
- psysh: which less
- mysql
- other maintenance scripts
- noc
- scap wikiversions-inuse
- Some things that are not used in WMF production:
- $wgAntivirus
- $wgExternalDiffEngine
- Installer::envCheckShellLocale()
- RandomImageGenerator
- FSFileBackend async mode
- cp, mv, test, chmod
Caller survey summary
Binaries used in the more tractable cases:
- convert
- ddjvu
- diff3
- djvudump
- djvutext
- exiftool
- exiv2
- ffmpeg
- fluidsynth
- gpg
- gs
- jpegtran
- lilypond
- pdfinfo
- pdftotext
- perl
- pnmtojpeg
- pygmentize
- rsvg
- ssh-keygen
- svn
- tiffinfo
- vips
- xvfb-run
More difficult or not applicable binaries
- git
- php
- ps
- kill
- stty
- readline
- ipconfig
A day in the life of a typical MediaWiki shell command
- Where is the binary? Usually it is statically configured but sometimes the PATH is used. If it’s not installed, I raise an error. Binary path configuration may be false or null, and if so, I do something else.
- I want to know the binary version, so I run the command with --version and cache the result.
- Here’s an input as a string. I write it to a temporary file.
- Here’s an input as a File object. I ask it for a local copy.
- Here’s an input which is a local file already. Easy!
- My command uses a lot of memory. I override the memory limits.
- I override some environment variables.
- My command needs to run with the working directory equal to the directory containing the input and output files. I call chdir().
- I run a command with some input files.
- The command generates some output files. I run another command that uses those output files as an input.
- I don’t know how many output files were generated. I need to find and gather them.
- One of the input files was modified by the command and now I want its new contents.
- What is the exit status?
- Something went wrong -- an output file I expected is not there. Maybe there is something in stderr or stdout that I can show to the user.
- All done, now I need to clean up the temporary files. Sometimes I know their names, sometimes I search for them.
Performance considerations
- Startup overhead impacts fast but common commands. We presumably win by forking a smaller process prior to exec() but lose by network stack overhead and container setup.
- Copying overhead impacts commands that run on large files
- OS-level containers are large and defeat library sharing. However, due to the way Docker works, the containers for the different routes will be very similar and will use the same base image, so will not use much additional disk space.
- Consider multipart encoding not ASCII.
Security considerations
- Can conduct timing attacks. RDTSC is unprivileged. This implies that you need either application-level security (disallow arbitrary machine code) or separate hardware.
- Linux is buggy and has a wide attack surface. Review the need for /proc, /sys, /dev.
- Read-only everything, or use an overlay
- Private /tmp
- Requests signed with a shared secret or asymmetric encryption
Credits
This proposal is based on @Joe's investigations into running MediaWiki on Kubernetes and originated with a conversation between @Joe and @tstarling.
To do
Work in progress is in the mediawiki/libs/Shellbox project in Gerrit.
Shellbox:
- Port FirejailCommandIntegrationTest and FirejailCommandTest from MediaWiki to Shellbox.
- Testing of Windows support
- Testing of systemd-run and firejail support.
- Enable PHPUnit tests in CI
- Parsing of shell commands to assist command validation.
- Per-route command validation.
- Decide whether to make Guzzle a hard dependency. If it is a hard dependency then we can provide a default HTTP client instead of just an interface.
MediaWiki integration: now tracked at T267532
Deployment:
- Routing
- Container creation
- ???
- wmf-config, service discovery etc.
- Score as a pilot