Page MenuHomePhabricator

RFC: Support PHP 7.4 preload
Open, MediumPublic

Description

  • Affected components: MediaWiki deployment at WMF. And maybe Scap and/or MediaWiki core.
  • Engineer for initial implementation: WMF Performance Team, WMF Service Ops
  • Code steward: Performance Team.

Motivation

Improve performance with lower backend latencies, by having PHP do less work on every web request.

The PHP version 7.4 provides a new way to tune performance in how the PHP code of an application is loaded and especially how caching of code of the application works. It introduces a way to preload PHP files that would otherwise be loaded on-demand when responding to web requests.

This RFC is for adding support for this feature in WMF's deployment of MediaWiki.

Requirements

The goal of this RfC is the following steps being implemented in MediaWiki:

  • Provide an entry point that can be referenced in the opcache.preload PHP ini setting, which will preload (some) of MediaWiki.
  • Make it possible to preload significant parts of MediaWiki core and MediaWiki extensions source code.

Exploration

Status quo

MediaWiki already greatly supports the use of many layers of caching, such as OpCache on the application code level, memcached and redis on the data level and so on. All of these layers are out of scope of this RfC. MediaWiki, as any other PHP application, still has the problem as described in the original php.net RfC for the preloading feature.

However, to support the solution of the PHP RfC, each administrator of a PHP server needs to add an option to their php.ini with a path to a file which specifies the files that should be pre-loaded. This requires them to know what this list is. Exactly at this point, this RfC is pointing at.

Considerations/Problems

There're some drawbacks from preloading an application or just parts of it into the php process:

  • Changes to the file system of pre-loaded files will not be reflected in the php-process memory, hence making the file system changes not being reflected when requesting the application. This makes updating MediaWiki or parts of it more difficult as a server restart is required.
  • Only one preload file can be specified, making different applications or versions of MediaWiki, running in the same php server, nearly impossible
  • When class names are conflicting when pre-loading them, opt's them out from being cached, making preloading only reasonable for one application, only
  • The implementation of this RfC needs to take into account, that code comes from MediaWiki core, as well as from extensions and also from composer and maybe other ways
  • when pre-loading all, MediaWiki core, extensions and composer:
    • Restarting php-fpm with preloading enabled may take a considerable longer time when all MediaWiki core, extensions and composer code is being preloaded
    • The default shared memory and max memory limit values may not be high enough to load all code

Proposals

There're many possible ways of how the preloading feature could be supported inside of MediaWiki.

Maintenance script (static preload file)

Using a maintenance script to generate the preload file (similar to how the autoload.php is updated in MediaWiki core) would allow a system administrator to run it right before deploying the MediaWiki installation to the production system. This allows the implementation to know exactly what files are present on the file system and therefore would just need to go over all the files in the MediaWiki installation and load them all into the cache.

Dynamic preload file

It seems it's possible to use a PHP class or function inside of the preload script, which then could go through the MediaWiki installation directory during the startup phase of the PHP server and provide all of the current files on the file system to the cache.

Composer

Composer has, as of December 2019, an outstanding issue where possible ways of autoloading code of dependencies. As long as the issue is not solved, or if it does not provide a compatible way, MediaWiki core would simply include the files from composer and third-party dependencies in one of the solutions above.


Other resources

  • Symfony already added support for preloading by generating a preload file into the cache folder of the framework
  • Composer is discussing about good ways how to support preloading

Event Timeline

Based on the script published in the original php RfC, I build a preload file for my local environment to see what's going on:

<?php
function _preload( $preload, string $pattern = "/\.php$/", array $ignore = [] ) {
	if ( is_array( $preload ) ) {
		foreach ( $preload as $path ) {
			_preload( $path, $pattern, $ignore );
		}
	} else {
		if ( is_string( $preload ) ) {
			$path = $preload;
			if ( !in_array( $path, $ignore ) && !strpos( $path, 'test' ) &&
				!strpos( $path, 'tests' ) && !strpos( $path, 'Test' ) &&
				!strpos( $path, 'Tests' ) && !strpos( $path, 'scripts' ) ) {
				if ( is_dir( $path ) ) {
					if ( $dh = opendir( $path ) ) {
						while ( ( $file = readdir( $dh ) ) !== false ) {
							if ( $file !== "." && $file !== ".." ) {
								_preload( $path . "/" . $file, $pattern, $ignore );
							}
						}
						closedir( $dh );
					}
				} else {
					if ( is_file( $path ) && preg_match( $pattern, $path ) ) {
						if ( !opcache_compile_file( $path ) ) {
							trigger_error( "Preloading Failed", E_USER_ERROR );
						}
					}
				}
			}
		}
	}
}

_preload( [
	__DIR__,
], "/\.php$/", [
	"/code/w/vendor/khanamiryan",
	"/code/w/preload.php",
	"/code/w/extensions/ConfirmEdit/FancyCaptcha/includes",
] );

The ignores at the end are mostly because of PHP notices, which seems to prevent the startup phase of fpm. Ignoring them for now is ok from my point of view in order to get the stuff running and then improve on top of it.

With this, the php server is pre-loading a bunch of functions and classes correctly, however, whenever I request the main page of my local wiki, the fpm worker SIGSEGVs. I haven't looked into it as of now. I'm also using the latest 7.4 release and also tried an unofficial build from the source in docker.

Based on the description here, it sounds pretty impossible to use this with hetdeploy, so is this about third parties only?

@Bawolff Hot deploying is complicated, but it really depends on what you want to pre-load and what not. If you, e.g., focus on deploying config changes only, then you can exclude the configuration paths from pre-loading and let the usual opcache handle them only.

Hot deploying bug fixes or new versions on the other hand is not possible _without_ restarting the php-fpm processes, however, there're maybe ways of working around that as well, depending on the deployment infrastructure (you do not need to restart all processes at the same time).

What I see as the biggest "blocker" for this to be used on wmf infrastructure, would be the fact, that you can not preload separate files which define the same class names or functions (as they would conflict obviously). So, using multiple different versions of MediaWiki within the same scope of a single php-fpm is not possible if you want to preload all versions. You would either need to decide if you want to preload only one version of MediaWiki or neither of them. I think that's kind of impossible to decide and, at the same time, get any benefit of preloading at all?

What I see as the biggest "blocker" for this to be used on wmf infrastructure, would be the fact, that you can not preload separate files which define the same class names or functions (as they would conflict obviously). So, using multiple different versions of MediaWiki within the same scope of a single php-fpm is not possible if you want to preload all versions. You would either need to decide if you want to preload only one version of MediaWiki or neither of them. I think that's kind of impossible to decide and, at the same time, get any benefit of preloading at all?

This is what I meant by hetdeploy. But reading the php rfc, you couldn't even preload just one version and rely on the other version being normal loaded, as its not just you can't preload two conflicting class names, but you can't even (normal) load a class name that conflicts with a preloaded classname, I think(?)

You're right, even if the same name is used from another version of the same application, the preloading class "wins" and may result in incompatibilities of the application code :/

so is this about third parties only?

To answer thi question: It seems, that with the current implementation of preloading in php, it is incompatible to the requirements and implementation of the wmf deployment, yes.

The model of MW-in-containers we're heading for will have each PHP process (within the container) running only one version of MW at a time ('hetdeploy' as it were will be done at a larger scale, with blue/green etc. container deployment), so this isn't out of scope for Wikimedia usage eventually.

Ok, on another system, I was now able to get a core dump from the exiting php-fpm worker:

Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fa8a7a60899 in __GI_abort () at abort.c:79
#2  0x00007fa8a7acb38e in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fa8a7bf43a5 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007fa8a7ad34dc in malloc_printerr (str=str@entry=0x7fa8a7bf60a0 "double free or corruption (out)") at malloc.c:5332
#4  0x00007fa8a7ad5170 in _int_free (av=0x7fa8a7c25b80 <main_arena>, p=0x559f93b4ea80, have_lock=<optimized out>) at malloc.c:4314
#5  0x0000559f923f3957 in zend_hash_do_resize (ht=0x559f93a14370) at ./Zend/zend_types.h:514
#6  zend_hash_do_resize (ht=0x559f93a14370) at ./Zend/zend_hash.c:1146
#7  0x0000559f923f3ab9 in _zend_hash_add_or_update_i (flag=2, pData=0x7ffe244b6f40, key=0x7fa89d0c4900, ht=0x559f93a14370) at ./Zend/zend_hash.c:772
#8  zend_hash_add (ht=0x559f93a14370, key=key@entry=0x7fa89d0c4900, pData=pData@entry=0x7ffe244b6f40) at ./Zend/zend_hash.c:874
#9  0x0000559f923d4301 in zend_hash_add_ptr (pData=0x7fa8a5805168, key=0x7fa89d0c4900, ht=<optimized out>) at ./Zend/zend_hash.h:591
#10 zend_compile_class_decl (ast=0x7fa8a58bcbc8, toplevel=<optimized out>) at ./Zend/zend_compile.c:6517
#11 0x0000559f923d4e37 in zend_compile_top_stmt (ast=0x7fa8a58bcbc8) at ./Zend/zend_compile.c:8424
#12 0x0000559f923d4e60 in zend_compile_top_stmt (ast=0x7fa8a58bb018) at ./Zend/zend_compile.c:8413
#13 0x0000559f923ac614 in zend_compile (type=type@entry=2) at Zend/zend_language_scanner.l:614
#14 0x0000559f923adcf2 in compile_file (file_handle=0x7ffe244b7660, type=8) at Zend/zend_language_scanner.l:648
#15 0x00007fa8a4d0365d in ?? () from /usr/lib/php/20190902/phar.so
#16 0x00007fa8a5a166c2 in ?? () from /usr/lib/php/20190902/opcache.so
#17 0x00007fa8a5a19321 in ?? () from /usr/lib/php/20190902/opcache.so
#18 0x0000559f92432608 in zend_include_or_eval (inc_filename=0x7fa89f9a3dc0, type=16) at ./Zend/zend_execute.c:4206
#19 0x0000559f9244e771 in ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER () at ./Zend/zend_vm_execute.h:4037
#20 0x0000559f92467a4d in execute_ex (ex=0x2) at ./Zend/zend_vm_execute.h:53857
#21 0x0000559f9246f0fb in zend_execute (op_array=0x7fa8a58750e0, return_value=0x0) at ./Zend/zend_vm_execute.h:57651
#22 0x0000559f923e6b7c in zend_execute_scripts (type=type@entry=8, retval=0x7fa89f9a3e10, retval@entry=0x0, file_count=-1518248896, file_count@entry=3) at ./Zend/zend.c:1663
#23 0x0000559f923868f0 in php_execute_script (primary_file=<optimized out>) at ./main/main.c:2619
#24 0x0000559f9225017b in main (argc=<optimized out>, argv=<optimized out>) at ./sapi/fpm/fpm/fpm_main.c:1944

Looks a bit like https://bugs.php.net/bug.php?id=78894
I'm going to compile php from source to see if the same error occurs again.

Latest master segfaults with:

#0  0x000055d1f20286e9 in zend_remove_ini_entries (el=0x55d1f43e09a0, arg=0x7ffd91ccd7cc) at /home/florian/php-src/Zend/zend_ini.c:40
#1  0x000055d1f202030e in zend_hash_apply_with_argument (ht=0x55d1f427d750, apply_func=apply_func@entry=0x55d1f20286e0 <zend_remove_ini_entries>, argument=argument@entry=0x7ffd91ccd7cc) at /home/florian/php-src/Zend/zend_hash.c:1839
#2  0x000055d1f2028ce4 in zend_unregister_ini_entries (module_number=<optimized out>) at /home/florian/php-src/Zend/zend_ini.c:271
#3  0x00007fa64e6c15af in zm_shutdown_zend_accelerator (type=<optimized out>, module_number=<optimized out>) at /home/florian/php-src/ext/opcache/zend_accelerator_module.c:422
#4  0x000055d1f20148df in module_destructor (module=module@entry=0x55d1f4442050) at /home/florian/php-src/Zend/zend_API.c:2382
#5  0x000055d1f200e780 in module_destructor_zval (zv=<optimized out>) at /home/florian/php-src/Zend/zend.c:769
#6  0x000055d1f201feda in _zend_hash_del_el_ex (prev=<optimized out>, p=<optimized out>, idx=33, ht=<optimized out>) at /home/florian/php-src/Zend/zend_hash.c:1308
#7  _zend_hash_del_el (p=0x55d1f42ec8f0, idx=33, ht=0x55d1f2cf9060 <module_registry>) at /home/florian/php-src/Zend/zend_hash.c:1331
#8  zend_hash_graceful_reverse_destroy (ht=ht@entry=0x55d1f2cf9060 <module_registry>) at /home/florian/php-src/Zend/zend_hash.c:1785
#9  0x000055d1f2013250 in zend_destroy_modules () at /home/florian/php-src/Zend/zend_API.c:1837
#10 0x000055d1f200f6ce in zend_shutdown () at /home/florian/php-src/Zend/zend.c:1069
#11 0x000055d1f1faed55 in php_module_shutdown () at /home/florian/php-src/main/main.c:2446
#12 0x000055d1f1fb0635 in php_module_shutdown () at /home/florian/php-src/main/main.c:2429
#13 0x000055d1f20a067d in fpm_php_cleanup (which=<optimized out>, arg=<optimized out>) at /home/florian/php-src/sapi/fpm/fpm/fpm_php.c:198
#14 0x000055d1f2098f2d in fpm_cleanups_run (type=type@entry=4) at /home/florian/php-src/sapi/fpm/fpm/fpm_cleanup.c:43
#15 0x000055d1f20a1144 in fpm_pctl_exit () at /home/florian/php-src/sapi/fpm/fpm/fpm_process_ctl.c:71
#16 fpm_pctl_action_last () at /home/florian/php-src/sapi/fpm/fpm/fpm_process_ctl.c:118
#17 0x000055d1f20a202b in fpm_pctl (action=2, new_state=0) at /home/florian/php-src/sapi/fpm/fpm/fpm_process_ctl.c:260
#18 fpm_pctl_child_exited () at /home/florian/php-src/sapi/fpm/fpm/fpm_process_ctl.c:260
#19 0x000055d1f2098a57 in fpm_children_bury () at /home/florian/php-src/sapi/fpm/fpm/fpm_children.c:266
#20 0x000055d1f209dc8a in fpm_event_fire (ev=0x55d1f2cf2600 <children_bury_timer>) at /home/florian/php-src/sapi/fpm/fpm/fpm_events.c:482
#21 fpm_event_loop (err=err@entry=0) at /home/florian/php-src/sapi/fpm/fpm/fpm_events.c:462
#22 0x000055d1f2098297 in fpm_run (max_requests=0x7ffd91ccdcac) at /home/florian/php-src/sapi/fpm/fpm/fpm.c:113
#23 0x000055d1f1d1ccee in main (argc=6, argv=0x7ffd91cce1d8) at /home/florian/php-src/sapi/fpm/fpm/fpm_main.c:1852

As I do not have any clue what to do here, I opened a bug: https://bugs.php.net/bug.php?id=78975 Let's see what they say?

The model of MW-in-containers we're heading for will have each PHP process (within the container) running only one version of MW at a time ('hetdeploy' as it were will be done at a larger scale, with blue/green etc. container deployment), so this isn't out of scope for Wikimedia usage eventually.

to be clear, this would be possible even outside of containers with some mangling of how we organize deployed code.

But in theory, we could run three different php-fpm pools, each related to a deployment-group, thus with only the right versions loaded.

OTOH, preloading would mostly break scap as it is.

Is the performance / memory tradeoff an issue, given that on a normal request most classes do not need to be loaded into memory? (Or maybe that would be offset by classes only being loaded once per server, not once per thread?)

That would be something that need to be proved first by seeing what the performance impact is (in an isolated setup). That's what I want to try to get done, to have data we can decide upon. However, there's the problem, that the php-fpm worker crashes when MediaWiki classes are preloaded (that's what I've reported as a PHP bug entry to see if that's really a bug or something I did wrong when setting up the preload).

However, even if not nearly all classes are used on a "normal" request, preloading could be of benefit, as these classes does not need to be rechecked with the fs if they need to be updated or not. Also, it would be possible to change the amount of preloaded classes as well (instead of "just" iterating over the filesystem and preload all classes), however, that would be an improvement, which probably needs a lot of more data and work to find out what classes should be preloaded and which not. And how this list may be maintained. However, that's, for me, a step after the first one, getting preloading working in the first place :D

The latest master of php (mentioned as 8.0.0-dev) seems to work with preloading MediaWiki, at least it does not segfault anymore, as of now. However, there seems to be some stuff that needs to be investigated, as I currently get a fatal error, that a function in WebStart can not be redeclared:

Fatal error: Cannot redeclare wfWebStartSetup() (previously declared in /var/www/html/w/includes/WebStart.php:73) in /var/www/html/w/includes/WebStart.php on line 70

Looking into it now :)

Krinkle updated the task description. (Show Details)
Krinkle moved this task from Under discussion to P2: Resource on the TechCom-RFC board.
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.
Krinkle renamed this task from Support PHP 7.4 preload to RFC: Support PHP 7.4 preload.Apr 4 2020, 2:34 AM

Change 585882 had a related patch set uploaded (by Florianschmidtwelzow; owner: Florianschmidtwelzow):
[mediawiki/core@master] [preload] Do not conditionally declare functions in WebStart.php

https://gerrit.wikimedia.org/r/585882

Ok, looking at the status of the opcache after the server startup, it looks like, that the function wfWebStartSetup is present in the cache, like one would expect. However, when WebStart.php is included from index.php or load.php, we evaluate, if a constant named MW_SETUP_CALLBACK is defined already (which is later called in Setup.php, which does not matter for the current case). This constant is, also like expected, not defined when running from index.php or load.php and MediaWiki tries to register the callback function wfWebStartSetup. The problem seems to be, that evaluating this code, MediaWiki defines the function conditionally inside of the defined check for the MW_SETUP_CALLBACK constant. This seems to result in PHP trying to define the wfWebStartSetup function again, even given it is already defined from the preloading run during startup.

Evidence for that:

  • If the function is only declared with a protecting function_exists check, the error disappears (while the function is still existing and correctly called from Setup.php. Example code:
// Custom setup for WebStart entry point
if ( !defined( 'MW_SETUP_CALLBACK' ) ) {
		if ( !function_exists( 'wfWebStartSetup' ) ) {
			function wfWebStartSetup() {
					// Initialise output buffering
					// Check for previously set up buffers, to avoid a mix of gzip and non-gzip output.
					if ( ob_get_level() == 0 ) {
							ob_start( 'MediaWiki\\OutputHandler::handle' );
					}
			}
		}

		define( 'MW_SETUP_CALLBACK', 'wfWebStartSetup' );
}
  • If the function is not declared conditionally, the error disappears as well (the function _seems_ to be declared once, during the preloading run at server startup, and cached in the opcache once. Example code:
function wfWebStartSetup() {
		// Initialise output buffering
		// Check for previously set up buffers, to avoid a mix of gzip and non-gzip output.
		if ( ob_get_level() == 0 ) {
				ob_start( 'MediaWiki\\OutputHandler::handle' );
		}
}

// Custom setup for WebStart entry point
if ( !defined( 'MW_SETUP_CALLBACK' ) ) {
		define( 'MW_SETUP_CALLBACK', 'wfWebStartSetup' );
}

-> I've added a patch for changing that here: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/585882 (as a reference, we probably should merge it only, if this task goes forward, otherwise it does not really bring any value, even if it does not make something bad though :D

Krinkle triaged this task as Medium priority.Sep 18 2020, 3:03 AM
Krinkle updated the task description. (Show Details)
Krinkle moved this task from P2: Resource to P3: Explore on the TechCom-RFC board.

Something to consider: If preload will encompass not just core code but e.g. extension classes, it may break extensions that use class_exists checks in conditionals to determine if some other extension is loaded, since class_exists will always return true for all preloaded classes, even if the extension providing them might not be enabled in the context of the current wiki. So these checks may need to be swapped out to use ExtensionRegistry::isLoaded or an equivalent instead.

Naive codesearch: https://codesearch.wmcloud.org/extensions/?q=class_exists&i=nope&files=&excludeFiles=&repos=

Most hits seem to be in SMW and its ecosystem, but there are a few in e.g. CentralAuth that'll need to be looked at.

Something to consider: If preload will encompass not just core code but e.g. extension classes, it may break extensions that use class_exists checks in conditionals to determine if some other extension is loaded, since class_exists will always return true for all preloaded classes, even if the extension providing them might not be enabled in the context of the current wiki.[…]

I think the first pass at this will be to figure out the MWMultiVersion aspect, because without that we can't even preload core classes. I'm thinking that we may want to transition WMF's setup for multiversion to handle it at the php-fpm or Apache level instead of multiversion. That way there's only one instance for a given deployment.

This was previously considered at T253673, but not needed at the time. The idea still makes sense I think, though, and also helps with two other issues:

  1. If I recall correctly from listening to @Joe in various tasks and conversations, I understand that our current appserver setup is bottlenecked in non-trivial ways by the load that a single php-fpm instance's master thread is able to handle. Thus we probably aren't making optimal use of our capacity today, and might be able to get better performance if we used multiple (smaller) fpm instances. As I understand it, this is also among other reason why in the future containerised cluster for MediaWiki, the individual pods will be smaller than what an app server currently does, and I believe we expect that to actually improve capacity and latency.
  2. The work needed to split php-fpm, as likely needed for php74-preload compatibility, would thus largely also serve as preparation work for containerization. This might help prioritise resourcing of that work.

If we go in this direction, I think an initial php74-preload roll out could use WebStart.php as starting point, which covers a pretty significant portion of initialization time for web requests (covers multiversion + wmf-config + etcd + autoloader + Setup + some or most of WebStart).

I'm not sure how much benefit we would get eagerly loading class definitions. I expect this to not matter so much since that's largely opcache's responsibility, and would matter even less after we disable opcache's revalidation-ttl's (thus no more file stats for mtimes).

Rather, the main benefit I'm hoping to get is from the global state snapshot after all the initialisation statements. Eagerly loading "all" classes could also backfire in that it might mean the memory footprint per-request could increase quite significantly due to static values all being in-process. Something we can measure for sure, but given that this would violate some expectations with regards to extension compatibility etc it might not more trouble than its worth.

I'm not sure how much benefit we would get eagerly loading class definitions. I expect this to not matter so much since that's largely opcache's responsibility, and would matter even less after we disable opcache's revalidation-ttl's (thus no more file stats for mtimes).

The preload RFC implies that one of the aims of this approach was to reduce some of the overhead that still exists even with opcache:

While storing files in an opcode cache eliminates the compilation overhead -- there is still cost associated with fetching a file from the cache and into a specific request's context. We still have to check if the source file was modified, copy certain parts of classes and functions from the shared memory cache to the process memory, etc. Notably, since each PHP file is compiled and cached completely independently from any other file, we can't resolve dependencies between classes stored in different files when we store the files in the opcode cache, and have to re-link the class dependencies at run-time on each request.

I am not familiar with PHP internals so it's difficult for me to judge what kind of speedup this would amount to in a real-life application. However, it could be worthwhile to test and measure if preloading some of the most used classes brought any benefit. I agree that preloading all classes would probably be more trouble than it's worth, especially in a larger deployment where some extensions are only enabled in a few places.

Change 585882 merged by jenkins-bot:
[mediawiki/core@master] WebStart: Avoid conditionally declared functions in this file

https://gerrit.wikimedia.org/r/585882