Reported in the #wikimedia-labs irc channel:

[23:25]  <Amitie_10g>	I got working HHVM in FastCGI by invoking the following in the following
[23:25]  <Amitie_10g>	"/usr/bin/hhvm --mode daemon -d hhvm.server.type=fastcgi -d hhvm.server.file_socket=/var/run/lighttpd/hhvm.socket.webarchivebot -d hhvm.server.source_root=/data/project/webarchivebot/public_html -d error_log=/data/project/webarchivebot/hhcv_error.log -d hhvm.log.file=/data/project/webarchivebot/hhvm_log.log"

If we try proxygen mode, the webservice backend will need to provide a way for the user to provide configuration settings for the HHVM process in some per-tool config file. This would be needed for things like setting up rewrite rules for tools that use a dispatcher script.

As the user who is interested in this, I have some question that has been not answered at the IRC. Based in my knlwledge, I see the following situations:

HHVM in Proxygen mode supports only numbered ports. Then, a Kubernetes container can run two instances of HHVM in Proxygen mode, listening the ports 80 and 443. Then, nginx proxy forwards the traffic originated to https://tools.wmflabs.org/**webarchivebot** to the HHVMs instances running inside the WebArchiveBOT's Kubernetes container. Everything (enabling SSL and setting the certificates paths) should be declared in the command line,or in the configuration file.

HHVM running in FastCGI mode supports both ports and UNIX sockets. As I know, based on my tests, HHVM is able to launch only one socket per instance,but it can listen in ports 80 and 443.

I created my own .lighttpd.conf (showing only the fastcgi.server section):

fastcgi.server += ( ".php" =>
        ((
                "bin-path" => "/usr/bin/hhvm -m daemon --user tools.webarchivebot -d hhvm.server.type=fastcgi -d hhvm.server.file_socket=/var/run/lighttpd/php.socket.{toolname} -d hhvm.log.use_log_file=true -d error_log=/data/project/{toolname}/hhvm_error_log -d hhvm.log.file=/data/project/{toolname}/hhvm_log",
                "socket" => "/var/run/lighttpd/php.socket.{toolname}",
                "max-procs" => 1,
                "bin-environment" => (
                        "PHP_FCGI_CHILDREN" => "1",
                        "PHP_FCGI_MAX_REQUESTS" => "500"
                ),
                "bin-copy-environment" => (
                        "PATH", "SHELL", "USER"
                ),
                "broken-scriptfilename" => "enable",
                "allow-x-send-file" => "enable"
         ))
)

Everything should be defined in the command line, or in the INI file.

This is I tried.

In T164161#3223888, @bd808 wrote:

If we try proxygen mode, the webservice backend will need to provide a way for the user to provide configuration settings for the HHVM process in some per-tool config file. This would be needed for things like setting up rewrite rules for tools that use a dispatcher script.

As I seen, that can be defined in $HOME/server.ini (where HHVM looks in). But Proxygen seems to be toolimited compared to FastCGI mode.

I suggest to create two Kubernetes Container variants: hhvm-proxygen (standalone server without rules, serving in both 80 and 443 ports with SSL and certificates configured in the command line or the HHVM configuration file) and hhvm-fastcgi (using lighttpd with a modified configuration of php-cgi).

In T164161#3224040, @Amitie_10g wrote:

HHVM in Proxygen mode supports only numbered ports. Then, a Kubernetes container can run two instances of HHVM in Proxygen mode, listening the ports 80 and 443. Then, nginx proxy forwards the traffic originated to https://tools.wmflabs.org/**webarchivebot** to the HHVMs instances running inside the WebArchiveBOT's Kubernetes container. Everything (enabling SSL and setting the certificates paths) should be declared in the command line,or in the configuration file.

Only one backend instance of the HHVM proxygen (or fcgi) server is needed. HTTPS requests are handled by the Tool Labs nginx proxy which has the proper SSL certificates. Requests from the nginx proxy to each tool's webservice are done via plain HTTP across the Labs network segment. An X-Forwarded-Proto header is added to requests sent through the proxy to ecah webservice so that the webservice can determine the protocol in use by the client.

In T164161#3224045, @bd808 wrote:

In T164161#3224040, @Amitie_10g wrote:

HHVM in Proxygen mode supports only numbered ports. Then, a Kubernetes container can run two instances of HHVM in Proxygen mode, listening the ports 80 and 443. Then, nginx proxy forwards the traffic originated to https://tools.wmflabs.org/**webarchivebot** to the HHVMs instances running inside the WebArchiveBOT's Kubernetes container. Everything (enabling SSL and setting the certificates paths) should be declared in the command line,or in the configuration file.

Only one backend instance of the HHVM proxygen (or fcgi) server is needed. HTTPS requests are handled by the Tool Labs nginx proxy which has the proper SSL certificates. Requests from the nginx proxy to each tool's webservice are done via plain HTTP across the Labs network segment. An X-Forwarded-Proto header is added to requests sent through the proxy to ecah webservice so that the webservice can determine the protocol in use by the client.

Good,this is exactly what I'm looking. At least,my tool does not require rewrite rules, so, the Proxygen flavour can be deployed first. If someone needs the FastCGI flavour,the user can request it, but for now I don't see it necessary.

I'm trying to run HHVM in Proxygen mode inside my tool account and I opened the page successfully. Following is the way to run HHVM:

hhvm -m daemon -c $HOME/server.ini

It looks for the server.ini file (located at $HOME). The following configuration is enough (but necessary) to run HHVM successfully:

hhvm.server.type = proxygen
hhvm.server.user = {username}
hhvm.server.source_root=$HOME/public_html
hhvm.log.use_log_file = true
error_log = $HOME/hhvm_server_error.log
hhvm.log.file = $HOME/hhvm_server.log
hhvm.pid_file = var/run/hhvm.www.{username}.pid
date.timezone=UTC
hhvm.enable_zend_sorting=1

However, there is a serious drawback: hhvm.server.default_document option can be set only once; if specified more than once,only the last instance will be taken.This mean that only "index.php" is taken as the default document, and, for example, index.html will be ignored if it is ommited when typing the URL:

For example, https://tools.wmflabs,org/webarchivebot will display the implicit "index.php", but when typing https://tools.wmflabs,org/webarchivebot/doc (that contains only plain HTML files) will throw a 404 error due the server haven't found the default document ("index.php"), even if there is an "index.html". Therefore, I sent a bug report.

For now, the only solution for that is using HHVM in FastCGI mode with lighttpd.

I spent some time exploring running HHVM in proxygen mode using webservice generic and a bootstrapping script that generates an HHVM ini file and starts HHVM in server mode. This does make a working webservice, but it has several drawbacks:

• Documentation for configuring HHVM's proxygen webserver is lacking upstream. Information can be found, but it requires a lot of digging.
• No obvious support for alias configuration to easily map https://tools.wmflabs.org/my-tool-name/ to the tool's $HOME/public_html. This can be worked around using hhvm.virtual_host[default][rewrite_rules] settings.
• No obvious way to do multi-view indexing as noted by @Amitie_10g in T164161#3224126

My experiments show that using proxygen mode is certainly possible, but based on the lack of documentation and that we don't use it in WMF production I would recommend against it for the average Tool Labs user. I will be looking more closely at fcgi mode configuration for integration with webservice.

For those who are curious, here's the script that I came up with to generate an appropriate configuration file and start the HHVM service using webservice --backend=gridengine generic start ${HOME}/hhvm-webservice.sh:

#!/usr/bin/env bash
# Run an HHVM webservice
#
# usage: webservice --backend=gridengine generic start hhvm-webservice.sh
set -e

TOOLNAME=${USER#tools.}

if [[ -z $PORT ]]; then
  echo "PORT environment variable not set." >&2
  echo "usage: webservice --backend=gridengine generic start $0" >&2
  exit 1
fi

/bin/cat << EOF > ${HOME}/server.ini
date.timezone = UTC
hhvm.enable_obj_destruct_call = true
hhvm.enable_zend_compat = true
hhvm.error_handling.call_user_handler_on_fatals = true
hhvm.hack.lang.iconv_ignore_correct = true
hhvm.jit = true
hhvm.log.always_log_unhandled_exceptions = true
hhvm.log.level = Warning
hhvm.log.native_stack_trace = false
hhvm.log.runtime_error_reporting_level = HPHP_ALL
hhvm.log.use_log_file = true
hhvm.log.use_syslog = false
hhvm.pcre_cache_type = lru
hhvm.pid_file =
hhvm.repo.central.path = /tmp/hhvm-${TOOLNAME}.hhbc
hhvm.server.apc.expire_on_sets = true
hhvm.server.apc.expire_on_sets = true
hhvm.server.apc.purge_frequency = 4096
hhvm.server.apc.table_type = concurrent
hhvm.server.apc.ttl_limit = 172800
hhvm.server.default_document = index.php
hhvm.server.dns_cache.enable = true
hhvm.server.dns_cache.ttl = 300
hhvm.server.exit_on_bind_fail = true
hhvm.server.port = ${PORT}
hhvm.server.source_root = ${HOME}/public_html
hhvm.server.stat_cache = true
hhvm.server.thread_count = 4
hhvm.server.type = proxygen
hhvm.virtual_host[default][rewrite_rules][0][pattern] = "^/${TOOLNAME}(.*)\$"
hhvm.virtual_host[default][rewrite_rules][0][qsa] = true
hhvm.virtual_host[default][rewrite_rules][0][to] = "\$1"
max_execution_time = 60
memory_limit = 128M
EOF

exec /usr/bin/hhvm -m server -c ${HOME}/server.ini -c ${HOME}/hhvm.ini

In T164161#3224443, @bd808 wrote:

I spent some time exploring running HHVM in proxygen mode using webservice generic and a bootstrapping script that generates an HHVM ini file and starts HHVM in server mode. This does make a working webservice, but it has several drawbacks:

• Documentation for configuring HHVM's proxygen webserver is lacking upstream. Information can be found, but it requires a lot of digging.
• No obvious support for alias configuration to easily map https://tools.wmflabs.org/my-tool-name/ to the tool's $HOME/public_html. This can be worked around using hhvm.virtual_host[default][rewrite_rules] settings.
• No obvious way to do multi-view indexing as noted by @Amitie_10g in T164161#3224126
  
  My experiments show that using proxygen mode is certainly possible, but based on the lack of documentation and that we don't use it in WMF production I would recommend against it for the average Tool Labs user. I will be looking more closely at fcgi mode configuration for integration with webservice.
  
  For those who are curious, here's the script that I came up with to generate an appropriate configuration file and start the HHVM service using `webservice --backend=gridengine generic start ${HOME}/hhvm-webservice.sh:

hhvm-webservice.sh

#!/usr/bin/env bash
# Run an HHVM webservice
#
# usage: webservice --backend=gridengine generic start hhvm-webservice.sh
set -e

TOOLNAME=${USER#tools.}

if [[ -z $PORT ]]; then
  echo "PORT environment variable not set." >&2
  echo "usage: webservice --backend=gridengine generic start $0" >&2
  exit 1
fi

/bin/cat << EOF > ${HOME}/server.ini
date.timezone = UTC
hhvm.enable_obj_destruct_call = true
hhvm.enable_zend_compat = true
hhvm.error_handling.call_user_handler_on_fatals = true
hhvm.hack.lang.iconv_ignore_correct = true
hhvm.jit = true
hhvm.log.always_log_unhandled_exceptions = true
hhvm.log.level = Warning
hhvm.log.native_stack_trace = false
hhvm.log.runtime_error_reporting_level = HPHP_ALL
hhvm.log.use_log_file = true
hhvm.log.use_syslog = false
hhvm.pcre_cache_type = lru
hhvm.pid_file =
hhvm.repo.central.path = /tmp/hhvm-${TOOLNAME}.hhbc
hhvm.server.apc.expire_on_sets = true
hhvm.server.apc.expire_on_sets = true
hhvm.server.apc.purge_frequency = 4096
hhvm.server.apc.table_type = concurrent
hhvm.server.apc.ttl_limit = 172800
hhvm.server.default_document = index.php
hhvm.server.dns_cache.enable = true
hhvm.server.dns_cache.ttl = 300
hhvm.server.exit_on_bind_fail = true
hhvm.server.port = ${PORT}
hhvm.server.source_root = ${HOME}/public_html
hhvm.server.stat_cache = true
hhvm.server.thread_count = 4
hhvm.server.type = proxygen
hhvm.virtual_host[default][rewrite_rules][0][pattern] = "^/${TOOLNAME}(.*)\$"
hhvm.virtual_host[default][rewrite_rules][0][qsa] = true
hhvm.virtual_host[default][rewrite_rules][0][to] = "\$1"
max_execution_time = 60
memory_limit = 128M
EOF

exec /usr/bin/hhvm -m server -c ${HOME}/server.ini -c ${HOME}/hhvm.ini

Well, I'm running my tool with the generic webserver you provided and works perfectly. I only need to tweak the links to point to the right index.html at the /doc.

Then, what is the state of the HHVM in FastCGI mode?

I attempted to use Repo Authoritative by adding the following to hhvm-webservice.sh:

hhvm.repo.authoritative=true
hhvm.repo.central.path=${HOME}/hhvm.hhbc

However, attempting to build the repo according to the Documentation, the process fails:

tools.webarchivebot@tools-bastion-03:~$ hhvm --hphp -t hhbc -v AllVolatile=true --keep-tempdir=1 --log=7 public_html/index.php public_html/.config.php public_html/phpinfo.php
running hphp...
creating temporary directory /tmp/hphp_7pAnay ...
parsing inputs...
parsing ./public_html/.config.php ...
parsing ./public_html/phpinfo.php ...
parsing ./public_html/index.php ...
analyzeProgram...
Analyzing Includes
Analyzing All
analyzeProgram took 0'00" (72165 us) wall time
parsing inputs took 0'00" (110944 us) wall time
pre-optimizing...
pre-optimizing took 0'00" (83357 us) wall time
creating binary HHBC files...
terminate called without an active exception
Core dumped: Aborted
Stack trace in /tmp/stacktrace.32346.log
hphp failed
running hphp took 0'01" (1703318 us) wall time

And generates an empty Stack trace (/tmp/stacktrace.32346.log is empty).

Also,if Repo generation is successful, should be a good idea to add hhvm --hphp -t hhbc -v AllVolatile=true --input-list master-file-list.txt before launching HHVM in server mode, to (re)generate the Repo every time the webservice is started (the master-file-list.txt should be filled by the user).

In T164161#3224547, @Amitie_10g wrote:

I attempted to use Repo Authoritative by adding the following to hhvm-webservice.sh:

hhvm.repo.authoritative=true
hhvm.repo.central.path=${HOME}/hhvm.hhbc

As @ori wrote upstream in https://github.com/facebook/hhvm/issues/6878, RepoAuth mode is difficult to get right. A tool would have to be very, very active before dealing with the complexity of RepoAuth would be remotely worth while. I'm not going to work on integrating it with webservice.

In T164161#3224562, @bd808 wrote:

In T164161#3224547, @Amitie_10g wrote:

I attempted to use Repo Authoritative by adding the following to hhvm-webservice.sh:

hhvm.repo.authoritative=true
hhvm.repo.central.path=${HOME}/hhvm.hhbc

As @ori wrote upstream in https://github.com/facebook/hhvm/issues/6878, RepoAuth mode is difficult to get right. A tool would have to be very, very active before dealing with the complexity of RepoAuth would be remotely worth while. I'm not going to work on integrating it with webservice.

This make sense, at least for my tool (and possibly all the PHP toolshosted at Labs). I'm just testing.

As HHVM in Proxygen mode worked properly, do you think is appropriate to document that at Wikitech as an experimental deployment (until the became official)? Also keep in mind my but report at GitHub.

For the record, I'm currently running the WebArchiveBOT's Web service as generic web service using the script provided by@bd808 since several hours ago.

In T164161#3224678, @Amitie_10g wrote:

As HHVM in Proxygen mode worked properly, do you think is appropriate to document that at Wikitech as an experimental deployment (until the became official)?

@Amitie_10g I don't have any strong objection to documenting the webservice generic method of running HHVM in proxygen mode on wikitech. It should be noted if added there that it will (a) only work with the grid engine backend, and (b) if it stops working you probably won't get a huge amount of help in debugging the problem from Tool Labs administrators. I think the term I've used before for such things is "suitable for intrepid adventurers only". ;)

Also keep in mind my but report at GitHub.

We only run LTS releases of HHVM, so any upstream fix will probably take quite a while to reach us, but yes having support for multiple default index files would be nice. It would also be nice for there to be some good documentation on configuring the proxygen server using the preferred ini-style configuration.

I'll document that tomorrow.

A good way to get debug information is using the logs, that I'll add to the configuration file. Also, should be a good idea to try HHVM in Daemon instead of the default Server mode (wich uses a PID file, and declaring the logs path is required. Just for experimenting.

In T164161#3224683, @Amitie_10g wrote:

I'll document that tomorrow.

A good way to get debug information is using the logs, that I'll add to the configuration file. Also, should be a good idea to try HHVM in Daemon instead of the default Server mode (wich uses a PID file, and declaring the logs path is required. Just for experimenting.

Running HHVM in demon mode is actually a bad idea. I had to clean up about 17 orphaned HHVM process that were started from your tool and the one I was testing from that were in demon mode on various job grid hosts. Processes running on the job grid should stay in the foreground. When they demonize they get "lost" and are not cleaned up when webservice stop is run.

In T164161#3224686, @bd808 wrote:

In T164161#3224683, @Amitie_10g wrote:

I'll document that tomorrow.

A good way to get debug information is using the logs, that I'll add to the configuration file. Also, should be a good idea to try HHVM in Daemon instead of the default Server mode (wich uses a PID file, and declaring the logs path is required. Just for experimenting.

Running HHVM in demon mode is actually a bad idea. I had to clean up about 17 orphaned HHVM process that were started from your tool and the one I was testing from that were in demon mode on various job grid hosts. Processes running on the job grid should stay in the foreground. When they demonize they get "lost" and are not cleaned up when webservice stop is run.

You're right. I needed to kill these process, too.

My request at GitHub has been answered: The HHVM developers don't have plans to add support for more than one index file. However, Pull requests are welcome.

If the motivation/use case for this is improved performance and/or parity with WMF production, then we should consider closing this in favour of T195689 and T188318, which request Toolforge support for PHP 7.x.

MediaWiki is planning to remove support for HHVM in a future release (T192166), which means by then the organisation will be spending its shared resources on PHP 7, not HHVM.

If the use case is to allow tools written in the Hack programming language, then this could be a request for that, but it's up to Cloud Services to decide whether they want to maintain separate support for that.