Add HHVM backend for webservice
Open, Stalled, LowPublic

Description

@Amitie_10g is interested in running a webservice on Toolforge using HHVM. It may be possible to do this with enough hacking of a tool's $HOME/.lighttpd.conf or webservice generic, but it would be much nicer for us to just add direct support in webservice and the Kubernetes containers.

In WMF production, HHVM runs in fcgi mode behind Apache2 web servers. The Tool Labs equivalent would use Lighttpd instead of Apache2. It is also possible to run HHVM in proxygen mode which is advertised as providing a full web server.

bd808 created this task.Apr 30 2017, 2:43 AM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptApr 30 2017, 2:43 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
bd808 added a comment.Apr 30 2017, 2:45 AM

Reported in the #wikimedia-labs irc channel:

[23:25]  <Amitie_10g>	I got working HHVM in FastCGI by invoking the following in the following
[23:25]  <Amitie_10g>	"/usr/bin/hhvm --mode daemon -d hhvm.server.type=fastcgi -d hhvm.server.file_socket=/var/run/lighttpd/hhvm.socket.webarchivebot -d hhvm.server.source_root=/data/project/webarchivebot/public_html -d error_log=/data/project/webarchivebot/hhcv_error.log -d hhvm.log.file=/data/project/webarchivebot/hhvm_log.log"
bd808 added a comment.Apr 30 2017, 2:52 AM

If we try proxygen mode, the webservice backend will need to provide a way for the user to provide configuration settings for the HHVM process in some per-tool config file. This would be needed for things like setting up rewrite rules for tools that use a dispatcher script.

As the user who is interested in this, I have some question that has been not answered at the IRC. Based in my knlwledge, I see the following situations:

HHVM in Proxygen mode supports only numbered ports. Then, a Kubernetes container can run two instances of HHVM in Proxygen mode, listening the ports 80 and 443. Then, nginx proxy forwards the traffic originated to https://tools.wmflabs.org/**webarchivebot** to the HHVMs instances running inside the WebArchiveBOT's Kubernetes container. Everything (enabling SSL and setting the certificates paths) should be declared in the command line,or in the configuration file.

HHVM running in FastCGI mode supports both ports and UNIX sockets. As I know, based on my tests, HHVM is able to launch only one socket per instance,but it can listen in ports 80 and 443.

I created my own .lighttpd.conf (showing only the fastcgi.server section):

fastcgi.server += ( ".php" =>
        ((
                "bin-path" => "/usr/bin/hhvm -m daemon --user tools.webarchivebot -d hhvm.server.type=fastcgi -d hhvm.server.file_socket=/var/run/lighttpd/php.socket.{toolname} -d hhvm.log.use_log_file=true -d error_log=/data/project/{toolname}/hhvm_error_log -d hhvm.log.file=/data/project/{toolname}/hhvm_log",
                "socket" => "/var/run/lighttpd/php.socket.{toolname}",
                "max-procs" => 1,
                "bin-environment" => (
                        "PHP_FCGI_CHILDREN" => "1",
                        "PHP_FCGI_MAX_REQUESTS" => "500"
                ),
                "bin-copy-environment" => (
                        "PATH", "SHELL", "USER"
                ),
                "broken-scriptfilename" => "enable",
                "allow-x-send-file" => "enable"
         ))
)

Everything should be defined in the command line, or in the INI file.

This is I tried.

Amitie_10g added a comment.EditedApr 30 2017, 4:26 AM

If we try proxygen mode, the webservice backend will need to provide a way for the user to provide configuration settings for the HHVM process in some per-tool config file. This would be needed for things like setting up rewrite rules for tools that use a dispatcher script.

As I seen, that can be defined in $HOME/server.ini (where HHVM looks in). But Proxygen seems to be toolimited compared to FastCGI mode.

I suggest to create two Kubernetes Container variants: hhvm-proxygen (standalone server without rules, serving in both 80 and 443 ports with SSL and certificates configured in the command line or the HHVM configuration file) and hhvm-fastcgi (using lighttpd with a modified configuration of php-cgi).

bd808 added a comment.Apr 30 2017, 4:34 AM

HHVM in Proxygen mode supports only numbered ports. Then, a Kubernetes container can run two instances of HHVM in Proxygen mode, listening the ports 80 and 443. Then, nginx proxy forwards the traffic originated to https://tools.wmflabs.org/**webarchivebot** to the HHVMs instances running inside the WebArchiveBOT's Kubernetes container. Everything (enabling SSL and setting the certificates paths) should be declared in the command line,or in the configuration file.

Only one backend instance of the HHVM proxygen (or fcgi) server is needed. HTTPS requests are handled by the Tool Labs nginx proxy which has the proper SSL certificates. Requests from the nginx proxy to each tool's webservice are done via plain HTTP across the Labs network segment. An X-Forwarded-Proto header is added to requests sent through the proxy to ecah webservice so that the webservice can determine the protocol in use by the client.

HHVM in Proxygen mode supports only numbered ports. Then, a Kubernetes container can run two instances of HHVM in Proxygen mode, listening the ports 80 and 443. Then, nginx proxy forwards the traffic originated to https://tools.wmflabs.org/**webarchivebot** to the HHVMs instances running inside the WebArchiveBOT's Kubernetes container. Everything (enabling SSL and setting the certificates paths) should be declared in the command line,or in the configuration file.

Only one backend instance of the HHVM proxygen (or fcgi) server is needed. HTTPS requests are handled by the Tool Labs nginx proxy which has the proper SSL certificates. Requests from the nginx proxy to each tool's webservice are done via plain HTTP across the Labs network segment. An X-Forwarded-Proto header is added to requests sent through the proxy to ecah webservice so that the webservice can determine the protocol in use by the client.

Good,this is exactly what I'm looking. At least,my tool does not require rewrite rules, so, the Proxygen flavour can be deployed first. If someone needs the FastCGI flavour,the user can request it, but for now I don't see it necessary.

Amitie_10g added a comment.EditedApr 30 2017, 6:33 AM

I'm trying to run HHVM in Proxygen mode inside my tool account and I opened the page successfully. Following is the way to run HHVM:

hhvm -m daemon -c $HOME/server.ini

It looks for the server.ini file (located at $HOME). The following configuration is enough (but necessary) to run HHVM successfully:

hhvm.server.type = proxygen
hhvm.server.user = {username}
hhvm.server.source_root=$HOME/public_html
hhvm.log.use_log_file = true
error_log = $HOME/hhvm_server_error.log
hhvm.log.file = $HOME/hhvm_server.log
hhvm.pid_file = var/run/hhvm.www.{username}.pid
date.timezone=UTC
hhvm.enable_zend_sorting=1

However, there is a serious drawback: hhvm.server.default_document option can be set only once; if specified more than once,only the last instance will be taken.This mean that only "index.php" is taken as the default document, and, for example, index.html will be ignored if it is ommited when typing the URL:

For example, https://tools.wmflabs,org/webarchivebot will display the implicit "index.php", but when typing https://tools.wmflabs,org/webarchivebot/doc (that contains only plain HTML files) will throw a 404 error due the server haven't found the default document ("index.php"), even if there is an "index.html". Therefore, I sent a bug report.

For now, the only solution for that is using HHVM in FastCGI mode with lighttpd.

bd808 added a comment.Apr 30 2017, 6:35 PM

I spent some time exploring running HHVM in proxygen mode using webservice generic and a bootstrapping script that generates an HHVM ini file and starts HHVM in server mode. This does make a working webservice, but it has several drawbacks:

  • Documentation for configuring HHVM's proxygen webserver is lacking upstream. Information can be found, but it requires a lot of digging.
  • No obvious support for alias configuration to easily map https://tools.wmflabs.org/my-tool-name/ to the tool's $HOME/public_html. This can be worked around using hhvm.virtual_host[default][rewrite_rules] settings.
  • No obvious way to do multi-view indexing as noted by @Amitie_10g in T164161#3224126

My experiments show that using proxygen mode is certainly possible, but based on the lack of documentation and that we don't use it in WMF production I would recommend against it for the average Tool Labs user. I will be looking more closely at fcgi mode configuration for integration with webservice.

For those who are curious, here's the script that I came up with to generate an appropriate configuration file and start the HHVM service using webservice --backend=gridengine generic start ${HOME}/hhvm-webservice.sh:

hhvm-webservice.sh
#!/usr/bin/env bash
# Run an HHVM webservice
#
# usage: webservice --backend=gridengine generic start hhvm-webservice.sh
set -e

TOOLNAME=${USER#tools.}

if [[ -z $PORT ]]; then
  echo "PORT environment variable not set." >&2
  echo "usage: webservice --backend=gridengine generic start $0" >&2
  exit 1
fi

/bin/cat << EOF > ${HOME}/server.ini
date.timezone = UTC
hhvm.enable_obj_destruct_call = true
hhvm.enable_zend_compat = true
hhvm.error_handling.call_user_handler_on_fatals = true
hhvm.hack.lang.iconv_ignore_correct = true
hhvm.jit = true
hhvm.log.always_log_unhandled_exceptions = true
hhvm.log.level = Warning
hhvm.log.native_stack_trace = false
hhvm.log.runtime_error_reporting_level = HPHP_ALL
hhvm.log.use_log_file = true
hhvm.log.use_syslog = false
hhvm.pcre_cache_type = lru
hhvm.pid_file =
hhvm.repo.central.path = /tmp/hhvm-${TOOLNAME}.hhbc
hhvm.server.apc.expire_on_sets = true
hhvm.server.apc.expire_on_sets = true
hhvm.server.apc.purge_frequency = 4096
hhvm.server.apc.table_type = concurrent
hhvm.server.apc.ttl_limit = 172800
hhvm.server.default_document = index.php
hhvm.server.dns_cache.enable = true
hhvm.server.dns_cache.ttl = 300
hhvm.server.exit_on_bind_fail = true
hhvm.server.port = ${PORT}
hhvm.server.source_root = ${HOME}/public_html
hhvm.server.stat_cache = true
hhvm.server.thread_count = 4
hhvm.server.type = proxygen
hhvm.virtual_host[default][rewrite_rules][0][pattern] = "^/${TOOLNAME}(.*)\$"
hhvm.virtual_host[default][rewrite_rules][0][qsa] = true
hhvm.virtual_host[default][rewrite_rules][0][to] = "\$1"
max_execution_time = 60
memory_limit = 128M
EOF

exec /usr/bin/hhvm -m server -c ${HOME}/server.ini -c ${HOME}/hhvm.ini
Amitie_10g added a comment.EditedApr 30 2017, 8:42 PM

I spent some time exploring running HHVM in proxygen mode using webservice generic and a bootstrapping script that generates an HHVM ini file and starts HHVM in server mode. This does make a working webservice, but it has several drawbacks:

  • Documentation for configuring HHVM's proxygen webserver is lacking upstream. Information can be found, but it requires a lot of digging.
  • No obvious support for alias configuration to easily map https://tools.wmflabs.org/my-tool-name/ to the tool's $HOME/public_html. This can be worked around using hhvm.virtual_host[default][rewrite_rules] settings.
  • No obvious way to do multi-view indexing as noted by @Amitie_10g in T164161#3224126

    My experiments show that using proxygen mode is certainly possible, but based on the lack of documentation and that we don't use it in WMF production I would recommend against it for the average Tool Labs user. I will be looking more closely at fcgi mode configuration for integration with webservice.

    For those who are curious, here's the script that I came up with to generate an appropriate configuration file and start the HHVM service using `webservice --backend=gridengine generic start ${HOME}/hhvm-webservice.sh:
hhvm-webservice.sh
#!/usr/bin/env bash
# Run an HHVM webservice
#
# usage: webservice --backend=gridengine generic start hhvm-webservice.sh
set -e

TOOLNAME=${USER#tools.}

if [[ -z $PORT ]]; then
  echo "PORT environment variable not set." >&2
  echo "usage: webservice --backend=gridengine generic start $0" >&2
  exit 1
fi

/bin/cat << EOF > ${HOME}/server.ini
date.timezone = UTC
hhvm.enable_obj_destruct_call = true
hhvm.enable_zend_compat = true
hhvm.error_handling.call_user_handler_on_fatals = true
hhvm.hack.lang.iconv_ignore_correct = true
hhvm.jit = true
hhvm.log.always_log_unhandled_exceptions = true
hhvm.log.level = Warning
hhvm.log.native_stack_trace = false
hhvm.log.runtime_error_reporting_level = HPHP_ALL
hhvm.log.use_log_file = true
hhvm.log.use_syslog = false
hhvm.pcre_cache_type = lru
hhvm.pid_file =
hhvm.repo.central.path = /tmp/hhvm-${TOOLNAME}.hhbc
hhvm.server.apc.expire_on_sets = true
hhvm.server.apc.expire_on_sets = true
hhvm.server.apc.purge_frequency = 4096
hhvm.server.apc.table_type = concurrent
hhvm.server.apc.ttl_limit = 172800
hhvm.server.default_document = index.php
hhvm.server.dns_cache.enable = true
hhvm.server.dns_cache.ttl = 300
hhvm.server.exit_on_bind_fail = true
hhvm.server.port = ${PORT}
hhvm.server.source_root = ${HOME}/public_html
hhvm.server.stat_cache = true
hhvm.server.thread_count = 4
hhvm.server.type = proxygen
hhvm.virtual_host[default][rewrite_rules][0][pattern] = "^/${TOOLNAME}(.*)\$"
hhvm.virtual_host[default][rewrite_rules][0][qsa] = true
hhvm.virtual_host[default][rewrite_rules][0][to] = "\$1"
max_execution_time = 60
memory_limit = 128M
EOF

exec /usr/bin/hhvm -m server -c ${HOME}/server.ini -c ${HOME}/hhvm.ini

Well, I'm running my tool with the generic webserver you provided and works perfectly. I only need to tweak the links to point to the right index.html at the /doc.

Then, what is the state of the HHVM in FastCGI mode?

I attempted to use Repo Authoritative by adding the following to hhvm-webservice.sh:

hhvm.repo.authoritative=true
hhvm.repo.central.path=${HOME}/hhvm.hhbc

However, attempting to build the repo according to the Documentation, the process fails:

tools.webarchivebot@tools-bastion-03:~$ hhvm --hphp -t hhbc -v AllVolatile=true --keep-tempdir=1 --log=7 public_html/index.php public_html/.config.php public_html/phpinfo.php
running hphp...
creating temporary directory /tmp/hphp_7pAnay ...
parsing inputs...
parsing ./public_html/.config.php ...
parsing ./public_html/phpinfo.php ...
parsing ./public_html/index.php ...
analyzeProgram...
Analyzing Includes
Analyzing All
analyzeProgram took 0'00" (72165 us) wall time
parsing inputs took 0'00" (110944 us) wall time
pre-optimizing...
pre-optimizing took 0'00" (83357 us) wall time
creating binary HHBC files...
terminate called without an active exception
Core dumped: Aborted
Stack trace in /tmp/stacktrace.32346.log
hphp failed
running hphp took 0'01" (1703318 us) wall time

And generates an empty Stack trace (/tmp/stacktrace.32346.log is empty).

Also,if Repo generation is successful, should be a good idea to add hhvm --hphp -t hhbc -v AllVolatile=true --input-list master-file-list.txt before launching HHVM in server mode, to (re)generate the Repo every time the webservice is started (the master-file-list.txt should be filled by the user).

bd808 added a subscriber: ori.Apr 30 2017, 9:59 PM

I attempted to use Repo Authoritative by adding the following to hhvm-webservice.sh:

hhvm.repo.authoritative=true
hhvm.repo.central.path=${HOME}/hhvm.hhbc

As @ori wrote upstream in https://github.com/facebook/hhvm/issues/6878, RepoAuth mode is difficult to get right. A tool would have to be very, very active before dealing with the complexity of RepoAuth would be remotely worth while. I'm not going to work on integrating it with webservice.

I attempted to use Repo Authoritative by adding the following to hhvm-webservice.sh:

hhvm.repo.authoritative=true
hhvm.repo.central.path=${HOME}/hhvm.hhbc

As @ori wrote upstream in https://github.com/facebook/hhvm/issues/6878, RepoAuth mode is difficult to get right. A tool would have to be very, very active before dealing with the complexity of RepoAuth would be remotely worth while. I'm not going to work on integrating it with webservice.

This make sense, at least for my tool (and possibly all the PHP toolshosted at Labs). I'm just testing.

bd808 triaged this task as Low priority.May 1 2017, 3:24 AM
bd808 changed the task status from Open to Stalled.
bd808 moved this task from Triage to Tools on the Cloud-Services board.
bd808 added a subscriber: yuvipanda.

I'm not having very good luck with experiments based on setting up HHVM in fastcgi mode automatically started by lighttpd. The problem is caused by assumptions in lighttpd's mod_fastcgi that don't match up with the implementation of HHVM's FastCGIServer class. When spawning a fastcgi worker, lighttpd first binds the socket (either TCP or UNIX) in the lighttpd process. Then it calls fork(), cleans up child process environment, and finally calls execve() to replace the child process with the configured fastcgi process. When the HHVM process starts it doesn't like that the socket is already open and refuses to start the fastcgi service.

This looks something like this in the logs:

2017-05-01 01:58:32: (log.c.166) server started
2017-05-01 01:58:32: (mod_fastcgi.c.1366) --- fastcgi spawning local
        proc: /usr/bin/hhvm -m server -c /data/project/bd808-test2/.hhvm.fcgi.ini -c /data/project/bd808-test2/.hhvm.ini
        port: 0
        socket /var/run/lighttpd/hhvm.socket.bd808-test2
        max-procs: 1
2017-05-01 01:58:32: (mod_fastcgi.c.1390) --- fastcgi spawning
        port: 0
        socket /var/run/lighttpd/hhvm.socket.bd808-test2
        current: 0 / 1
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0501 01:58:32.696315 28375 fastcgi-server.cpp:105] 98failed to bind to async server socket: /var/run/lighttp/hhvm.socket.bd808-test2-0: Address already in use
2017-05-01 02:35:12: (log.c.166) server started
2017-05-01 02:35:12: (mod_fastcgi.c.1366) --- fastcgi spawning local
        proc: /usr/bin/hhvm -m server -c /data/project/bd808-test2/.hhvm.fcgi.ini -c /data/project/bd808-test2/.hhvm.ini
        port: 31337
        socket
        max-procs: 1
2017-05-01 02:35:12: (mod_fastcgi.c.1390) --- fastcgi spawning
        port: 31337
        socket
        current: 0 / 1
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0501 02:35:13.248257 22009 fastcgi-server.cpp:105] 98failed to bind to async server socket: 127.0.0.1:31337: Address already in use

Patching ether lighttpd or HHVM to make them play nicely together is a much bigger project that I hoped to undertake here. It also makes me wonder if @yuvipanda had already discovered this problem when he abandoned T78783: Make HHVM based webservices available on toollabs and T84984: Add --server=hhvm to webservice2.

For now, I think the answer for folks who really want HHVM is the script I posted in T164161#3224443. I don't really want to make the proxygen mode more official as it looks like a long term support problem. I also don't want to spend a lot of time trying to make a robust supervisor process that webservice can execv() that would manage starting both HHVM and a fronting web server. As @yuvipanda noted in T84984 we may be able to reevaluate this when we resolve T136264: Evaluate Kubernetes based workflow replacement options for SGE by selecting a Kubernetes based platform as a service layer.

bd808 moved this task from Triage to Backlog on the Toolforge board.May 1 2017, 3:24 AM
Amitie_10g added a comment.EditedMay 1 2017, 3:36 AM

As HHVM in Proxygen mode worked properly, do you think is appropriate to document that at Wikitech as an experimental deployment (until the became official)? Also keep in mind my but report at GitHub.

For the record, I'm currently running the WebArchiveBOT's Web service as generic web service using the script provided by@bd808 since several hours ago.

bd808 added a comment.May 1 2017, 3:59 AM

As HHVM in Proxygen mode worked properly, do you think is appropriate to document that at Wikitech as an experimental deployment (until the became official)?

@Amitie_10g I don't have any strong objection to documenting the webservice generic method of running HHVM in proxygen mode on wikitech. It should be noted if added there that it will (a) only work with the grid engine backend, and (b) if it stops working you probably won't get a huge amount of help in debugging the problem from Tool Labs administrators. I think the term I've used before for such things is "suitable for intrepid adventurers only". ;)

Also keep in mind my but report at GitHub.

We only run LTS releases of HHVM, so any upstream fix will probably take quite a while to reach us, but yes having support for multiple default index files would be nice. It would also be nice for there to be some good documentation on configuring the proxygen server using the preferred ini-style configuration.

I'll document that tomorrow.

A good way to get debug information is using the logs, that I'll add to the configuration file. Also, should be a good idea to try HHVM in Daemon instead of the default Server mode (wich uses a PID file, and declaring the logs path is required. Just for experimenting.

bd808 added a comment.May 1 2017, 4:10 AM

I'll document that tomorrow.

A good way to get debug information is using the logs, that I'll add to the configuration file. Also, should be a good idea to try HHVM in Daemon instead of the default Server mode (wich uses a PID file, and declaring the logs path is required. Just for experimenting.

Running HHVM in demon mode is actually a bad idea. I had to clean up about 17 orphaned HHVM process that were started from your tool and the one I was testing from that were in demon mode on various job grid hosts. Processes running on the job grid should stay in the foreground. When they demonize they get "lost" and are not cleaned up when webservice stop is run.

I'll document that tomorrow.

A good way to get debug information is using the logs, that I'll add to the configuration file. Also, should be a good idea to try HHVM in Daemon instead of the default Server mode (wich uses a PID file, and declaring the logs path is required. Just for experimenting.

Running HHVM in demon mode is actually a bad idea. I had to clean up about 17 orphaned HHVM process that were started from your tool and the one I was testing from that were in demon mode on various job grid hosts. Processes running on the job grid should stay in the foreground. When they demonize they get "lost" and are not cleaned up when webservice stop is run.

You're right. I needed to kill these process, too.

My request at GitHub has been answered: The HHVM developers don't have plans to add support for more than one index file. However,pull requests are welcome.