php7.3 webservice type unable to run PHP
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	LucasWerkmeister
	Aug 1 2025, 4:59 PM

Description

Steps to replicate the issue (include links if applicable):

In a tool’s public_html/, create an HTML file and a PHP file (e.g. hello.html and hello.php in ~lucaswerkmeister-test/public_html/)
webservice php7.3 start (note: php7.4 is not affected)
Load the HTML file and the PHP file

What happens?:
The HTML file loads; the PHP file yields 403 Forbidden.

What should have happened instead?:
The PHP file should also work.

Other information (browser name/version, screenshots, etc.):
I initially noticed this in T400995, with the pagepile tool. Earlier today, a rolling restart still worked; however, when I tried to start the webservice from scratch, the tool broke and I could only fix it by upgrading to php7.4. I subsequently reproduced the same behavior in the lucaswerkmeister-test tool.

The php7.3 webservice type is deprecated, but still, this seems like a bad state of affairs for tool authors who haven’t upgraded to PHP 7.4 yet.

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	lighttpd: Use "lighttpd" as webservice type	repos/cloud/toolforge/webservice-cli!80	bd808	work/bd808/legacy-php	main

Customize query in GitLab

Related Objects

Mentioned Here: T400995: pagepile.toolforge.org 504 timed out

Event Timeline

LucasWerkmeister created this task.Aug 1 2025, 4:59 PM

Restricted Application added a project: cloud-services-team. · View Herald TranscriptAug 1 2025, 4:59 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

You can currently see the 403 at https://lucaswerkmeister-test.toolforge.org/hello.php (compare with https://lucaswerkmeister-test.toolforge.org/hello.html), though I’ll take that webservice down / replace it with a different test at some point ^^

Let's see if we can figure out what is broken in the image.

$ ssh dev.toolforge.org
$ become bd808-test2
tools.bd808-test2@tools-bastion-12:~$ cat public_html/index.php
<?php phpinfo();
tools.bd808-test2@tools-bastion-12:~$ webservice php7.3 start
DEPRECATED: 'php7.3' type is deprecated.
  See https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes
  for currently supported types.
Starting webservice...
tools.bd808-test2@tools-bastion-12:~$ curl -D - https://bd808-test2.toolforge.org/
HTTP/2 403
server: nginx/1.22.1
date: Fri, 01 Aug 2025 17:16:46 GMT
content-type: text/html
content-length: 341
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()
content-security-policy-report-only: default-src 'self' 'unsafe-eval' 'unsafe-inline' blob: data: filesystem: mediastream: *.toolforge.org wikibooks.org *.wikibooks.org wikidata.org *.wikidata.org wikimedia.org *.wikimedia.org wikinews.org *.wikinews.org wikipedia.org *.wikipedia.org wikiquote.org *.wikiquote.org wikisource.org *.wikisource.org wikiversity.org *.wikiversity.org wikivoyage.org *.wikivoyage.org wiktionary.org *.wiktionary.org *.wmcloud.org *.wmflabs.org wikimediafoundation.org mediawiki.org *.mediawiki.org wss://bd808-test2.toolforge.org; report-uri https://csp-report.toolforge.org/collect;

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>403 Forbidden</title>
 </head>
 <body>
  <h1>403 Forbidden</h1>
 </body>
</html>
tools.bd808-test2@tools-bastion-12:~$ kubectl get po
NAME                          READY   STATUS    RESTARTS   AGE
bd808-test2-5469fcbc7-ghsb7   1/1     Running   0          5m17s
tools.bd808-test2@tools-bastion-12:~$ kubectl exec -it bd808-test2-5469fcbc7-ghsb7 -- /bin/bash
tools.bd808-test2@bd808-test2-5469fcbc7-ghsb7:~$ grep php /var/run/lighttpd/bd808-test2
index-file.names = ( "index.php", "index.html", "index.htm" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )

That doesn't look right. The generated lighttpd config should have a config block that looks a lot like:

fastcgi.server += ( ".php" =>
  ((
    "bin-path" => "/usr/bin/php-cgi",
    "socket" => "/tmp/lighttpd-php.sock",
    "max-procs" => 2,
    "bin-environment" => (
      "PHP_FCGI_CHILDREN" => "2",
      "PHP_FCGI_MAX_REQUESTS" => "500"
    ),
    "broken-scriptfilename" => "enable",
    "allow-x-send-file" => "enable"
  ))
)

In modern images we generate this config with https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/toollabs-images/+/refs/heads/master/shared/lighttpd/webservice-runner. In the olden days of php 7.3 though it was the responsibility of the webservice-runner entrypoint of the toolforge-webservice package (https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/toollabs-images/+/2c64e77e9cd420755f38195ece012e5139a9785c/php73-sssd/web/Dockerfile.template).

The entry point of the Pod looks like this:

spec:
  containers:                                                                     
  - command:
    - /usr/bin/webservice-runner
    - --type
    - lighttpd-plain
    - --port
    - "8000"

The version of toolforge-webservice in the php7.3 container is 0.130.1. That's https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/tree/debian/0.103.1. At that version, webservice-runner expected to be called with --type lighttpd to trigger adding the missing fastcgi.server config block to the generated lighttpd configuration on startup.

The argument passed to --type is toolsws.backends.kubernetes.KubernetesBackend.get_types()["php7.3"]["cls"](tool, extra_args).name. toolsws.backends.kubernetes.KubernetesBackend.get_types()["php7.3"]["cls"] is toolsws.wstypes.lighttpd.LighttpdWebService. The name member there changed from "lighttpd" to "lighttpd-plain" in 04d9f03 when @taavi removed the legacy GridEngine support. I think if we switch that back to "lighttpd" things will work as expected again for legacy php images that still care about the --type ... argument's value.

That would explain why the rolling restart still worked – the container spec (+ command) still stayed the same then.

bd808 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/80

lighttpd: Use "lighttpd" as webservice type

bd808 merged https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/80

lighttpd: Use "lighttpd" as webservice type

Maintenance_bot removed a project: Patch-For-Review.Aug 6 2025, 4:31 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-06T17:33:06Z] <bd808@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli (T401014)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-06T17:34:18Z] <bd808@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli (T401014)

I stopped+started the webservice in the lucaswerkmeister-test tool and now https://lucaswerkmeister-test.toolforge.org/hello.php is working \o/ thanks @bd808!

Edit: that’s because the webservice start defaulted to php7.4 🤦 don’t mind me, I’ll let you figure out these cookbook failures in peace…

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-06T17:38:47Z] <bd808@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli (T401014)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-06T17:39:05Z] <bd808@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli (T401014)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-06T17:41:19Z] <bd808@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli (T401014)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-06T17:41:34Z] <bd808@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli (T401014)

$ become bd808-test2
tools.bd808-test2@tools-bastion-12:~$ webservice php7.2 restart
DEPRECATED: 'php7.2' type is deprecated.
  See https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes
  for currently supported types.
Restarting...
tools.bd808-test2@tools-bastion-12:~$ curl -ID - https://bd808-test2.toolforge.org/
HTTP/2 200
HTTP/2 200
server: nginx/1.22.1
server: nginx/1.22.1
date: Wed, 06 Aug 2025 18:10:43 GMT
date: Wed, 06 Aug 2025 18:10:43 GMT
content-type: text/html; charset=UTF-8
content-type: text/html; charset=UTF-8
strict-transport-security: max-age=31622400
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()
permissions-policy: browsing-topics=()
content-security-policy-report-only: default-src 'self' 'unsafe-eval' 'unsafe-inline' blob: data: filesystem: mediastream: *.toolforge.org wikibooks.org *.wikibooks.org wikidata.org *.wikidata.org wikimedia.org *.wikimedia.org wikinews.org *.wikinews.org wikipedia.org *.wikipedia.org wikiquote.org *.wikiquote.org wikisource.org *.wikisource.org wikiversity.org *.wikiversity.org wikivoyage.org *.wikivoyage.org wiktionary.org *.wiktionary.org *.wmcloud.org *.wmflabs.org wikimediafoundation.org mediawiki.org *.mediawiki.org wss://bd808-test2.toolforge.org; report-uri https://csp-report.toolforge.org/collect;
content-security-policy-report-only: default-src 'self' 'unsafe-eval' 'unsafe-inline' blob: data: filesystem: mediastream: *.toolforge.org wikibooks.org *.wikibooks.org wikidata.org *.wikidata.org wikimedia.org *.wikimedia.org wikinews.org *.wikinews.org wikipedia.org *.wikipedia.org wikiquote.org *.wikiquote.org wikisource.org *.wikisource.org wikiversity.org *.wikiversity.org wikivoyage.org *.wikivoyage.org wiktionary.org *.wiktionary.org *.wmcloud.org *.wmflabs.org wikimediafoundation.org mediawiki.org *.mediawiki.org wss://bd808-test2.toolforge.org; report-uri https://csp-report.toolforge.org/collect;

Restricted Application added a project: User-bd808. · View Herald TranscriptAug 6 2025, 6:11 PM

php7.3 webservice type unable to run PHPClosed, ResolvedPublicBUG REPORTActions

Description

Details

Related Objects

Event Timeline

php7.3 webservice type unable to run PHP
Closed, ResolvedPublicBUG REPORT
Actions