$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(39b7a663af) gitpuppet - varnish: Implement new direct routing for mobile views'
Notice: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]: Unscheduling refresh on Service[prometheus-varnishkafka-exporter]
Notice: /Stage[main]/Profile::Cache::Haproxy/Package[lua5.3-maxminddb]/ensure: created
Notice: /Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content:
--- /etc/haproxy/haproxy.cfg    2025-08-26 14:53:32.376260599 +0000
+++ /tmp/puppet-file20250826-2210500-18xjwaw    2025-08-26 22:49:18.635209927 +0000
@@ -10,6 +10,7 @@
     nbthread 4
     cpu-map 1/1- 0 1 2 3

+    lua-load-per-thread /etc/haproxy/lua/maxmind-lookup.lua

     ssl-default-bind-options ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3
     ssl-default-bind-ciphers -ALL:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256

Notice: /Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content: content changed '{sha256}661341f7682593551490bb9b215436ce82edb12b5a61e11a93940d60f15497fd' to '{sha256}130d203201a917ded8781cd381a3fcf61844984ec34bac284d41ed58396bd750'
Info: /Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]: Scheduling refresh of Service[haproxy]
Notice: /Stage[main]/Profile::Cache::Haproxy/File[/etc/haproxy/lua/maxmind-lookup.lua]/ensure: defined content as '{sha256}b48a7074c612bf70b4840b80d9c098ffb8ff62dd7c344c016f7993ff02c1686d'
Info: /Stage[main]/Profile::Cache::Haproxy/File[/etc/haproxy/lua/maxmind-lookup.lua]: Scheduling refresh of Service[haproxy]
Notice: /Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content:
--- /etc/haproxy/conf.d/tls.cfg 2025-08-26 14:53:32.492261151 +0000
+++ /tmp/puppet-file20250826-2210500-17noyf6    2025-08-26 22:49:18.755210641 +0000
@@ -123,8 +123,41 @@
     # Allow OPTIONS method only with Origin header
     http-request deny deny_status 405 hdr X-Cache %[var(txn.x_cache)] if { method OPTIONS } !{ hdr(Origin) -m found }

-    # fallback to validate wikimedia_trust if we are not using X-Provenance
-    http-request set-var(req.trusted_request) str(A) if wikimedia_trust
+    # Set x-provenance
+    # First, we check if the request comes from our ring of trust, or is internal. We set:
+    # * net=wikimedia-trust if the request is coming from the wikimedia-trust subnet
+    # * net=internal if the request is coming from a private network
+    # If neither condition is met, the value will be looked up a map containing all
+    # requestctl-defined ipblocks, resulting in:
+    # * abuse=<value> if the request is coming from a known abuser
+    # * client=<value> if the request is coming from a known client ipblock
+    # * cloud=<value> if the request is coming from a known cloud
+    # If all of the above fails, we look up the ip in maxmind to fetch an isp value
+    # * isp=<value> if the IP matches the isp lookup in maxmind
+    # If this also fails, we have no information on the request and we'll set
+    # * net=unknown
+    # We only set the variable once
+    # Set X-Trusted-Request
+    # Provide a trust score from A to F, currently solely based on the source of the request
+    # A for net=wikimedia_trust|internal
+    # F for abuse=
+    # E otherwise
+    http-request set-var(req.provenance,ifnotexists) str('net=wikimedia-trust') if wikimedia_trust
+    acl is_private_network src 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 127.0.0.0/8 ::1
+    http-request set-var(req.provenance,ifnotexists) str('net=internal') if is_private_network
+    # AWS Elastic IPs used by the Wikimedia Enterprise project reported in the following tasks over time:
+    # T255524 T294798 T370294
+    acl is_wme_client src 3.23.12.83/32 3.211.48.168/32 44.206.140.241/32 35.168.168.219/32 35.172.30.169/32 3.222.74.115/32
+    http-request set-var(req.provenance,ifnotexists) str('net=wme') if is_wme_client
+    http-request set-var(req.trusted_request) str(A) if { var(req.provenance) -m found }
+    # check if the IP is included in one of our ipblocks
+    http-request set-var(req.provenance,ifnotexists,ifnotempty) src,map_ip(/etc/haproxy/ipblocks.d/all.map)
+    http-request set-var(req.trusted_request,ifnotexists) str(F) if { var(req.provenance) -m beg "abuse=" }
+    # If everything else failed, find an isp in maxmind
+    http-request set-var(req.provenance,ifnotexists,ifnotempty) lua.fetch_isp,lower,bytes(0,64)
+    # lookup failed
+    http-request set-var(req.provenance,ifnotexists) str('net=unknown')
+    http-request set-header X-Provenance %[var(req.provenance)]
     acl is_trusted_request var(req.trusted_request) -m str A
     http-request set-var(req.bearer) http_auth_bearer
     http-request set-var(req.jwt_alg) var(req.bearer),jwt_header_query('$.alg')

Notice: /Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content: content changed '{sha256}1b7c18517e375d245bec427a53d77635990f71a1cbdd42161184189cf7ada07b' to '{sha256}241e40d0c7972feceb71d580ee7a8e6d5b68cc07b01f70fc7fe3d45d3f0ebe66'
Info: /Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]: Scheduling refresh of Service[haproxy]
Error: /Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]: Failed to call refresh: Systemd restart for haproxy failed!
journalctl log for haproxy:
-- Journal begins at Tue 2025-08-19 14:48:25 UTC, ends at Tue 2025-08-26 22:49:20 UTC. --
Aug 26 22:49:20 deployment-cache-text08 systemd[1]: Reloading HAProxy Load Balancer.
Aug 26 22:49:20 deployment-cache-text08 systemd[1]: haproxy.service: Control process exited, code=exited, status=1/FAILURE
Aug 26 22:49:20 deployment-cache-text08 systemd[1]: Reload failed for HAProxy Load Balancer.

Error: /Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]: Systemd restart for haproxy failed!
journalctl log for haproxy:
-- Journal begins at Tue 2025-08-19 14:48:25 UTC, ends at Tue 2025-08-26 22:49:20 UTC. --
Aug 26 22:49:20 deployment-cache-text08 systemd[1]: Reloading HAProxy Load Balancer.
Aug 26 22:49:20 deployment-cache-text08 systemd[1]: haproxy.service: Control process exited, code=exited, status=1/FAILURE
Aug 26 22:49:20 deployment-cache-text08 systemd[1]: Reload failed for HAProxy Load Balancer.

Info: Class[Haproxy]: Unscheduling all events on Class[Haproxy]
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 21.46 seconds