Page MenuHomePhabricator
Create Task
Maniphest T398176

Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep
Closed, ResolvedPublic
Actions

Description

Common information

  • summary: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep
  • alertname: PuppetAgentFailure
  • instance: deployment-cache-upload08
  • job: node
  • project: deployment-prep
  • severity: warning

Firing alerts

Event Timeline

wmcs-alerts created this task.Jun 30 2025, 9:26 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 30 2025, 9:26 AM
wmcs-alerts updated the task description. (Show Details)Jun 30 2025, 9:41 AM
bd808 closed this task as Invalid.Jun 30 2025, 5:20 PM
bd808 moved this task from To Triage to Puppet errors on the Beta-Cluster-Infrastructure board.
bd808 subscribed.

Fixed itself on the next run apparently

taavi reopened this task as Open.Jun 30 2025, 5:27 PM
taavi subscribed.

The diff was to HAProxy config, which is now failing to reload:

Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content)
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content) --- /etc/haproxy/haproxy.cfg        2025-06-30 08:39:59.651778658 +0000
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content) +++ /tmp/puppet-file20250630-1139447-abiwnz        2025-06-30 09:09:54.399538194 +0000
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content) @@ -12,10 +12,7 @@
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content)
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content)  defaults
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content)      mode       http
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content) -
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content) -    log-format "%rt %Tr %Tw %Tc %ST {%[capture.req.hdr(0)]} {%[capture.res.hdr(0)]} %ts"
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content) -    log-format-sd %{+E}o\ [haproxykafka@0\ server_pid=\"%pid\"\ ip=\"%ci\"\ sequence=\"%ID\"\ dt=\"%tr\"\ time_backend_response=\"%Tr\"\ http_status=\"%ST\"\ response_size=\"%B\"\ termination_state=\"%ts\"\ uri_host=\"%[capture.req.hdr(0)]\"\ referer=\"%[capture.req.hdr(1)]\"\ user_agent=\"%[capture.req.hdr(2)]\"\ accept_language=\"%[capture.req.hdr(3)]\"\ range=\"%[capture.req.hdr(4)]\"\ accept=\"%[capture.req.hdr(5)]\"\ tls=\"%[capture.req.hdr(6)]\"\ cache_status=\"%[capture.res.hdr(0)]\"\ content_type=\"%[capture.res.hdr(1)]\"\ x_analytics=\"%[var(txn.x_analytics)]\"\ x_cache=\"%[capture.res.hdr(2)]\"\ backend=\"%[capture.res.hdr(3)]\"\ http_method=\"%HM\"\ uri_path=\"%HPO\"\ uri_query=\"%HQ\"]
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content) -
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content) +    log-format "%rt %Tr %Tw %Tc %ST %hr %hs %ts"
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content)      option     dontlognull
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content)      option     accept-invalid-http-request
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content)      option     accept-invalid-http-response
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]/content) content changed '{sha256}84c89c2ed14dbb6524ec4119e796bec5e860b959e432692a34bc48f379e0b813' to '{sha256}e076dc5f0ef150cf27faf37d7acaa41d03d9be733fc6842db56eab9526642f93'
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/File[/etc/haproxy/haproxy.cfg]) Scheduling refresh of Service[haproxy]
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) --- /etc/haproxy/conf.d/tls.cfg        2025-06-30 08:39:59.699779000 +0000
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) +++ /tmp/puppet-file20250630-1139447-1ia4i8d        2025-06-30 09:09:54.447538493 +0000
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) @@ -53,28 +53,8 @@
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      # timeout used after upgrading a connection (websockets) or after the first response when no keepalive/close option is specified
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      timeout tunnel 3600s
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    unique-id-format "%rt"
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    # capture slot 0 for X-Cache-Status
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    declare capture response len 10
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    # capture slot 1 for Content-Type
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    declare capture response len 128
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    # capture slot 2 for X-Cache
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    declare capture response len 96
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    # capture slot 3 Server
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    declare capture response len 64
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) +    capture response header X-Cache-Status len 10
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      capture request header Host len 255
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    capture request header Referer len 1024
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    capture request header User-Agent len 1024
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    capture request header Accept-Language len 1024
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    capture request header Range len 10
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    capture request header Accept len 64
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    # capture headers in pre-defined slots
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    http-response capture hdr(Content-Type) id 1
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    http-after-response capture hdr(X-Cache) id 2
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    http-after-response capture hdr(Server) id 3
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      # Host normalization
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      # Use the host_only converter to remove eventual trailing port and
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) @@ -118,17 +98,11 @@
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-request set-header X-Analytics-TLS "vers=%[ssl_fc_protocol];keyx=UNKNOWN;auth=%[var(req.auth)];ciph=%[var(req.ciph)];prot=%[var(req.h2s)];sess=%[var(req.sess)]"
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-request set-header X-Requestctl " "
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    http-request capture hdr(X-Analytics-TLS) len 96
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-request del-header X-Request-Id if !wikimedia_trust
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-request del-header tracestate if !wikimedia_trust
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-request del-header traceparent if !wikimedia_trust
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-request del-header X-Experiment-Enrollments
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    # Copy X-Analytics hdr into var to safely log it after deletion
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    http-response set-var(txn.x_analytics) res.hdr(X-Analytics)
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-response del-header X-Analytics if missing_xwd
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-response del-header Backend-Timing if missing_xwd
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-response del-header X-ATS-Timestamp if missing_xwd
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-response del-header X-Envoy-Upstream-Service-Time if missing_xwd
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) @@ -192,29 +166,10 @@
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      bind :80
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      bind :::80 v6only
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    unique-id-format "%rt"
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    # capture slot 0 for X-Cache-Status
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    declare capture response len 10
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    # capture slot 1 for Content-Type
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    declare capture response len 128
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    # capture slot 2 for X-Cache
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    declare capture response len 96
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    # capture slot 3 Server
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    declare capture response len 64
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      # Needed for logging purposes
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      capture request header Host len 255
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    capture request header Referer len 1024
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    capture request header User-Agent len 1024
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    capture request header Accept-Language len 1024
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    capture request header Range len 10
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    capture request header Accept len 64
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    http-after-response capture hdr(X-Cache-Status) id 0
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    http-response capture hdr(Content-Type) id 1
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    http-after-response capture hdr(X-Cache) id 2
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -    http-after-response capture hdr(Server) id 3
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) +    # declare a capture slot with id 0 for X-Cache-Status
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) +    declare capture response len 10
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      # we can be rather aggresive regarding timeouts here as only impact http to https redirections
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      # time to wait for a complete HTTP request, It only applies to the header part of the HTTP request (unless option http-buffer-request is used)
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) @@ -241,7 +196,7 @@
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      # Provide X-Cache headers
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-after-response set-header X-Cache "deployment-cache-upload08 int"
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-after-response set-header X-Cache-Status "int-tls"
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) -
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) +    http-after-response capture hdr(X-Cache-Status) id 0
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      # METH_GET is a predefined ACL that includes GET and HEAD requests
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      # http://docs.haproxy.org/2.6/configuration.html#7.4
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content)      http-request redirect scheme https code 301 if METH_GET
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]/content) content changed '{sha256}a2402e4a3fd96af04c0979095050566baf2a404bf17c8585580ccb42ba3085b5' to '{sha256}9e9ac4fff3f7c191371c4c5815259496217f542a57b4b7b250db4d841f63f278'
Jun 30 09:09:54 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Profile::Cache::Haproxy/Haproxy::Site[tls]/File[/etc/haproxy/conf.d/tls.cfg]) Scheduling refresh of Service[haproxy]
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]) Failed to call refresh: Systemd restart for haproxy failed!
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]) journalctl log for haproxy:
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]) -- Journal begins at Fri 2025-06-06 06:25:01 UTC, ends at Mon 2025-06-30 09:09:57 UTC. --
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]) Jun 30 09:09:57 deployment-cache-upload08 systemd[1]: Reloading HAProxy Load Balancer.
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]) Jun 30 09:09:57 deployment-cache-upload08 systemd[1]: haproxy.service: Control process exited, code=exited, status=1/FAILURE
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]) Jun 30 09:09:57 deployment-cache-upload08 systemd[1]: Reload failed for HAProxy Load Balancer.
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]) Systemd restart for haproxy failed!
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]) journalctl log for haproxy:
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]) -- Journal begins at Fri 2025-06-06 06:25:01 UTC, ends at Mon 2025-06-30 09:09:57 UTC. --
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]) Jun 30 09:09:57 deployment-cache-upload08 systemd[1]: Reloading HAProxy Load Balancer.
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]) Jun 30 09:09:57 deployment-cache-upload08 systemd[1]: haproxy.service: Control process exited, code=exited, status=1/FAILURE
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (/Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]) Jun 30 09:09:57 deployment-cache-upload08 systemd[1]: Reload failed for HAProxy Load Balancer.
Jun 30 09:09:57 deployment-cache-upload08 puppet-agent[1139447]: (Class[Haproxy]) Unscheduling all events on Class[Haproxy]

And indeed:

taavi@deployment-cache-upload08:~$ sudo systemctl restart haproxy
Job for haproxy.service failed because the control process exited with error code.
See "systemctl status haproxy.service" and "journalctl -xe" for details.
taavi@deployment-cache-upload08:~$ sudo haproxy -c -V -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d/
[NOTICE]   (1282889) : haproxy version is 2.6.17-1~bpo11+1
[NOTICE]   (1282889) : path to executable is /usr/sbin/haproxy
[ALERT]    (1282889) : config : parsing [/etc/haproxy/conf.d//tls.cfg:70] : 'filter' : unknown keyword 'bwlim-out'. Registered keywords :
    [CACHE] cache
    [COMP] compression
    [FCGI] fcgi-app
    [SPOE] spoe
    [TRACE] trace
[ALERT]    (1282889) : config : parsing [/etc/haproxy/conf.d//tls.cfg:72] : 'filter' : unknown keyword 'bwlim-out'. Registered keywords :
    [CACHE] cache
    [COMP] compression
    [FCGI] fcgi-app
    [SPOE] spoe
    [TRACE] trace
[ALERT]    (1282889) : config : parsing [/etc/haproxy/conf.d//tls.cfg:137]: 'http-request' expects 'wait-for-handshake', 'set-log-level', 'set-nice', 'use-service', 'sc-inc-gpc(*)', 'sc-inc-gpc0(*)', 'sc-inc-gpc1(*)', 'sc-set-gpt(*)', 'sc-set-gpt0(*)', 'send-spoe-group', 'do-resolve(*)', 'cache-use', 'add-acl(*)', 'add-header', 'allow', 'auth', 'capture', 'del-acl(*)', 'del-header', 'del-map(*)', 'deny', 'disable-l7-retry', 'early-hint', 'normalize-uri', 'redirect', 'reject', 'replace-header', 'replace-path', 'replace-pathq', 'replace-uri', 'replace-value', 'return', 'set-header', 'set-map(*)', 'set-method', 'set-path', 'set-pathq', 'set-query', 'set-uri', 'strict-mode', 'tarpit', 'track-sc(*)', 'set-timeout', 'wait-for-body', 'set-var-fmt(*)', 'set-var(*)', 'unset-var(*)', 'set-dst', 'set-dst-port', 'set-mark', 'set-src', 'set-src-port', 'set-tos', 'silent-drop', 'set-priority-class', 'set-priority-offset', but got 'set-bandwidth-limit'.
[ALERT]    (1282889) : config : Error(s) found in configuration file : /etc/haproxy/conf.d//tls.cfg
bd808 added a comment.Jun 30 2025, 6:24 PM

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1152083 looks to have been the most recent change to touch the filter bwlim-out lines. It seems to have landed quite a while ago though? CommitDate: Thu Jun 5 04:58:39 2025 +0000

For a filter bwlim-out to work we first need some http-request set-bandwidth-limit ... config lines. In the Puppet config this is gated by @feature_flags['bwlimit'] in modules/profile/templates/cache/haproxy/tls_terminator.cfg.erb:

<%- if @feature_flags['bwlimit'] %>
    # TODO: move in the filtering backend so we can remove the condition
    http-request set-bandwidth-limit limit-by-source if !wikimedia_trust
    http-request set-bandwidth-limit limit-by-path if !wikimedia_trust
<%- end -%>

This flag is set in modules/profile/manifests/cache/haproxy.pp:

## HAProxy configuration
# per cluster feature flags
$feature_flags = $cache_cluster ? {
    'upload' => { 'bwlimit' => true },
    default  => { 'bwlimit' => false }
}

That is set under the same @cache_cluster == 'upload' condition that the output filter bwlim-out ... lines are guarded with. And indeed those lines are in the generated config:

bd808@deployment-cache-upload08:~$ grep set-bandwidth-limit /etc/haproxy/conf.d/tls.cfg
    http-request set-bandwidth-limit limit-by-source if !wikimedia_trust
    http-request set-bandwidth-limit limit-by-path if !wikimedia_trust

What am I missing?

wmcs-alerts updated the task description. (Show Details)Jun 30 2025, 6:26 PM
taavi added a comment.Jun 30 2025, 6:31 PM

deployment-prep might be behind in haproxy versions? deployment-cache-upload08 is running 2.6, while the Puppet code defaults to 2.8.

bd808 added a comment.Jun 30 2025, 8:02 PM
In T398176#10960242, @taavi wrote:

deployment-prep might be behind in haproxy versions? deployment-cache-upload08 is running 2.6, while the Puppet code defaults to 2.8.

bd808@deployment-cache-upload08:~$ sudo apt-get upgrade haproxy
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
haproxy is already the newest version (2.6.17-1~bpo11+1).
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libvarnishapi2 python3-debconf
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
bd808@deployment-cache-upload08:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

Oh hey look -- profile::cache::haproxy::version: haproxy26 is in the deployment-cache prefix puppet. The default is value for that is haproxy28 in the ::profile::cache::haproxy manifest.

Stashbot added a comment.Jun 30 2025, 8:03 PM

Mentioned in SAL (#wikimedia-releng) [2025-06-30T20:03:29Z] <bd808> Remove profile::cache::haproxy::version: haproxy26 from deployment-cache Prefix Puppet (T398176)

bd808 added a comment.Jun 30 2025, 8:06 PM
bd808@deployment-cache-upload08:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-cache-upload08.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(42e91e5800) gitpuppet - puppetserver: check for rebase in puppetserver-deploy-code'
Notice: /Stage[main]/Apt/File[/etc/apt/sources.list.d/thirdparty-haproxy26-apt.wikimedia.org-wikimedia-bullseye-wikimedia.list]/ensure: removed
Notice: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]: Unscheduling refresh on Service[prometheus-varnishkafka-exporter]
Notice: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Apt::Repository[thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia]/File[/etc/apt/sources.list.d/thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia.list]/ensure: defined content as '{sha256}33f447bbcb5c7045bc916c1d6257202f5f17c09a366ee32c010ea12cedc6dc37'
Info: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Apt::Repository[thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia]/File[/etc/apt/sources.list.d/thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia.list]: Scheduling refresh of Exec[apt_repository_thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia]
Notice: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Apt::Repository[thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia]/Exec[apt_repository_thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia]: Triggered 'refresh' from 1 event
Info: Apt::Repository[thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia]: Scheduling refresh of Exec[apt_package_from_component_haproxy]
Notice: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Apt::Pin[apt_pin_haproxy]/File[/etc/apt/preferences.d/apt_pin_haproxy.pref]/content:
--- /etc/apt/preferences.d/apt_pin_haproxy.pref 2023-06-16 14:05:02.093849739 +0000
+++ /tmp/puppet-file20250630-1329429-n7v0ui     2025-06-30 20:04:15.669673639 +0000
@@ -1,3 +1,3 @@
 Package: haproxy
-Pin: release c=thirdparty/haproxy26
+Pin: release c=thirdparty/haproxy28
 Pin-Priority: 1002

Notice: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Apt::Pin[apt_pin_haproxy]/File[/etc/apt/preferences.d/apt_pin_haproxy.pref]/content: content changed '{sha256}3f50b2a62bc75fe15fecbbccd6bbd05cf0369ac64bf5d18e7ad1cd771b853241' to '{sha256}f03dde54d9094782f329bad46d73d9b9816d7ca4a1bf9d2ae2907f0dd0640153'
Info: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Apt::Pin[apt_pin_haproxy]/File[/etc/apt/preferences.d/apt_pin_haproxy.pref]: Scheduling refresh of Exec[apt_package_from_component_haproxy]
Info: Apt::Pin[apt_pin_haproxy]: Scheduling refresh of Exec[apt_package_from_component_haproxy]
Notice: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Exec[apt_package_from_component_haproxy]: Triggered 'refresh' from 3 events
Error: Systemd start for haproxy failed!
journalctl log for haproxy:
-- Journal begins at Fri 2025-06-06 06:25:01 UTC, ends at Mon 2025-06-30 20:04:20 UTC. --
Jun 30 20:04:20 deployment-cache-upload08 systemd[1]: Starting HAProxy Load Balancer...
Jun 30 20:04:20 deployment-cache-upload08 systemd[1]: haproxy.service: Control process exited, code=exited, status=1/FAILURE
Jun 30 20:04:20 deployment-cache-upload08 systemd[1]: haproxy.service: Failed with result 'exit-code'.
Jun 30 20:04:20 deployment-cache-upload08 systemd[1]: Failed to start HAProxy Load Balancer.

Error: /Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]/ensure: change from 'stopped' to 'running' failed: Systemd start for haproxy failed!
journalctl log for haproxy:
-- Journal begins at Fri 2025-06-06 06:25:01 UTC, ends at Mon 2025-06-30 20:04:20 UTC. --
Jun 30 20:04:20 deployment-cache-upload08 systemd[1]: Starting HAProxy Load Balancer...
Jun 30 20:04:20 deployment-cache-upload08 systemd[1]: haproxy.service: Control process exited, code=exited, status=1/FAILURE
Jun 30 20:04:20 deployment-cache-upload08 systemd[1]: haproxy.service: Failed with result 'exit-code'.
Jun 30 20:04:20 deployment-cache-upload08 systemd[1]: Failed to start HAProxy Load Balancer.
 (corrective)
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 23.27 seconds

A second puppet run has similar failures.

wmcs-alerts updated the task description. (Show Details)Jun 30 2025, 8:11 PM
bd808 added a comment.Jun 30 2025, 8:11 PM

Puppet didn't actually update the package.

bd808@deployment-cache-upload08:~$ sudo apt-get upgrade haproxy
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libvarnishapi2 python3-debconf
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
  haproxy
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2390 kB of archives.
After this operation, 857 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/thirdparty/haproxy28 amd64 haproxy amd64 2.8.14-1~bpo11+1 [2390 kB]
Fetched 2390 kB in 0s (27.5 MB/s)
(Reading database ... 47267 files and directories currently installed.)
Preparing to unpack .../haproxy_2.8.14-1~bpo11+1_amd64.deb ...
Unpacking haproxy (2.8.14-1~bpo11+1) over (2.6.17-1~bpo11+1) ...
Setting up haproxy (2.8.14-1~bpo11+1) ...
Processing triggers for rsyslog (8.2102.0-2+deb11u1) ...
Processing triggers for man-db (2.9.4-2) ...
bd808@deployment-cache-upload08:~$ sudo haproxy -c -V -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d
[NOTICE]   (1335040) : haproxy version is 2.8.14-1~bpo11+1
[NOTICE]   (1335040) : path to executable is /usr/sbin/haproxy
[WARNING]  (1335040) : config : log format ignored for frontend 'stats' since it has no log address.
Warnings were found.
Configuration file is valid
bd808@deployment-cache-upload08:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-cache-upload08.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(42e91e5800) gitpuppet - puppetserver: check for rebase in puppetserver-deploy-code'
Notice: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]: Unscheduling refresh on Service[prometheus-varnishkafka-exporter]
Notice: Applied catalog in 16.87 seconds
Stashbot added a comment.Jun 30 2025, 8:12 PM

Mentioned in SAL (#wikimedia-releng) [2025-06-30T20:12:59Z] <bd808> Upgraded haproxy to 2.8.14-1~bpo11+1 on deployment-cache-upload08 (T398176)

Stashbot added a comment.Jun 30 2025, 8:17 PM

Mentioned in SAL (#wikimedia-releng) [2025-06-30T20:17:11Z] <bd808> Upgraded haproxy to 2.8.14-1~bpo11+1 on deployment-cache-text08 (T398176)

bd808 added a comment.Jun 30 2025, 8:17 PM
bd808@deployment-cache-text08:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(42e91e5800) gitpuppet - puppetserver: check for rebase in puppetserver-deploy-code'
Notice: /Stage[main]/Apt/File[/etc/apt/sources.list.d/thirdparty-haproxy26-apt.wikimedia.org-wikimedia-bullseye-wikimedia.list]/ensure: removed
Notice: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]: Unscheduling refresh on Service[prometheus-varnishkafka-exporter]
Notice: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Apt::Repository[thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia]/File[/etc/apt/sources.list.d/thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia.list]/ensure: defined content as '{sha256}33f447bbcb5c7045bc916c1d6257202f5f17c09a366ee32c010ea12cedc6dc37'
Info: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Apt::Repository[thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia]/File[/etc/apt/sources.list.d/thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia.list]: Scheduling refresh of Exec[apt_repository_thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia]
Notice: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Apt::Repository[thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia]/Exec[apt_repository_thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia]: Triggered 'refresh' from 1 event
Info: Apt::Repository[thirdparty-haproxy28-apt.wikimedia.org-wikimedia-bullseye-wikimedia]: Scheduling refresh of Exec[apt_package_from_component_haproxy]
Notice: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Apt::Pin[apt_pin_haproxy]/File[/etc/apt/preferences.d/apt_pin_haproxy.pref]/content:
--- /etc/apt/preferences.d/apt_pin_haproxy.pref 2023-06-22 11:58:55.815126370 +0000
+++ /tmp/puppet-file20250630-617787-leou46      2025-06-30 20:14:29.918810202 +0000
@@ -1,3 +1,3 @@
 Package: haproxy
-Pin: release c=thirdparty/haproxy26
+Pin: release c=thirdparty/haproxy28
 Pin-Priority: 1002

Notice: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Apt::Pin[apt_pin_haproxy]/File[/etc/apt/preferences.d/apt_pin_haproxy.pref]/content: content changed '{sha256}3f50b2a62bc75fe15fecbbccd6bbd05cf0369ac64bf5d18e7ad1cd771b853241' to '{sha256}f03dde54d9094782f329bad46d73d9b9816d7ca4a1bf9d2ae2907f0dd0640153'
Info: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Apt::Pin[apt_pin_haproxy]/File[/etc/apt/preferences.d/apt_pin_haproxy.pref]: Scheduling refresh of Exec[apt_package_from_component_haproxy]
Info: Apt::Pin[apt_pin_haproxy]: Scheduling refresh of Exec[apt_package_from_component_haproxy]
Notice: /Stage[main]/Profile::Cache::Haproxy/Apt::Package_from_component[haproxy]/Exec[apt_package_from_component_haproxy]: Triggered 'refresh' from 3 events
Notice: Applied catalog in 30.30 seconds
bd808@deployment-cache-text08:~$ sudo apt-get update
Hit:1 http://apt.wikimedia.org/wikimedia bullseye-wikimedia InRelease
Hit:2 http://mirrors.wikimedia.org/debian bullseye InRelease
Hit:3 http://mirrors.wikimedia.org/debian bullseye-updates InRelease
Hit:4 http://deb.debian.org/debian-debug bullseye-debug InRelease
Hit:5 http://security.debian.org/debian-security bullseye-security InRelease
Reading package lists... Done
bd808@deployment-cache-text08:~$ sudo apt-get upgrade haproxy
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libdigest-hmac-perl libnet-ip-perl libvarnishapi2 python3-debconf
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
  haproxy
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2390 kB of archives.
After this operation, 857 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/thirdparty/haproxy28 amd64 haproxy amd64 2.8.14-1~bpo11+1 [2390 kB]
Fetched 2390 kB in 0s (26.5 MB/s)
(Reading database ... 50144 files and directories currently installed.)
Preparing to unpack .../haproxy_2.8.14-1~bpo11+1_amd64.deb ...
Unpacking haproxy (2.8.14-1~bpo11+1) over (2.6.17-1~bpo11+1) ...
Setting up haproxy (2.8.14-1~bpo11+1) ...
Processing triggers for rsyslog (8.2102.0-2+deb11u1) ...
Processing triggers for man-db (2.9.4-2) ...
bd808@deployment-cache-text08:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(42e91e5800) gitpuppet - puppetserver: check for rebase in puppetserver-deploy-code'
Notice: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]: Unscheduling refresh on Service[prometheus-varnishkafka-exporter]
Notice: /Stage[main]/Haproxy/Systemd::Service[haproxy]/Systemd::Unit[haproxy]/File[/lib/systemd/system/haproxy.service]/content:
--- /lib/systemd/system/haproxy.service 2025-02-03 08:07:49.000000000 +0000
+++ /tmp/puppet-file20250630-620180-1dv4590     2025-06-30 20:16:04.919312182 +0000
@@ -2,39 +2,22 @@
 Description=HAProxy Load Balancer
 Documentation=man:haproxy(1)
 Documentation=file:/usr/share/doc/haproxy/configuration.txt.gz
-After=network-online.target rsyslog.service
-Wants=network-online.target
+After=network-online.target syslog.service
+Wants=network-online.target syslog.service

 [Service]
+Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy/haproxy.pid"
 EnvironmentFile=-/etc/default/haproxy
-EnvironmentFile=-/etc/sysconfig/haproxy
-BindReadOnlyPaths=/dev/log:/var/lib/haproxy/dev/log
-Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid" "EXTRAOPTS=-S /run/haproxy-master.sock"
-ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS
-ExecReload=/usr/sbin/haproxy -Ws -f $CONFIG -c -q $EXTRAOPTS
+ExecStartPre=/usr/local/sbin/tls-check /etc/haproxy-tls-check.cfg
+ExecStartPre=/usr/sbin/haproxy -f ${CONFIG} -c -q $EXTRAOPTS
+ExecStart=/usr/sbin/haproxy -Ws -f ${CONFIG} -p $PIDFILE $EXTRAOPTS
+ExecReload=/usr/sbin/haproxy -f ${CONFIG} -c -q $EXTRAOPTS
 ExecReload=/bin/kill -USR2 $MAINPID
 KillMode=mixed
 Restart=always
 SuccessExitStatus=143
 Type=notify
-
-# The following lines leverage SystemD's sandboxing options to provide
-# defense in depth protection at the expense of restricting some flexibility
-# in your setup (e.g. placement of your configuration files) or possibly
-# reduced performance. See systemd.service(5) and systemd.exec(5) for further
-# information.
-
-# NoNewPrivileges=true
-# ProtectHome=true
-# If you want to use 'ProtectSystem=strict' you should whitelist the PIDFILE,
-# any state files and any other files written using 'ReadWritePaths' or
-# 'RuntimeDirectory'.
-# ProtectSystem=true
-# ProtectKernelTunables=true
-# ProtectKernelModules=true
-# ProtectControlGroups=true
-# If your SystemD version supports them, you can add: @reboot, @swap, @sync
-# SystemCallFilter=~@cpu-emulation @keyring @module @obsolete @raw-io
+LimitNOFILE=500000

 [Install]
 WantedBy=multi-user.target

Notice: /Stage[main]/Haproxy/Systemd::Service[haproxy]/Systemd::Unit[haproxy]/File[/lib/systemd/system/haproxy.service]/content:

Notice: /Stage[main]/Haproxy/Systemd::Service[haproxy]/Systemd::Unit[haproxy]/File[/lib/systemd/system/haproxy.service]/content: content changed '{sha256}551aecbd5987826002836240b207e9310ff252ef8f6c5de954671da7d913427c' to '{sha256}c9a6a9cd6d12d48ed6ec716009d92efe7020f50c7e70800c9e56a960239a12db' (corrective)
Notice: /Stage[main]/Haproxy/Systemd::Service[haproxy]/Systemd::Unit[haproxy]/File[/lib/systemd/system/haproxy.service]/mode: mode changed '0644' to '0444' (corrective)
Info: /Stage[main]/Haproxy/Systemd::Service[haproxy]/Systemd::Unit[haproxy]/File[/lib/systemd/system/haproxy.service]: Scheduling refresh of Exec[systemd daemon-reload for haproxy.service (haproxy)]
Info: /Stage[main]/Haproxy/Systemd::Service[haproxy]/Systemd::Unit[haproxy]/File[/lib/systemd/system/haproxy.service]: Scheduling refresh of Exec[systemd daemon-reload for haproxy.service (haproxy)]
Notice: /Stage[main]/Haproxy/Systemd::Service[haproxy]/Systemd::Unit[haproxy]/Exec[systemd daemon-reload for haproxy.service (haproxy)]: Triggered 'refresh' from 2 events
Notice: Applied catalog in 24.91 seconds
bd808@deployment-cache-text08:~$ sudo haproxy -c -V -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d
[NOTICE]   (620959) : haproxy version is 2.8.14-1~bpo11+1
[NOTICE]   (620959) : path to executable is /usr/sbin/haproxy
[WARNING]  (620959) : config : log format ignored for frontend 'stats' since it has no log address.
Warnings were found.
Configuration file is valid
bd808@deployment-cache-text08:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(42e91e5800) gitpuppet - puppetserver: check for rebase in puppetserver-deploy-code'
Notice: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]: Unscheduling refresh on Service[prometheus-varnishkafka-exporter]
Notice: Applied catalog in 24.45 seconds
bd808 closed this task as Resolved.Jun 30 2025, 8:21 PM
bd808 claimed this task.
In T398176#10960242, @taavi wrote:

deployment-prep might be behind in haproxy versions? deployment-cache-upload08 is running 2.6, while the Puppet code defaults to 2.8.

This was the important thing I had missed. I had not paid attention to the info box at the top of https://www.haproxy.com/documentation/haproxy-configuration-tutorials/performance/traffic-shaping/ that says "Available since HAProxy 2.7" or checked versions. Thanks for spotting that @taavi and for reopening when I assumed that the bot blanking the "Firing alerts" section meant things had fixed themselves.

Restricted Application added a project: User-bd808. · View Herald TranscriptJun 30 2025, 8:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits