the assessment is OK and the link can be removed safely

We are now rate-limiting non thumbnail steps requests for cache misses when certain X-Is-Browser thresholds are met

From SREBatchRunnerBase __reboot_action():

a quick check using https://en.wikipedia.org/wiki/Main_Page?vgutierrez=tg resulted in telegram bot visiting https://en.m.wikipedia.org/wiki/Main_Page?vgutierrez=tg and retrying after getting a 301 instead of following the redirect

It does but acme-chief also respects the staging time so it's probably under /new instead of /live or still blocked cause it's waiting the staging time till a previous version gets deployed

downtiming for prometheus alertmanager seems broken to me. What we are seeing here looks like this:

• The metric was in an alerting state from 18:45:30 o 18:49:00 per https://grafana.wikimedia.org/goto/BShBVVgDg?orgId=1
• Downtime (silence) was present, so even if the alert condition was true for 3m, notifications were suppressed.
• At 18:49:00 the alert condition cleared and at 18:49:19 the downtime was removed.
• Prometheus’s next evaluation (at 18:49:25) saw that the alert had been pending and satisfied for: 3m, so it fired the alert. This can be verified with this query: https://grafana.wikimedia.org/goto/uJUkS4Rvg?orgId=1 that shows the alert firing till 18:49:30

This is pretty weird, according to RFC 9562 for UUID v4, the third block should always start with a 4.

wdqs-internal-main still has traffic on port 80 in codfw:

TCP  10.2.1.93:80 mh (mh-port)
  -> 10.192.0.85:80               Tunnel  10     67         73        
  -> 10.192.32.155:80             Tunnel  10     46         87        
  -> 10.192.32.156:80             Tunnel  10     52         86

we need to double check that HAProxy supports EdDSA for JWT verification purposes

following up on my last comment, the webm file size is 1660261 bytes, so a request asking for a range starting at 1660261 should probably trigger a 416 Range Not Satisfiable response instead of a 503 but it still doesn't look like a valid request for me, it I use a valid range like 1000-1024, the request gets a valid response every time but inconsistent, the first one it gets a 1660261 bytes response back, and the following ones a 25 bytes response as requested on the Range header

Taking a second look at the curl reproducer, I'm seeing the following behavior:

Do we know what the current behavior is for layers that set X-Request-ID?

yes, it's still hapenning https://grafana.wikimedia.org/goto/SHdP6s3HR?orgId=1:

you got a nice mix of uses cases there @A_smart_kitten.

In T390813#11161805, @ayounsi wrote:

@Vgutierrez @ssingh could that be a good opportunity to see how drmrs handles the loss of a switch/rack ?

With the site depooled, and while one ToR switch is upgrading, maybe we could see if the other rack could handle all the traffic properly ?

probably unrelated but I've found what it could be a HAProxy bug related to %rt being increased twice per request: https://github.com/haproxy/haproxy/issues/3107

local tests show that HAProxy issued 46410639 but it never reached the kafka cluster, probably because haproxykafka failed to parse it for some reason, if this happens systematically after a BADREQ I think we could have a bug on HPK

puppet is now happy on deployment-cache-text08.

puppet is happier in deployment-cache-text08 but not 100%:

Sep  5 10:22:01 deployment-cache-text08 puppet-agent[1978923]: (/Stage[main]/Profile::Cache::Haproxy/File[/usr/share/GeoIP/datacenter.mmdb]) Could not evaluate: Could not retrieve information from environment production source(s) puppet:///volatile/datacenter_vendors/datacenter.mmdb

FWIW HAProxy provides UUIDv4 out of the box so it should be as easy as http-request set-header X-Request-Id %[uuid()].

assigning the task to @JMeybohm, he is the SRE on clinic duty this week

That's great, thanks @brouberol

I think we have 3 upcoming DAGs:

• the one covered by this task
• probenet data for the GeoDNS pipeline (T380626)
• Curate lists of well-known JA3N hashes (part of T400270)

@brouberol hey! it looks like we split airflow airflow instances by team and we don't currently have an instance for SRE so I'm guessing we would need to create it as well?

In T400119#11136788, @DavidBrooks wrote:

Re the comment: "Allow user-agents with contact information" - implies blocking UAs with no contact information. Is this referring to a subset of queries? I understood from earlier that a legacy client-side app with a UA modeled on a browser UA would be OK (unless it runs into a rate limit). Still true?

You're totally right, that's referring to library defaults UAs like python-requests

In T400119#11134249, @Don-vip wrote:

I still have an error in my unit tests from Gitlab CI when trying to access https://upload.wikimedia.org/wikipedia/commons/d/d2/Epichlorhydrin_vzorec.webp

[ERROR]   ImageUtilsTest.testReadImage:29 » IO GET /wikipedia/commons/d/d2/Epichlorhydrin_vzorec.webp => Forbidden -- [content-length:"92", content-type:"text/plain", x-analytics:"", server:"HAProxy", x-cache:"cp1105 int", x-cache-status:"int-tls"] -- Please set a user-agent and respect our robot policy https://w.wiki/4wJS. See also T400119.

The same test works for two other URLs with the same user agent, why?

• https://upload.wikimedia.org/wikipedia/commons/7/78/Empty_200x1.jpg
• https://upload.wikimedia.org/wikipedia/commons/e/e2/Artemis_program_%28original_with_wordmark%29.svg

In T400119#11134002, @DavidBrooks wrote:

Re-upping a question I had earlier - will the servers' "Retry-After" header use seconds, or http-date, or potentially either? Of course it would be easy to figure out in my code, but it would still be good to know.

In T400119#11131904, @CKoerner_WMF wrote:

@Joe Could I ask for a two week exemption for diff.wikimedia.org until we have our next sprint with our devs? Right now folks can't log into the community blog to share their stories and updates and I won't have developer time until the second week of September.

I'm closing this since we've fixed the wrong behavior on HAProxy regarding sequence numbers, please feel to re-open it if you're still detecting issues on your side with sequence numbers.

In T400119#11119143, @Urbanecm_WMF wrote:

Our goal is to block all traffic from unidentified clients and not coming from authorized actors, like toolsforge or our internal APIs.

FWIW, this seems to not be the case 100%. It seems one of our GitLab pipelines was affected by this, see T402801: CI in schemas-event-secondary fails because the tests do not follow WMF's Robot policy. Fortunately, fixing the UA string was simple enough (and reasonable anyway, in case someone runs the test locally), but we do seem to block requests coming from our own infra?

In T352245#11112038, @Scott_French wrote:

I'd like to get this moving again early next week, ideally Tuesday or Wednesday during Europe / Americas overlap hours (starting ~ 14:00 UTC or before).

@Vgutierrez - If you might be available to assist in verifying Liberica control-plane health on either of those days, that would be greatly appreciated.

From Special:Version it appears Lingua Libre is running mediawiki/oauthclient 1.1.0. The ability to set a custom User-Agent (via Config::setUserAgent) was added in 1.2.0 (commit 81edea5f545ef551cb1fb3e8937fd81c549fa94b, task T293609).

@Yug thanks, no need to keep appending user reports till @mickeybarber reports back. This is a well-known and expected behavior of the CDN as announced on https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/APW6FQBGIVLCEN7WZ65D4NVZ6XQIWGCW/ back in 2025-07-30

sampling on webrequest_sampled has been fixed by merging https://gerrit.wikimedia.org/r/1181033

looks good, I took care of merging it, thanks for the patch @Dzahn

change has been merged, please allow 30 minutes to let puppet apply the changes on the required systems. Thanks

flagging as high cause this is already making the downsampling in benthos fail (nice catch by @CDanis):

root = if this.ip != "-" && this.sequence != "-" && this.sequence % env("SAMPLING").number() != 0 { deleted() }

Right now we get the sequence number from haproxy %rt log format, that's request_counter (HTTP req or TCP session) according to its documentation. On early stages of the TCP connection it seems like the request counter isn't accesible so it gets logged as 0.

I think I've identified the issue, right now haproxy always log sequence: 0 for <BADREQ> requests

SSH key verified out of band via https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1180839

@Dima_Koushha_WMDE could you create a gerrit change that contains your public SSH key (it can be immediately abandoned)? we can use that as a way of verifying the SSH key out-of-band, thanks

The change granting access to the requested groups has been merged, please allow up to 30 minutes to let puppet apply the changes on the impacted servers. Thanks

In T402191#11101054, @dang wrote:

@Vgutierrez Yes you can re-use it, it's better that way, and that's my account anw

I'm already seeing an account (https://ldap.toolforge.org/user/dang) requested on T288355 with some privileges:

dang:
  ensure: present
  realname: Tien Dat Nguyen
  email: dat.nguyen@wikimedia.de
  uid: 32183
  gid: 500
  ssh_keys:
    - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICnlB6UtmPKPJZOXl/2fkAC88ccb9dn15upi0SsifFg5 dang@C353

I'm seeing you have 3 LDAP accounts at the moment:

• https://ldap.toolforge.org/user/dang
• https://ldap.toolforge.org/user/datwmde
• https://ldap.toolforge.org/user/datnguyen

@dang could you create a CR on gerrit with your public SSH key to confirm it? thanks!

sorry for the delay @Mayakp.wiki.

Thanks, please feel free to re-open it if needed

you got that available as part of the sre.loadbalancer.migrate-service-ipip cookbook on https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/cookbooks/sre/loadbalancer/migrate-service-ipip.py#131:

def _ipip_traffic_accepted(self,  *,
                           outer_src_ip: str, outer_dst_ip: str,
                           inner_src_ip: str, inner_dst_ip: str,
                           dport: int) -> bool:
    """Send a single SYN packet using IPIP encapsulation"""
    s = socket(AF_INET, SOCK_STREAM)
    s.bind((inner_src_ip, 0))
    sport = s.getsockname()[1]
    syn_packet = (
        IP(src=outer_src_ip, dst=outer_dst_ip) /
        IP(src=inner_src_ip, dst=inner_dst_ip) /
        TCP(sport=sport, dport=dport, flags="S", seq=1000)
    )
    response = sr1(syn_packet, timeout=3, verbose=self.dry_run)
    s.close()
    return response is not None

Do you know which specific hostname the volunteer is asking about?

In T352956#11016142, @akosiaris wrote:

Yup, scheduling it for the weeks of either August 11th or August 18th.

https://phabricator.wikimedia.org/P80962 for future reference