Per NEL data it looks like that this issue was fixed on Sept 17th!

Ah, I see the problem, you weren't added to the wmf LDAP group.

Try visiting https://idp.wikimedia.org/logout and then logging back in?

Weird, but happy to hear it :)

Interesting, I haven't encountered this before.

In T293536#7433184, @Legoktm wrote:

For known slow queries, this would be useful in addition to the automated query killer to limit runtime to <= 60 seconds.

This could also work in the opposite direction, have a global limit that's low, but increase it for known slow queries.

BTW, I think this fired today in the releng IRC channel?

Access granted to the wmf group in LDAP, so you should be able to access web-based tools.

also @DAbad please confirm you approve as well

@Ottomata or @odimitrijevic please approve, thanks!

@lbowmaker Great, thanks. Please paste somewhere in this task the public part of the key (a line of text that will start with something like ssh-ed25519 or ssh-rsa) and I'll get you set up.

Thanks Andrew! @DAbad if you want shell access, please generate and paste a public key: https://wikitech.wikimedia.org/wiki/SRE/Production_access#Generating_your_SSH_key

@Ottomata or @odimitrijevic can you please approve? Thanks!

@Ottomata or @odimitrijevic can you please approve? Thanks!

Access granted!

Shell account created and it will be live in ~30 minutes.

Just a quick update that, after a year, this is still happening, on both intake-analytics (50x/minute =~ 0.8x/sec) and intake-logging (1-2x/minute).

@MunizaA can you confirm that this wikitech user is you? https://ldap.toolforge.org/user/mnz

In T292870#7415867, @ayounsi wrote:

I'd wary of the complexity of the setup.

@MunizaA can you please sign the L3 agreement by clicking on L3 ? thanks!

Access granted!

Hi Natalia, just a friendly reminder that this ticket is still awaiting an SSH public key from you.

Access granted!

Access granted!

Andrew, can you approve? Thanks!

Andrew, can you approve? Thanks!

For posterity:

Because the only data available in the API is the timestamp of the most recent successfully-processed datapoint, this unfortunately left a gap in the data for two metrics:

To aid in the Puppetization and deployment: P16741 is a private-to-SRE paste that contains a configuration file suitable for use in production.

In T202061#7176114, @CDanis wrote:

In T202061#7175886, @Legoktm wrote:

In T202061#7175872, @lmata wrote:

https://wikimedia.statuspage.io/ is live and will continue to see more development, usage and integration. Further efforts to be tracked in the Observability Workboard.

Is there documentation about this somewhere? E.g. what does "Editing" mean, which API is "API" about, what developer tools are being watched, etc.?

The section headers are not finalized -- honestly I'm not happy with them, despite having chose them. These were just the ideas that we hated the least. Documentation to be written, likely after some more discussion and some experience.

As a fun curiosity, here's today's datacenter switchover as shown by Statuspage:

As a fun curiosity, here's today's datacenter switchover as shown by Statuspage:

In T202061#7175886, @Legoktm wrote:

In T202061#7175872, @lmata wrote:

https://wikimedia.statuspage.io/ is live and will continue to see more development, usage and integration. Further efforts to be tracked in the Observability Workboard.

Is there documentation about this somewhere? E.g. what does "Editing" mean, which API is "API" about, what developer tools are being watched, etc.?

Unfortunately I won't have time to work on this before going on leave, but it seems like it might not be a bad task for @cmooney to learn some about RIPE Atlas. Or failing that perhaps @ayounsi will have some time for it.

RIPE Atlas probes support sending DoT queries; however, the option is not exposed anywhere in the measurement creation web UI, nor in the official ripe-atlas-tools distribution. The unofficial blaeu tools for creating one-off measurements support sending the option, and there is also a pull request against ripe-atlas-tools (lingering almost two years now).

Totally agree with all of the above! Filed a placeholder task: T283614: RIPE Atlas monitoring of reachability & latency towards anycasted Wikidough IP

Some very initial results running against the anycast NS at https://atlas.ripe.net/frames/measurements/30366093#!probes

Thanks very much for all the helpful comments!

Right now the WMCS NSes ({ns0,ns1}.openstack.eqiad1.wikimediacloud.org) are behind CF, but nothing else from WMCS is.

There is evidence in our NEL data that suggests this problem still exists.

@fdans @JAllemandou New map entry should be ready for Analytics to set up in Turnilo :)

Moving to AM sounds good to me. But if needed, in the interim we could change the magic string we use in check_librenms to something else instead of hash page, which I chose for simplicity but has maybe just caused more confusion.

I'm not sure if we tend to use IP addresses directly in Mediawiki out of latency concerns, reliability concerns, PHP/other client bugginess concerns, or DNS recursor capacity concerns, or some mix of all of the above. (I've seen all of the above occur at different times.)

ASN, ISP/organization, country, & subdivision are now visible in Logstash!