I believe this is related to https://sal.toolforge.org/log/YZV29XcBa_6PSCT9KHrZ

I'm pretty sure the baud rate is 19200

Did this cause any actual issue?

In T274888#6861566, @BBlack wrote:

4. sh hashing - I think @CDanis already worked on some patches to transition us to maglev hashing a quarter or two ago, but it wasn't ever deployed to all production LVSes. Need to take a look at the state of affairs on that and see if we're at a point where we can safely deploy it (also, re-check assumptions about whether it relies on a weight=0 depooling strategy?).

wmf-utils is a little-used repo, with only two scripts in it.

Caches have filled, and there's no more fetch failures due to "LRU limited": https://grafana.wikimedia.org/d/000000352/varnish-failed-fetches?orgId=1&var-datasource=eqsin%20prometheus%2Fops&var-cache_type=upload&var-server=All&var-layer=frontend&from=1613576456135&to=1613678433179

@Ottomata just one more question for you!

Email address sent privately.

In T269324#6794983, @Marostegui wrote:

In T269324#6794656, @Krinkle wrote:

nightmare […] It all starts with having to depool them via a MW commit, […]

I'm confident we can avoid this for mainstash db (x2).

The fact that this requires a MW commit today for parser cache is afaik "just" because we haven't moved more of db-related config to Etcd. As with previous moves, we need to be very careful and aware of the contractual expectations and needs, which for parser cache are indeed non-trivial, but ultimately it is just a primitive array structure that has no inherent need for being in PHP or MW config. As far as I'm concerned parser cache config can and should, too, be moved to Etcd if there are no other stakeholders raising concerns against that.

I would love to see that, but not sure how doable it is. That's probably for @CDanis to estimate.

@hnowlan just a heads up that it looks like the depool of maps1001 left maps@eqiad underprovisioned:
https://grafana.wikimedia.org/d/000000305/maps-performances?orgId=1&from=1612777514651&to=1612819057362

In T273602#6811137, @Ottomata wrote:

@CDanis nda LDAP is also needed for Jupyter access. Pretty much all users should get LDAP access if they are getting any access at all.

I contacted Carol on Slack; this request is approved.

Glad to hear, thanks!

Hi, just wanted to check in if anything more was needed here?

@KFrancis Can you please get an NDA signed with this WMDE staff member? Thanks!

@jrobell Can you please confirm? Thanks!

The wmf group does not require manager approval -- only verification that staff is staff :)

@AikoChou should have shell access within half an hour.

Checked in with Miriam on IRC and Turnilo/Superset access isn't needed, but Kerberos is. Doing that now.

Thank you!

I think the change pushed today in the puppet private repo (hash b1b32d4ab) broke the meta-monitoring validation script.

Waiting on @AikoChou to complete prerequisites and also for @Ottomata to approve from Analytics.

looks good now, thanks!

Hi Maya,

I think something is still wrong? LibreNMS is showing the port on the asw receiving about 7kbps of errors: https://librenms.wikimedia.org/device/device=149/tab=port/port=12410/view=graphs/

The first few cycles of logs from the ulsfo side:

Jan 29 20:21:46  cr4-ulsfo bfdd[16019]: BFD Session fe80::827f:f800:43:6b66 (IFL 75) state Up -> Down LD/RD(159/26) Up time:4d 18:26 Local diag: CtlExpire Remote diag: None Reason: Detect Timer Expiry.
Jan 29 20:21:46  cr4-ulsfo bfdd[16019]: BFDD_TRAP_SHOP_STATE_DOWN: local discriminator: 159, new state: down, interface: gr-0/0/0.1, peer addr: fe80::827f:f800:43:6b66
Jan 29 20:21:46  cr4-ulsfo rpd[16292]: RPD_OSPF_NBRDOWN: OSPF neighbor fe80::827f:f800:43:6b66 (realm ipv6-unicast gr-0/0/0.1 area 0.0.0.0) state changed from Full to Down due to InActiveTimer (event reason: BFD session timed out and neighbor was declared dead)
Jan 29 20:21:46  cr4-ulsfo rpd[16292]: RPD_OSPF_NBRUP: OSPF neighbor fe80::827f:f800:43:6b66 (realm ipv6-unicast gr-0/0/0.1 area 0.0.0.0) state changed from Init to ExStart due to 2WayRcvd (event reason: neighbor detected this router)
Jan 29 20:21:47  cr4-ulsfo bfdd[16019]: BFDD_TRAP_SHOP_STATE_UP: local discriminator: 159, new state: up, interface: gr-0/0/0.1, peer addr: fe80::827f:f800:43:6b66
Jan 29 20:22:09  cr4-ulsfo rpd[16292]: RPD_OSPF_NBRUP: OSPF neighbor fe80::827f:f800:43:6b66 (realm ipv6-unicast gr-0/0/0.1 area 0.0.0.0) state changed from Exchange to Full due to ExchangeDone (event reason: DBD exchange of master completed)
Jan 29 20:22:39  cr4-ulsfo bfdd[16019]: BFD Session fe80::827f:f800:43:6b66 (IFL 75) state Up -> Down LD/RD(159/26) Up time:00:00:52 Local diag: CtlExpire Remote diag: None Reason: Detect Timer Expiry.
Jan 29 20:22:39  cr4-ulsfo bfdd[16019]: BFDD_TRAP_SHOP_STATE_DOWN: local discriminator: 159, new state: down, interface: gr-0/0/0.1, peer addr: fe80::827f:f800:43:6b66
Jan 29 20:22:39  cr4-ulsfo rpd[16292]: RPD_OSPF_NBRDOWN: OSPF neighbor fe80::827f:f800:43:6b66 (realm ipv6-unicast gr-0/0/0.1 area 0.0.0.0) state changed from Full to Down due to InActiveTimer (event reason: BFD session timed out and neighbor was declared dead)
Jan 29 20:22:39  cr4-ulsfo rpd[16292]: RPD_OSPF_NBRUP: OSPF neighbor fe80::827f:f800:43:6b66 (realm ipv6-unicast gr-0/0/0.1 area 0.0.0.0) state changed from Init to ExStart due to 2WayRcvd (event reason: neighbor detected this router)
Jan 29 20:22:41  cr4-ulsfo rpd[16292]: RPD_OSPF_NBRUP: OSPF neighbor fe80::827f:f800:43:6b66 (realm ipv6-unicast gr-0/0/0.1 area 0.0.0.0) state changed from Exchange to Full due to ExchangeDone (event reason: DBD exchange of master completed)
Jan 29 20:22:43  cr4-ulsfo bfdd[16019]: BFDD_TRAP_SHOP_STATE_UP: local discriminator: 159, new state: up, interface: gr-0/0/0.1, peer addr: fe80::827f:f800:43:6b66
Jan 29 20:22:50  cr4-ulsfo bfdd[16019]: BFD Session fe80::827f:f800:43:6b66 (IFL 75) state Up -> Down LD/RD(159/26) Up time:00:00:06 Local diag: CtlExpire Remote diag: None Reason: Detect Timer Expiry.

Here are overall, whole-cluster mean latency milliseconds, summed across all machines and then broken down by kernel version.

Can you please provide a complete dump of a "null response", with both the complete response headers and the raw response body?

It seems the User-Agent being used is Peachy MediaWiki Bot API Version 2.0 (alpha 8) (which ideally should be updated to comply with the User-Agent policy).

What is the originating IP address of these requests?

In T272633#6768020, @JMeybohm wrote:

Unfortunately, upstream was not very responsive on my question about adding Cache-Control (https://github.com/helm/chartmuseum/issues/368). I wonder why this problem arised only recently as we ran this configuration from the start.

As ChartMuseum does an internal caching of the repo objects as well, I think we can switch to 'pass' for now. Will prepare a patch.

In T272633#6766881, @Dzahn wrote:

An easy way to do this would be to just switch 'normal' to 'pass' here. Then there would be no caching at all. We do the same for other services like cxserver, API, blubberoid, config-master, debmonitor and others.

Would that be ok? If not then it probably needs a new type of caching besides the existing: normal, pass, pipe and websockets?

Sorry for the truly baffling error message.

In T272539#6763461, @CDanis wrote:

There's one related problem, which is that enable-puppet should check the given message both with and without appending - $SUDO_USER, as perhaps you set a disable-puppet from a context where that wasn't present, for example https://sal.toolforge.org/log/8BlxIXcBgTbpqNOm5XlV

There's one related problem, which is that enable-puppet should check the given message both with and without appending - $SUDO_USER, as perhaps you set a disable-puppet from a context where that wasn't present, for example https://sal.toolforge.org/log/8BlxIXcBgTbpqNOm5XlV

John has it right -- I wanted to lower the bar, and ensure all deployers / folks with any sort of shell access have Klaxon access. I wasn't sure that set was covered by wmf/wmde/nda; thanks to John for proving that is indeed the case :)

In T263496#6744057, @Ottomata wrote:

The long term solution here is still not clear and is very tied up with some other yet undefined long term projects, like Data Governance.

Let's just make this happen in the short term just like we have for client ip and user agent.

Hey @Ottomata, I meant to get around to this last quarter but didn't. Would very much like to get some mechanism in place soon -- do you have any ideas on the best path forward? Maybe we should set up a quick meeting to chat through options?

Thanks for the writeup with all the background! And for the cleanup patches so far :)

In T270391#6719023, @ayounsi wrote:

A downside, for example with Google is that it will most likely include crawlers IPs

Thanks to observability for initial feedback, and to @Joe @jbond and especially @RLazarus for code reviews!

Now live: https://klaxon.wikimedia.org/
and successfully tested today!

Thanks for the report.

In T269324#6705379, @Marostegui wrote:

@CDanis I was taking a look at the section values and I have seen: flavour which is sort of new to me and seems to accept regular or external. But I haven't been able to find what are each for on https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/conftool/+/refs/heads/master/conftool/extensions/dbconfig/README.md

sX seem to be regular and x1, esX seem to be external, what's the difference between them?

I'll put this in puppet-private once I have the other puppetization ready :)

Not totally sure about the best fit here vs Beta, but you might also
consider using the 'staging' k8s cluster.

So I guess we need a separate task for paging and the check_librenms
deprecation?

Does this mean we can deprecate the check_librenms Icinga integration?

Sure, always happy to help :)

Tagging @BBlack in the hopes he knows anything offhand about shipping new GeoLite2 or full GeoIP2 files to Beta Cluster

signed

The max lifetime of any object in the Traffic CDN is 24 hours. Are you sure they're being cached there? Can you give a full example URL?