deploy1002 will need to be scheduled well in advance and/or failed over to deploy2002 as it is the canonical deployment host.

Many thanks for this!

In T306649#7934722, @elukey wrote:

I merged two changes for the ml-serve-eqiad cluster, and now the concerns expressed in T306649#7881940 should be gone:

elukey@ml-serve1005:~$ ss -tulpna | grep ":179" | sort
tcp   ESTAB     0      0                       10.64.131.3:43807                   10.64.131.1:179         
tcp   ESTAB     0      0      [2620:0:861:10a:10:64:131:3]:60061           [2620:0:861:10a::1]:179         
tcp   LISTEN    0      8                           0.0.0.0:179                         0.0.0.0:*           
tcp   LISTEN    0      8                              [::]:179                            [::]:*

ml-serve1005 is in row E and it doesn't peer with cr{1,2} anymore. Of course the configuration needs to be improved and automated as described above, but for the moment:

• ml-serve100[5-8] peer only with their ToRs in row E/F
• ml-serve100[1-4] and ml-serve-ctrl100[1,2] peer with cr{1,2}-eqiad

Gitlab wise, I can create a personal repo easily, but my guess says that's not ideal either. I guess I shouldn't just jump the gun and create a repo on my own under the non-personal hierarchy, right?

In T306649#7933266, @JMeybohm wrote:

In T306649#7931058, @akosiaris wrote:

Regarding the "fake nodes": I think that could be done with adding the leafs as GlobalNetworkSet to the K8s/Calico API. That should make them easily selectable via peerSelectors without creating the confusion fake nodes would create.

Reading that, I am under the impression it won't work cause it only applies for Network policies. Can't hurt to try though.

Yeah. Some other docs suggest they can be used in selector fields an general, though.

In T306649#7931058, @akosiaris wrote:

They have the node-role.kubernetes.io/master:NoSchedule taint so nothing (aside from calico-node) will be scheduled on them, we can skip them for now while we test our setup, I expect (famous last words) it won't hurt (much) if they don't peer with cr{1,2}.

Not 100% true currently (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/777364).

Ah, cool! Thanks for the update!

In T306397#7927205, @Tsevener wrote:

@akosiaris cool, thanks! My instinct is that it feels a bit low - I wonder if pushes are getting dropped somewhere. It would be cool if we could somehow check how many Echo notifications were generated by accounts with subscribed device tokens in the last 7 days and see if that number is close to 552. Although I know the service batches pushes which may throw the numbers off too much for a direct comparison. Tagging @JMinor and @SNowick_WMF to see if this is worth investigating.

Regarding the "fake nodes": I think that could be done with adding the leafs as GlobalNetworkSet to the K8s/Calico API. That should make them easily selectable via peerSelectors without creating the confusion fake nodes would create.

The 50% bump in capacity didn't make any noticeable difference this time around. :-(

In T306649#7927197, @elukey wrote:

helm 2 did have a different structure regarding these things, it's definitely fine to revisit and re-evaluate the approach.

In T306649#7921343, @cmooney wrote:

Even in the legacy setup (pre row e/f) adding new nodes requires manual error-prone gerrit changes like this one 35b0c9a4832068d08

Yes I agree totally. One thing I had previously looked at, and thought it was not an option for us, is the Juniper "dynamic neighbor" feature for BGP peers. Basically this allows you to define an IP range/subnet as the peer, after which it will form a peering session with any remote device on the subnet that tries to establish a session. I'd previously tested this and it works great, however until recently it wouldn't work as you could only configure a single peer-as for the "dynamic-peer", and we have various peer ASNs connected off all our private subnets. I've looked again at this today though, and Juniper have introduced a "peer as-list" command in version 20 which seems to allow for it:

https://phabricator.wikimedia.org/P27787

So we could basically add that kind of config to all top-of-rack switches by default, and no further per-server configuration would be needed on the network side. You can configure a password on the sessions too if that tightens it up security wise. I'd need to run that by @ayounsi but it seems to me to be a good way forward.

we should consider if it makes sense to make an exception and renumber these hosts from 2037 so they are in par with eqiad.

Note that when above https://commons.wikimedia.org/wiki/Special:Upload is mentioned, we are focusing on a specific subset of Special:Upload. We are focusing on Copy by URL which is only triggered via the respective radio box. See image for a visual explanation

For what is worth, I think we 've peaked.

Thanks for the update. Load has somewhat increased on our side, albeit minimally.

In T306181#7920083, @BTullis wrote:

I have now deployed the change to double the number of replica pods for eventgate-analytics-external. Monitoring for any changes to the p90 and p99 response times.

In T306649#7916593, @cmooney wrote:

If there is any kind of anycast with the k8s prefixes (same prefix advertised from multiple locations), we should also prepend the AS once on the core routers to keep path lengths consistent across the infra.

Yeah I had considered that alright, in checking I found what Alex has confirmed (we don't have it right now).

My only gripe is that it's going to be a pretty long and cryptic config. But at least it's not going to be changing often. We 'll have to deduplicate it a bit too across cluster before it becomes unyieldy.

Yeah apologies for that. Happy to help in whatever way we can to assist getting this config together. At the very least as we add new racks/switches we should be able to complete the definition/add the IP. The peer on each subnet will be the default GW configured on the host too, and we can probably ensure it's the first usable IP on the subnet. I suspect that doesn't help much but want to mention it.

Just to note a semantic thing here:

In T306181#7917390, @BTullis wrote:

We have proposed creating a new bucket for requests that take up to 10 seconds but that hasn't happened yet.

Looking at https://grafana-rw.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&viewPanel=37&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos&editPanel=37 and temporarily adding a legend with max to the right:

@bd808, just for greater visibility, as I said in https://gerrit.wikimedia.org/r/c/773994, you can proceed and self-merge https://gerrit.wikimedia.org/r/c/773994 and https://gerrit.wikimedia.org/r/c/773995 and do the first deploy of it. We are unfortunately short-handed right now, we can't properly review the chart. The rest of the changes have been merged, so we shouldn't be a blocker for getting developer-portal deployed. Don't hesitate however to reach out if you hit any roadblocks or need any help!

In T303049#7903184, @Ottomata wrote:

FWIW, we hope that Datahub will one day be a service for more than just analytics data, but for now, it isn't, and can be considered part of the 'analytics cluster' which only exists in eqiad. What that means for DNS names and DC failover for now I'll leave up to yall :)

In T306797#7906932, @Ottomata wrote:

Oh, just saw this response, was just chatting with @BBlack about this in IRC.

I'd guess if > 100ms is a rare occasion, Kafka stretch would still be fine.

Just for greater visibility and awareness, there is T301505 for the upstream connect error or disconnect/reset before headers. reset reason: overflow". error. As pointed out in that task that error message is a symptom and not the cause.

Been reading some blogs and talks about this 'stretched' Kafka cluster idea, and it sure would make Multi DC apps much easier. From the linked Kafka talk though, it recommends not doing it unless you can guarantee latency between DCs of <= 100ms. I have a feeling that our SREs would not like to guarantee that.

In T303049#7902665, @BTullis wrote:

In T303049#7902578, @JMeybohm wrote:

I was under the impression that datahub should only run/be used in the active datacenter because it relies on state in MySQL and other datastores which are not equally available in both DCs.

Thanks. You're right that the stateful components only exist in eqiad, so running datahub in codfw is going to be slower than eqiad. However, the network policies are in place so that it should work in codfw.
I'm happy to take advice on whether this should be set up as an active/passive or active/active service. Do you think active/passive would be better, if our preferred service is eqiad?

In T306649#7883435, @elukey wrote:

We have discussed this issue in the serviceops channel yesterday, and the idea is to indeed use labels. The ML clusters do not define failure-domain.beta.kubernetes.io at the moment, but we are going to add them asap (I'd say manually, with a comment in Hiera to be deployed when the nodes are reimaged).
The idea for the new ToR-specific configuration is to add a label called wikimedia.org/node-location (or something similar), for example:

lsw1-f3-eqiad-ipv6:
  nodeSelector: wikimedia.org/node-location == lsw1-f3-eqiad
  asNumber: 14907
  peerIP: 2620:0:861:10f::1

In T306797#7880771, @Ottomata wrote:

We should talk to @dcausse and @akosiaris about their experience deploying Flink and what they would do differently.

As a starting point: @jhathaway noted that we're running ffmpeg at niceness -19, which is quite assertive; raising that value might be an easy way to relieve the pressure. I don't have historical context for why it is that way, but if we can change it safely, it might be a good first step.

In T305899#7889812, @bd808 wrote:

In T305899#7861232, @bd808 wrote:

In T305899#7861063, @akosiaris wrote:

which just doesn't look right. I wonder what prometheus does in this case.

That 301 redirect is basically the same issue that we saw with /healthz checks in T294072. I'll put up a patch to make the same fix for /metrics.

After fixing the redirect metrics dropped down to levels that make more logical sense (<1 rps). There must have been something very strange happening because of the redirect for the traffic to have been reporting 2000% higher than reality.

https://gerrit.wikimedia.org/r/787708 removes the puppet lvm module which is GPL-2 and incompatible with apache 2.0. So that removes an interesting blocker in adopting a cross repo license.

In T297140#7886240, @Dzahn wrote:

Hi @akosiaris @JMeybohm

@Eevans, @hnowlan let me know if you have any ideas on how to fix this.

So, prometheus follows redirects by default. In https://github.com/prometheus/prometheus/commit/646556a2632700f7fca42cec51d0100294d43c52 support for disabling that functionality was added. I 'd say that for production we want to default to not following redirects, it should help us avoid weird edge cases like this one.

In T305899#7858965, @bd808 wrote:

I have attempted to fix the Traffic and Errors rows on https://grafana.wikimedia.org/d/wJHvm8Ank/toolhub?orgId=1&refresh=1m. I'm honestly not convinced that I have the sum(rate(...)) things quite right. The shape of the curves all look reasonable, but the amplitude seems too high even after I filtered out all of the /healthz and /metrics requests from the time series queries.

These are the queries that I used:

• Traffic / Total: sum(rate(django_http_requests_total_by_view_transport_method_total{app="$service",view!="healthz",view!="prometheus-django-metrics"}[5m]))
• Traffic / by HTTP method: sum(rate(django_http_requests_total_by_view_transport_method_total{app="$service",view!="healthz",view!="prometheus-django-metrics"}[5m])) by (method)
• Traffic / by endpoint: sum(rate(django_http_requests_total_by_view_transport_method_total{app="$service",view!="healthz",view!="prometheus-django-metrics"}[5m])) by (view)
• Traffic / by HTTP status: sum(rate(django_http_responses_total_by_status_view_method_total{app="$service",view!="healthz",view!="prometheus-django-metrics"}[5m])) by (status)
• Errors / HTTP errors: sum(rate(django_http_responses_total_by_status_view_method_total{app="$service",status=~"4..|5..",view!="healthz",view!="prometheus-django-metrics"}[5m])) by (status)
• Errors / current error rate: sum(rate(django_http_responses_total_by_status_view_method_total{app="$service",status=~"4..|5..",view!="healthz",view!="prometheus-django-metrics"}[5m]))
• Errors / current error %: sum(rate(django_http_responses_total_by_status_view_method_total{app="$service",status=~"4..|5..",view!="healthz",view!="prometheus-django-metrics"}[5m])) / sum(rate(django_http_responses_total_by_status_view_method_total{app="$service",view!="healthz",view!="prometheus-django-metrics"}[5m]))

I think that you are right. Those numbers don't look ok. And I do notice the following

In T301471#7840496, @Michaelcochez wrote:

I merged the pull request on github now.

I do not have rights to push to the gerrit repository, it might just be my limited knowledge of how gerrit works.

I am reopening this. Due to https://wikitech.wikimedia.org/wiki/Incidents/2022-03-27_api we had to lower the concurrency by half in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/774462

Some more information regarding this. With the exception of the warnings stanza the response by mathoid for queries `\land xcc' and '\and xcc' is identical

Adding @Eevans and @hnowlan. They are the only 2 people in the WMF that I know that might be able to help debugging this.

@Wurgl, nailed it!

In T305613#7850547, @Physikerwelt wrote:

OK, I am sending the output of the curl -v in private paste as suggested.

• The monitoring: stanza can't be added as having that without lvs: breaks icinga. Can potentially be ignored (T291946), see above.

Stalling until T306121 is done.

I had a quick lock at openstack browser. Only VM that is stretch is packager01.packaging.eqiad1.wikimedia.cloud. I only use that one for building the etherpad-lite Debian packages cause they are unfortunately unbuildeable in our production infrastructure. I just tested I can build them on packager02.packaging.eqiad1.wikimedia.cloud, so feel free to delete packager01.

@Papaul: mc2023 and kubestage2002 have been downtimed again (for 2days) and I 've just powered them off. The should be ready to be moved.

In T238751#7851690, @Addshore wrote:

@Joe (Also pinging @akosiaris as I know joe is out right now).
It seems like the ideal solution of T239392: Applications and scripts need to be able to understand the pooled status of servers in our load balancers. might not happen for some time.
Would it be possible to resolve this for now with https://gerrit.wikimedia.org/r/c/operations/puppet/+/553097 which I believe would have been "fine" TM for the last 2.5 years and decreased humans touching things and also decreased the number of issues users end up seeing around delayed / broken but depooled wdqs hosts?

In T290357#7846886, @bd808 wrote:

In T290357#7405708, @akosiaris wrote:

That leaves the use cases of interactively using the Django REPL (or any other interactive tool requiring a PTY), which we are still discussing if and how we will support.

@akosiaris has there been any more discussion of this use case in your team? One of the findings from T303889: Toolhub broken in prod by memcached client library change was that "run maintenance actions from Bryan's laptop" (T290357#7578866) is slow and hindered our recovery from the outage.

As pointed out in T305902, those metrics are already scraped by prometheus for a pretty long time now. What's left is to actually alter the dashboard to reference those metrics and not the service-runner ones.

In T305902#7849329, @bd808 wrote:

In T305902#7849062, @Raymond_Ndibe wrote:

How can one get started working on this @bd808 ? any pointers? especially the kubernetes scraping part

This is yet another area where Toolhub is an early adopter and there does not yet seem to be strong documentation on how to proceed. I think the answer is going to be something like "make a patch to operations/puppet.git". https://github.com/wikimedia/puppet/blob/production/modules/profile/files/prometheus/rules_k8s.yml might be the right place, but I'm not sure.

I would recommend trying to contact folks like @fgiunchedi from SRE Observability or @akosiaris from serviceops to ask for some advice.

Tentatively resolving this.

I don't see an alert since Apr 5 in Icinga for either codfw or eqiad. I am gonna re-enabling paging.

In T305613#7848440, @Physikerwelt wrote:

I recall that some requests were cached forever. The person who cleaned the cache last time was @GWicke so some time has passed since then. Do you know what the current lifetime of the cache is https://github.com/wikimedia/restbase/blob/ecef17bda6f4efc0d6e187fb05b1eeb389bf7120/sys/mathoid.js#L176

Thanks for the ping, wouldn't have seen it otherwise. Re-opening and I 'll have a look.

In T305613#7847731, @Physikerwelt wrote:

@akosiaris thank you. Thank you. The check endpoint does not work with a hash but the actual TeX string.

In T305613#7847243, @Physikerwelt wrote:

@Wurgl I reached the end of my options. I checked that no warning is emitted in the source code by adding tests. Maybe this is a caching problem. Here you need support from someone with access to the restbase server that can delete the cache for the land command from Cassandra. As you can see from the edits on https://www.mediawiki.org/w/index.php?title=Extension:Math/T305613&action=history it seems to be a caching issue. I don't know what to do next. Maybe @akosiaris can help?

I marked rdb2008, kubestage2002 and mc2023 as YES in the table. rdb2008 is the secondary, not the primary, kubestage2002 is for the staging cluster anyway and mc2023 will be handled by mcrouter's configuration and shard05 should be moved to mc-gp* hosts (gutter pool).

In T301471#7837692, @Addshore wrote:

So a summary comment as promised!

In T301471#7799393, @Michaelcochez wrote:

@QChris I noticed the addition of the .gitreview file on gerrit. Is this file needed? If so, we would merge it into our github repository, so we can keep the active development there and synchronize with gerrit.

Sounds like this would work to me, and would be fine from the WMDE side of things.
@akosiaris would syncing the code from Github to Gerrit via a Gerrit change whenever it would need to be updated be acceptable on your side?

I am resolving this followup from https://wikitech.wikimedia.org/wiki/Incident_documentation/2022-03-27_api. This will decrease errors from mobileapps in the future and responses will be given to changeprop more promptly, lowering the amount of retries.

And now that enough time has passed, indeed the rate of errors is lower (I 've arbitrarily drawn a couple of lines at around the 90+th percentile to showcase it easily. There is also less variation so this isn't exactly scientific, but I 'd say good enough.

After patch was merged and deployed we have happier graphs!

In T303803#7778484, @herron wrote:

The reasoning for checking these via the proxy is because the prometheus hosts can't reach all of the watchrat checked URLs directly, and it's simpler to have one blackbox exporter configuration that uses a proxy and works for all cases than to split the config out between proxied/non-proxied urls. Here's the current config https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/prometheus/templates/blackbox_exporter/common.yml.erb$25-34

I agree it would be nice to not need the proxy, and IMO also worth considering if it'd be worthwhile for prometheus hosts to have public addresses so this kind of config would do the right thing without proxy.

If it's not causing a problem on the proxies but more a question of why then I think we're ok as configured, but happy to adjust if needed.

For posterity's sake, alert histograms from icinga for the 2 instances of zotero.

In T303803#7778903, @jbond wrote:

Putting aside if we should split the config or provide an external ip address. i wonder if https://wikitech.wikimedia.org/wiki/Url-downloader should be preferred for things like this, @akosiaris?

In T291707#7401997, @Mvolz wrote:

In terms of a get endpoint, would swagger docs suffice? I started to do something like that but never got around to finishing it :/ https://github.com/zotero/translation-server/issues/76

That would work fine. Many thanks for working on that!

In review -> https://github.com/zotero/translation-server/pull/131

In T288375#7804357, @BTullis wrote:

Could we deploy the GeoIP databases to the kube-workers and then mount it to the mw pods as a readonly hostpath volume?

This has been done, resolving!