Just as an FYI, everything looks ok on this end, but there's a train freeze this week, so we have to wait before deploying this. Patches are up and waiting to be merged on Monday the 17th

In T220402#5243209, @Tarrow wrote:

This should now be fixed. Sadly this was due to a mismatch between the code in wikibase master and that deployed on Wikidata.org

In T224603#5243200, @Papaul wrote:

@ayounsi I am planning on installing those new servers in row c and row D and I don't have the "interface-range ganeti" in both of those rows Is it okay for me to go ahead and create "interface-range ganeti" on asw-c-codfw and asw-d-codfw?

sessionstore.discovery.wmnet is now around and should be the canonical DNS used to address the service.

In T199219#5234979, @Smalyshev wrote:

But won't we lose use of the varnish cache if we use the internal endpoint?

Indeed this was fixed. However another regression has crept up it's head

@Tarrow, @WMDE-leszek
Hi, sorry for taking so long to answer to this, it's been really busy.

In T225064#5240772, @thcipriani wrote:

In T225064#5240522, @mobrovac wrote:

In T225064#5240480, @hashar wrote:

Nice, thank you for the explanation :-] Left to figure out in a different task is how to test Citoid together with Zotero, but I guess that is for another task.

Euh, no, that's what this task is for :) We were able to build images before, now we are not.

Not since adding the magic file that makes helm test work: https://gerrit.wikimedia.org/r/#/c/mediawiki/services/citoid/+/506107/

The build is gone from Jenkins (since it was six weeks old), but the failure comment in gerrit is still there.

In T198901#5237895, @Ottomata wrote:

I don't think eventstreams is in k8s, is it?

In T199219#5234041, @Smalyshev wrote:

There's a change though that WDQS no longer uses nocache for cache-busting in most common cases (see T217897 for more details). So I am not sure using internal endpoint now makes sense.

In T199219#4452041, @Smalyshev wrote:

@BBlack I am getting rather strange result with appservers-ro.discovery.wmnet - if I call the URL you provided, the call takes a lot of time:

real 0m4.270s

while if I call to www.wikidata.org, I get:

real 0m0.127s

Same with api-ro. appservers-rw is a bit faster:

real 0m0.320s

But still 3x from going through frontend (and it's not caching - I changed the URL, result is the same, and varnish settings all say "miss").

And LVS done today.

Patch has been applied to our own packages and has been deployed and tested. Marking this as resolved, thanks!

Fix by upstream in https://github.com/OTRS/otrs/commit/7ab33e51a4db9f712e979040f644d0d0c39ff0af for 5.x (which we run). Has also been fixed in our package for OTRS in https://gerrit.wikimedia.org/r/#/c/operations/software/otrs/+/514230

In T220401#5230202, @Eevans wrote:

In T220401#5228040, @akosiaris wrote:

In T220401#5226623, @Eevans wrote:

In T220401#5226531, @akosiaris wrote:

One minor question. Given per T220401#5128786 1 kask instance is able to handle ~300req/s, how many instances will we require? I am unsure of the current rate of sessions requests to/from redis.

What was the test environment used there? When I tested using the sessionstore Cassandra cluster nodes, I got at least two orders of magnitude higher throughput.

An admittedly underpowered minikube environment with a probably untuned cassandra. Some values for cassandra itself are in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/kask/values.yaml#114. It makes absolute sense that a well tuned and more powered cassandra cluster would be able to serve more req/s.

Now, to answer my question, and by looking at T221292, I 'll assume a single instance for production should be able to serve some 30k req/s (I am rounding down from the lowest score in that table just to be on the safe side). So 1 instance would probably not cover it, we would need at least 2 instances. Adding 2x rack row redundancy means 4 instances. Looks like that's our number for now. We can always increase it ofc.

FWIW, we've been bouncing around a target throughput of 30k/sec in production based on Redis metrics, but as was later noted in T212129, that number includes everything in Mainstash, only a fraction of which is sessions (we're moving sessions over separately of the rest). IOW, sessions should be something considerably less 30k/s, even if we don't know exactly what.

In T220401#5226623, @Eevans wrote:

In T220401#5226531, @akosiaris wrote:

One minor question. Given per T220401#5128786 1 kask instance is able to handle ~300req/s, how many instances will we require? I am unsure of the current rate of sessions requests to/from redis.

What was the test environment used there? When I tested using the sessionstore Cassandra cluster nodes, I got at least two orders of magnitude higher throughput.

One minor question. Given per T220401#5128786 1 kask instance is able to handle ~300req/s, how many instances will we require? I am unsure of the current rate of sessions requests to/from redis.

I think so, let's wait for @fsero though

And this uncovered now that prometheus can't talk to it (cause it expects HTTP I guess?). /me looking into it (more deeply this time around).

In T220401#5226287, @Eevans wrote:

In T220401#5225728, @akosiaris wrote:

One thing that I just met is that kask stops accepting HTTP connections if kask cert/key pair is configured. That's fine normally, but there is a very interesting repercussion. Kubernetes readiness probes to the /healthz endpoint now fail. kask logs

2019/05/31 09:36:00 http: TLS handshake error from 10.64.0.247:55194: tls: first record does not look like a TLS handshake

One thing that I just met is that kask stops accepting HTTP connections if kask cert/key pair is configured. That's fine normally, but there is a very interesting repercussion. Kubernetes readiness probes to the /healthz endpoint now fail. kask logs

We might be able to get away with reusing the redis misc servers (rdb1005/rdb1009). That should give us more memory and allow us to use the empty databases on those hosts and deprecate/remove these VMs

Indeed. I added it under TBD. The exact way this will be done will need to be investigated.

In T198901#5220959, @Krenair wrote:

Is apertium part of the cxserver migration?

https://wikitech.wikimedia.org/wiki/Ganeti#VMs_without_DRBD_disk_template has been added to address the drawback needing to be communicated and documented.

Hidden and marked as security by upstream.

Upstream bug at https://bugs.otrs.org/show_bug.cgi?id=14568

https://gerrit.wikimedia.org/r/512875 seems to have fixed this. Ticket #2019052810003632 as a demonstration. I 'll resolve this as successfully. @Reedy, thanks for correcting me, it really helped!

I just reproduced it. This is related to a sandboxed <iframe> that is embedded in the page to showcase the content of the email in a safe way. That being said, leaking session info in the url is indeed bad practice, due to all the copy pasting that can happen. In fact, just adding a valid OTRSAgentInterface to any OTRS url seems to be enough to allow assuming the OTRS identity of a user. On the plus side, those aren't guessable at least. The fun part is that

We 've upgraded to 5.0.35. Resolving this. Thanks!

In T224404#5215640, @Reedy wrote:

https://meta.wikimedia.org/wiki/Special:Contact/Stewards is used in the Wikimedia Cluster and as such is using the WikimediaMaintenance extension [1] which per [2] sets the SMTP Precedence header.

I note WikimediaMaintenance is just a repo full of WMF specific maintenance scripts...

ContactPage does not use sendBulkEmail.php from that repo, at all.

That would split even worse though the configuration. Instead of having it at least in different but related parts of the infrastructure, now it's across different infrastructures (mediawiki vs email).

In T224041#5201573, @thcipriani wrote:

It seems that the cassandra subchart already exists for cask (via https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/509102/ );

I am lowering to High, just in the interest of not abusing Unbreak Now!, since this task has been in this state since Apr 23. That being said, this indeed needs to be resolved ASAP.

needs an extra stanza

Every deployment that uses statsd-exporter (namely zotero & blubberoid don't) in kubernetes has been upgraded. Resolving this. Many thanks!

In T220235#5183451, @Krenair wrote:

krenair@deployment-docker-cxserver01:~$ sudo /usr/bin/docker run -p 8080:8080 -v /etc/mediawiki-services-cxserver/:/etc/mediawiki-services-cxserver --name alex-test docker-registry.wikimedia.org/wikimedia/mediawiki-services-cxserver:2019-05-08-064536-production -c /etc/mediawiki-services-cxserver/config.yaml
 Error during DHT setup undefined
 krenair@deployment-docker-cxserver01:~$

Anyone know what that means?

In T223345#5183583, @Joe wrote:

It's working in production because we connect to external URIs via a proxy, hence we don't need ca-certificates.

In T220402#5180031, @Pablo-WMDE wrote:

Hi @akosiaris,

thanks for taking the time to explain the way the Host header is intended to be used.
If I understand correctly the goal is to ensure that requests originating from our service bear a header Host: (www.)wikidata.org and reach which ever IP(s) appservers.discovery.wmnet resolves to on the system running it. This sounds like a name resolution challenge and a case for HostAliases or, more traditionally, a CNAME record.

In T220402#5177862, @Pablo-WMDE wrote:

Hi @akosiaris - thanks for getting back to us.

sending a Host: HTTP for the identification of the exact project. Would it be possible to add that functionality?

It certainly is possible and depending on operational needs we certainly can make this happen. We quickly discussed this in the team and would like to first truly understand the goal to make sure we don't mix up the different layers of our proverbial sausage pizza without a valid reason. The service would be run in a container inside a k8s pod, controlling its DNS - why not use this option to make sure requests reach the intended endpoint? The correct host would then come "for free" per the host part configured in WIKIBASE_REPO.

With respect to the end point checks it would be great to hear what we are trying to achieve with them. Our service depends on the availability of another service. If the examples are to act as smoke tests then their reliability depends on the upstream service; a dependency which would need to be configured (are we going to point it against prod for this?) & modeled (how to express service inter-dependency in the config?) in order to be able to make sense of the information down the line (i.e. "no need to be alarmed that this service reported 500 while the mw api was down").

In T222899#5179002, @Dzahn wrote:

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=lvs2003&service=PyBal+IPVS+diff+check

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=lvs2003&service=PyBal+connections+to+etcd

^ some new Icinga alerts apparently caused by this. Probably just needs pybal restart? But not sure enough.

In T220235#5179194, @Joe wrote:

In T220235#5176630, @Ottomata wrote:

Could we use image version: latest in beta hiera? And somehow pull down the new latest and restart the image whenever a new version is created and uploaded to the registry?

Sure, you just have to add a more complex script to exec to service::docker I guess, so that you can properly check that 'latest' or any other similar metatag are correctly respected.

Be my guest, I'll happily review the change!

Bacula & puppet databases are not going to exhibit any problems anyway. Puppet is literally used only by servermon and this is to be uninstalled pretty soon and backups don't happen during that timewindow.
etherpad, given the software, is a best-effort service, so no guarantees there. it will probably crash anyway, be restarted by systemd (as it anyway does every couple of days), users will be reconnected.

In T222962#5172806, @Ottomata wrote:

Hm, question. Currently mediawiki-config ProductionServices.php has:

'eventgate-analytics' => 'http://eventgate-analytics.discovery.wmnet:31192',

If we change LVS port to 33192, this will just change the LVS monitoring/pooling/depooling, right?

In T222795#5176181, @Pchelolo wrote:

Thank you for an impressive level of details :) There's a bunch of other places where we abuse the timing metric within services exactly for the reason that we've needed to have percentiles, so the decision we make here should probably be adopted elsewhere.

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/377269/ had fallen through the cracks. It's now merged, right before a SWAT window, in order to identify issues as fast as possible

@Tarrow , @WMDE-leszek I 've noticed 3 things while working on the above