In T224041#5201573, @thcipriani wrote:

It seems that the cassandra subchart already exists for cask (via https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/509102/ );

I am lowering to High, just in the interest of not abusing Unbreak Now!, since this task has been in this state since Apr 23. That being said, this indeed needs to be resolved ASAP.

needs an extra stanza

Every deployment that uses statsd-exporter (namely zotero & blubberoid don't) in kubernetes has been upgraded. Resolving this. Many thanks!

In T220235#5183451, @Krenair wrote:

krenair@deployment-docker-cxserver01:~$ sudo /usr/bin/docker run -p 8080:8080 -v /etc/mediawiki-services-cxserver/:/etc/mediawiki-services-cxserver --name alex-test docker-registry.wikimedia.org/wikimedia/mediawiki-services-cxserver:2019-05-08-064536-production -c /etc/mediawiki-services-cxserver/config.yaml
 Error during DHT setup undefined
 krenair@deployment-docker-cxserver01:~$

Anyone know what that means?

In T223345#5183583, @Joe wrote:

It's working in production because we connect to external URIs via a proxy, hence we don't need ca-certificates.

In T220402#5180031, @Pablo-WMDE wrote:

Hi @akosiaris,

thanks for taking the time to explain the way the Host header is intended to be used.
If I understand correctly the goal is to ensure that requests originating from our service bear a header Host: (www.)wikidata.org and reach which ever IP(s) appservers.discovery.wmnet resolves to on the system running it. This sounds like a name resolution challenge and a case for HostAliases or, more traditionally, a CNAME record.

In T220402#5177862, @Pablo-WMDE wrote:

Hi @akosiaris - thanks for getting back to us.

sending a Host: HTTP for the identification of the exact project. Would it be possible to add that functionality?

It certainly is possible and depending on operational needs we certainly can make this happen. We quickly discussed this in the team and would like to first truly understand the goal to make sure we don't mix up the different layers of our proverbial sausage pizza without a valid reason. The service would be run in a container inside a k8s pod, controlling its DNS - why not use this option to make sure requests reach the intended endpoint? The correct host would then come "for free" per the host part configured in WIKIBASE_REPO.

With respect to the end point checks it would be great to hear what we are trying to achieve with them. Our service depends on the availability of another service. If the examples are to act as smoke tests then their reliability depends on the upstream service; a dependency which would need to be configured (are we going to point it against prod for this?) & modeled (how to express service inter-dependency in the config?) in order to be able to make sense of the information down the line (i.e. "no need to be alarmed that this service reported 500 while the mw api was down").

In T222899#5179002, @Dzahn wrote:

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=lvs2003&service=PyBal+IPVS+diff+check

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=lvs2003&service=PyBal+connections+to+etcd

^ some new Icinga alerts apparently caused by this. Probably just needs pybal restart? But not sure enough.

In T220235#5179194, @Joe wrote:

In T220235#5176630, @Ottomata wrote:

Could we use image version: latest in beta hiera? And somehow pull down the new latest and restart the image whenever a new version is created and uploaded to the registry?

Sure, you just have to add a more complex script to exec to service::docker I guess, so that you can properly check that 'latest' or any other similar metatag are correctly respected.

Be my guest, I'll happily review the change!

Bacula & puppet databases are not going to exhibit any problems anyway. Puppet is literally used only by servermon and this is to be uninstalled pretty soon and backups don't happen during that timewindow.
etherpad, given the software, is a best-effort service, so no guarantees there. it will probably crash anyway, be restarted by systemd (as it anyway does every couple of days), users will be reconnected.

In T222962#5172806, @Ottomata wrote:

Hm, question. Currently mediawiki-config ProductionServices.php has:

'eventgate-analytics' => 'http://eventgate-analytics.discovery.wmnet:31192',

If we change LVS port to 33192, this will just change the LVS monitoring/pooling/depooling, right?

In T222795#5176181, @Pchelolo wrote:

Thank you for an impressive level of details :) There's a bunch of other places where we abuse the timing metric within services exactly for the reason that we've needed to have percentiles, so the decision we make here should probably be adopted elsewhere.

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/377269/ had fallen through the cracks. It's now merged, right before a SWAT window, in order to identify issues as fast as possible

@Tarrow , @WMDE-leszek I 've noticed 3 things while working on the above

@WMDE-leszek. Yes I did.

I guess this can be resolved? Nice work on the validation btw! I just witnessed it. Thanks!

@Clarakosi @Eevans. I 've updated the chart to also conditionally install a minimal cassandra for use in minikube. In my tests I was able to use and even run some benchmarks on it. On our side we are ready to proceed with deployment to staging and then production as soon as https://gerrit.wikimedia.org/r/#/c/mediawiki/services/kask/+/507397/ is merged.

Since the docker container will be the same as the one running in production, I don't think the environmental differences will be more than they already are in beta. The config for the service will be manually specified in hiera (either in Horizon or in operations/puppet), and here you can get as close to or as far from prod as you like.

I think the main problem will come from the fact that in production the service configs live in Helm charts now, rather than in Puppet. This means we can't easily reuse config defaults between beta and production. If we eventually end up somehow rendering Helm release values files from Puppet (@akosiaris mentioned this might happen), then it might be easier to build some puppet abstraction on top of role::beta::docker_services that makes it a little easier to share configs with beta.

In T218346#5161770, @Ottomata wrote:

@akosiaris I think https://wikitech.wikimedia.org/wiki/Kubernetes/Helm#Install_a_new_release_for_a_specific_service needs an update to include https://gerrit.wikimedia.org/r/c/operations/puppet/+/508371 as a step.

In T221529#5143984, @jbond wrote:

In T218346#5137946, @Ottomata wrote:

kubernetes_namespace might be right, but will app or service be? If we leave it as is, I believe those will be set to .Chart.Name, which will now be just 'eventgate'.

In T218346#5137518, @Ottomata wrote:

I 'd advise against having the namespace referenced in charts.

I'm mostly just trying fix the fact that the service and the metrics are named after .Chart.Name currently. We don't currently use the release (e.g. staging, production) in these names, which is good since they can vary even more than that, as you say. How should I set

# Statsd metrics reporter
metrics:
  name: {{ .Chart.Name }}

If I don't use the namespace?

In T218346#5136013, @Ottomata wrote:

@fsero @akosiaris - moving discussion about eventgate-main patch here, will be easier to communicate.

I'm going to try to make a single chart that can be used for all deployments of EventGate, and I think I need a little help figuring out how to organize that. Right now in prod we have:

• chart: eventgate-analytics
• releases:
  • production (in CLUSTER=eqiad and CLUSTER=codfw)
  • staging (in CLUSTER=staging).

I think what we want is:

• chart: eventgate
• releases:
  • analytics-production (in CLUSTER=eqiad and CLUSTER=codfw)
  • analytics-staging (in CLUSTER=staging)
  • main-production (in CLUSTER=eqiad and CLUSTER=codfw)
  • main-staging (in CLUSTER=staging)
    
    And also eventually as part of T217142 something like:
• releases:
  • logging-production (in CLUSTER=eqiad and CLUSTER=codfw)
  • logging-staging (in CLUSTER=staging)

CenturyLink sent a summary and a notification they 'll close the issues as resolved on their end.

New update says:

@Eevans @Clarakosi chart has been merged and is published. The only thing missing before we can move on to the deployment is the swagger/openapi spec so that service-checker[1] can run and monitor this service.

Received information for Level3

@Tarrow, @WMDE-leszek. I 've been working on the termbox helm chart and while the service seems to be up and running, I see no /_info endpoint nor a swagger/openapi[1] spec published under /?spec. Both are crucial for deploying, as the former is used as a kubernetes readiness probe (aka if an instance of the app can't serve that for any reason it will temporarily not see new traffic) and the latter is used by our monitoring, so we can't proceed without those. Could you please have a look at it and add them? Thanks!

Just for posterity's sake, at ~1500 artificially simulated users the service started to crumble and started returning

I did some benchmarking and here's some first (rather impressive numbers) for kask

kubernetes1005, kubernetes1006, kubernetes2005, kubernetes2006 added with specific taints in order to have only kask being scheduled on them. Resolving

In T220822#5128635, @ayounsi wrote:

One typo:
codfw has 10.64.32.18 and 2620:0:861:103:10:64:32:18

In T218750#5126229, @Urbanecm wrote:

The same could be said of the ssh key. Temporary access to a browser session would enable someone to add a key to perform actions on behalf of a user. The gerrit command line tools have mostly the same capabilities as the http token.

That's really good point.

Change merged, thanks!

This is almost done. That only thing missing seems to be the peering with the juniper routers.

Before we move forward and enable this, let's make sure we have understood the security repercussions and have mitigated them (and if we find it impossible to do so, avoid it).

Host is up and running but as @Cmjohnson points out in T196478#4976375

In T200832#5120806, @Krenair wrote:

In T200832#5061718, @akosiaris wrote:

In T200832#5051312, @Krenair wrote:

deployment-mathoid still exists and has been failing puppet runs since December 3rd when profile::mathoid got removed.

I 'd delete the VM, profile::mathoid isn't coming back. If anything the VMs with the role::beta::docker_services role applied can probably handle the service now.

Unfortunately MediaWiki is still expecting the VM to exist:
wmf-config/LabsServices.php: 'mathoid' => 'http://deployment-mathoid.eqiad.wmflabs:10042',

The two hosts with that role are deployment-eventgate-analytics-1 and deployment-sessionstore01 so neither of them are really appropriate for this, plus they both run jessie.

In T219556#5120409, @Ottomata wrote:

ganeti1001, 1002 and 2001 have been installed. I dunno what's up with ganeti2002. gnt-instance console schema2002.codfw.wmnet never shows anything.

Just noting that at 10:41 UTC the circuit was still down per

In T220661#5116643, @Ottomata wrote:

Been doing a lot to get more data, including enabling node profiling and connecting to node inspector. I didn't learn much, except confirmed that I don't think Kafka CPU culprit. I did see good number of cycles being spent generating the x-request-id header using cassandra-uuid.TimeUuid. This won't happen if we can ever get the x-request-id header to be set by varnish eventually (T201409).

But I didn't see anything particularly abnormal, just a lot of time being spent by express parsing the JSON request bodies, which I guess makes sense.

Just to try, I increased k8s resource requests/limits to:

requests:
  cpu: 1000m
  memory: 100Mi
limits:
  cpu: 2000m
  memory: 200Mi

In T165795#5108897, @bd808 wrote:

I see and understand the concerns from @akosiaris in T165795#5092381. I personally believe this un-indexed lookup is an acceptable overhead in the short term because it will only be used in the Wikitech account login flow. This is not a high rate of change event in our current system.

In T220661#5107329, @Pchelolo wrote:

@akosiaris no, spanning up a new worker takes no time, the problem here is actually hilling old worker.

The heartbeat limit currently is 7.5 seconds, after which we do not just SIGKILL the worker, we optimistically try to properly shut it down and if that does not happen, we SIGKILL it after 1 minute. For 'normal' operation with many workers, this is a right approach - master stops dispatching requests to the worker immediately after ordering it to shut down gracefully, with less incoming pressure the worker might recover a bit and get a chance to finish up requests and do it's pre-shutdown housekeeping.