Change merged, thanks!

This is almost done. That only thing missing seems to be the peering with the juniper routers.

Before we move forward and enable this, let's make sure we have understood the security repercussions and have mitigated them (and if we find it impossible to do so, avoid it).

Host is up and running but as @Cmjohnson points out in T196478#4976375

In T200832#5120806, @Krenair wrote:

In T200832#5061718, @akosiaris wrote:

In T200832#5051312, @Krenair wrote:

deployment-mathoid still exists and has been failing puppet runs since December 3rd when profile::mathoid got removed.

I 'd delete the VM, profile::mathoid isn't coming back. If anything the VMs with the role::beta::docker_services role applied can probably handle the service now.

Unfortunately MediaWiki is still expecting the VM to exist:
wmf-config/LabsServices.php: 'mathoid' => 'http://deployment-mathoid.eqiad.wmflabs:10042',

The two hosts with that role are deployment-eventgate-analytics-1 and deployment-sessionstore01 so neither of them are really appropriate for this, plus they both run jessie.

In T219556#5120409, @Ottomata wrote:

ganeti1001, 1002 and 2001 have been installed. I dunno what's up with ganeti2002. gnt-instance console schema2002.codfw.wmnet never shows anything.

Just noting that at 10:41 UTC the circuit was still down per

In T220661#5116643, @Ottomata wrote:

Been doing a lot to get more data, including enabling node profiling and connecting to node inspector. I didn't learn much, except confirmed that I don't think Kafka CPU culprit. I did see good number of cycles being spent generating the x-request-id header using cassandra-uuid.TimeUuid. This won't happen if we can ever get the x-request-id header to be set by varnish eventually (T201409).

But I didn't see anything particularly abnormal, just a lot of time being spent by express parsing the JSON request bodies, which I guess makes sense.

Just to try, I increased k8s resource requests/limits to:

requests:
  cpu: 1000m
  memory: 100Mi
limits:
  cpu: 2000m
  memory: 200Mi

In T165795#5108897, @bd808 wrote:

I see and understand the concerns from @akosiaris in T165795#5092381. I personally believe this un-indexed lookup is an acceptable overhead in the short term because it will only be used in the Wikitech account login flow. This is not a high rate of change event in our current system.

In T220661#5107329, @Pchelolo wrote:

@akosiaris no, spanning up a new worker takes no time, the problem here is actually hilling old worker.

The heartbeat limit currently is 7.5 seconds, after which we do not just SIGKILL the worker, we optimistically try to properly shut it down and if that does not happen, we SIGKILL it after 1 minute. For 'normal' operation with many workers, this is a right approach - master stops dispatching requests to the worker immediately after ordering it to shut down gracefully, with less incoming pressure the worker might recover a bit and get a chance to finish up requests and do it's pre-shutdown housekeeping.

Change merged and shepherded into production. I am lowering priority but not resolving as we probably want to evaluate this more

In T220808#5107076, @akosiaris wrote:

I am thinking about excluding exec_sync operations for a while from the checks to restore faith in the alerts.

I am thinking about excluding exec_sync operations for a while from the checks to restore faith in the alerts.

A breakdown of the alerts per host follows starting from 2019-03-26 to 2019-04-12 follows

Today (2019-04-12), I 've raised the possibility that T220661 is related to the reason these alerts are flapping so much.

FYI, note the pod restarts below. Seems like the worker isn't ready in the timespan of the kubelet liveness probes (3x 10s => 30s) and eventually the pod is killed. Does it really take that long to initialize a new worker? We can definitely tune those numbers but 30s for initalizing a worker sounds like a lot.

dummy private repo updated, so is the actual private repo. Resolving, thanks!

In T188281#5099397, @Harej wrote:

@akosiaris Should we still pursue this?

In T219556#5098760, @Ottomata wrote:

@akosiaris, I'd love to do this sooner rather than later. It'd make some configuration/deployment stuff in the Hive/Hadoop world much easier.

Am I allowed to follow https://wikitech.wikimedia.org/wiki/Ganeti#Create_a_VM and do this myself?

FWIW the ganeti cluster uses exactly the approach outlined by @BBlack for this (among other, even more important) reasons:

Firejail seems a nice extra for robustness, but @akosiaris seemed to suggest in T217724#5008302 that that might introduce some overhead. AIUI the sandbox is discarded anyway after every PDF render, to reduce the likelihood of problems caused by state leaking over to the next render, but I might be misunderstanding.

@WMDE-leszek Hi, sorry for not answering any sooner, last few weeks have been crazy indeed.

Stalling until we have some sane solution.

In T165795#5075683, @bd808 wrote:

In T165795#5075101, @chasemp wrote:

Can we tell ldap to enforce non-case sensitivity?

Yes, with the :caseExactMatch: matching rule. See T165795#4110716.

In T219923#5086476, @Pchelolo wrote:

Apparently graphoid is still using service::node::config and not the config template in the deployment repo.

Nice! Thanks so much!

In T198939#5085736, @Volans wrote:

And when we do, can we also drop the package_updates custom fact?

I can say I haven't in a pretty long time. If @faidon also doesn't I think we can shut it down.

In T220003#5084386, @MoritzMuehlenhoff wrote:

In T220003#5084382, @akosiaris wrote:

Currently the /etc/apt/sources.list for the pbuilder base images are missing entries for the security suites. Theses files should be updated and managed by puppet.

Why though? When creating that puppet module I avoided that on purpose, relying on the fact that security updates would anyway be making it to our hosts and package names for that remain constant. That assumption might not be true anymore, or I may very well have erred back then, but I 'd like to know which of the 2 (or something else entirely) it is.

This is needed for jessie, it's in LTS stage, so packages only get updated/added to jessie-security (the original jessie is frozen). Jessie LTS added a clang-4.0 package (used to build the Rust-based Firefox packages), which we need for a different package, but with the current setup pbuilder doesn't see it as it's only in jessie-security.

Currently the /etc/apt/sources.list for the pbuilder base images are missing entries for the security suites. Theses files should be updated and managed by puppet.

In T219708#5084339, @Ladsgroup wrote:

In T219708#5082329, @akosiaris wrote:

Configs need to be overhauled into one big yaml file that's going to be checked out into helm charts

That's not necessary. We can ship more than 1 config files via helm charts. We can ship config directories as well.

Nice, thank you for letting me know!

Assets ores use, they also need to pulled down when starting the service.

I am not so sure about this one. Very often assets are tightly coupled with the deployed code version and usually relatively small in size and fine to ship with the application.

We have an asset for ores that is 1.3 GB (word2vec dictionary)

Culprit identified.

Just some notes, I agree on premise with most of the above.

@bd808, I 've submitted https://gerrit.wikimedia.org/r/500681 for review and updated https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/497866 to match.

One minor correction

Now that I 've automated the tests it was easy to combine the 2 approaches and fix the drawbacks. User5 in P8321 is subject to both approaches and the only drawback I could find out is a race of 1s between an admin resets the password (by mistake, or following some old and outdated process) and the user managing to authenticate to a service. This is an attack vector that requires 2 humans to perform an action and thus not something that can be somehow automated. I think we are safe from this one.

In the interest of fully figuring this out I 've updated my testing openldap vagrant env [1] with a Makefile[2] that automates and runs some tests[2]. I 've pasted results for it in P8321. The test suite can (and maybe should) be extended to make sure we got all our bases covered.

@Gilles, I am a bit unclear as to what remains to be done for this. Could you shed some light?

In T219332#5060344, @Ottomata wrote:

And most specifically:
https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate/Administration
^ @akosiaris plz review :)

Copying from https://gerrit.wikimedia.org/r/497684 (and adding some extra stuff)

In T218680#5063374, @Ottomata wrote:

Thanks for tips, Petr clued me into the fact that JSONSchema itself has an examples field, so we can add example events in schemas. I'm going to try to do something with this. I'm writing a nodejs script that will exist in the EventGate code that will take a schema URI, get it, get the examples from it, and POST it to the service. Could we exec this script for the readinessProbe? This would be like:

readinessProbe:
  exec:
    command: [/srv/service/scripts/post-events examples /test/event/0.0.3 http://localhost:8192/v1/events]
  initialDelaySeconds: 2

In T218680#5054738, @Ottomata wrote:

it might make more sense to create a specialized GET /healthz endpoint that does just produces (and deletes if required/prudent?) a hardcoded test event in kafka.

I don't like this idea so much, mainly because it requires that we include WMF specific schemas in the API routing code. Thus far I've been able to keep any WMF specific stuff in the wikimedia-eventgate specific implementation.

In T200832#5051312, @Krenair wrote:

deployment-mathoid still exists and has been failing puppet runs since December 3rd when profile::mathoid got removed.

Resolving, feel free to reopen

In T218268#5049090, @Ottomata wrote:

apt-get update couldn't connect to the apt source IPv6 addresses.

In T212189#5044451, @Addshore wrote:

In T212189#5020187, @akosiaris wrote:

Thanks for the understanding. We are drafting next quarter goals this week, I 'll make sure to add this.

Just poking to double check that this was added (I would hate to see it missed).

Incident report updated. The only actionable followup task is T122676

In T218305#5040830, @Ottomata wrote:

@akosiaris, @fgiunchedi when you get a chance I'd appreciate a lookover of this dashboard:

https://grafana.wikimedia.org/d/ePFPOkqiz/eventgate-analytics-otto0?refresh=1m&orgId=1&from=1553097349177&to=1553100949177&var-dc=eqiad%20prometheus%2Fk8s-staging&var-service=eventgate-analytics&var-kafka_producer_type=All&var-kafka_broker=All&var-kafka_topic=All

I went ahead and put the Kafka graphs I added into the existing rows (sorry Filippo if you really don't like I can move back to a Kafka specific one!).

Well I have my reservations for sure. As I said we are talking about a service-checker run every 10s (tunable, but it's a sensible default). While tunable, the command should not take long. The timeout is tunable but the default is 1s. I think it's a sensible value for a web service and I don't think service-checker doing all the endpoints + a POST that will end up in kafka is going to be THAT fast. It also requires having service-checker on every node out there. While fine in production, the development environments is definitely not gonna have it. We could have the probe in values.yaml (we do for all other services, only eventgate is an exception) and override it in production and that's probably fine, but it should be documented.

The readiness probe can't really be POST. The ref is here https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/#probe-v1-core, it only allows httpGet, tcpSocket and exec. Exec could be used for this and call the service-checker but it adds a dependency on having service-checker on the nodes (it isn't there currently) and instrumentation (getting the pod IP essentially, I am not sure it's exposed there, will need to check) to make it happen. It might also be a tad heavy to run service-checker every 10 secs for every pod.

In T217650#5036256, @Eevans wrote:

I don't like it either; This is what prompted these questions! The image entrypoint was /etc/kask/config.yaml, but I didn't see a way to alter where the configuration would be written via docker_services, and an override_cmd won't work so long as the image entrypoint is passing all of the arguments.

In T217650#5033562, @Eevans wrote:

Kask has now been setup for session storage in deployment-prep using docker_services (deployment-sessionstore01.deployment-prep.eqiad.wmflabs); I have a few questions about how this all will work in production (and presumably deployment-prep, at some point in the future).

• The name used in docker_services is a normalization of the Git repo name (i.e. mediawiki-services-kask here), will this also be the case when deployed to k8s in production?

I think the intention is to (somewhat ?) limit the impact the vandal is trying to achieve (at least by removing the capability to link to those comments). While it's, as you point out due to the email notifications, impossible to fully mitigate that, and potentially causing other issues, I still consider it a prudent course of action. Other than that, please wait for a formal announcement (one that a link to will be posted to this task as well).