In T217650#5035879, @akosiaris wrote:

In T217650#5033562, @Eevans wrote:

Kask has now been setup for session storage in deployment-prep using docker_services (deployment-sessionstore01.deployment-prep.eqiad.wmflabs); I have a few questions about how this all will work in production (and presumably deployment-prep, at some point in the future).

• The name used in docker_services is a normalization of the Git repo name (i.e. mediawiki-services-kask here), will this also be the case when deployed to k8s in production?

More or less. It will be the name of the image that is used in the helm charts. But not in anything else.

Kask has now been setup for session storage in deployment-prep using docker_services (deployment-sessionstore01.deployment-prep.eqiad.wmflabs); I have a few questions about how this all will work in production (and presumably deployment-prep, at some point in the future).

In T215533#5024386, @EvanProdromou wrote:

Should this task go to @BPirkle now?

In T132632#5023075, @mobrovac wrote:

@Eevans @fgiunchedi should we go ahead with this?

From an IRC conversation:

For my part, I'm Good with either approach; I really like the idea of not having an exceptional deployment for session storage. However, one of the objectives here was isolation, and while I think I'm satisfied we're still achieving that, the latter of the two proposed strategies would seem to trade-away the total isolation we'd have if everything were on dedicated iron.

There appears to be some possibility that instead of running the Kask service stand-alone on the session storage cluster, we may run it inside a security-fenced k8s environment.

To summarize a (IRC) discussion w/ @Joe today:

Kask now implements a logger that is a simple wrapper around syslog, w/ CEE compatible messages. A question has come up though about which log levels/severities to implement, and when/how they should be used.

In T209110#4880850, @fgiunchedi wrote:

My two cents WRT logging would be that messages should end up on syslog, either by way of syslog() (i.e. writing to /dev/log) or emitted to stdout/stderr, which then would be picked up by journald (or k8s/docker) and ultimately received by rsyslog and ingested into the logging pipeline.

For posterity sake, here are the Prometheus metrics provided by default:

In T212418#4870100, @Cmjohnson wrote:

@Eevans I am going to have to power it back on and let it go for a few days to see if the error returns, will that present an issue for you?

In T212418#4869959, @Cmjohnson wrote:

I need to move DIMM around and do standard troubleshooting. Is this server able to be powered off and down in icinga?

We're currently in the process of force-removing these instances. We'll need to coordinate when the host comes back up, as we'll have to re-bootstrap all 3 instances.

Is there any status update, or ETA on this?

Not long back, we were alarmed to see a very high rate of range-slice requests (a type of query our app does not perform). I wasn't able to find a ticket, but this turned out to be the driver using a SELECT * FROM ... on a small system table as a sort of heartbeat. The queries themselves were harmless, but the number of them was shocking until we realized that every Node worker had it's own connection pool, and that ${workers} * ${hosts} was a very large number.

In T212129#4834329, @mark wrote:

I am getting the impression here that some things are being rushed and finalized without time for a proper discussion between people/teams about the different possible solutions and their impact, after this new discovery. Is that because goals are due to be posted now?

In T212129#4832455, @Joe wrote:

[ ... ]
This needs a thorough discussion ASAP.

In T206010#4820737, @kchapman wrote:

TechCom has approved this, noting performance discussions are outside of the scope of this RFC.