All instances bootstrapped, and cleanups in corresponding rack are complete; Closing

In T220246#5333382, @Volans wrote:

First of all I've a some questions due to my lack of knowledge of Cassandra specifics:

• Is there any configuration that could require a code change in the application? Can those be changed at will without any coordination with the application?

In T227776#5326474, @daniel wrote:

In T227776#5326455, @Anomie wrote:

Kask,[1] accessed via RESTBagOStuff?

Probably not Kask, but perhaps something similar, or a derivative or successor of Kask.

Do you have a link to the documentation? What is the likelihood that any documentation written now may change after going through code-review, and should we leave this open until then?

This can't be done, it's dependent on moving the project to Gerrit.

Current snafu: None of data volumes are mounted (entries are missing from fstab). @fgiunchedi I seem to (vaguely) remember this as a thing, was the solution to add them manually?

I guess it was left open because it has sub-tickets still open (of which T222960: Fix restbase1017's physical rack is still unresolved).

The task description has been updated with information on a working prototype created by @holger.knust. We'd be very curious to hear any feedback about the general approach (including the workflow it's meant to empower), particular from folks in SRE (@Joe are you still monitoring this ticket? :))

Any word on when we'll be imaging this machine?

In T222960#5318560, @Cmjohnson wrote:

@Eevans We did a test run for an install and the server was able to reach the installer without an issue. I did see on IRC something about stretch. I will leave that up to you if you like and the server can be installed whenever you need it.

In T222960#5318812, @Dzahn wrote:

restbase1017 is shown as down in Icinga and has no downtime or comment . would be appreciated if you can schedule downtimes for planned maintenance. thanks
https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&style=hostdetail&hoststatustypes=4&hostprops=2097162

In T209110#5251135, @Eevans wrote:

[ ... ]
Also, there are some exceptional situations where the Go http module (and perhaps gocql) as well do their own unstructured logging to stdout (and/or stderr?). I don't believe there are any situations were these errors modes wouldn't be handled (and logged) appropriately, but we should probably make sure this output doesn't cause harm elsewhere.

After several failed experiments (editing /srv/scap-helm/sessionstore/sessionstore-staging-values.yaml and upgrading), I restored the original configuration (that was previously not working), and it now works.

All 3 Cassandra instances are decommissioned; We are ready to begin

In T198787#5310026, @WDoranWMF wrote:

@Eevans Can we close this?

In T178839#5309937, @WDoranWMF wrote:

@Eevans Do want to move this along or has it stalled?

In T209393#5309839, @WDoranWMF wrote:

@Eevans Is this closable?

From IRC, 2019-07-03T16:57:04-05:00:

This is now complete, and Kask v1.0.0 has been release.

In T222960#5289614, @Cmjohnson wrote:

@Eevans Do you still want to move this server? Let's coordinate a day/time

In T222099#5292237, @Eevans wrote:

In T222099#5290383, @BPirkle wrote:

[ ... ]
Note: during the transition, testwiki will still be using a TTL ($wgObjectCacheSessionExpiry) value of 3600. The Kask TTL setting must therefore be at least 3600 seconds. If the Kask setting is longer, the transition will still be okay. But we will do more writes to Kask than we would with a longer settings, because MediaWiki will think the sessions need to be rewritten.

I will work on getting the sessionstore cluster reconfigured for a TTL of 3600 (currently set to 86400).

In T221986#5289449, @EvanProdromou wrote:

In T221986#5284687, @Eevans wrote:

If we just consider the session ID, I can see these potential things that a villain could do by carefully crafting a session ID value:

• Kask to Cassandra (CQL, mostly?)
  • retrieve data that's not authorized

If an attacker could utilize the session ID of another, then they'd be able to to do anything the owner of the session would, there is no defense against this other than keeping other user's session a secret.

Again, I don't know CQL, but I was thinking about code that crafts SQL queries like

sprintf("SELECT * FROM session WHERE id = '%s'", id)

...so that an injected ID string like "IGNORED' OR 'constant' = 'constant" will generate "SELECT * FROM session WHERE id = 'IGNORED' OR 'constant' = 'constant'" which should get all rows in the table.

In T222099#5290383, @BPirkle wrote:

[ ... ]
Note: during the transition, testwiki will still be using a TTL ($wgObjectCacheSessionExpiry) value of 3600. The Kask TTL setting must therefore be at least 3600 seconds. If the Kask setting is longer, the transition will still be okay. But we will do more writes to Kask than we would with a longer settings, because MediaWiki will think the sessions need to be rewritten.

In T226666#5287824, @BPirkle wrote:

Currently, if decoding fails, RESTBagOStuff will simply log the error without any extended information. This would include only:

• a message describing the operation that was being attempted (ex. "Failed to store XXXX")
• the HTTP error code (ex. 500)
• the HTTP response string (ex. "Internal Server Error")

So we would have a record of the error,. But if the body contained error information beyond the HTTP code/string, that information would be lost. In the example from the task description, we would not lose anything.
Are there cases where the body will contain useful text/plain information beyond the HTTP response string?

Responding (in-line) to those items I'm familiar with...

• setting $wgObjectCacheSessionExpiry to the same value as is configured for kask (9 * 3600?)

In T219831#5251350, @sbassett wrote:

@Eevans - I'm not seeing anything for this particular review, though I might dig a little deeper into the code and attempt some dynamic-scanning this week, as mentioned in T219831#5173498. But none of this should block resolving the task or deployment IMO.

@sbassett is there anything remaining here before we close/resolve this?

I believe this issue is basically complete, but was been left open because we weren't certain whether or not we needed the @cee token to be prepended to log messages; @akosiaris is logging working as expected in k8s?

I believe the current status here to be: