In T222099#5504837, @Reedy wrote:

In T222099#5504605, @Reedy wrote:

As far as we can see, it's only seemingly broken on testwiki (something put into session data doesn't come out the same). And may have been broken for a while, as I imagine enabling 2FA on testwiki isn't how most people would do it
The class in question going in is https://github.com/wikimedia/mediawiki-extensions-OATHAuth/blob/master/src/Key/TOTPKey.php and apparently comes out as an empty array...

TTBMK, this is the only remaining blocker to fully moving sessionstore to production.

restbase2012 is decommissioned and can be reimaged at any time.

After startup, Kask doesn't log much outside of exceptional errors; To invoke log output try issuing a HEAD request:

And here is what latency looks like after r/537539 has been deployed (see also: direct link to Grafana dashboard).

restbase2011 is fully decommissioned and ready to be reimaged.

In T229697#5471288, @akosiaris wrote:
[ ... ]

The p99 looks suspiciously close to the cross DC latency, is there any way it is related?

restbase2010 is ready to be reimaged.

restbase2009 is fully decommissioned and ready to be reimaged.

restbase1018 is decommissioned and ready to be reimaged.

In T229697#5471288, @akosiaris wrote:

In T229697#5470248, @Eevans wrote:

P9046 wrk.sh

1	eevans@deploy1001:~$ KASK_URL=https://sessionstore.svc.eqiad.wmnet:8081 ./wrk.sh
2	+ WRK_ARGS=--latency -d10m
3	+ LUA_CPATH=/usr/lib/x86_64-linux-gnu/lua/5.1/?.so
4	+ KASK_URL=https://sessionstore.svc.eqiad.wmnet:8081
5	+ dirname ./wrk.sh
6	+ cd .
7	+ [ ! -f multi-request-json.lua ]
8	+ [ ! -f requests.json ]
9	+ LUA_CPATH=/usr/lib/x86_64-linux-gnu/lua/5.1/?.so wrk --latency -d10m -s multi-request-json.lua https://sessionstore.svc.eqiad.wmnet:8081
10	multiplerequests: Found 101000 requests
11	multiplerequests: Found 101000 requests
12	multiplerequests: Found 101000 requests
13	Running 10m test @ https://sessionstore.svc.eqiad.wmnet:8081
14	2 threads and 10 connections
15	Thread Stats Avg Stdev Max +/- Stdev
16	Latency 19.96ms 18.05ms 270.12ms 64.20%
17	Req/Sec 251.88 37.75 360.00 59.98%
18	Latency Distribution
19	50% 22.14ms
20	75% 37.76ms
21	90% 37.95ms
22	99% 38.88ms
23	301034 requests in 10.00m, 50.53MB read
24	Non-2xx or 3xx responses: 66355
25	Requests/sec: 501.67
26	Transfer/sec: 86.23KB
27	------------------------------
28	10%,1703
29	20%,1815
30	30%,1920
31	40%,2057
32	50%,22144
33	60%,37591
34	70%,37706
35	80%,37811
36	90%,37946
37	99%,38882
38	duration (micros),600062009
39	requests,301034
40	bytes,52984874
41	connect errors,0
42	read errors,0
43	write errors,0
44	status errors,66355
45	timeout errors,0
46	------------------------------
47	eevans@deploy1001:~$ KASK_URL=https://sessionstore.svc.codfw.wmnet:8081 ./wrk.sh
48	+ WRK_ARGS=--latency -d10m
49	+ LUA_CPATH=/usr/lib/x86_64-linux-gnu/lua/5.1/?.so
50	+ KASK_URL=https://sessionstore.svc.codfw.wmnet:8081
51	+ dirname ./wrk.sh
52	+ cd .
53	+ [ ! -f multi-request-json.lua ]
54	+ [ ! -f requests.json ]
55	+ LUA_CPATH=/usr/lib/x86_64-linux-gnu/lua/5.1/?.so wrk --latency -d10m -s multi-request-json.lua https://sessionstore.svc.codfw.wmnet:8081
56	multiplerequests: Found 101000 requests
57	multiplerequests: Found 101000 requests
58	multiplerequests: Found 101000 requests
59	Running 10m test @ https://sessionstore.svc.codfw.wmnet:8081
60	2 threads and 10 connections
61	Thread Stats Avg Stdev Max +/- Stdev
62	Latency 56.33ms 18.03ms 301.04ms 55.91%
63	Req/Sec 88.94 12.77 131.00 79.11%
64	Latency Distribution
65	50% 73.43ms
66	75% 74.26ms
67	90% 74.50ms
68	99% 74.99ms
69	106474 requests in 10.00m, 13.35MB read
70	Requests/sec: 177.44
71	Transfer/sec: 22.78KB
72	------------------------------
73	10%,38066
74	20%,38210
75	30%,38338
76	40%,38497
77	50%,73429
78	60%,74046
79	70%,74193
80	80%,74327
81	90%,74495
82	99%,74989
83	duration (micros),600072527
84	requests,106474
85	bytes,14000072
86	connect errors,0
87	read errors,0
88	write errors,0
89	status errors,0
90	timeout errors,0
91	------------------------------
92	eevans@deploy1001:~$

The p99 looks suspiciously close to the cross DC latency, is there any way it is related?

restbase-dev1005 has been decommissioned and is ready to be reimaged.

Part of what has made this so odd is that it only ever occurred to this one instance, so good/bad news, this has now been observed on 2009-b as well.

I've started the decommission of -dev1005-b quite late in my evening; It should be complete by EU morning. If there is no output from running ssh restbase-dev1004.eqiad.wmnet -- c-any-nt status -r | grep 1005, then the node can be taken down for reimage.

In T231027#5470702, @Joe wrote:

In T231027#5452798, @Eevans wrote:

If the systemd syslog output is to believed, the service exited with status code 3 (...status=3/NOTIMPLEMENTED). AFAICT, Java would exit with a status code > 127 if the JVM shutdown as the result of a signal, otherwise this would have to be the result of a java.lang.System.exit(3) call. However, I can find no explanation of this in Cassandra's code.

I think we can trust systemd on that. I checked and it correctly identifies java as the mainpid, which was the only doubt I had.
Very dumb question: which version of the cassandra code did you check? Did you include debian patches we apply, if any?

In T224554#5470427, @Dzahn wrote:

@Eevans I recreated the certs for restbase-dev1004 through restbase-dev1006 and committed in the private repo. Please try again now.

In T229697#5468075, @akosiaris wrote:

https://gerrit.wikimedia.org/r/534613 merged and deployed fine.

I wasn't either. I ended up doing a helmfile destroy && helmfile apply to fix it, but we will need to have a better look at why this happened, since even manually fixing the underlying cause did not make the release move from FAILED to DEPLOYED

helmfile sync fixed it though after all.

In T224554#5467470, @MoritzMuehlenhoff wrote:

restbase-dev1004 has been reinstalled as Stretch. @Eevans, you can bootstrap 1004 in Cassandra and decom 1005 in Cassandra, then I'll proceed with reimaging 1005.

In T224554#5467470, @MoritzMuehlenhoff wrote:

restbase-dev1004 has been reinstalled as Stretch. @Eevans, you can bootstrap 1004 in Cassandra and decom 1005 in Cassandra, then I'll proceed with reimaging 1005.

restbase-dev1004 has been decommissioned and can come down for a re-image at any time.

In T229697#5464398, @akosiaris wrote:

[ ... ]

@akosiaris are we OK to merge this? Happy to do so and deploy, just making sure it's there.

Yes, feel free. I can deploy it as well/be around when you do.

In T229697#5459187, @gerritbot wrote:

Change 533922 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] sessionstore: Bump limits and requests
https://gerrit.wikimedia.org/r/533922

In T229697#5459204, @akosiaris wrote:

As fan FYI, the 3 different stanzas of data in https://grafana.wikimedia.org/d/000001590/sessionstore?orgId=1&var-dc=eqiad%20prometheus%2Fk8s-staging&var-service=sessionstore&from=1567432126930&to=1567435225421 are 3 distinct benchmarkings, limits are respectively 150m, 1500m, 2500m. Some observations off the top of my head and in no particular order:

In T231027#5451242, @Eevans wrote:

This instance has gone down again, we should dig deeper.
[ ... ]

The timeline of events goes like this:

This instance has gone down again, we should dig deeper.

This imbalance seems to resolved itself.

I do not know how it came to pass that machines are getting setup without reserved space, but given how long this issue has been open (and since I'm still unsure how to best go about Puppetizing this), I think we should accept this as a gift and close the issue.

Interestingly, reserved space on the main data volumes in the production cluster already have zero reserved blocks.

In T224995#5250739, @Eevans wrote:

In T224995#5234039, @Eevans wrote:

In the short-term, production configuration lives in deploy1001:/srv/scap-helm/sessionstore/sessionstore-{codfw,eqiad,staging}-values.yaml. I've updated each of these files with the following comment.

# WARNING: The value of $wgObjectCacheSessionExpiry in MediaWiki must
# correspond to the TTL defined here; If you alter default_ttl, update
# MediaWiki accordingly or problems with session renewal/expiry may occur.
default_ttl: 86400

Longer-term, these files will be version-controlled as part of operations/deployment-charts repository (and will be initialized from the above files).

FYI; Just leaving this ticket open to followup later and ensure that these comments make into the Git repository.

Host has been rebooted and there are no ACPI errors present.

In T198787#5439421, @WDoranWMF wrote:

@Eevans Who could look at this, would this be a good task for @Clarakosi? Not sure if you've done deb packaging yet Clara

In T92471#5439574, @Eevans wrote:

[ ... ]
This is no longer the case (again?); RMI is (again?) bound only to the IPv4 loopback.
So to summarize the risk of sticking with the status quo:
Anyone with local access to the node can a) issue any command nodetool is capable of, in addition to b) executing arbitrary code as the Cassandra user (only network access is needed)
If we enabled password authentication of RMI, we would restrict this level of access to anyone capable of reading the credentials file (root, presumably).

OK, time for our yearly update of this ticket!

We're up to 3.5.0 of the driver at this time; Let's close this unless we can establish that there is more to do.

Let's just do this already.

I used some 10% time to have a (cursory) look at the Datastax PHP driver. Some observations:

In T224553#5431986, @Eevans wrote:

In T224553#5431770, @MoritzMuehlenhoff wrote:

T208087, T223976 and T222960 are fixed. Could we get restbase2009-restbase2012, restbase1018 (and the two remaining -dev servers) migrated to Stretch in the next 1-2 months?
cassandra/restbase is the only remaining use case for our custom OpenJDK 8 backports in jessie-wikimedia and it would be fantastic not to spend more time on this when the October Java security release gets released.

Assuming we're adhering to the process of decommissioning, re-imaging, and bootstrapping, then I suspect the only blocker to doing so would be SRE resources. :)

I've only looked at this briefly, but some observations:

• The JVM seems to have spontaneously and uncerimoniously self-destructed:
  • No logged (fatal) exceptions
  • No crash log or heap dump
• None of the usual signs of distress preceding the event
  • StatusLogger frequency
  • Major GCs
  • Load avg, cpu, IO
• High latency immediately preceding the event shared by 2009-b