At approximately 11:00 this morning (2017-11-29), a full compliment of tables with the legacy grouping and naming convention were created in the new Cassandra cluster. To the best of my knowledge, this is pretty wrong.
The timing would seem to put this during the bootstrap of 1007-c, and before the bootstrap of 1012-a (which is currently failing as a result of missing schema for one of these tables).
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Eevans | T181689 New RESTBase Cassandra cluster has legacy tables | |||
| Resolved | fgiunchedi | T181964 Clean metrics for restbase erroneus legacy tables from cassandra 3 cluster |
I think I know what happened here. At approximately the same time, SystemD decided to start RESTBase back up on restbase1012. Since puppet was disabled on that node for a long time, the list of seeds was likely wrong in its configuration, so it issued CREATE KEYSPACE queries to a Cassandra3 node (1007 would be my bet). This would also explain the errors seen in the Cassandra logs about cluster name clashes. Since RB is no longer present on that node, and the instance on restbase1014 (which was also started by SystemD at some point) has been masked, I think it's safe to just remove these keyspaces and move on :/
I just linked another side effect that might be related: namely the graphite machines for cassandra (graphite2002 / graphite1003) were low on disk space and will need a cleanup.
I managed to find the table creation logs which confirm the theory that they were created by rb1012 (albeit the logs don't tell us the destination of these queries, but it's safe to assume that was directed at the new Cassandra cluster). The strange part is that these tables hold data, e.g. I can see this on restbase1008:
$ c-cqlsh a --request-timeout=240 -e 'select count(*) from "local_group_wikipedia_T_title__revisions".data;' count ------- 80 (1 rows)
This confirms that these tables have not just been taken from the Cassandra 2 cluster, but it raises the question: how did any data manage to get written there? Conftool/Pybal say that these nodes are not pooled, but the data there is recent:
$ c-cqlsh a --request-timeout=240 -e 'select * from "local_group_wikipedia_T_title__revisions".data limit 10;' _domain | title | rev | tid | latest_rev | page_deleted | _del | comment | latest_tid | namespace | nextrev_tid | page_id | redirect | renames | restrictions | tags | timestamp | user_id | user_text ------------------+----------------------------+-----------+--------------------------------------+------------+--------------+------+---------------------------------------------------------------------+------------+-----------+-------------+----------+----------+---------+--------------+------------------------------------+---------------------------------+----------+---------------- en.wikipedia.org | Joseph_Stalin | 811977685 | eaa68870-d503-11e7-a897-f0fcc5ebdd83 | null | null | null | /* Famine and the Great Terror: 1932–1939 */ | null | 0 | null | 15641 | False | null | null | null | 2017-11-25 04:47:51.000000+0000 | 13518729 | LittleJerry
(eaa68870-d503-11e7-a897-f0fcc5ebdd83 is Wednesday, November 29, 2017 12:50:45 PM GMT which falls in the window during which RESTBase was up on rb1012).
+1 on dropping them, but since there are still logs about cluster name disagreements, perhaps it would be good to first do a rolling restart of the Cass3 cluster @Eevans ?
The name disagreements are caused by hosts in the legacy cluster that have not been restarted since others legacy hosts were moved to the new cluster (so they continue to gossip to them). "Fixing" that means restarting the legacy cluster, and to keep these messages from reappearing, we'd need to do a rolling restart prior to each new bootstrap (see also: T178845: Cassandra cluster name mismatch warnings).
Mentioned in SAL (#wikimedia-operations) [2017-11-30T16:44:45Z] <urandom> drop (erroneous) legacy tables from -ng cassandra cluster - T181689
Update: the process of dropping these keyspaces continues; Using the (conservative) algorithm in [[ https://github.com/eevans/cassandra-schema-nodejs/blob/master/bin/mkschema | mkschema ]], these DROPs are happening at a rate of ~9/hour. At this time we are ~75% complete.
Once complete, the state of 1012-a can be cleared (by removing the contents of /srv/sda4/cassandra-a/data, /srv/sdb4/cassandra-a/data, /srv/sdc4/cassandra-a/data, /srv/sdd4/cassandra-a/data, /srv/sde4/cassandra-a/data, and /srv/cassandra/instance-data/cassandra-a/{commitlog,hints,saved_caches}), the systemd unit unmasked, and the service started. This may fail and require that -Dcassandra.replace_address=10.64.32.202 be passed (for example, by temporarily disabling Puppet, and placing the parameter in /etc/cassandra-a/jvm.options).
Mentioned in SAL (#wikimedia-operations) [2017-12-01T10:21:01Z] <godog> initial purge of old table metrics from graphite2002 - T181689
Mentioned in SAL (#wikimedia-operations) [2017-12-04T09:32:07Z] <godog> clear erroneous table metrics from graphite1003 / graphite2002 - T181689