One easy improvement would be to serialize schema changes on a single nodeCassandra schema creation in RESTBase has always been somewhat flaky, but it seems to have gotten worse in the new cluster. Presumably this would be the result of either changes in the newer version of Cassandra (currently 3.11.0), our new application code, or the differing conditions/load imposed by the new code on the cluster (or even some combination).
When RESTBase issues Cassandra schema changes, it uses a general purpose connection pool created at startup, one that is populated with all nodes in the cluster. This means that each statement could be executed on a different hostQueries to Cassandra upstream would seem to suggest that no one else is having this issue in the 3.11.x series, which is not recommended;and while this doesn't preclude the possibility of a regression, Schema changes should be serialized on a single hostit should be enough to warrant a thorough assessment of what can be done on our side before pursuing it further (we have had serious bugs in this code on more than one occasion).
It should be possible to create and use a separate pool forOne straightforward improvement would be to serialize schema changes on a single node. Currently, when RESTBase issues Cassandra schema changes, one that is configured with a [[ http://docs.datastax.com/en/developer/nodejs-driver/3.2/api/module.policies/module.loadBalancing/class.WhiteListPolicy/ | whitelist policy ]] to restrict execution to a single hostit uses a general purpose connection pool created at startup, one that is populated with all nodes in the cluster. This means it is possible that each schema altering statement could be executed on a different host, which is not recommended.
== Update: ==[[ https://github.com/eevans/cassandra-schema-nodejs | Some prototype code was put together ]] that uses a single connection pool ala a WhiteListPolicy to ensure that schema changes are serialized to one host. [[ https://github.com/eevans/cassandra-schema-nodejs/blob/master/bin/mkschema | A script ]] was created to use this to issue DDL statements as defined in a YAML formatted file. Additionally, the script allows for the max agreement wait time to be increased, as well as performing a separate validation of agreement before continuing with subsequent statements. This was tested by manually creating the tables necessary for some [[ https://phabricator.wikimedia.org/T179416 | recent migrations ]]. The results, while still not perfect, are very much improved over what we were seeing when RESTBase auto-generated schema.
[[ https://github.com/eevans/cassandra-schema-nodejs | Some prototype code was put together ]] for nodejs that uses a single connection pool ala a WhiteListPolicy to ensure that schema changes are serialized on a single node. [[ https://github.com/eevans/cassandra-schema-nodejs/blob/master/bin/mkschema | A script ]] was created to use this to issue DDL statements as defined in a YAML formatted file. This was tested by manually creating the tables necessary for [[ https://phabricator.wikimedia.org/T179416 | recent migrations ]]. The results, while still not perfectAt a minimum, are very much improved over what we were seeing when RESTBase auto-generated schemawe should update RESTBase's schema creation and migration code to incorporate these techniques.