As a result of T136340: Staging / Test environment(s) for RESTBase, we now have 3 new nodes we can use to perform testing under more realistic workloads.

There are a number of use-cases for this environment in the pipeline, but some initial work is needed to get the cluster setup, and to have the means of synthesizing storage and workload more closely matching that of production.

The requirements for this issue are:

• Establish baseline configuration for restbase-test100[1-3]
  • Cassandra 2.2.6 (as a starting point)
  • RESTBase
• Implementation of basic workload simulation
  • Sampled request replay of varnish cache misses
  • Sampled request replay of change updates
• Historical data (inducing conditions that develop over time)
  • Revision import of problematic partitions

Workload simulation

Varnish

It should be straightforward to query Hive for web requests that are forwarded to RESTBase from Varnish. For example:

USE wmf;

SELECT
  dt, cache_status, uri_host, uri_path
FROM
  webrequest
WHERE
  uri_path like '%api/rest_v1%'
  AND cache_status IN ('miss', 'pass')
  AND year=2016
  AND month=12
  AND day=01

The results from such queries can be sampled and replayed accordingly.

Change propagation

Ideally, rMSCP mediawiki-services-change-propagation code could be reused to create something that could consume from arbitrary offsets in Kafka, and replay sampled requests according to a production-like configuration. It may however be simpler (read: more expedient), to simply implement a basic consumer for this purpose.

Historical data

Given a corpus of titles (perhaps harvested from the blacklist), recreate a document's revision history in RESTBase. These imports should reflect the outcome of revision retention policies on the underlying Cassandra storage.

Considerations

Requests sent to this test environment will in turn issues requests to the action API and parsoid. We need to either a) establish that these traffic levels are acceptable, or b) come up with alternatives (for example, teach RESTBase to forward Parsoid requests to the production RESTBase instance).