For realistic load testing of various services, it would be useful to be able to replay actual requests at varying rates. This task is to serve as a reminder to look into this, and as a point to collect options.
## Considerations
### RESTBase
- It would be ideal to be able to use a corpus of historical transactions for replay (e.g. past varnish logs, past events in kafka), however when re-rendering for transclusions, we use the latest version and do not persist if the output is unchanged. If may be easier to avoid this problem and track current events only, at least to begin with.
- Dev/test/staging environments will always be smaller (have less capacity) than production, so sampling will be necessary. However, assuming a test environment with 20% the capacity of production, it wouldn't be sufficient to simply drop every 4 in 5 events, since that would adversely effect the distribution of updates by title; It will be necessary to instead sample by title (using consistent hashing, or similar).
- When replaying update requests, a corresponding request load will be generated on the Parsoid and API clusters. We will need to establish whether the additional request load will be a problem for these systems. If so, one possibility would be the addition of a RESTBase endpoint to emulate Parsoid using cached responses.
### General
- A solution using [[ https://github.com/buger/goreplay | `goreplay` ]] or [[ http://tcpreplay.synfin.net/ | `tcpreplay` ]] would be more widely applicable, but would need to be deployed centrally/strategically, where packet capture of the applicable traffic could be performed. Such a solution could be considered more invasive, would need to take PII concerns into account, and would require more buy-in and effort on the part of the Ops team.
## Candidates (please add)
- https://www.varnish-cache.org/docs/3.0/reference/varnishreplay.html: Replays Varnish logs.
- https://github.com/buger/goreplay Open-source tool to capture and mimic the traffic. We could use it to load staging.
- `tcpdump` [[ http://tcpreplay.synfin.net/ | and `tcpreplay` (et al) ]]
- [[ https://www.mediawiki.org/wiki/Change_propagation | Change-propagation ]] (with support added for sampling) (RESTBase-specific)