EventGate is not primarily a bulk data endpoint, so let's be sure not to overwhelm it with scraper events. Put a rate-limiting wrapper on the producer event_gate.ex, with an initial setting of 500 events/sec.
Verify that back-pressure is not slowing down the scrape eg. by profiling a smaller sample with the rate limit, vs. to /dev/null .
Monitor the EventGate Grafana dashboard when we run, to watch for any impact on server health eg. lag or request latency of other streams, or memory usage.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T396752 [Epic] Metrics for sub-referencing | |||
| Open | None | T398277 [Epic] Scraping data dumps and scraper enhancements | |||
| Open | None | T396731 Scraper: Build a new dashboard for the updated scraper data | |||
| Open | None | T410719 Persist scraper outputs to Hive | |||
| Resolved | None | T412227 Rate limit EventGate output to 500 events/second |
Code to review: https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/merge_requests/159
No library was needed, I found a trick: zip the output stream with a timer so that batches of a known size are emitted a known time apart. This might prove to be too bursty: a period of slow processing would let the timer build up. On the other hand, it still rate-limits to an *average* throughput.