@EBernhardson, according to @JMeybohm there is no way to limit the IP ranges of pod/service/namespace to associated them closely with an application (SUP).

I brought up this discussion with @JMeybohm and as it turns out, rate-limiting is a long-wanted feature for the MW API anyways, see T248543. Service/Ops is willing to discuss implementing it the envoy-way: Envoy supports local and remote/distributed rate limits, as described here. The least invasive approach to test this would be the following:

What is required for the test environment? Currently I would follow those steps:

Here's a list of metrics and alerts based on them:

• alert if the combined kafka message-in-rate (see grafana panel) is 0 for more than 5 minutes, for the following topics
  • eqiad.cirrussearch.update_pipeline.update.rc0
  • codfw.cirrussearch.update_pipeline.update.rc0
• alert if the combined kafka message-in-rate (see grafana panel) is 0.1 for more than 5 minutes, for the following topics
  • eqiad.cirrussearch.update_pipeline.fetch_error.rc0
  • codfw.cirrussearch.update_pipeline.fetch_error.rc0

@Jdforrester-WMF, I looked at the code. The CirrusSearch SearchEngine does not handle Special:… searches itself, instead it delegates to the (core) implementation which in turn delegates to the default TitlePrefixSearch where searches inside NS_SPECIAL are processed in PrefixSearch::specialSearch( $search, $limit, $offset ). This performs an in-memory search against pages listed in SpecialPageFactory::CORE_LIST.

As of today, all non-private wikis featuring the cirrussearch extension publish page_rerender events by default.

For reference, here's a screenshot of more kafka metrics around enabling compaction:

Partitions + keyed records are already enabled on kafka-test, see T354064.

consumer-devnull (no resource request/limit overrides) correctly gets ioThreadCount=1, see logs:

One of the key findings of the backfill tests was a lower-than-expected throughput, see T353460. That is mainly caused by a bug inside the flink consumer but is still a unexpectedly high rate of timeouts (from enovys perspective) see T354289.

Yes, you are right, Janis just told me. Alternatively, we can send a header x-envoy-max-retries: 0, see docs.

@JMeybohm, thanks! That brought down the throttling.

@Ottomata, we considered this but but decided against it since

@Ottomata, yes, this was intended to a) save disk space and b) reduce the number of records that have to be processed in case of a back fill. Events representing "page X has been re-rendered" seemed a perfect use case where only the latest event is meaningful and worth to be kept.

@elukey, we have an updated estimate of the expected topic size increment per wiki we publish page_rerender records for: https://docs.google.com/spreadsheets/d/1Fp44MdLxUVlxi03MBD_64m0zQErny-9jUD5C6RGf_bU/edit#gid=670687915

Upstream elasticsearch bug