Change 462042 had a related patch set uploaded (by HaeB; owner: HaeB):
[operations/mediawiki-config@master] Increase sampling ratio for ReadingDepth

https://gerrit.wikimedia.org/r/462042

Change 462535 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[operations/mediawiki-config@master] Increate ReadingDepthSamplingRate to 0.1

https://gerrit.wikimedia.org/r/462535

Change 462535 abandoned by Pmiazga:
Increate ReadingDepthSamplingRate to 0.1

Reason:
Abandoned in favour of I7501c3ff20b73af140b74f1297221aded950df1e

https://gerrit.wikimedia.org/r/462535

Monitor ReadingDepth event traffic post-deploy and that it matches expectations.

Per rOMWC6bf9cdc796ba: wme: Set ReadingDepth sampling rate to 0.1%:

The current sampling rate, 0.5% (the default), results in a peak rate of ~2500 events/minute.

So we should expect fewer events than that. Regardless, we saw a peak rate of over 18,000 events/minute for the Page Previews instrumentation, which seemed to be handled by EventLogging pipeline (ingesting into Hadoop, not MariaDB) just fine.

Change 462042 merged by jenkins-bot:
[operations/mediawiki-config@master] Increase sampling ratio for ReadingDepth

https://gerrit.wikimedia.org/r/462042

Mentioned in SAL (#wikimedia-operations) [2018-09-25T11:20:31Z] <pmiazga@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:462042|Increase sampling ratio for ReadingDepth (T205176)]] (duration: 00m 50s)

@Tbayer @Groceryheist - the new sampling rate is live

Monitor ReadingDepth event traffic post-deploy and that it matches expectations.

Per https://grafana.wikimedia.org/dashboard/db/eventlogging-schema?orgId=1&var-schema=ReadingDepth&from=1537874100000&to=now, we're seeing ~550 events/minute and 0 errors. For the latter, I've also been tailing the eventlogging_Error Kafka topic [0] and I've seen no errors there either.

kafkacat -q -C -b kafka-jumbo1001.eqiad.wmnet -t eventlogging_EventError | grep ReadingDepth

Further to T205176#4614616:

Between 12 AM and 6 AM today we saw an error rate of between 0.0005% and 0.0067%. The upper bound assumes that all erroneous events that couldn't be identified as belonging to any schema are ReadingDepth events.

select
    count(*) as n
from
    event.readingdepth
where
    year = 2018 and
    month = 9 and
    day = 26 and

    hour >= 0 and hour < 6
;

+-----------+
|     n     |
+-----------+
| 10754243  |
+-----------+

select
    event.schema as schema,
    count(*) as n
from
    event.eventerror
where
    year = 2018 and
    month = 9 and
    day = 26 and

    hour >= 0 and hour < 6 and

    event.schema in ("ReadingDepth", "unknown")
group by
    event.schema
;

+---------------+------+
|    schema     |  n   |
+---------------+------+
| ReadingDepth  | 56   |
| unknown       | 661  |
+---------------+------+

In T205176#4618506, @phuedx wrote:

Further to T205176#4614616:

Between 12 AM and 6 AM today we saw an error rate of between 0.0005% and 0.0067%.

[...]
Thanks! Does this mean we can consider the AC "Analyse any errors that are introduced in the EventLogging pipeline relating to this change" fulfilled?

In T205176#4624033, @Tbayer wrote:

Thanks! Does this mean we can consider the AC "Analyse any errors that are introduced in the EventLogging pipeline relating to this change" fulfilled?

Sure! I've checked it off and I'll update the task description.