As part of the Search Update pipeline work, we want to be able to measure impact and quantify the problem that we are trying to resolve.
More general discussion about the potential definition of SLO are at https://docs.google.com/document/d/1gYROXo8Fl7JSxReHAVI22EhcPvG-INVkq79a1C3tfK0/edit
AC:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Gehel | T317045 [Epic] Re-architect the Search Update Pipeline | |||
| Resolved | RKemper | T328330 Create SLI / SLO on Search update lag | |||
| Resolved | dcausse | T320408 Monitor CirrusSearch update lag | |||
| Declined | None | T363795 SUP: Add metric to determine deduplication window length |
There was a suggestion that we could use the average number of Sanetizer corrections over the Nov 2020 - Nov 2021 period: https://grafana.wikimedia.org/d/2DIjJ6_nk/cirrussearch-saneitizer-historical-fix-rate?orgId=1&from=now-2y&to=now
dcausse opened https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/21
import_cirrus_indexes: add index_name to the dataset
dcausse opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/314
search: add a new index_name fields to cirrus dumps
dcausse merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/314
search: add a new index_name fields to cirrus dumps
dcausse opened https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/25
WIP: Add check_cirrus_index_consistency.py
dcausse opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/427
Draft: search: schedule cirrus_consistency_check
ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/25
Add check_cirrus_index_consistency.py
dcausse merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/427
search: schedule cirrus_consistency_check
still blocked on gathering more data, sadly a lot of typos were made in this monthly dag (https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/465)
We should split this ticket between error rate and lag. Lag is easy to complete, but error will take longer.
dcausse updated https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/147
Add metric to report lag from the consumer
dcausse merged https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/147
Add metric to report lag from the consumer
dcausse opened https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/148
Do not override producer events dt
dcausse merged https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/148
Do not override producer events dt
Change #1060076 had a related patch set uploaded (by DCausse; author: DCausse):
[operations/deployment-charts@master] cirrus-streaming-updater: bump image to v20240806085845-f838190
Change #1060076 merged by jenkins-bot:
[operations/deployment-charts@master] cirrus-streaming-updater: bump image to v20240806085845-f838190
Change #1060150 had a related patch set uploaded (by DCausse; author: DCausse):
[operations/grafana-grizzly@master] search: add search update lag SLO
Dashboard is up at https://grafana.wikimedia.org/d/8xDerelVz/search-update-lag-slo?orgId=1
A patch to grafana-grilly is upload at https://gerrit.wikimedia.org/r/c/operations/grafana-grizzly/+/1060150 (but not sure how to move forward with it)
Change #1060150 merged by Bking:
[operations/grafana-grizzly@master] search: add search update lag SLO
Mentioned in SAL (#wikimedia-operations) [2024-09-05T21:57:55Z] <inflatador> bking@grafana1002 apply grizzly SLO dashboard updates slo-Search added slo-apigw updated P68729 T328330
Grizzly dashboard was deployed last week: https://grafana.wikimedia.org/d/slo-Search/search-slo-s?orgId=1