| pfischer | |
| Aug 27 2024, 4:22 PM |
| F57678074: image.png | |
| Nov 4 2024, 10:19 AM |
Possible approach: Compare the number of events coming in via page_change_weighted_tags with the number of bulk actions modifying the weighted_tags multimap.
AC:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | pfischer | T366253 Create a generic stream to populate CirrusSearch weighted_tags | |||
| Resolved | dcausse | T373459 SUP: set up alerting for page_change_weighted_tags ingestion |
dcausse opened https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/164
Monitor weithed_tags
dcausse merged https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/164
Monitor weighted_tags
Change #1081172 had a related patch set uploaded (by DCausse; author: DCausse):
[operations/deployment-charts@master] cirrus-streaming-updater: bump to v20241017132903-67693a7
Change #1081172 merged by jenkins-bot:
[operations/deployment-charts@master] cirrus-streaming-updater: bump to v20241017132903-67693a7
What is the state of this? I'm asking, because looking at things on our end, it seems that the "database-and-search-index"-pairs are missing the search-index part. And the staircase-pattern makes me suspect that something is going wrong when the maintenance script every couple of hours is trying to add new records both to the database and to the search-index. Adding to the database succeeds, adding to the search-index seems to fail, thus the dangling record.
Our code is supposed to cancel the transaction adding the db-row if \CirrusSearch\WeightedTagsUpdater::updateWeightedTags throws, but maybe that is not happening?
(Or maybe the error is somewhere in the deleting step after all. But why the pattern then?)
On Wednesday October 30 we switched to using event-gate for weighted tags emitted from CirrusSearch at this point we started collecting some metrics under flink_taskmanager_job_task_operator_weighted_tags_tag_prefix_clear & flink_taskmanager_job_task_operator_weighted_tags_tag_prefix_set.
The status of this ticket is that now we have to create a dashboard based on this data so that it might help better correlate possible errors.
I'll be setting this dashboard asap so that we have a better understanding of what's happening.
I'm asking, because looking at things on our end, it seems that the "database-and-search-index"-pairs are missing the search-index part. And the staircase-pattern makes me suspect that something is going wrong when the maintenance script every couple of hours is trying to add new records both to the database and to the search-index. Adding to the database succeeds, adding to the search-index seems to fail, thus the dangling record.
If you noticed that the issues started right after October 10 then it might be very possible that it's related to T377150.
Our code is supposed to cancel the transaction adding the db-row if \CirrusSearch\WeightedTagsUpdater::updateWeightedTags throws, but maybe that is not happening?
(Or maybe the error is somewhere in the deleting step after all. But why the pattern then?)
@pfischer/@Michael a dashboard is up at: https://grafana-rw.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1
I'm not sure how to craft an alert based on these numbers, please let me know if you have ideas. Moving to Blocked/Waiting while we decide on if we want to set up an alert and how.
Thanks, this is useful!
I think my recommendation would be: let's fix T378983: Add Link recommendation are not being processed by CirrusSearch (November 2024) first, then we wait a week or two to see what kind of numbers we're working with, and then we maybe can set some lower-bounds alerts.
@Michael I think this is relatively stable now, since search is not owning the individual sources of tags I think it might be better to have more fine-grained alerts (per tag?) on your side if you want, on our side I might set a very broad alert to capture only obvious problems (i.e. no tags updated in the last hour) but it might not cover failures specific to a particular tag.
Change #1111300 had a related patch set uploaded (by DCausse; author: DCausse):
[operations/alerts@master] search: add alerts for weighted_tags indexing throughput
Change #1111300 merged by jenkins-bot:
[operations/alerts@master] search: add alerts for weighted_tags indexing throughput