Page MenuHomePhabricator
Create Task
Maniphest T239120

Services downstream from RESTBase should not separately alert due to RESTBase production monitoring timeouts
Closed, DeclinedPublic
Actions

Description

When RESTBase fails production monitoring checks due to timeouts, a number of other services downstream from it also alert; typically these include at least the Page Content Service (mobileapps) and the Recommendation API service (recommendation-api). These downstream services should not also alert because that creates noise and makes it harder to identify the true cause of instability.

Proposal: In downstream services, maintain HTML fixtures for monitoring requests that are used in lieu of /page/html requests to RESTBase when a monitoring request parameter is set.

Event Timeline

• Mholloway created this task.Nov 25 2019, 4:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 25 2019, 4:10 PM
LGoto triaged this task as Medium priority.Nov 27 2019, 4:37 PM
LGoto moved this task from Needs triage to Backlog on the Product-Infrastructure-Team-Backlog-Deprecated board.
• mobrovac closed this task as Declined.Dec 4 2019, 7:53 PM
• mobrovac subscribed.

RESTBase (and other services) are alerting because we want to be sure that we catch problems visible by external users. While it does sometimes introduce false positives (the inverse is true too - RB alerts if back-end services alert), it is better to have more alerts than none due to convoluted logic or fixtures.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits