Page MenuHomePhabricator
Create Task
Maniphest T172492

Database alerting
Closed, ResolvedPublic
Actions

Description

This is an epic task to gather all alerting (not including monitoring of trends/graphing, only potential emergencies a.k.a. icinga) of databases. There is currently too many false positives, and some gaps on the alerting, so tools and model has to change.

Related incident: https://wikitech.wikimedia.org/wiki/Incident_documentation/2017-07-28_s5_(WikiData_and_dewiki)_read-only

Details

SubjectRepoBranchLines +/-
mariadb: Default monitor disk & process paging to falseoperations/puppetproduction+56 -17
monitoring: remove usages of 'dba' contact groupoperations/puppetproduction+12 -24
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT172492 Database alerting
 ResolvedjcrespoT172490 Monitor swap/memory usage on databases
 ResolvedjcrespoT172489 Monitor read_only on all databases, make it page on masters
 DeclinedNoneT112473 Better mysql monitoring for number of connections and processlist strange patterns
 ResolvedaaronT177778 Improve database application performance monitoring visibility
 ResolvedjcrespoT177782 Reduce false positives on database pages
 ResolvedDzahnT178008 ensure that services on labtest machines never create SMS from Icinga (not send sms pages for labtest* things to non-cloud folks)
 ResolvedCDanisT197126 Create tool to handle the state of database configuration in MediaWiki in etcd
 ResolvedVolansT197531 Data model for dbconfig
 ResolvedCDanisT229070 phased rollout of dbctl, etcd-backed database configuration in Mediawiki
 ResolvedCDanisT229631 noc.wm.o/db.php: remove hosts information, or fetch it from etcd somehow
 ResolvedMarosteguiT200509 Make sure multi-instance slaves page
 OpenNoneT205628 Handle object metadata backups and compare it with stored database object inventory
 ResolvedjcrespoT237927 Add replication lag (and other checks) to misc all hosts
 OpenNoneT252679 Move paging from individual databases to database service "groups"
 Resolved• KormatT253120 Create prometheus alert to detect lag spikes
 ResolvedjcrespoT57406 Monitor database hosts' disk performance
 ResolvedhasharT38994 [OPS] Add disk I/O to ganglia reports
 DeclinedLSobanskiT274513 Investigate intermittent replica lag alarms
 OpenNoneT277174 Update DB read_only alert to represent correct state
 Resolved• KormatT233684 Make primary DB masters page on HOST DOWN alert
 ResolvedMarosteguiT322987 db2173 crashed and didn't alert
 ResolvedPapaulT322988 db2173 HW errors
 OpenNoneT281833 Introduce alerting to monitor mediawiki databases QPS rate of change
 Resolved• KormatT290591 Database read_only alert has a changing service description

Event Timeline

jcrespo created this task.Aug 4 2017, 8:41 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2017, 8:41 AM
jcrespo triaged this task as Medium priority.Aug 4 2017, 8:41 AM
jcrespo moved this task from Triage to Meta/Epic on the DBA board.
jcrespo added subtasks: T172490: Monitor swap/memory usage on databases, T172489: Monitor read_only on all databases, make it page on masters, T112473: Better mysql monitoring for number of connections and processlist strange patterns.Aug 4 2017, 8:43 AM
jcrespo mentioned this in T133062: Fix m1 replication icinga checks.Aug 4 2017, 8:53 AM
Luke081515 awarded a token.Aug 4 2017, 1:40 PM
greg added a project: Wikimedia-Incident.Aug 4 2017, 9:18 PM
greg moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.
Aklapper added a project: Epic.Aug 4 2017, 11:23 PM
jcrespo created subtask T177778: Improve database application performance monitoring visibility.Oct 9 2017, 3:35 PM
jcrespo removed a subtask: T177778: Improve database application performance monitoring visibility.Oct 9 2017, 3:43 PM
jcrespo added a subtask: T177778: Improve database application performance monitoring visibility.
jcrespo created subtask T177782: Reduce false positives on database pages.Oct 9 2017, 4:19 PM
Marostegui closed subtask T200509: Make sure multi-instance slaves page as Resolved.Sep 19 2018, 9:23 AM
jcrespo changed the status of subtask T172489: Monitor read_only on all databases, make it page on masters from Open to Stalled.Sep 19 2018, 1:07 PM
Marostegui removed a project: Wikimedia-Incident.Apr 11 2019, 1:23 PM
jcrespo added a subtask: T205628: Handle object metadata backups and compare it with stored database object inventory.Dec 2 2019, 10:04 AM
jcrespo changed the status of subtask T172489: Monitor read_only on all databases, make it page on masters from Stalled to Open.May 7 2020, 7:03 AM
jcrespo closed subtask T172489: Monitor read_only on all databases, make it page on masters as Resolved.May 7 2020, 2:04 PM
jcrespo added a subtask: T237927: Add replication lag (and other checks) to misc all hosts.May 7 2020, 4:01 PM
gerritbot added a comment.May 8 2020, 10:39 AM

Change 595149 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] monitoring: remove usages of 'dba' contact group

https://gerrit.wikimedia.org/r/595149

gerritbot added a project: Patch-For-Review.May 8 2020, 10:39 AM
gerritbot added a comment.May 8 2020, 11:23 AM

Change 595153 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Default monitor disk paging to false

https://gerrit.wikimedia.org/r/595153

gerritbot added a comment.May 13 2020, 7:04 AM

Change 595149 merged by Jcrespo:
[operations/puppet@production] monitoring: remove usages of 'dba' contact group

https://gerrit.wikimedia.org/r/595149

gerritbot added a comment.May 13 2020, 2:06 PM

Change 595153 merged by Jcrespo:
[operations/puppet@production] mariadb: Default monitor disk & process paging to false

https://gerrit.wikimedia.org/r/595153

jcrespo closed subtask T237927: Add replication lag (and other checks) to misc all hosts as Resolved.May 13 2020, 2:10 PM
Maintenance_bot removed a project: Patch-For-Review.May 13 2020, 2:11 PM
jcrespo closed subtask T177782: Reduce false positives on database pages as Resolved.May 13 2020, 2:42 PM
jcrespo closed subtask T177778: Improve database application performance monitoring visibility as Resolved.May 13 2020, 2:46 PM
Marostegui added a subtask: T253120: Create prometheus alert to detect lag spikes.May 19 2020, 1:07 PM
Marostegui closed subtask T112473: Better mysql monitoring for number of connections and processlist strange patterns as Declined.
• Kormat changed the status of subtask T253120: Create prometheus alert to detect lag spikes from Open to Stalled.Jun 29 2020, 7:47 AM
fgiunchedi moved this task from Inbox to Radar on the observability board.Jul 20 2020, 1:16 PM
jcrespo closed subtask T172490: Monitor swap/memory usage on databases as Resolved.Aug 10 2020, 4:56 PM
Marostegui changed the status of subtask T253120: Create prometheus alert to detect lag spikes from Stalled to Open.Aug 12 2020, 11:21 AM
Marostegui closed subtask T253120: Create prometheus alert to detect lag spikes as Resolved.Aug 13 2020, 8:21 AM
jcrespo added a subtask: T57406: Monitor database hosts' disk performance.Jan 27 2021, 1:02 PM
jcrespo added a subtask: T274513: Investigate intermittent replica lag alarms.Feb 11 2021, 11:52 AM
jcrespo mentioned this in T277498: DB alerting updates.May 14 2021, 7:01 AM
LSobanski added a subtask: T277174: Update DB read_only alert to represent correct state.May 14 2021, 10:28 AM
LSobanski added a subtask: T233684: Make primary DB masters page on HOST DOWN alert.
LSobanski added a subtask: T281833: Introduce alerting to monitor mediawiki databases QPS rate of change.
LSobanski merged a task: T277498: DB alerting updates.
LSobanski edited projects, added Data-Persistence; removed DBA.
LSobanski added subscribers: LSobanski, Peachey88.
LSobanski moved this task from Inbox to Epic on the Data-Persistence board.May 14 2021, 10:33 AM
LSobanski renamed this task from Improve database alerting (tracking) to Database alerting.May 14 2021, 10:37 AM
jcrespo closed subtask T57406: Monitor database hosts' disk performance as Resolved.May 14 2021, 10:40 AM
LSobanski moved this task from Epic to Epic - Database on the Data-Persistence board.May 14 2021, 5:11 PM
LSobanski closed subtask T274513: Investigate intermittent replica lag alarms as Declined.May 20 2021, 1:35 PM
LSobanski added a project: User-LSobanski.May 24 2021, 10:19 AM
LSobanski removed a project: User-LSobanski.May 24 2021, 2:02 PM
RLazarus added a subtask: T290591: Database read_only alert has a changing service description.Sep 8 2021, 11:38 PM
Krinkle added a project: Sustainability (Incident Followup).Sep 28 2021, 11:44 PM
Krinkle updated the task description. (Show Details)
• Kormat closed subtask T290591: Database read_only alert has a changing service description as Resolved.Oct 11 2021, 11:40 AM
• Kormat closed subtask T233684: Make primary DB masters page on HOST DOWN alert as Resolved.Nov 15 2021, 4:49 PM
Joe added a project: SRE-Sprint-Week-Sustainability-March2023.Mar 20 2023, 11:38 AM
Joe moved this task from Backlog to E_TOO_BIG_MAYBE_OKR? on the SRE-Sprint-Week-Sustainability-March2023 board.
lmata added a project: SRE Observability.Mar 28 2023, 8:32 PM
lmata edited projects, added Observability-Alerting; removed SRE Observability.
lmata moved this task from Inbox to Radar on the Observability-Alerting board.
Ladsgroup closed this task as Resolved.Apr 19 2023, 4:30 PM
Ladsgroup subscribed.

Database alerting in general needs improvements and we made a lot of progress since this ticket was created. But tickets usually shouldn't be kept open forever. The subtickets are open and that should be enough. We can always create a new ticket for specific or goal-oriented cases ("switch all alerts to prometheus" as an example) if needed and close them once done.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits