Page MenuHomePhabricator
Create Task
Maniphest T383854

cxserver: SectionMapping DB timeout
Closed, ResolvedPublic2 Estimated Story PointsBUG REPORT
Actions

Description

The cxserver log has entries with a DB error when accessing the section mapping database on m5.

Log

Error: Could not connect to section mapping databaseError: connect ETIMEDOUT

https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2025.01.15?id=VOATaZQBLmySI1N_eieU

Error is from cxserver/lib/suggestion/SectionSuggester.js where it fails to connect with database.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Update cxserver to 2025-01-16-103443-productionoperations/deployment-chartsmaster+1 -1
SectionSuggester: Open db connection in the same execution contextmediawiki/services/cxservermaster+10 -6
Customize query in gerrit

Related Objects

Event Timeline

KartikMistry created this task.Jan 16 2025, 3:42 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 16 2025, 3:42 AM
gerritbot added a comment.Jan 16 2025, 6:53 AM

Change #1111675 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/services/cxserver@master] SectionSuggester: Open db connection in the same execution context

https://gerrit.wikimedia.org/r/1111675

gerritbot added a project: Patch-For-Review.Jan 16 2025, 6:53 AM
Nikerabbit triaged this task as Medium priority.Jan 16 2025, 8:40 AM
Nikerabbit added projects: LPL Essential (LPL Essential 2024 Nov-Jan), Unplanned-Sprint-Work.
Nikerabbit moved this task from Backlog to Maintenance and operational issues on the CX-cxserver board.
Marostegui subscribed.Jan 16 2025, 9:11 AM

Was this a one time thing? m5 databases look up and healthy

gerritbot added a comment.Jan 16 2025, 10:32 AM

Change #1111675 merged by jenkins-bot:

[mediawiki/services/cxserver@master] SectionSuggester: Open db connection in the same execution context

https://gerrit.wikimedia.org/r/1111675

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2025, 11:30 AM
gerritbot added a comment.Jan 16 2025, 11:32 AM

Change #1111948 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update cxserver to 2025-01-16-103443-production

https://gerrit.wikimedia.org/r/1111948

gerritbot added a project: Patch-For-Review.Jan 16 2025, 11:32 AM
KartikMistry added a comment.Jan 16 2025, 11:34 AM
In T383854#10465605, @Marostegui wrote:

Was this a one time thing? m5 databases look up and healthy

Getting more logs with some fixes in logging. Let's see!

gerritbot added a comment.Jan 16 2025, 11:40 AM

Change #1111948 merged by jenkins-bot:

[operations/deployment-charts@master] Update cxserver to 2025-01-16-103443-production

https://gerrit.wikimedia.org/r/1111948

Stashbot mentioned this in T377966: Make cxserver Logstash logs readable and reliable.Jan 16 2025, 11:49 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-16T11:49:28Z] <kart_> Updated cxserver to 2025-01-16-103443-production (T383854, T377966)

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2025, 12:30 PM
Marostegui moved this task from Triage to Refine on the DBA board.Jan 16 2025, 2:40 PM
Nikerabbit set the point value for this task to 2.EditedJan 16 2025, 3:22 PM
Nikerabbit moved this task from Backlog to Need QA on the LPL Essential (LPL Essential 2024 Nov-Jan) board.
Nikerabbit subscribed.

I think I'm only seeing errors before the latest deployment but might to be too early to tell.

Nikerabbit assigned this task to abi_.Jan 20 2025, 8:51 AM
KartikMistry added a comment.Jan 20 2025, 1:00 PM

With the latest cxserver logs (sadly, not available to logstash as of now due to a different issue), I don't see the above error CC @abi_

abi_ closed this task as Resolved.Jan 23 2025, 8:41 AM
abi_ moved this task from Need QA to Done on the LPL Essential (LPL Essential 2024 Nov-Jan) board.

Not seeing this error anymore.

Marostegui added a comment.Jan 23 2025, 8:43 AM

Thanks!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits