Page MenuHomePhabricator
Create Task
Maniphest T341117

cxserver: Section Mapping Database (m5) not accessible by certain region
Closed, ResolvedPublic
Actions

Description

cxserver's section mapping database (m5) seems down for certain region (ie Europe), while works well in India.

Logstash has some entries of the error:

Error: Could not connect to section mapping databaseError: connect ETIMEDOUT
    at Connection._handleConnectTimeout (/srv/service/node_modules/mysql/lib/Connection.js:409:13)
    at Object.onceWrapper (node:events:627:28)
    at Socket.emit (node:events:513:28)
    at Socket._onTimeout (node:net:526:8)

To test,

https://cxserver.wikimedia.org/v2/suggest/sections/United_States/en/de

should return sections.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
cxserver: Migrate to the new MariaDB egress functionalityoperations/deployment-chartsmaster+3 -12
cxserver: Bump to networkpolicy_1.1.0.tploperations/deployment-chartsmaster+85 -54
modules: Add a new networkpolicy for base modulesoperations/deployment-chartsmaster+127 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

KartikMistry created this task.Jul 5 2023, 9:30 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 5 2023, 9:30 AM
Ladsgroup added a comment.Jul 5 2023, 9:32 AM

Is it active/active? We don't have have anything in other datacenters, only between eqiad and codfw. I highly doubt it's a database issue.

KartikMistry updated the task description. (Show Details)Jul 5 2023, 9:35 AM
Ladsgroup edited projects, added Kubernetes, serviceops-deprecated; removed DBA.Jul 5 2023, 9:37 AM
Ladsgroup added a subscriber: akosiaris.

The databases are working fine, I think k8s can't reach the new proxies again. cc. @akosiaris

Marostegui subscribed.Jul 5 2023, 9:41 AM

This might be related to T337812 I have failed back to dbproxy1017 until it is investigated which FW are needed for that new host.
Please revert https://gerrit.wikimedia.org/r/935692 once it is fixed.

@KartikMistry allow 5-10 minutes for the DNS to spread.

Marostegui added a comment.Jul 5 2023, 9:56 AM

Confirmed this works now: https://cxserver.wikimedia.org/v2/suggest/sections/United_States/en/de
@akosiaris could you figure out what is needed to get dbproxy1027 to work too? Thanks

Marostegui added a parent task: T340843: WikiKube: Investigate how to abstract misc Mariadb clusters host/ip information so that no deployment of apps is needed when a master is failed over.Jul 5 2023, 9:56 AM
akosiaris claimed this task.Jul 5 2023, 10:24 AM
Marostegui mentioned this in T341121: Decommission dbproxy10[12-17].Jul 5 2023, 10:52 AM
KartikMistry added a comment.Jul 5 2023, 11:19 AM
In T341117#8989921, @Marostegui wrote:

This might be related to T337812 I have failed back to dbproxy1017 until it is investigated which FW are needed for that new host.
Please revert https://gerrit.wikimedia.org/r/935692 once it is fixed.

@KartikMistry allow 5-10 minutes for the DNS to spread.

Working fine in the Europe region now.

Marostegui added a comment.Jul 5 2023, 12:02 PM

Cool, once the FW has been changed, we'd need to revert that patch and confirm it keeps working as expected. Thanks for the heads up and sorry for the inconveniences!

gerritbot added a comment.Jul 5 2023, 1:47 PM

Change 935746 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] modules: Add a new networkpolicy for base modules

https://gerrit.wikimedia.org/r/935746

gerritbot added a project: Patch-For-Review.Jul 5 2023, 1:47 PM
gerritbot added a comment.Jul 5 2023, 1:54 PM

Change 935748 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] cxserver: Bump to networkpolicy_1.1.0.tpl

https://gerrit.wikimedia.org/r/935748

gerritbot added a comment.Jul 5 2023, 1:54 PM

Change 935749 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] cxserver: Migrate to the new MariaDB egress functionality

https://gerrit.wikimedia.org/r/935749

akosiaris moved this task from Incoming 🐫 to Doing 😎 on the serviceops-deprecated board.Jul 6 2023, 3:23 PM
KartikMistry changed the task status from Open to In Progress.Jul 10 2023, 8:21 AM
KartikMistry moved this task from Quarter Backlog to In Progress on the Language-Team (Language-2023-July-September) board.
Nikerabbit edited projects, added Language-Team; removed Language-Team (Language-2023-July-September).Aug 14 2023, 9:05 AM
Nikerabbit moved this task from Backlog to Other teams/Watching on the Language-Team board.
Nikerabbit moved this task from Backlog to Other teams on the CX-cxserver board.Aug 14 2023, 10:42 AM
gerritbot added a comment.Aug 29 2023, 2:38 PM

Change 935746 merged by jenkins-bot:

[operations/deployment-charts@master] modules: Add a new networkpolicy for base modules

https://gerrit.wikimedia.org/r/935746

gerritbot added a comment.Aug 29 2023, 2:43 PM

Change 935748 merged by jenkins-bot:

[operations/deployment-charts@master] cxserver: Bump to networkpolicy_1.1.0.tpl

https://gerrit.wikimedia.org/r/935748

gerritbot added a comment.Aug 29 2023, 4:07 PM

Change 935749 merged by jenkins-bot:

[operations/deployment-charts@master] cxserver: Migrate to the new MariaDB egress functionality

https://gerrit.wikimedia.org/r/935749

Stashbot added a comment.Aug 29 2023, 4:09 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-29T16:09:43Z] <akosiaris> deploy cxserver mariadb egress functionality. T341117

Maintenance_bot removed a project: Patch-For-Review.Aug 29 2023, 4:10 PM
akosiaris closed this task as Resolved.Aug 29 2023, 4:36 PM

Fix merged and deployed. Some hiccups aside, it works fine across all 3 environments (staging, production eqiad, production codfw). I 'll resolve this one.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits