Page MenuHomePhabricator

Connecting to Wiki Replicas from whgi.wikidumpparse.eqiad.wmflabs intermittently gives SSL error
Open, Needs TriagePublic

Description

A self-tracking bug.
From VPS whgi.wikidumpparse.eqiad.wmflabs. Occassionally, for a period of a few days at a time, I am unable to connect to databases replicas, receiving errors: ERROR 2026 (HY000): SSL connection error: unknown error number. Then it will fix itself.

Twice this has happened.

  • in 2019/4 (approx 3 days)
  • in 2019/6/19 - 2019/6/24
    • during this period adding --ssl-mode Disabled fixed the problem as in, mysql -h enwiki.analytics.db.svc.eqiad.wmflabs -u u2443 -p --ssl-mode Disabled

Using this task to track more outages, to see if I can find a pattern. It may be local to the machine, not sure.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 25 2019, 12:48 AM

Hi @notconfusing. Can you please associate at least one active project with this task (via the Add Action...Change Project Tags dropdown). This will allow others to get notified and see this task when looking at the corresponding project workboard. Thanks.

Hi @notconfusing I think this might be related to the mysql-community-client 5.7.25-1debian9 package you have installed on whgi. The mysql community builds are bundled with yaSSL[0] and the wiki replicas are using 10.1.39-MariaDB which is using openSSL.

I tested today and the whgi server is currently unable to connect due to STARTTLS handshake errors, which leads me to the SSL library compatibility issue. However, this doesn't really explain why it's intermittently working. It could be related to the upgrades and maintenance activity on the wiki replica servers this month[1].

[0] https://dev.mysql.com/doc/refman/5.7/en/ssl-libraries.html
[1] T224852 https://wikitech.wikimedia.org/wiki/Server_Admin_Log

bd808 renamed this task from Connecting to Databases Replicas intermittently gives SSL error to Connecting to Wiki Replicas from whgi.wikidumpparse.eqiad.wmflabs intermittently gives SSL error.Jul 1 2019, 3:38 AM
bd808 edited projects, added VPS-Projects; removed Cloud-VPS.
bd808 updated the task description. (Show Details)