Page MenuHomePhabricator
Create Task
Maniphest T349961

PhutilAggregateException: #2006: MySQL server has gone away
Closed, ResolvedPublicBUG REPORT
Actions

Description

Steps to replicate the issue (include links if applicable):

What happens?:

PhutilAggregateException
Encountered a processing exception, then another exception when trying to build a response for the first exception.
- PhabricatorClusterStrandedException: Unable to establish a connection to any database host (while trying "phabricator_search"). All masters and replicas are completely unreachable.

AphrontConnectionLostQueryException: #2006: MySQL server has gone away

This error may occur if your configured MySQL "wait_timeout" or "max_allowed_packet" values are too small. This may also indicate that something used the MySQL "KILL <process>" command to kill the connection running the query.
- PhabricatorClusterStrandedException: Unable to establish a connection to any database host (while trying "phabricator_search"). All masters and replicas are completely unreachable.

AphrontConnectionLostQueryException: #2006: MySQL server has gone away

This error may occur if your configured MySQL "wait_timeout" or "max_allowed_packet" values are too small. This may also indicate that something used the MySQL "KILL <process>" command to kill the connection running the query.
PhabricatorClusterStrandedException
Unable to establish a connection to any database host (while trying "phabricator_search"). All masters and replicas are completely unreachable.

AphrontConnectionLostQueryException: #2006: MySQL server has gone away

This error may occur if your configured MySQL "wait_timeout" or "max_allowed_packet" values are too small. This may also indicate that something used the MySQL "KILL <process>" command to kill the connection running the query.
PhabricatorClusterStrandedException
Unable to establish a connection to any database host (while trying "phabricator_search"). All masters and replicas are completely unreachable.

AphrontConnectionLostQueryException: #2006: MySQL server has gone away

This error may occur if your configured MySQL "wait_timeout" or "max_allowed_packet" values are too small. This may also indicate that something used the MySQL "KILL <process>" command to kill the connection running the query.

HTML body:

<body class="unhandled-exception"><div class="unhandled-exception-detail"><h1 class="unhandled-exception-title">PhutilAggregateException</h1><div class="unhandled-exception-body">Encountered a processing exception, then another exception when trying to build a response for the first exception.<br>
    - PhabricatorClusterStrandedException: Unable to establish a connection to any database host (while trying "phabricator_search"). All masters and replicas are completely unreachable.<br>
      <br>
      AphrontConnectionLostQueryException: #2006: MySQL server has gone away<br>
      <br>
      This error may occur if your configured MySQL "wait_timeout" or "max_allowed_packet" values are too small. This may also indicate that something used the MySQL "KILL &lt;process&gt;" command to kill the connection running the query.<br>
    - PhabricatorClusterStrandedException: Unable to establish a connection to any database host (while trying "phabricator_search"). All masters and replicas are completely unreachable.<br>
      <br>
      AphrontConnectionLostQueryException: #2006: MySQL server has gone away<br>
      <br>
      This error may occur if your configured MySQL "wait_timeout" or "max_allowed_packet" values are too small. This may also indicate that something used the MySQL "KILL &lt;process&gt;" command to kill the connection running the query.</div></div><div class="unhandled-exception-detail"><h1 class="unhandled-exception-title">PhabricatorClusterStrandedException</h1><div class="unhandled-exception-body">Unable to establish a connection to any database host (while trying "phabricator_search"). All masters and replicas are completely unreachable.<br>
<br>
AphrontConnectionLostQueryException: #2006: MySQL server has gone away<br>
<br>
This error may occur if your configured MySQL "wait_timeout" or "max_allowed_packet" values are too small. This may also indicate that something used the MySQL "KILL &lt;process&gt;" command to kill the connection running the query.</div></div><div class="unhandled-exception-detail"><h1 class="unhandled-exception-title">PhabricatorClusterStrandedException</h1><div class="unhandled-exception-body">Unable to establish a connection to any database host (while trying "phabricator_search"). All masters and replicas are completely unreachable.<br>
<br>
AphrontConnectionLostQueryException: #2006: MySQL server has gone away<br>
<br>
This error may occur if your configured MySQL "wait_timeout" or "max_allowed_packet" values are too small. This may also indicate that something used the MySQL "KILL &lt;process&gt;" command to kill the connection running the query.</div></div>
</body>

What should have happened instead?:
Phabricator

Event Timeline

AlexisJazz created this task.Oct 28 2023, 5:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 28 2023, 5:51 PM
AlexisJazz updated the task description. (Show Details)Oct 28 2023, 5:53 PM
Aklapper renamed this task from PhutilAggregateException to PhutilAggregateException: #2006: MySQL server has gone away.Oct 28 2023, 6:24 PM
Kizule subscribed.Oct 28 2023, 6:27 PM

I've faced this as well while I was using search, but after I refreshed the page, it went away.
As I'm not the only one who has faced this issue, there must be something with MySQL.

Lucas_Werkmeister_WMDE added subscribers: Dreamy_Jazz, Lucas_Werkmeister_WMDE.Oct 30 2023, 4:54 PM

Still happening, I got it twice today:

image.png (926×1 px, 148 KB)

@Dreamy_Jazz also reported it in the operations channel.

AlexisJazz added a comment.EditedOct 30 2023, 6:38 PM

Taking a shot in the dark: maybe in the transition from Phabricator to Phorge (or some other major change, but this was relatively recent) one of the servers didn't get the memo, so if the loadbalancer sends you to that server you get the error, but it resolves when you refresh the page as the loadbalancer takes you elsewhere?

Just thinking out loud, no offense taken if the next comment debunks this theory.

I haven't seen it again, but I'm not super active here. Btw, despite the error saying "while trying phabricator_search", I just tried to visit the homepage. I wasn't searching. (but perhaps the homepage executes phabricator_search)

thcipriani subscribed.Oct 30 2023, 6:45 PM

Thanks for the task! Doubtful it's Phorge since nothing has changed there recently and really it's just one big server serving everyone.

Either something's up with mysql or there's an abnormal amount of traffic. We are seeing some traffic spikes: https://grafana.wikimedia.org/d/000000587/phabricator?orgId=1&from=now-2d&to=now&viewPanel=23

Still investigating.

thcipriani closed this task as Resolved.Oct 30 2023, 7:46 PM
thcipriani assigned this task to RLazarus.

Added a requestctl rule to block some aggressive/expensive crawling that accounted for about ⅓ of our daily traffic and > ½ our traffic over the past few hours.

Traffic graphs look much happier in the immediate aftermath (far right of the graph where traffic halves):

2023-10-30_crawler-traffic.png (358×727 px, 40 KB)

Optimistically calling this resolved while we work on other mitigations. Please re-open if you keep seeing this error.

kostajh subscribed.Oct 30 2023, 7:49 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL