Page MenuHomePhabricator
Create Task
Maniphest T358020

Not receiving posts or moderation messages
Closed, ResolvedPublic
Actions

Description

The last message I received was at Feb 19, 09:56 +0000, but I can see posts after that in list archives.

Email provider: Gmail

Previous: T331633: Not receiving posts or moderation messages

Related Objects

Event Timeline

JJMC89 created this task.Feb 20 2024, 4:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 20 2024, 4:46 PM
JJMC89 added a project: SRE.Feb 20 2024, 4:55 PM
JJMC89 added subscribers: Ladsgroup, Legoktm.
Yahya subscribed.Feb 21 2024, 4:20 PM
Faendalimas subscribed.Feb 21 2024, 4:55 PM
MdsShakil subscribed.Feb 21 2024, 5:00 PM
Legoktm added a comment.Feb 22 2024, 12:39 AM

queue runner seems to have crashed, based on https://grafana.wikimedia.org/d/GvuAmuuGk/mailman3?orgId=1&viewPanel=2&from=now-2d&to=now

Screenshot 2024-02-21 at 19-38-57 View panel - Mailman3 - Dashboards - Grafana.png (940×1 px, 128 KB)

trying to flag down an SRE

RLazarus closed this task as Resolved.Feb 22 2024, 12:47 AM
RLazarus claimed this task.
RLazarus subscribed.

Restarted mailman3 at 00:43, icinga alerts are cleared, and the graph in T358020#9565952 is trending down again. Thanks @JJMC89 for the report and thanks @Legoktm for the ping.

Optimistically resolving, though it'll take some time to churn through the queue.

JJMC89 added a project: Wikimedia-Incident.Feb 22 2024, 12:47 AM

Looks the same as a previous Wikimedia-Incident T331626: Mailman hasn't delivered emails since 2023-03-07 14 UTC (was: reviewer-bot is not working)

jnuche mentioned this in T358009: No email received for latest train presync.Feb 22 2024, 10:24 AM
taavi merged a task: T358198: Message-ID: <CACNAgmwRgLo4Uy0w9ZtAi07kB_rA9wxC5T3p+kr+ftWDUq7hWg@mail.gmail.com> delayed by 3 days.Feb 22 2024, 10:49 AM
taavi added a subscriber: saper.
saper added a comment.Feb 22 2024, 11:17 AM
In T358020#9565952, @Legoktm wrote:

queue runner seems to have crashed

Do we know why did it crash? In T331626 it seems the software lost access to MySQL due to the network reconfiguration.

bking subscribed.Feb 22 2024, 3:43 PM
matmarex merged a task: T358198: Message-ID: <CACNAgmwRgLo4Uy0w9ZtAi07kB_rA9wxC5T3p+kr+ftWDUq7hWg@mail.gmail.com> delayed by 3 days.Feb 22 2024, 4:08 PM
matmarex subscribed.
JJMC89 moved this task from Active investigation to Awaiting report on the Wikimedia-Incident board.Mar 2 2024, 7:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL