generate_vrts_aliases failing on mx-in1001
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	LSobanski
	Jun 24 2024, 11:12 AM

Project Tags

Referenced Files

None

Subscribers

Description

Despite recent changes in T284145: Clean up OTRS/Znuny addresses handles by gsuite generate_vrts_aliases is still failing on mx-in1001 almost daily at different times.

[2024-06-24 10:53:48] <jinxer-wm> FIRING: [2x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[2024-06-24 11:08:48] <jinxer-wm> FIRING: [2x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[2024-06-24 11:15:49] <jinxer-wm> RESOLVED: [2x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	vrts_aliases: add retry logic	JHathaway	operations/puppet	production	+59 -43
	Filter out addresses that cannot be removed from VRTS	LSobanski	operations/puppet	production	+2 -0

Customize query in gerrit

Related Objects

Mentioned In: T373485: Dissociate/release old iOS and Android support email addresses (currently VRTS queues)
Mentioned Here: T373485: Dissociate/release old iOS and Android support email addresses (currently VRTS queues)
T284145: Clean up OTRS/Znuny addresses handles by gsuite

Event Timeline

LSobanski created this task.Jun 24 2024, 11:12 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 24 2024, 11:12 AM

jhathaway claimed this task.Jun 24 2024, 1:57 PM

jhathaway triaged this task as Medium priority.Jun 24 2024, 2:45 PM

LSobanski moved this task from Incoming to Consultation on the Collaboration-Services board.Jun 24 2024, 3:13 PM

I was able to capture this traceback:

Traceback (most recent call last):
  File "/home/jhathaway/./vrts_aliases", line 162, in main
    if verify_email(row[0], config["DEFAULT"]["smtp_server"]):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jhathaway/./vrts_aliases", line 41, in verify_email
    smtp.connect(smtp_server)
  File "/usr/lib/python3.11/smtplib.py", line 343, in connect
    (code, msg) = self.getreply()
                  ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/smtplib.py", line 405, in getreply
    raise SMTPServerDisconnected("Connection unexpectedly closed")
smtplib.SMTPServerDisconnected: Connection unexpectedly closed

I'm not sure why are connection is getting closed, or why this would only appear on mx-in hosts, perhaps the bookworm upgrade? We may want to add some retry logic.

It would be useful to know why it failed (maybe on the server's logs?), but +1 to adding a retry logic regardless.

Since August 30th the script is failing once every 2-5 minutes.

@LSobanski the current failures are because there are 2 email addresses in the config that are managed by gsuite (I've redacted some part):

vrts_aliases[1485203]: ERROR:/usr/local/bin/vrts_aliases:email is handled by gsuite: ****-wikipedia@wikimedia.org
vrts_aliases[1485203]: ERROR:/usr/local/bin/vrts_aliases:email is handled by gsuite: ****-wikipedia@wikimedia.org

Change #1070222 had a related patch set uploaded (by LSobanski; author: LSobanski):

[operations/puppet@production] Filter out addresses that cannot be removed from VRTS

https://gerrit.wikimedia.org/r/1070222

gerritbot added a project: Patch-For-Review.Sep 3 2024, 10:30 AM

Change #1070222 merged by Dzahn:

[operations/puppet@production] Filter out addresses that cannot be removed from VRTS

https://gerrit.wikimedia.org/r/1070222

In T368257#10112246, @Volans wrote:

@LSobanski the current failures are because there are 2 email addresses in the config that are managed by gsuite (I've redacted some part):

vrts_aliases[1485203]: ERROR:/usr/local/bin/vrts_aliases:email is handled by gsuite: **-wikipedia@wikimedia.org
vrts_aliases[1485203]: ERROR:/usr/local/bin/vrts_aliases:email is handled by gsuite: **-wikipedia@wikimedia.org

This is T373485 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1070222 was just merged.

But I would like to point out this check was already alerting before for some other reason as well.

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2024, 5:31 PM

Looks like the last alert was at 2:42 UTC so the patch likely wasn't needed after all.

The patch was needed, the last error was at Sep 03 17:08:32. After that it run smoothly except for one run at Sep 03 20:10:57 that failed with ERROR:/usr/local/bin/vrts_aliases:Connection unexpectedly closed.

So looks like we're back to the original problem only.

Change #1070671 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] vrts_aliases: add retry logic

https://gerrit.wikimedia.org/r/1070671

gerritbot added a project: Patch-For-Review.Sep 4 2024, 9:28 PM

Dzahn mentioned this in T373485: Dissociate/release old iOS and Android support email addresses (currently VRTS queues).Sep 5 2024, 6:21 AM

Change #1070671 merged by JHathaway:

[operations/puppet@production] vrts_aliases: add retry logic

https://gerrit.wikimedia.org/r/1070671

retry logic has been added which should resolve the issue, please reopen if it occurs again.

Maintenance_bot removed a project: Patch-For-Review.Sep 9 2024, 4:31 PM