Page MenuHomePhabricator

OTRS/mail: investigate why "T=remote_smtp_signed: all hosts for 'ticket.wikimedia.org' have been failing for a long time"
Open, LowPublic

Description

found by @faidon during investigation for unrelated incident T297127:

2021-12-03 23:25:17 1msFUH-0050x2-0l ** otrs@ticket.wikimedia.org R=dnslookup  T=remote_smtp_signed: all hosts for 'ticket.wikimedia.org' have been failing for a long time (and retry time not reach

investigate how this is related to OTRS renaming to VTRS / Znuny (T280400 ?) and whether this is still an issue and needs a fix somewhere

Event Timeline

My understanding per T225623#5253119 is that @ticket.wikimedia.org is not a valid domain we accept mail on.

Had a quick look at that. It is true that we never have received email on @ticket.wikimedia.org. It is only the web host that has that identifier. However, that identifier is also know to the system (it's the FQDN Znuny/OTRS setting[1]). And it looks like the Znuny/OTRS daemon has been periodically mailing us various errors that it has been meeting. I have found quite a few of e.g.

Loaded DateTime::TimeZone::Europe::Moscow, which is from a different versio=
n (2021e) of the Olson database than this installation of DateTime::TimeZon=
e (2021b).

which essentially says that the daemon hasn't been reloaded since the upgrades have been applied. I had found those out via an unrelated path (the Znuny System Log) some 38 days ago (oldest such mail in the queue is 43d) and figured out the issue and reloaded the daemon fixing the issue. But alas, the email remained in the mx1001's queue, eventually frozen. There is also 1 about reinstall the various packages, a result of a Znuny upgrade.

I 've removed the messages from the queue by now, but this will probably happen again. It won't cause issues to the infrastructure, that much is clear, but we probably don't want to see those at all.

I would expect the AdminEmail config setting to be the recipient for these, which we have set to otrs-admins@lists.wikimedia.org, but the emails I found have a recipient of znuny_lts@ticket.wikimedia.org, which is puzzling.

I 've started digging at the code to figure out what is going on.

[1] https://doc.otrs.com/doc/manual/admin/6.0/en/html/ConfigReference_Section_Core.html

FQDN
Defines the fully qualified domain name of the system. This setting is used as a variable, OTRS_CONFIG_FQDN which is found in all forms of messaging used by the application, to build links to the tickets within your system.

This setting can not be deactivated.
akosiaris moved this task from Incoming to Backlog on the Znuny board.

Code found. https://github.com/znuny/Znuny/blob/dev/Kernel/System/Daemon/DaemonModules/BaseTaskWorker.pm#L57-L71

The config vars NotificationSenderName and NotificationSenderEmail are relevant to our case and they are set to values that don't match up to what I found on mx1001.

That is we have them set up to:

NotificationSenderName: VRTS Notifications
NotificationSenderEmail: vrts@<OTRS_CONFIG_FQDN>

whereas what I found on mx1001 was From: Znuny LTS Notifications <znuny_lts@ticket.wikimedia.org>, which seems to be the default values. Someone must have changed the config since the time we witnessed that.

I am inclined to lower priority and monitor this for a while in order to catch it in a more recent and easy to reason about situation.