Page MenuHomePhabricator

fr-log-announcer reporting logins to payments2xxx servers multiple times
Closed, ResolvedPublic

Description

I was doing updates this morning and noticed that the fr-log-announcer was announcing logins to payments2xxx hosts twice instead of just a single time like the other codfw hosts. It should only announce these once.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 27 2020, 6:03 PM

OK, I think I have tracked this down. In the codfw payments hosts, they are configured to send their logs to the local and remote central syslog servers. Payments hosts in eqiad only send to their local central syslog server. Thus, when you log in to a payments2xxx host, the tagged message is sent to each central syslog host and both send the message on to the location where fr-log-announcer picks it up. Will need to think about what the right answer is here to ensure we have logs in the appropriate places and aren't getting duplicate data/reports.

Jgreen closed this task as Resolved.Tue, Feb 23, 2:52 PM
Jgreen claimed this task.
Jgreen moved this task from Backlog to Done on the fundraising-tech-ops board.
Jgreen added a subscriber: Jgreen.

This was fixed by ceasing the duplicate logging to both loggers, which was a hack to support orphan rectification when we were running on the codfw payments servers, which is not our normal mode of operation. There's a separate task T272150: review civi orphan rectifier log use to review log use in orphan rectification.