I was doing updates this morning and noticed that the fr-log-announcer was announcing logins to payments2xxx hosts twice instead of just a single time like the other codfw hosts. It should only announce these once.
Description
Related Objects
- Mentioned Here
- T272150: review civi orphan rectifier log use
Event Timeline
OK, I think I have tracked this down. In the codfw payments hosts, they are configured to send their logs to the local and remote central syslog servers. Payments hosts in eqiad only send to their local central syslog server. Thus, when you log in to a payments2xxx host, the tagged message is sent to each central syslog host and both send the message on to the location where fr-log-announcer picks it up. Will need to think about what the right answer is here to ensure we have logs in the appropriate places and aren't getting duplicate data/reports.
This was fixed by ceasing the duplicate logging to both loggers, which was a hack to support orphan rectification when we were running on the codfw payments servers, which is not our normal mode of operation. There's a separate task T272150: review civi orphan rectifier log use to review log use in orphan rectification.