Page MenuHomePhabricator

Andrew gets pages many hours too late
Closed, ResolvedPublic

Description

When icinga pages, I don't get the pages for a long time, often 5, 6 or 7 hours late. Not only are these pages useless, but they tend to arrive in the middle of the night since I have my paging hours set to daytime.

Event Timeline

Andrew raised the priority of this task from to Needs Triage.
Andrew updated the task description. (Show Details)
Andrew added a project: SRE.
Andrew subscribed.

Note from the attached screenshot -- the pages themselves are dated 1:17 UTC, but they are arriving at my phone at 1:31AM CST, which is UTC-5. So, 5:10 minutes after they fired.

Note from the attached screenshot -- the pages themselves are dated 1:17 UTC, but they are arriving at my phone at 1:31AM CST, which is UTC-5. So, 5:10 minutes after they fired.

Just an uneducated guess; you don't have "Low power mode" or somthing like that turned on? My phone delevers my mail ~1-2 hours late due to me having low-power mode turned on 24/7 to double it's battery life.

What path are you using? I use icinga -> smtp -> imap -> k9 mail. Are you using an smtp to sms gateway?

The path used by our US staff is icinga dispatches an email to the phone.number@cellularprovider.to.sms.address. This allows us to send the notices for no cost to WMF (as the US cellular providers provide that gateway as part of their services.) When we send pages to our non USA based opsen, we do so via AQL. AQL does the same thing (email to sms gateway) but we have to pay per message.

@Andrew is using the same provider gateway as I am, and I'm not receiving any pages late (that I've seen.) I'd advise this issue is specific to his phone #, phone, or coverage area.

@Andrew: If you send an email to yourfullcellnumber@tmomail.net (if tmobile, or @txt.att.net if att, or @vtext.com if verizon) it should hit your phone. You can test to see if it is delayed via those means, please advise?

Example: If you phone is on tmoble with the # of +1.222.333.4444 it would be 12223334444@tmomail.net.

Andrew claimed this task.

@RobH, you set me on the right path. I recently switched to google fi, which apparently still gets pages sent to the tmomail gateway but not for a long time. I switched to msg.fi.google.com and anticipate better service.

Oh, notably, msg.fi.google.com didn't exist until last week, approximately when my pages starting being delayed. Surely no coincidence; maybe google degraded their support of the tmo gateway when they set up their own.