Page MenuHomePhabricator

Add maint-announce@ to Equinix's recipient list for eqsin incidents
Closed, ResolvedPublic


During the T206861 incident, it was discovered that the scheduled maintenance notices were only addressed to the personal email addresses of a handful of people (Mark, Brandon, myself, Rob, Arzhel).

These notices should go to maint-announce@ instead. I didn't find an obvious way to do this in the panel, so unless I missed it, we should probably ask Equinix about it.

(h/t to @ArielGlenn for double checking and discovering this!)

Event Timeline

faidon triaged this task as High priority.Oct 16 2018, 9:14 AM
faidon created this task.
RobH added a comment.Oct 16 2018, 5:37 PM

Ok, this is an easy fix.

I logged into the portal, and then selected modify users, selected the maint announce user, and can see it had notifications set for DC6, CH2, DA1, but not SG3 or SV9, I set it for ALL of them (except SV9 which shows in the list but is not check box selectable, due to being just peering?)

RobH added a comment.Oct 16 2018, 5:37 PM

seems I'm not set to modify permissions for SV9, thats why I cannot adjust. I'll email our rep about it I suppose, but its our odd peering link.

RobH closed this task as Resolved.Oct 17 2018, 4:27 PM

synced with @faidon, sv9 is the virtual site for the eq peering link and thus won't have real notices. the notices for any issues on it are sent with the IX notices.

So maint announce is now listed for all our sites with EQ from my update on the 16th.

faidon reopened this task as Open.Oct 25 2018, 1:07 PM

I see emails for SG3 that (as far as I can tell) haven't made it to maint-announce, e.g.

Date: Thu, 25 Oct 2018 13:01:14 +0000 (UTC)
From: Equinix Maintenance NO-REPLY <>
Subject: REMINDER - Scheduled UPS Power Capacity Upgrade at the SG3 IBX  [5-168456259310]
ema moved this task from Triage to General on the Traffic board.Oct 26 2018, 8:58 AM
RobH added a comment.Nov 5 2018, 4:16 PM

So the above screen shot is the settings for maint announce's user in the eq portal permissions, however, under the notification preferences, it was lacking IBX notices!

I hadn't really noticed this second tab, so I'm also reviewing everyone else's settings to normalize them.

RobH added a comment.Nov 5 2018, 4:17 PM

Actually, I take that back, IBX advisory is what we want and it is checked...

RobH added a comment.Nov 5 2018, 4:30 PM

I emailed our rep:

We currently have all our maint announcements sent over to <redacted>. However, while we're getting our Equinix North America notices, we don't seem to be getting emails from SG3 to <redacted>, while I am still getting them to my direct email <redacted>.
I like getting them directly, but we need to fix it so our generic inbox gets them as well. I've previously checked off IBX notifications in the portal for the <redacted> user, but we have not gotten notices to it that I've gotten directly into my inbox.
Example: I recently received an email from EQ SG3 subject "REMINDER - Scheduled UPS Power Capacity Upgrade at the SG3 IBX [5-168456259310]", but the <redacted> user did not receive it, when it has the same notification settings as I do in the portal.
Can you advise if I missed a checkbox somewhere, or what can be fixed so the <redacted> user also gets all notices for SG3 that I get?
Please advise,

RobH added a comment.Nov 5 2018, 8:00 PM

Please be informed that today (6th Nov 2018) is a Public Holiday in Singapore and i have very limited access to my email. So, please expect some delay in my response/reply.
For any urgent matter, please whatsapp/call me at <redacted>.

RobH changed the task status from Open to Stalled.Nov 9 2018, 4:13 PM
RobH lowered the priority of this task from High to Low.

Vivian @ EQ Singapore fixed it, adding in maint announce to their alerts. We should get the next alert going forward: should be able to receive all network notifications moving forward.

I'll leave this open, low priority, and stalled until we see a notice come in.

RobH added a comment.Nov 15 2018, 6:36 PM

Today I recieved a notice about "COMPLETED - Scheduled Generator Capacity Upgrade at the SG3 IBX [5-168459376275]" to my email, but we did not receive it to
Can you advise why this did not arrive to

RobH added a comment.Nov 16 2018, 3:58 PM

Hi Rob
I’m raising to our IT to check on this.
Please bear with us while they investigate on it.
Best regards
Vivien Ho (Ms)

RobH added a comment.Nov 21 2018, 4:48 PM

There has been no notices from eq singapore to test this, and last note was from Vivian's update on Nov 16th. I emailed a reply in just now asking for update:

RobH added a comment.EditedDec 6 2018, 11:11 PM

So, as of December 6th, there are no new emails to/from (everyone else on the list is bcc) since our last email/update request to Vivian.

Once we have one, we should see it hit maint announce as well as my inbox.

There were a few notices on the 15th and 16th of December. Did these arrive to maint-announce@?

RobH closed this task as Resolved.Dec 18 2018, 4:45 PM

So yesterday eqsin sent out the email subject: COMPLETED - Non-Critical Upstream Provider Maintenance-SG Metro Area Network Maintenance -16-DEC-2018 [5-179367337587] and it arrived to the maint announce group successfully. Resolving this task since now we get these notices to both my inbox (like before) and now properly receive independently to maint-announce group.