Add maint-announce@ to Equinix's recipient list for eqsin incidents
Open, Stalled, LowPublic

Description

During the T206861 incident, it was discovered that the scheduled maintenance notices were only addressed to the personal email addresses of a handful of people (Mark, Brandon, myself, Rob, Arzhel).

These notices should go to maint-announce@ instead. I didn't find an obvious way to do this in the panel, so unless I missed it, we should probably ask Equinix about it.

(h/t to @ArielGlenn for double checking and discovering this!)

faidon created this task.Oct 16 2018, 9:14 AM
faidon triaged this task as High priority.
RobH added a comment.Oct 16 2018, 5:37 PM

Ok, this is an easy fix.

I logged into the portal, and then selected modify users, selected the maint announce user, and can see it had notifications set for DC6, CH2, DA1, but not SG3 or SV9, I set it for ALL of them (except SV9 which shows in the list but is not check box selectable, due to being just peering?)

RobH added a comment.Oct 16 2018, 5:37 PM

seems I'm not set to modify permissions for SV9, thats why I cannot adjust. I'll email our rep about it I suppose, but its our odd peering link.

RobH closed this task as Resolved.Oct 17 2018, 4:27 PM

synced with @faidon, sv9 is the virtual site for the eq peering link and thus won't have real notices. the notices for any issues on it are sent with the IX notices.

So maint announce is now listed for all our sites with EQ from my update on the 16th.

faidon reopened this task as Open.Oct 25 2018, 1:07 PM

I see emails for SG3 that (as far as I can tell) haven't made it to maint-announce, e.g.

Date: Thu, 25 Oct 2018 13:01:14 +0000 (UTC)
From: Equinix Maintenance NO-REPLY <no-reply@equinix.com>
To: EquinixMaintenance.SG@ap.equinix.com
Subject: REMINDER - Scheduled UPS Power Capacity Upgrade at the SG3 IBX  [5-168456259310]
ema moved this task from Triage to General on the Traffic board.Oct 26 2018, 8:58 AM
RobH added a comment.Nov 5 2018, 4:16 PM

So the above screen shot is the settings for maint announce's user in the eq portal permissions, however, under the notification preferences, it was lacking IBX notices!

I hadn't really noticed this second tab, so I'm also reviewing everyone else's settings to normalize them.

RobH added a comment.Nov 5 2018, 4:17 PM

Actually, I take that back, IBX advisory is what we want and it is checked...

RobH added a comment.Nov 5 2018, 4:30 PM

I emailed our rep:

Shaun,

We currently have all our maint announcements sent over to <redacted>. However, while we're getting our Equinix North America notices, we don't seem to be getting emails from SG3 to <redacted>, while I am still getting them to my direct email <redacted>.

I like getting them directly, but we need to fix it so our generic inbox gets them as well. I've previously checked off IBX notifications in the portal for the <redacted> user, but we have not gotten notices to it that I've gotten directly into my inbox.

Example: I recently received an email from EQ SG3 subject "REMINDER - Scheduled UPS Power Capacity Upgrade at the SG3 IBX [5-168456259310]", but the <redacted> user did not receive it, when it has the same notification settings as I do in the portal.

Can you advise if I missed a checkbox somewhere, or what can be fixed so the <redacted> user also gets all notices for SG3 that I get?

Please advise,

RobH added a comment.Nov 5 2018, 8:00 PM
Please be informed that today (6th Nov 2018) is a Public Holiday in Singapore and i have very limited access to my email. So, please expect some delay in my response/reply.

For any urgent matter, please whatsapp/call me at <redacted>.

Thanks,
Shaun

RobH changed the task status from Open to Stalled.Nov 9 2018, 4:13 PM
RobH lowered the priority of this task from High to Low.

Vivian @ EQ Singapore fixed it, adding in maint announce to their alerts. We should get the next alert going forward:

maint-announce@wikimedia.org should be able to receive all network notifications moving forward.

I'll leave this open, low priority, and stalled until we see a notice come in.

RobH added a comment.Thu, Nov 15, 6:36 PM

Vivian,

Today I recieved a notice about "COMPLETED - Scheduled Generator Capacity Upgrade at the SG3 IBX [5-168459376275]" to my rhalsell@wikimedia.org email, but we did not receive it to maint-announce@wikimedia.org

Can you advise why this did not arrive to maint-announce@wikimedia.org?

RobH added a comment.Fri, Nov 16, 3:58 PM

Hi Rob

I’m raising to our IT to check on this.

Please bear with us while they investigate on it.

Best regards

Vivien Ho (Ms)

RobH added a comment.Wed, Nov 21, 4:48 PM

There has been no notices from eq singapore to test this, and last note was from Vivian's update on Nov 16th. I emailed a reply in just now asking for update:

RobH added a comment.EditedThu, Dec 6, 11:11 PM

So, as of December 6th, there are no new emails to/from (everyone else on the list is bcc) EquinixMaintenance.SG@ap.equinix.com since our last email/update request to Vivian.

Once we have one, we should see it hit maint announce as well as my inbox.