@TAndic from reading their docs, I think they only support sending from a subdomain of wikimedia.org:

Attention: Adding a custom FROM domain will allow any user within your organization to distribute emails from any email from that domain. Main company domains (e.g. mycompany.com) should not be used as a custom FROM domain in Qualtrics. We require only one MX record be associated with the custom from domain, so a subdomain (e.g. surveys.mycompany.com) is almost always required. Adding the Qualtrics MX record to your main company domain may affect deliverability.

so perhaps qualtrics.wikimedia.org?

If using a subdomain is not desired the other route would be to use their SMTP relay option, which would require some changes to our mail server config. I don't think those changes would be too difficult, but we don't currently have authenticated relay setup in our mail server config, so their may be some tricky bits.

@jhathaway Good catch! I've set up one for surveys.wikimedia.org and updated the configuration in the sheet I shared, starting on line 11.

However, after reading the SMTP documentation, I'm wondering if it would make sense to use SMTP (or set up both?) as it would allow control on our end for which WMF Qualtrics accounts are able to use a wikimedia.org email as their sending email (as we have both staff accounts and community user accounts).

In T314815#8139092, @TAndic wrote:

@jhathaway Good catch! I've set up one for surveys.wikimedia.org and updated the configuration in the sheet I shared, starting on line 11.

thanks

However, after reading the SMTP documentation, I'm wondering if it would make sense to use SMTP (or set up both?) as it would allow control on our end for which WMF Qualtrics accounts are able to use a wikimedia.org email as their sending email (as we have both staff accounts and community user accounts).

It is unclear to me from reading their documentation whether their is a difference in user access control between the SMTP and Custom FROM Domain options. My read is that with either setup any user could send an email originating from the address we set, e.g. noreply@surveys.wikimedia.org for a Custom FROM Domain or noreply@wikimedia.org with the SMTP option.

For the difference, I'm specifically looking at Step 13 of https://www.qualtrics.com/support/survey-platform/distributions-module/email-distribution/using-a-custom-from-address/#SettingUpAnSMTPRelay for SMTP, which seems to imply you can selectively choose users who can use this. Whereas in the Custom Email domains setting on my end there is a note: "Send emails through Qualtrics using your organization’s custom email address; domains can’t be restricted from user to user".

I apologize that this is becoming a bigger rabbit hole than anticipated -- the SMTP introduction linked also notes that "Before January 26, 2022, all SMTP credentials were set up by representatives at Qualtrics. SMTP credentials added by a Qualtrics representative will not appear on the Extensions page."
I can also see in our setting that there is an addition by Qualtrics for backwards compatibility:

I'm wondering if it's possible that we had SMTP set up by Qualtrics (our account was created, to my knowledge, about 10 years ago?), and now it's a different update needed that isn't the Custom From Domain?

This entire adventure started with an email from Qualtrics noting:

Hello,

This is an email regarding an upcoming change to the Qualtrics data center IP space in the Canada region. You are receiving this email because your brand is currently located in our Toronto data center region in Canada (ca1.qualtrics.com).

What is the change?
On August 26th, the Qualtrics data center in Canada will be consolidated into the Montreal facility and out of the legacy Toronto facility. During this migration, you will experience an outage of services in the CA1 data center totaling 60 minutes during a 2-hour window from 10:00PM PT on August 25th, to 12:00AM PT on August 26th.

What action do I need to take?
If your company implements outgoing or incoming network access control lists, you will need to make sure the new IP address ranges are included in the network allowlist managed by your IT administrator. You will need to add the 139.60.152.0/22, 64.69.212.0/24, and 162.247.216.0/22 ranges before August 8th, 2022, and the 98.97.248.0/21 IP range to your firewalls before August 25th, 2022.

Which we thought didn't apply to us as, to our knowledge, we've never had to do this. Could this be something we need to do on an SMTP side rather than what we're pursuing by my potentially faulty information? (again, apologies, I have no idea what anything is on your end and am coming from a very vague understanding of what the differences between SMTP and custom domain are on the technical side -- happy to meet if it would be easier to screen-share through what I'm able to look at to determine how this was all set up.)

I don't know if they're relevant, but just in case, here are some past tickets related to Qualtrics emails from our domain:

• T164424: Allow Qualtrics to send @wikimedia.org emails using an SPF record or an SMTP relay
• T176666: Qualtrics cannot send email to wikimedia.org addresses

Thank you so much for this, @nshahquinn-wmf -- that seems super relevant! From these tickets it appears that we do use SMTP rather than Custom Email Domain.

Though I'm unsure of how to proceed from here, as I can't tell whether the email Qualtrics sent us about updates was related to SMTP or to Custom Email Domain. Likewise, I'm confused as to why we started having issues before August 8th -- the issue was first spotted on the 4th when I attempted to send a test email, see New and Old for differences:

New (seems phishy, signed by qemailserver.com rather than wikimedia.org):

Old:

Is the issue we're seeing possibly unrelated to any of this, and it's a coincidence that something on our end that could have made a difference (eg. could it be that something happened to the qualtrics@wikimedia.org email account?)? Many thanks to anyone who chimes in with more insight and how we might proceed.

@TAndic would it be possible to attach the original or raw email messages of your new and old samples? I want to view the email headers to see how the emails were routed.

From looking at the headers that @TAndic sent me the change in behavior appears as follows:

Old Route

Qualtrics sends to gmail first, which then sends the email to the recipient. We have authorized gmail to send on Wikimedia’s behalf, so this email passes spam checks from email providers.

qualtrics -> gmail ->  recipient

New Route

Qualtrics sends directly to the recipient. Since we have not authorized Qualtrics to send email on Wikimedia’s behalf email providers flag these messages as suspicious spam.

qualtrics ->  recipient

I believe the old route was using the method described in T164424, namely authenticating to gmail’s smtp server with the user qualtrics@wikimedia.org and setting the reply-to to surveys@wikimedia.org. I wonder if that configuration was removed on the Qualtrics side?

Options:

1. Engage with ITS to see if the qualtrics user is still setup, if it is re-add the config on the Qualtrics side.
2. Modify our mail servers to accept authenticated SMTP connections and use those for relaying from Qualtrics rather than gmail’s.

Status update: I've reached out to techsupport@ as we had a thread on this previous to contacting SRE (#92751 on zendesk), hoping they can help with clarifying Option 1.

I welcome input from anyone with more experience/expertise/background info on this as well! @KCVelaga_WMF has access to our Qualtrics accounts as well in case anyone cannot reach me while I'm out.

@TAndic thanks for the update. I also did some investigation into option (2) and it seems pretty feasible, probably a day or two of work. So let me know if we want to go that route. One advantage of option (2) is that it might be less likely to be removed since it would be part of our infrastructure code.

@TAndic @jhathaway

Thanks for the additional background. I was unaware that we had any SMTP relay rules set up for Qualtrics, but it looks like we do (screenshot). Should I add the additional IP ranges Qualtrics noted in their email to this SMTP relay rule?

@bcampbell from their docs it appears they also require SMTP Authentication, does the qualtrics@wikimedia.org user exist and do we have its password documented in some secret store?

@jhathaway qualtrics@wikimedia.org exists as a Google Group, but not a Google user.

In T314815#8144089, @bcampbell wrote:

@jhathaway qualtrics@wikimedia.org exists as a Google Group, but not a Google user.

how about, surveys@wikimedia.org?

how about, surveys@wikimedia.org?

surveys@ is also a Google Group. I see that a user survey@ exists, though.

@TAndic I am happy to hop on a call with ITS to explore solutions, let me know how you want to proceed when you return.

@bcampbell, @TAndic, and I had a meeting to try and suss out how this was working previously. As far as we were able to ascertain it appears gmail was setup to allow relaying from specific Qualtircs IP addresses, without SMTP authentication. @bcampbell added the additional IP addresses requested by Qualtrics and @TAndic is going to test and see if that resolves the problem. If it does not @TAndic is going to reach out to Qualtrics support to determine if we have any legacy configuration that is not viewable in their web interface and if we do not, what would be the best option to configure, SMTP Relay or subdomain.

Hi @bcampbell and @jhathaway -- it looks like adding the IPs has not worked to resolve the issue and I called Qualtrics support to see what the next steps would be.

They are also unable to see anything on their end as far as a legacy configuration; it appears the same as my end, a note for setup for backwards compatibility but no additional information, seemingly nothing done that wasn't automated.

They recommended that the best route would be redo our configuration from the start, which should override any previous configuration. I also asked whether they've ever had issues with SMTP via Google Workspace vs. regular servers and they said no, Google should work fine.

My proposal is that (very open to alternatives, of course):

1. We first attempt to redo our configuration through the STMP extension (Qualtrics) via Google Workspace.

• the instructions are available here: https://www.qualtrics.com/support/survey-platform/distributions-module/email-distribution/using-a-custom-from-address/#SettingUpAnSMTPRelay
• For this I will need the information below from @bcampbell (and a clarification, does this mean we need an account for this specifically, as we've been discussing above with qualtrics@wikimedia.org, surveys@ and potentially survey@? see the "BEFORE YOU SET UP AN SMTP CONNECTION" section in the link above.)

• On ITS's side we'll again need to whitelist the IP addresses

2. If this doesn't work, we could consider modifying our mail servers? Or generally going back to the drawing board.

Thanks @TAndic I'll leave this in @bcampbell's hands unless I hear otherwise!

@bcampbell - It seems the last action still has not resolved this issue. Is there a next step that we should be trying or following back up with Qualtrics?

Hey @JAnstee_WMF, I've been working with Tanja more on this on our Zendesk ticket. The next step I proposed was to set up a meeting with us and a Qualtrics engineer to troubleshoot further, because our SMTP relay is seemingly set up correctly on the Google side. I think Tanja is out this week, so we will pick it back up when Tanja returns.

I checked today following the switch overnight - It seems we are still able to send invites, and it is still sending to spam via qualtrics@wikimedia.org via qemailserver.com 

I am not sure if it has had any other effects - Maybe this is unrelated but I will note it here also for when Tanja is back at her desk - When logged in, I am experiencing strange time-outs and long lags in data retrieval and also 503 errors when trying to navigate through the admin interface, it is not appearing a consistent issue though as the refreshing gets me there.

Hi all -- I think it's time to pursue Option 2. Modify our mail servers to accept authenticated SMTP connections and use those for relaying from Qualtrics rather than gmail’s.

We've been unable to make the Gmail SMTP work with Qualtrics (likewise with LimeSurvey, another survey software we use) after exhausting every possible configuration, which has led me to the conclusion that the issue is likely with how Gmail SMTP is functioning. Though we're still waiting on Qualtrics to inspect our account to see what the legacy setup was, I think having a reliable SMTP that doesn't depend on the quirks of Google would be beneficial at this point.

Hi all, sorry for the second update today --

Qualtrics support suggested that we should set up the Custom From Domain (like the start of this ticket) and that that should solve our SMTP setup problem -- not with a subdomain, but with our regular no-sub wikimedia.org domain. Is it worth a shot before we go on with Option 2? If this is the issue, it may need to happen whether or not we use the Gmail SMTP? @jhathaway @bcampbell

I've added a configuration if you think it's worth trying

Change 830948 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] mail::mx: Add support for PLAIN auth over tls

https://gerrit.wikimedia.org/r/830948

Change 830948 merged by JHathaway:

[operations/puppet@production] mail::mx: Add support for PLAIN auth over tls

https://gerrit.wikimedia.org/r/830948

@TAndic this configuration has now been deployed to prod and tested. I can provide you the credentials so you can setup the Qualtrics side.

Thank you @jhathaway -- crossing fingers it works!