Sendmail integration could perhaps be tested on the beta cluster, but it would need manual patching.

@Jrbranaa JR, this subject came up as part of our discussion of T207621. We'd like to have a simple way to do automated integration testing of email infrastructure.

My thumbnail sketch would be something like a Kubernetes configuration to launch MW with an associated SMTP and IMAP server(s), plus test scripts to trigger email sending in MW and to check for proper delivery in the IMAP server. It could be part of a more comprehensive integration-testing service or tool.

@EvanProdromou, that sounds like a good thing to pursue. Definitely something worth investigating. Let me poke Tyler/Dan to see the feasibility for using K8s for that today. I know that there's some work in this space that could be leveraged.

In T215918#4967428, @Jrbranaa wrote:

@EvanProdromou, that sounds like a good thing to pursue. Definitely something worth investigating. Let me poke Tyler/Dan to see the feasibility for using K8s for that today. I know that there's some work in this space that could be leveraged.

I think there are some foundational pieces still missing before k8s testing with MediaWiki happens through the Pipeline project. We're still working on creating a full Deployment Pipeline for Services (that's our focus for the remainder of this fiscal year). That being said, the technologies we're currently using to run e2e smoke-tests for Services may be useful here.

We're using Helm as an abstraction over managing the individual k8s resources. Helm provides a "chart" that is a templated set of k8s resources. We're currently deploying a few Services into production using helm. Helm provides a way to test a chart that basically allows you to define some acceptance tests after standing up your pods, services, etc (cf: Chart Tests docs). Most of the current tests are simple smoke tests of a service (using service-checker) for example here's the chart test for Mathoid: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/charts/mathoid/templates/tests/test-service-checker.yaml I would be interested to see more complex testing using helm test though.

In a previous job (a decade or so ago) I did a lot of testing of sending e-mail. What I've learned is that e-mail servers are pretty good at their job, so that part doesn't need to be tested. What needs to be tested is e-mail content, and that's probably what you're changing.

How it worked then was that a web application would create an .eml text file and put it in a folder that was monitored by a SMTP server. When a file appeared there, SMTP server would send it.

I was testing e-mail message content either disabling SMTP server or configuring the app to store mail in a different folder, I don't remember. (SMTP server would delete the file after sending the e-mail, I wanted to avoid that.)

I wrote tests that would trigger the app to send mail (using something like Selenium, or API). The test would monitor the folder where the e-mail file should appear. If the e-mail did not appear, an assertion would break. If the e-mail appeared but didn't have expected content, an assertion would break.

As far as I remember, I was only testing text-only portion of the mail, but that was catching all problems. I don't think there was ever a problem when text part was fine, but there was a problem with HTML version.

I would keep all eml files and, after a test run, run a script that would open all files in an e-mail client and take screenshots (https://stackoverflow.com/q/970025/17469). I think I just took a quick look at screenshots, in case anything looked really strange. I don't think I had a script at the time that would compare the screenshots between test runs and warn if any of them have significant changes, but that could be done. I guess rendering HTML portion of the e-mail could be done with a tool like Selenium (https://www.mediawiki.org/wiki/Selenium/Node.js).

I don't know how mediawiki sends mail, so this might not be useful. I do think that separating creating e-mail content, storing it in a file, and then sending the file using SMTP server is a good architecture. It made testing e-mail content simple.

Excuse me, I'm butting in with another opinion.

I've done tests that send email for real, using SMTP, and test that it
arrives, with IMAP. This is not too difficult, as Python has modules
to do both in its standard library (see
http://git.liw.fi/server-yarns/tree/pieni.net.yarn#n206 and
http://git.liw.fi/server-yarns/tree/lib.py). However, it's tricky
enough in practice that I'd recommend having a small (micro)service
specifically for sending email. For whatever reason, dealing with
restful-ish HTTP APIs tends to be easier than the legacy protocolas
for email. (Real email systems sometimes have inexplicable emails, for
example.)

That service can be tested on its own, and the same instance can be
used by all MediaWiki instances. The MediaWiki side can then use a
mock instance of the mail sending service to test the MediaWiki parts,
without having to actually send anything. Or there can be a separate
test instance just for running MediaWiki tests, perhaps running a mode
where it only ever accepts email to its local addresses.

Such a microservice should be fairly simple to write and operate, and
once written and deployed it should not often require changes, I
think. It would also be easy to set it up to run in a container, which
meshes will the plans we have for the delivery pipeline, and local
development environments.

The above may not be a great idea, if I'm missing something about the
requirements MediaWiki testing has for sending emails. If so, it might
be good to get the requirements written down somewhere, or point to
that place in this ticket so that nobody else butts in with
ill-informed opinions, and I can save face by amending my suggestion
to be more workable.