Page MenuHomePhabricator

Puppetize mailman3
Closed, ResolvedPublic

Description

This task is done when the puppetized mailman3 is deployed in beta cluster and merged to production.

Event Timeline

Change 608163 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Start mailman3

https://gerrit.wikimedia.org/r/608163

Change 608878 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] role::idp: disable X-frame-options for icinga

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608878

Change 608878 merged by Jbond:
[operations/puppet@production] role::idp: disable X-frame-options for icinga

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608878

jcrespo triaged this task as Medium priority.Jul 6 2020, 2:55 PM

Change 617842 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] lists: Use hiera value instead of hard-coded value "lists.wikimedia.org"

https://gerrit.wikimedia.org/r/617842

Mentioned in SAL (#wikimedia-cloud) [2020-08-08T20:14:57Z] <Amir1> made mailman-puppetmaster a stand-alone puppetmaster and made mailman-mailman02 a client (T258365 T256536)

So https://gerrit.wikimedia.org/r/c/operations/puppet/+/608163 is cherry-picked on the standalone puppet master and it works to some degree which is nice. This is going to be lots of bits and pieces so I rather get it done in small parts.

BTW I put these hiera values in the mailman puppetmaster (/var/lib/git/labs/private/hieradata/labs/mailman/common.yaml)

profile::mailman3::db_host: mailman-db.mailman.eqiad1.wikimedia.cloud
profile::mailman3::db_password: <redacted>
profile::mailman3::api_password: <redacted>

Change 608163 merged by Legoktm:
[operations/puppet@production] mailman3: Start mailman3

https://gerrit.wikimedia.org/r/608163

Change 655203 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Add parts for Postorius (web interface)

https://gerrit.wikimedia.org/r/655203

Change 655203 merged by Legoktm:
[operations/puppet@production] mailman3: Add parts for Postorius (web interface)

https://gerrit.wikimedia.org/r/655203

What's left: MTA integration.

For production-ready puppet (TLS termination and acme, logging, monitoring, SpamAssassin, etc.). I'll make a separate ticket.

Change 669182 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Add exim4 configuration

https://gerrit.wikimedia.org/r/669182

Change 670971 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Add let's encrypt parts for labs

https://gerrit.wikimedia.org/r/670971

Change 669182 merged by Legoktm:
[operations/puppet@production] mailman3: Add exim4 configuration

https://gerrit.wikimedia.org/r/669182

Change 670971 merged by Legoktm:
[operations/puppet@production] mailman3: Add let's encrypt parts for labs

https://gerrit.wikimedia.org/r/670971

So far everything is fine except hyperkitty archiver not being able to archive things. The error says because it fails to make requests internally and yup, it fails like this:

root@mailman-mailman02:/var/log/mailman3# curl http://localhost/hyperkitty/api/mailman/urls
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

on non-puppetized node, it works just fine.

The reason is /etc/apache2/conf-enabled/50-server-status.conf (added for T113090) overrides all requests locally and 404s for internal requests. Strangely increasing priority of the main config didn't fix this issue but deleting the file fixes the issue. I'm done for now, will work more on it later.

Looks like the same issue as T190111: VirtualHost for mod_status breaks debugging Apache/MediaWiki from localhost (on jobrunners). Based on the debugging there, I was able to get it to work with:

root@mailman-mailman02:~# curl -H "Host: localhost" "http://mailman-mailman02.mailman.eqiad.wmflabs/hyperkitty/api/" -I
HTTP/1.1 301 Moved Permanently
Date: Fri, 12 Mar 2021 05:41:00 GMT
Server: Apache
Content-Type: text/html; charset=utf-8
Location: https://localhost/hyperkitty/api/
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
X-Frame-Options: DENY
Vary: Accept-Language,Cookie
Content-Language: en

Of course, then it runs into the LE redirect (which I think we should disable because the VPS proxy enforces HTTPS so only internal traffic can hit it on HTTP).

root@mailman-mailman02:~# curl -H "Host: localhost" "https://mailman-mailman02.mailman.eqiad.wmflabs/hyperkitty/api/lists/?format=json" -k
[{"url":"https://localhost/hyperkitty/api/list/test@lists.wmcloud.org/?format=json","name":"test@lists.wmcloud.org","display_name":"Test","description":"","subject_prefix":"[Test] ","archive_policy":"public","created_at":"2020-07-11T20:10:55+02:00","threads":"https://localhost/hyperkitty/api/list/test@lists.wmcloud.org/threads/?format=json","emails":"https://localhost/hyperkitty/api/list/test@lists.wmcloud.org/emails/?format=json"},{"url":"https://localhost/hyperkitty/api/list/test2@lists.wmcloud.org/?format=json","name":"test2@lists.wmcloud.org","display_name":"Test2","description":"","subject_prefix":"[Test2] ","archive_policy":"public","created_at":"2021-03-11T02:59:30+01:00","threads":"https://localhost/hyperkitty/api/list/test2@lists.wmcloud.org/threads/?format=json","emails":"https://localhost/hyperkitty/api/list/test2@lists.wmcloud.org/emails/?format=json"},{"url":"https://localhost/hyperkitty/api/list/test3@lists.wmcloud.org/?format=json","name":"test3@lists.wmcloud.org","display_name":"Test3","description":"","subject_prefix":"[Test3] ","archive_policy":"public","created_at":"2021-03-12T00:17:26+01:00","threads":"https://localhost/hyperkitty/api/list/test3@lists.wmcloud.org/threads/?format=json","emails":"https://localhost/hyperkitty/api/list/test3@lists.wmcloud.org/emails/?format=json"}]

(which I think we should disable because the VPS proxy enforces HTTPS so only internal traffic can hit it on HTTP).

AFAIK this is not behind the VPS proxy. The DNS is bound to our public IP (due to exim4 reasons). Hyperkitty can enforce HTTPS though, there's an option for it. Didn't try it yet.

If we do:

 ALLOWED_HOSTS = [
     "localhost",  # Archiving API from Mailman, keep it.
     'lists.wmcloud.org',
+    'mailman-mailman02.mailman.eqiad.wmflabs',
     '0.0.0.0',
 ]

And then make requests using that domain instead of localhost, API requests with curl work fine. Will send a patch shortly.

Change 671279 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] mailman3: Don't talk to hyperkitty over localhost

https://gerrit.wikimedia.org/r/671279

Change 671279 merged by Legoktm:
[operations/puppet@production] mailman3: Don't talk to hyperkitty over localhost

https://gerrit.wikimedia.org/r/671279

Change 673632 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] mailman3: Parameterize MariaDB dbname and username

https://gerrit.wikimedia.org/r/673632

Change 673641 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] mailman3: Use acme-chief, unify Apache configuration

https://gerrit.wikimedia.org/r/673641

Change 674393 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Add ferm

https://gerrit.wikimedia.org/r/674393

Change 674393 merged by Legoktm:
[operations/puppet@production] mailman3: Configure ferm

https://gerrit.wikimedia.org/r/674393

Change 673632 merged by Legoktm:
[operations/puppet@production] mailman3: Parameterize MariaDB dbname and username

https://gerrit.wikimedia.org/r/673632

Change 673641 merged by Legoktm:
[operations/puppet@production] mailman3: Use acme-chief, unify Apache configuration

https://gerrit.wikimedia.org/r/673641

Change 674476 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] hyperkitty: Install whoosh

https://gerrit.wikimedia.org/r/674476

Change 674477 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Clean up the web frontend

https://gerrit.wikimedia.org/r/674477

Change 674476 merged by Legoktm:
[operations/puppet@production] hyperkitty: Install whoosh

https://gerrit.wikimedia.org/r/674476

Change 674477 merged by Legoktm:
[operations/puppet@production] mailman3: Clean up the web frontend

https://gerrit.wikimedia.org/r/674477

This is done. Some follow up stuff for production are being done in T277286: Make puppet for mailman3 ready for production

Change 617842 abandoned by Ladsgroup:
[operations/puppet@production] lists: Use hiera value instead of hard-coded value "lists.wikimedia.org"

Reason:
Already implemented in a better way.

https://gerrit.wikimedia.org/r/617842