Page MenuHomePhabricator
Create Task
Maniphest T256536

Puppetize mailman3
Closed, ResolvedPublic
Actions

Description

This task is done when the puppetized mailman3 is deployed in beta cluster and merged to production.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Ladsgroup created this task.Jun 27 2020, 3:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2020, 3:40 PM
Ladsgroup removed a project: Security-Team.Jun 27 2020, 3:41 PM
Paladox subscribed.Jun 27 2020, 3:50 PM
gerritbot added a comment.Jun 27 2020, 5:01 PM

Change 608163 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Start mailman3

https://gerrit.wikimedia.org/r/608163

gerritbot added a project: Patch-For-Review.Jun 27 2020, 5:01 PM
gerritbot added a comment.Jul 1 2020, 1:22 PM

Change 608878 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] role::idp: disable X-frame-options for icinga

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608878

gerritbot added a comment.Jul 2 2020, 11:43 AM

Change 608878 merged by Jbond:
[operations/puppet@production] role::idp: disable X-frame-options for icinga

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608878

jcrespo triaged this task as Medium priority.Jul 6 2020, 2:55 PM
jcrespo moved this task from Backlog to Mailman v3 on the Wikimedia-Mailing-lists board.Jul 8 2020, 10:06 AM
gerritbot added a comment.Aug 1 2020, 9:57 PM

Change 617842 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] lists: Use hiera value instead of hard-coded value "lists.wikimedia.org"

https://gerrit.wikimedia.org/r/617842

Stashbot mentioned this in T258365: Setup Mailman3 in Cloud VPS.Aug 8 2020, 8:14 PM

Mentioned in SAL (#wikimedia-cloud) [2020-08-08T20:14:57Z] <Amir1> made mailman-puppetmaster a stand-alone puppetmaster and made mailman-mailman02 a client (T258365 T256536)

Ladsgroup added a comment.Oct 3 2020, 11:46 PM

So https://gerrit.wikimedia.org/r/c/operations/puppet/+/608163 is cherry-picked on the standalone puppet master and it works to some degree which is nice. This is going to be lots of bits and pieces so I rather get it done in small parts.

Ladsgroup added a comment.Oct 3 2020, 11:50 PM

BTW I put these hiera values in the mailman puppetmaster (/var/lib/git/labs/private/hieradata/labs/mailman/common.yaml)

profile::mailman3::db_host: mailman-db.mailman.eqiad1.wikimedia.cloud
profile::mailman3::db_password: <redacted>
profile::mailman3::api_password: <redacted>
gerritbot added a comment.Jan 9 2021, 12:31 AM

Change 608163 merged by Legoktm:
[operations/puppet@production] mailman3: Start mailman3

https://gerrit.wikimedia.org/r/608163

gerritbot added a comment.Jan 9 2021, 5:03 PM

Change 655203 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Add parts for Postorius (web interface)

https://gerrit.wikimedia.org/r/655203

gerritbot added a comment.Jan 20 2021, 6:47 PM

Change 655203 merged by Legoktm:
[operations/puppet@production] mailman3: Add parts for Postorius (web interface)

https://gerrit.wikimedia.org/r/655203

Ladsgroup mentioned this in T274953: Access group for Gitlab contractors.Feb 23 2021, 9:28 PM
Ladsgroup closed subtask T256542: Puppetize mailman3 web and hyperkitty (mailman archiver) as Resolved.Mar 4 2021, 11:39 PM
Ladsgroup added a comment.Mar 4 2021, 11:42 PM

What's left: MTA integration.

For production-ready puppet (TLS termination and acme, logging, monitoring, SpamAssassin, etc.). I'll make a separate ticket.

gerritbot added a comment.Mar 6 2021, 9:50 PM

Change 669182 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Add exim4 configuration

https://gerrit.wikimedia.org/r/669182

Ladsgroup mentioned this in T276686: Requesting a test VM in production for mailman3.Mar 6 2021, 11:05 PM
gerritbot added a comment.Mar 11 2021, 10:59 PM

Change 670971 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Add let's encrypt parts for labs

https://gerrit.wikimedia.org/r/670971

gerritbot added a comment.Mar 11 2021, 11:37 PM

Change 669182 merged by Legoktm:
[operations/puppet@production] mailman3: Add exim4 configuration

https://gerrit.wikimedia.org/r/669182

gerritbot added a comment.Mar 11 2021, 11:41 PM

Change 670971 merged by Legoktm:
[operations/puppet@production] mailman3: Add let's encrypt parts for labs

https://gerrit.wikimedia.org/r/670971

Ladsgroup added a comment.Mar 12 2021, 1:04 AM

So far everything is fine except hyperkitty archiver not being able to archive things. The error says because it fails to make requests internally and yup, it fails like this:

root@mailman-mailman02:/var/log/mailman3# curl http://localhost/hyperkitty/api/mailman/urls
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

on non-puppetized node, it works just fine.

The reason is /etc/apache2/conf-enabled/50-server-status.conf (added for T113090) overrides all requests locally and 404s for internal requests. Strangely increasing priority of the main config didn't fix this issue but deleting the file fixes the issue. I'm done for now, will work more on it later.

Legoktm subscribed.Mar 12 2021, 5:45 AM

Looks like the same issue as T190111: VirtualHost for mod_status breaks debugging Apache/MediaWiki from localhost (on jobrunners). Based on the debugging there, I was able to get it to work with:

root@mailman-mailman02:~# curl -H "Host: localhost" "http://mailman-mailman02.mailman.eqiad.wmflabs/hyperkitty/api/" -I
HTTP/1.1 301 Moved Permanently
Date: Fri, 12 Mar 2021 05:41:00 GMT
Server: Apache
Content-Type: text/html; charset=utf-8
Location: https://localhost/hyperkitty/api/
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
X-Frame-Options: DENY
Vary: Accept-Language,Cookie
Content-Language: en

Of course, then it runs into the LE redirect (which I think we should disable because the VPS proxy enforces HTTPS so only internal traffic can hit it on HTTP).

root@mailman-mailman02:~# curl -H "Host: localhost" "https://mailman-mailman02.mailman.eqiad.wmflabs/hyperkitty/api/lists/?format=json" -k
[{"url":"https://localhost/hyperkitty/api/list/test@lists.wmcloud.org/?format=json","name":"test@lists.wmcloud.org","display_name":"Test","description":"","subject_prefix":"[Test] ","archive_policy":"public","created_at":"2020-07-11T20:10:55+02:00","threads":"https://localhost/hyperkitty/api/list/test@lists.wmcloud.org/threads/?format=json","emails":"https://localhost/hyperkitty/api/list/test@lists.wmcloud.org/emails/?format=json"},{"url":"https://localhost/hyperkitty/api/list/test2@lists.wmcloud.org/?format=json","name":"test2@lists.wmcloud.org","display_name":"Test2","description":"","subject_prefix":"[Test2] ","archive_policy":"public","created_at":"2021-03-11T02:59:30+01:00","threads":"https://localhost/hyperkitty/api/list/test2@lists.wmcloud.org/threads/?format=json","emails":"https://localhost/hyperkitty/api/list/test2@lists.wmcloud.org/emails/?format=json"},{"url":"https://localhost/hyperkitty/api/list/test3@lists.wmcloud.org/?format=json","name":"test3@lists.wmcloud.org","display_name":"Test3","description":"","subject_prefix":"[Test3] ","archive_policy":"public","created_at":"2021-03-12T00:17:26+01:00","threads":"https://localhost/hyperkitty/api/list/test3@lists.wmcloud.org/threads/?format=json","emails":"https://localhost/hyperkitty/api/list/test3@lists.wmcloud.org/emails/?format=json"}]
Ladsgroup mentioned this in T277286: Make puppet for mailman3 ready for production.Mar 12 2021, 12:51 PM
Ladsgroup added a comment.Mar 12 2021, 7:11 PM

(which I think we should disable because the VPS proxy enforces HTTPS so only internal traffic can hit it on HTTP).

AFAIK this is not behind the VPS proxy. The DNS is bound to our public IP (due to exim4 reasons). Hyperkitty can enforce HTTPS though, there's an option for it. Didn't try it yet.

Legoktm added a comment.Mar 12 2021, 9:56 PM

If we do:

 ALLOWED_HOSTS = [
     "localhost",  # Archiving API from Mailman, keep it.
     'lists.wmcloud.org',
+    'mailman-mailman02.mailman.eqiad.wmflabs',
     '0.0.0.0',
 ]

And then make requests using that domain instead of localhost, API requests with curl work fine. Will send a patch shortly.

gerritbot added a comment.Mar 12 2021, 10:02 PM

Change 671279 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] mailman3: Don't talk to hyperkitty over localhost

https://gerrit.wikimedia.org/r/671279

gerritbot added a comment.Mar 12 2021, 11:16 PM

Change 671279 merged by Legoktm:
[operations/puppet@production] mailman3: Don't talk to hyperkitty over localhost

https://gerrit.wikimedia.org/r/671279

gerritbot added a comment.Mar 20 2021, 1:53 AM

Change 673632 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] mailman3: Parameterize MariaDB dbname and username

https://gerrit.wikimedia.org/r/673632

gerritbot added a comment.Mar 20 2021, 1:53 AM

Change 673641 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] mailman3: Use acme-chief, unify Apache configuration

https://gerrit.wikimedia.org/r/673641

gerritbot added a comment.Mar 23 2021, 4:05 PM

Change 674393 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Add ferm

https://gerrit.wikimedia.org/r/674393

gerritbot added a comment.Mar 24 2021, 12:08 AM

Change 674393 merged by Legoktm:
[operations/puppet@production] mailman3: Configure ferm

https://gerrit.wikimedia.org/r/674393

gerritbot added a comment.Mar 24 2021, 12:23 AM

Change 673632 merged by Legoktm:
[operations/puppet@production] mailman3: Parameterize MariaDB dbname and username

https://gerrit.wikimedia.org/r/673632

gerritbot added a comment.Mar 24 2021, 5:49 AM

Change 673641 merged by Legoktm:
[operations/puppet@production] mailman3: Use acme-chief, unify Apache configuration

https://gerrit.wikimedia.org/r/673641

gerritbot added a comment.Mar 24 2021, 6:15 AM

Change 674476 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] hyperkitty: Install whoosh

https://gerrit.wikimedia.org/r/674476

gerritbot added a comment.Mar 24 2021, 6:15 AM

Change 674477 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Clean up the web frontend

https://gerrit.wikimedia.org/r/674477

gerritbot added a comment.Mar 24 2021, 6:24 AM

Change 674476 merged by Legoktm:
[operations/puppet@production] hyperkitty: Install whoosh

https://gerrit.wikimedia.org/r/674476

gerritbot added a comment.Mar 24 2021, 6:29 AM

Change 674477 merged by Legoktm:
[operations/puppet@production] mailman3: Clean up the web frontend

https://gerrit.wikimedia.org/r/674477

Ladsgroup closed this task as Resolved.Mar 24 2021, 6:34 AM

This is done. Some follow up stuff for production are being done in T277286: Make puppet for mailman3 ready for production

gerritbot added a comment.Mar 27 2021, 11:30 AM

Change 617842 abandoned by Ladsgroup:
[operations/puppet@production] lists: Use hiera value instead of hard-coded value "lists.wikimedia.org"

Reason:
Already implemented in a better way.

https://gerrit.wikimedia.org/r/617842

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits