FTR, I'm currently working on automating the various MediaWiki accesses (group membership, accounts on private wikis, etc.), but I plan on proceeding with Mailman lists reasonably soon, so I'm filling this task to get the discussion on this started.

T279023: Expose mailman3 internal REST API inside Wikimedia production network

Sorry for the late response. What @Peachey88 linked to is basically still up to date on the lack of API access inside the prod network. I would suggest that said onboarding tool publishes a list of stewards/email addresses somewhere, and then a script that runs on the lists100X server can fetch said list and make the necessary changes.

Following up on the last comment by Legoktm. I am also in favor of this option and would be happy to make the relevant puppet change to add a systemd timer for this on the lists server.

Could you export the list of stewards/email addresses on the stewards* machines locally and let me know where they are and then I take a look at the rest?

Specifically the mailman-wrapper syncmembers command seems like the right thing for this:

syncmembers    Add and delete members as necessary to syncronize a list's...

@Urbanecm What do you think? Would it be reasonable to create a text file per list on the stewards* machines that contains just an email address per line of all the users who _should_ be subscribed?

I am thinking:

• you create text files on stewards* automatically
• we rsync them to lists* automatically
• we run "syncmembers" commands automatically

This will both add new users and remove offboarded users.

[lists1001:~] $ sudo mailman-wrapper syncmembers --help
Usage: mailman syncmembers [OPTIONS] FILENAME LISTSPEC

  Add and delete members as necessary to syncronize a list's membership with
  an input file.  FILENAME is the file containing the new membership, one
  member per line.  Blank lines and lines that start with a '#' are ignored.
  Addresses in FILENAME which are not current list members will be added to
  the list with delivery mode as specified with -d/--delivery.  List members
  whose addresses are not in FILENAME will be removed from the list.
  FILENAME can be '-' to indicate standard input.

Options:
  -d, --delivery [regular|mime|plain|summary|disabled]
                                  Set the added members delivery mode to
                                  'regular', 'mime', 'plain', 'summary' or
                                  'disabled'.  I.e., one of regular, three
                                  modes of digest or no delivery.  If not
                                  given, the default is regular.
  -w, --welcome-msg / -W, --no-welcome-msg
                                  Override the list's setting for
                                  send_welcome_message to added members.
  -g, --goodbye-msg / -G, --no-goodbye-msg
                                  Override the list's setting for
                                  send_goodbye_message to deleted members.
  -a, --admin-notify / -A, --no-admin-notify
                                  Override the list's setting for
                                  admin_notify_mchanges.
  -n, --no-change                 Don't actually make the changes.  Instead,
                                  print out what would be done to the list.
  -h, --help                      Show this message and exit.

That sounds perfect to me @Dzahn! Thanks for the suggestion.

In T351202#9639225, @Dzahn wrote:

Following up on the last comment by Legoktm. I am also in favor of this option and would be happy to make the relevant puppet change to add a systemd timer for this on the lists server.

Could you export the list of stewards/email addresses on the stewards* machines locally and let me know where they are and then I take a look at the rest?

Done:

[urbanecm@stewards1001 /srv/repos/onboarding-system (main|u=)]$ ls /home/urbanecm/data/
[urbanecm@stewards1001 /srv/repos/onboarding-system (main|u=)]$ export STEWARD_ONBOARDER_CONFIG=/home/urbanecm/etc/steward-onboarder.yaml  # this config file should actually be in /etc
[urbanecm@stewards1001 /srv/repos/onboarding-system (main|u=)]$ python3 onboarder.py update
== Updating mailman_list
[urbanecm@stewards1001 /srv/repos/onboarding-system (main|u=)]$ ls /home/urbanecm/data/mailman_list/lists.wikimedia.org
stewards-l  stewards-usergroup
[urbanecm@stewards1001 /srv/repos/onboarding-system (main|u=)]$ grep Urbanec /home/urbanecm/data/mailman_list/lists.wikimedia.org/stewards-l
Martin Urbanec <martin.urbanec@wikimedia.cz>
[urbanecm@stewards1001 /srv/repos/onboarding-system (main|u=)]$

@Dzahn: Can you please make a dry-run of syncmembers and share it as a private past on this task, so that I can take a look at the changes it would make?

We also need to move the list of users from ~urbanecm/config/users.yaml to a better location. Filled T361547 to track that. Also, we should create a more reasonable export path than somewhere in my home. Maybe /srv/exports?

Once we settle on those, we should be able to move /home/urbanecm/etc/steward-onboarder.yaml under `/etc/.

Thank you, will do! Yes, let's use /srv/.

Change #1016439 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] stewards: let puppet create /srv/exports

https://gerrit.wikimedia.org/r/1016439

Change #1016441 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] stewards: puppetize steward-onboarder config file and paths

https://gerrit.wikimedia.org/r/1016441

In T351202#9678542, @Urbanecm wrote:

@Dzahn: Can you please make a dry-run of syncmembers and share it as a private past on this task, so that I can take a look at the changes it would make?

Done. Please see P59236

In T351202#9682286, @Dzahn wrote:

In T351202#9678542, @Urbanecm wrote:

@Dzahn: Can you please make a dry-run of syncmembers and share it as a private past on this task, so that I can take a look at the changes it would make?

Done. Please see P59236

Thanks! Unfortunately, this doesn't look good (it changes e-mail addresses of several stewards to another one, which is not intended. I changed users.yaml and re-ran the script; the dry run should now be empty. If it is, it should be ready for setting up the rsync and timer (for the stewards-l list only for now).

In the meantime, I'll be working on the other maillists (with the script now working, it should be relatively easy).

Change #1016439 abandoned by Dzahn:

[operations/puppet@production] stewards: let puppet create /srv/exports

Reason:

replaced by I67cad69044d0306de

https://gerrit.wikimedia.org/r/1016439

Change #1016441 merged by Dzahn:

[operations/puppet@production] stewards: add config and export dirs, steward onboarder config

https://gerrit.wikimedia.org/r/1016441

In T351202#9682432, @Urbanecm wrote:

..it changes e-mail addresses of several stewards to another one, which is not intended. I changed users.yaml and re-ran the script; the dry run should now be empty.

confirmed.

[lists1001:~] $  sudo mailman-wrapper syncmembers -n ./stewards-l stewards-l@lists.wikimedia.org
Nothing to do

Change #1022193 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] lists: start a class for automating certain subscriptions

https://gerrit.wikimedia.org/r/1022193

Change #1023505 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/puppet@production] stewards-onboarder: Add mediawiki_api to the config

https://gerrit.wikimedia.org/r/1023505

Change #1023505 merged by Dzahn:

[operations/puppet@production] stewards-onboarder: Add mediawiki_api to the config

https://gerrit.wikimedia.org/r/1023505