Page MenuHomePhabricator

Provide mbox archives and add missing lists to Gmane
Closed, ResolvedPublic

Description

Once again after bug 25105, we need to add to Gmane a bunch of mailing lists which were not added on time (i.e. as soon as they were created).
From what I can see, we have 268 publicly archived mailing lists (http://lists.wikimedia.org/pipermail/) and only 189 lists on Gmane (http://dir.gmane.org/search.php?match=gmane.org.wikimedia + http://dir.gmane.org/index.php?prefix=gmane.science.linguistics.wikipedia ), so there are about 80 to add.* We need an op to make the mbox files available somewhere from which they can be wget'ed and then someone to fill http://gmane.org/subscribe.php for each of the lists.

(*) [OT] Sigh, why do we keep creating so many mailing lists? :( [/OT]


Version: unspecified
Severity: enhancement
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=58957

Details

Reference
bz57246

Event Timeline

bzimport raised the priority of this task from to Low.Nov 22 2014, 2:23 AM
bzimport set Reference to bz57246.
bzimport added a subscriber: Unknown Object (MLST).
Nemo_bis created this task.Nov 19 2013, 2:57 PM
JohnLewis added a subscriber: JohnLewis.

Will look into this with consideration comments from @Dzahn

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 19 2015, 4:52 PM
Restricted Application added a subscriber: Matanya. · View Herald TranscriptOct 19 2015, 4:57 PM
Dzahn added a comment.EditedOct 19 2015, 5:29 PM

This has been done before but a long time ago. It's basically a) check if list is really public b) copy .mbox file from mailman dir to a (password protected) dir on webserver or c) upload directly to gmane

@Nemo_bis do you have the diff for "268 publicly archived mailing lists and only 189 lists on Gmane" ?

Sent an email to Lars.

Also the stats in this ticket are *significantly* out of date. The description states 268 but in reality the number is much much closer to 600, also remember the distinction of public vs private on the listinfo page is really wrong. Lists there are private and some public lists aren't even listed there.

[some is a bad word. could be 10s to 100s]

Dzahn added a comment.Oct 19 2015, 6:04 PM

Yes, i agree, the definition of private vs. public should be based on our script "remove_from_private" that is also for offboarding.

Also, haha @ 300 more lists since then..wow.

Thanks for mailing Lars

Nemo_bis added a comment.EditedOct 23 2015, 9:29 PM

There isn't any need to email Lars now. First the mailing lists need to be created on Gmane per the usual process, *then* the mbox files can be provided and will be imported by Lars. See instructions on gmane.org. The task description is correct and details the steps to follow.

I was only counting the publicly listed public mailing lists, in case the others are unlisted for a reason. I doubt 300 more mailing lists were created, but for sure a lot of mailing lists were created incorrectly.

Even the instructions are unclear and seem fairly directed at '1 or 2' lists. When you get into the boundaries of near a hundred, just emailing gets complicated, length and annoying for everyone involved.

Furthermore, we have no list of what lists exist and which don't in Gmane so we need that firstly and the additional hope was they could provide a full list of everything that is archived, unless you wish to do that.

When you get into the boundaries of near a hundred, just emailing gets complicated, length and annoying for everyone involved.

I'm not sure what you're trying to say. I said that emails are not necessary and you reply saying that emails are annoying? Then we agree. :)

Furthermore, we have no list of what lists exist and which don't in Gmane

That's not quite correct. The main directory of mailing lists has such information: https://meta.wikimedia.org/wiki/Mailing_lists/Overview
While subscribing new mailing lists with http://gmane.org/subscribe.php , we will find out whether some are already there.

I have actually just received an email from a Gmane administrator. They've provided me a list of all lists that are currently being monitored and clarity on how they want the mboxes. I will follow up from there.

See P2224 which is a full list of all lists that don't exist in Gmane but should.

I'll have these made in Gmane and then put the mboxes behind a password protected location on lists so they can be recovered.

RobH added a subscriber: RobH.Oct 23 2015, 11:31 PM

I'm interested to know if there are any considerable downsides to setting all public lists to also have publicly accessible mbox files? This would prevent future followups for this request, and allow our full public mail archives to be easily archived by third parties.

chasemp closed this task as Declined.Oct 28 2015, 6:59 PM
chasemp added a subscriber: chasemp.

There aren''t resources or clear list participant interest across the scope of lists for this. We also don't have the manpower to keep up with it. List admins are welcome to request this for their list if they want to index/gmane/go public. No one in Operations has a desire to police this but also without clear interest from list participants and admins we cannot justify pursuing it. Advice to this effect is being added to https://wikitech.wikimedia.org/wiki/Mailman#Step-by-step_procedure

wikimedia-mailman participants have indicated this is a per list consideration

Nemo_bis reopened this task as Open.EditedOct 28 2015, 9:34 PM

There is no need of ops manpower for [most of?] this task.

chasemp set Security to None.
JohnLewis closed this task as Declined.Oct 28 2015, 9:42 PM

The point still stands however that forcing an opt-in is inappropriate. We're not in a place to judge whether or not a lists archives should (at this stage now) forcibly put into Gmane. Plus considering the number of lists being 100, it is neither worth the time unless a list individually requests it.

Furthermore following a discussion earlier, the 'put all lists in Gmane' part that was initially on the check list should only be on request. Distributing archives without a straight forward requests has more fall out than advantages when it comes later to operations acting on archive related issues.

Please let's not throw this into a war. Seek consensus and/or open tickets about individual lists rather than a global one (notably without consensus, unless you have a link to an actual consensus in which point - do share).

Dzahn added a comment.Oct 28 2015, 9:46 PM

I would like to add this:

I went to the upstream mailman IRC channel and asked about a feature that just let's list admins download the .mbox files of their own lists, after they logged in with their list admin password. So that they can do this self-service if they want to and don't need to ask ops.

And i got this response from Mark Sapiro [1] .

< msapiro> mutante: it already exists. Set PUBLIC_MBOX = Yes in mm_cfg.py
< msapiro> Also even without that, urls like http://example.com/mailman/private/LIST.mbox/LIST.mbox work if you are already authenticated.

[1] http://wiki.list.org/msapiro (release manager and close to the sole maintainer of the Mailman 2.1 branch)

So we can say that:

  • each list admin can login and export the mbox file of their list if they want to
  • they could then ask any WMF deployer to use people.wikimedia.org to upload it to a webserver
  • and then mail Lars of Gmane to import it from there

We are all comfortable with this being an opt-in self-service thing, but don't think it should be a default. And we don't want it to be a centralized task where possible just because of the sheer scale of it as well.

Nemo_bis reopened this task as Open.Oct 28 2015, 9:49 PM

The point still stands however that forcing an opt-in is inappropriate.

It has always been done and nobody ever complained.

JohnLewis closed this task as Declined.Oct 28 2015, 9:54 PM

Please find consensus or request this for individual lists instead of re-opening this one. The overall objective of this ticket has been declined and will not be done. The continuous dispute over it is not useful either.

The point still stands however that forcing an opt-in is inappropriate.

It has always been done and nobody ever complained.

I question the original introduction of it honestly. No one complained likely because most people never notice it. On the same ground, I've not done and no one has ever complained either.

I'm interested to know if there are any considerable downsides to setting all public lists to also have publicly accessible mbox files?

Sure, this is already possible. The trick (which Daniel mentioned above) is documented at T27105 and requires subscribing to the list. It's just a bit tedious.

Nemo_bis reopened this task as Open.Oct 28 2015, 9:55 PM
Dzahn added a comment.Oct 28 2015, 9:55 PM

where "always" is one single time in the entire ticket history since RT though, afaict

Nemo_bis added a comment.EditedOct 28 2015, 9:55 PM

I question the original introduction of it honestly.

You are free to open a discussion to change the policy.

On the same ground, I've not done and no one has ever complained either.

This is not correct, every now and then someone complains they can't access/find a certain list on Gmane. Sure, it's not that frequent.

Nemo_bis claimed this task.Oct 28 2015, 9:56 PM

I question the original introduction of it honestly.

You are free to open a discussion to change the policy.

I don't see why I need to open a discussion to change a line that was only listed on the guideline creation instructions followed only by Ryan? They were changed to reflect reality today anyway.

@Nemo_bis I suggest you keep it open then but assigned to you and if you see a complaint of a user about missing lists in gmane, contact the individual list admins, ask them if they want to be added or not.

If the list admins agree, any authenticated user should be able to download the mbox file of their list and it can be uploaded to a deployers home dir on people.wm, from where Lars can import.

ldap-admins and restricted users should also be able to put files in people.wikimedia.org, fwiw

agree, in fact i just said that earlier. people.wm should/could come with "bastion" i guess. it's just mixed with the mw maintenance server currently. maybe that would have to change.

Nemo_bis closed this task as Resolved.Feb 11 2016, 7:55 AM

All the lists were added to the Meta list, added to Gmane, warned at their owner's address (a couple decided to not use Gmane). The mbox files were sent to Gmane, their call whether to import them or not.