Page MenuHomePhabricator

Interwiki support for Newsletter extension
Open, MediumPublic

Description

The goal is to offer the Newsletter features to all Wikimedia users while keeping one centralized list of newsletters.

There is an underlying assumption that different wikis could host different newsletters locally, decentralizing the creation/administration of those newsletters and offering a local experience for different languages and projects. However, there might be other approaches to accomplish the same goal, i.e. having all newsletters hosted in one wiki and resolve localization and interwiki sync in other ways, like Commons or Wikidata does.

User stories (draft)

  • As an anonymous user, I can easily find a list with all the newsletters available in Wikimedia.
  • As a registered user, I can subscribe/unsubscribe easily to any of the newsletters featured in the list.
  • As a registered user, I can request the creation of a newsletter in my own wiki / in my own language.
  • As a subscriber of newsletters, I can receive web or email notifications containing the summary and the link of the issue announced.
  • As a publisher of newsletters, I can easily access my newsletter and announce new issues regardless of their location.
  • As a wiki administrator (or whoever has permissions to create newsletters) I can review/create/remove newsletters hosted in my wiki.

Background

In an ideal Wikimedia world, the Newsletter extension would have interwiki support. There would be a catalog of Newsletters in Meta (or replicated in every wiki?) and users could browse and subscribe to newsletters regardless of their primary wiki and their primary language. This implementation is most probably complex through, both in terms of UI and backend. During the GSoC phase it was clear that such feature was out scope. The question is, should the first implementation in Wikimedia include interwiki support?

Another way to look at this problem is that Newsletter extension in a single wiki already provides enough improvements, and the model needs to be tested and consolidated anyway. We would start with mediawiki.org (usual first destination of technical innovations) or Meta (home of Tech News and cross-project collaboration) and interwiki support would come at a later stage. We would need to decide whether the extension would be enabled in a single wiki only, or whether different wikis could have it installed, with a plan to merge all that data in a later stage.

Event Timeline

Qgil raised the priority of this task from to Medium.
Qgil updated the task description. (Show Details)
Qgil added subscribers: Aklapper, Qgil, Tinaj1234 and 7 others.

The backend part might not be as complicated as it would appear to be at first glance. Depends on how integrated you want the interwiki support. Having notices delivered to foreign wikis doesn't sound to hard. But having some sort of unified subscription center that combines all the newsletters from all the wikis, might be more challenging. That said, I would suggest holding this back for a second iteration, after the extension has gotten used by real people in Wikimedia.

Model for sending notices to foreign wikis could be something like:

  • You have the full newsletter extension installed anywhere you want to be able to create newsletter (I suspect that newsletters targeting specific wikis, are likely to want to stay local. I doubt you'll convince the signpost to move to meta)
  • You always subscribe at the wiki
  • As part of the subscription process, user can specify what wiki they want to receive notifications on (Or maybe it should be a setting in Special:Preferences)
  • The user subscribing + preferred wiki gets stored in db at newsletter wiki
  • On creation of a new issue, this triggers a job for each foreign wiki that has subscribers. This job is created in the foreign wiki's job queue.
  • When job at foreign wiki executes, it sends the notice to all users local to that wiki who have subscribed.
Tinaj1234 raised the priority of this task from Low to High.Apr 1 2016, 8:17 AM
Qgil lowered the priority of this task from High to Low.Dec 19 2016, 10:21 AM
Qgil raised the priority of this task from Low to Medium.Sep 6 2017, 9:15 PM
Qgil moved this task from Confirmed to Needs discussion on the MediaWiki-extensions-Newsletter board.

A pity, but you are the boss, of course.

I'm far from it. I don't code. Whoever can code the patches needed is boss. :)

  • How long do you think it will take? Because it's quite a difficulte to use foreign wiki - we don't have a local notification description, and wrong url.

I have no idea. I have tried to push the technical discussion, but (almost) nobody technical has followed up so far. Without a technical plan, we cannot have a technical implementation.

  • The common list you mentioned is much less than all the interwiki problem solving, can't it be solved standalone?

Maybe. Maybe not. Depends on the technical plan... that we don't have.

  • Why do you think it's wrong to have a common list for a while, if it allows a normal work? Because all the problem can take months. You can add a message on the special page, that meanwhile there are some irrelevant items here. Please.

Look at Wikimedia. How many times various local solutions have been implemented while Wikimedia-wide solutions were to be decided, and then we got all this fragmentation at all these levels. While I don't know how to solve the interwiki support problem and now, at least I know how not to make the problem bigger: not allowing newsletters beyond MediaWiki.org before having a clear interwiki plan.

I know, it sucks (we have good use cases for Meta as well). But I believe this is the right decision in the long term.

Thank you for the answer.

While I don't know how to solve the interwiki support problem and now, at least I know how not to make the problem bigger: not allowing newsletters beyond MediaWiki.org before having a clear interwiki plan.

How exactly the partial deployment can make the problem bigger? The code is the same, and when the problem will be solved, the extension characteristics will change. What is the difference between existing on MediaWiki and on other wiki? Both are parts of centralauth, not private.

@Bawolff I am not sure the interwiki notifications are the problem. Currently users are already getting Newsletter notifications in their wikis when they are not MediaWiki support, thanks to plain interwiki Notifications. T175279: Add the newsletter description to interwiki bundle notification is a problem, but maybe the solution lies in the Notifications framework, not in the interwiki support for Newsletter extension.

I think the main problem is how to get one list of newsletters shared across all the Wikimedia wikis. This way users of any wiki could check the list in their wiki and learn about all the newsletters available. With that single list in place, it would be ok if users would then jump to wiki X in order to subscribe to the newsletter they are interested about. This might even help solving the localization problem, because newsletters hosted in French Wikipedia would logically have titles and description in French, and the UI there would be in French as well.

Not having this common list of newsletters means having soon dozens of separate lists causing disconnections and duplication.

How exactly the partial deployment can make the problem bigger?

Look at the situation with gadgets and templates, where very similar instances exist in different projects, forked or rewritten, without having a central location. We have a very similar risk with newsletters if in addition to MediaWiki.org we enable the extension in Meta, Wikidata, Commons, and English Wikipedia. Who tells you that nobody will create a Newsletter about media in Hebrew at Commons while you do the same in Hebrew Wikipedia, or who knows whether someone will create a newsletter in Hebrew Wikisource that Hebrew Wikipedia will not even know about?

You might think "this will not happen to us", and you might be perfectly right. But I think there is enough proof in human history (and Wikimedia history specifically) showing that if you enable an extension in hundreds of projects, "this will happen" to someone indeed.

You might also think "but we haven't got hundreds of projects requesting the extension". True, but if we accept the deployment to a second wiki, what argument do we have to stop a request for a third wiki, and a forth?

Who tells you that nobody will create a Newsletter about media in Hebrew at Commons while you do the same in Hebrew Wikipedia, or who knows whether someone will create a newsletter in Hebrew Wikisource that Hebrew Wikipedia will not even know about?
Sorry, I just don't get. If somebody else will do this too, it's wonderful. I'll send them a huge thanks. What's the problem?

Plain users and plain publishers not finding each other. A single list with all the newsletters reduces that risk.

Plain users and plain publishers not finding each other. A single list with all the newsletters reduces that risk.

That's all? That's a little problem (if any, because it can be solved in other ways, but let it be). The non-existing newsletter, because there is no extension, is ways worse.

It's not that users needing urgently a newsletter don't have options. You have created a newsletter in MediaWiki.org that is serving Hebrew Wikipedians, and you can also create a newsletter in he.wiki distributed with MassMessage, like newsletters in Wikimedia have been doing and continue to do.

The single catalog of newsletters is a worthy goal, at least according to the maintainers of this project. Please let us try.

I see. No, these options do not help. Old massmessage is exactly the problem that newsletter solves - redundant text on talk pages. Mediawiki newsletter is good, but it does not provide a description or link to the new issue. I still do not know why it's a bad idea to do this now, but I have no choice, so I'll wait.

@01tonythomas @Bawolff @Addshore I have tried to define the interwiki support feature in way that allows developers to discuss the technical implementation. Do you think that it is clear enough?

And for that discussion, do you think that we (the people currently subscribed to this task) can come up with that plan, or do we need help from the wikitech-l audience, or even TechCom? I am happy to help bringing attention to this conversation, or doing whatever it takes to have a plan in place. With a plan, I can help finding someone or some way to work on the implementation.

Wikimedia at large needs these newsletters! :)

eranroz subscribed.

Tagging as a possible project that might interesting Wikimedia-Israel-Hackers . It would be great if someone familiar with the underlaying infrastructure (Echo/newsletter extension) can break-down this task to sub tasks with more clear and technical details so it will be easier to get into this task.

I'm not familiar with this extension so maybe the below is bullshit but I think what we need to get it simple (and somewhat quick and dirty) is:

  1. API for getting the newsletters (currently the API only supports subscribe https://www.mediawiki.org/w/api.php?action=help&modules=newslettersubscribe )
  2. Write [[Special:Newsletters]] to query with AJAX the API for existing newsletter (either locally or from remote wiki). (then we can deploy it on all wiki)

This is simple solution that can work well with no need for jobs and heavy support in backend.
drawbacks:

  1. JS support needed in client but RCFilters and VE which are more core functionality have same limitation
  2. Possible CORS issues, but could be easily overcome using mw.ForeignApi

@IKhitron Note that as some mentioned Wikidata above, I have doubt about how the Newsletters can be helpful for WD users, and just linking P31 (instance of) = Q264238 (the Newsletter article) doesn't provide any useful informations for me. Would you please tell me the benefit of doing that? (also posted at Wikidata talk:Notability)

@IKhitron Note that as some mentioned Wikidata above, I have doubt about how the Newsletters can be helpful for WD users, and just linking P31 (instance of) = Q264238 (the Newsletter article) doesn't provide any useful informations for me. Would you please tell me the benefit of doing that? (also posted at Wikidata talk:Notability)

Sorry. I don't understand you at all. Is there a chance you wanted to ask QuimGil?

I see that this is the task blocking our deploy strategy.

We have couple of devs here at Wikimania-Hackathon-2018 on this. Lets see what we can do. I guess solving this within these days would be a great goal to set.

Change 446590 had a related patch set uploaded (by 01tonythomas; owner: Addshore):
[mediawiki/extensions/Newsletter@master] WIP DNM TODOs & pointers for global newsletter stuff?

https://gerrit.wikimedia.org/r/446590

Tentative plan after meeting with @Addshore:

  1. We install the extension on all wikis, but 1 database table at meta. All installation are pointed to the meta table. The table has a $wgSiteId so we know on which wiki the newsletter was originally made on.
  2. A publisher can create a newsletter and, while creation - a checkbox: [x] Make this newsletter available globally on all wikis
  3. There is a query on the meta table each time someone opens Special:Newsletter (should we cache something here, etc) ?
  4. On Special:Newsletter, there are no other changes - but (+ should we add a filter for [] this wiki [] a particular wiki)
  5. On Special:Newsletter, the newsletter links directly point to the corresponding page from the $wiki its part of
  6. Announce and notification do not change as Echo can handle interwiki notifications without any changes.

Thoughts, questions.

Tentative plan after meeting with @Addshore:
Announce and notification do not change as Echo can handle interwiki notifications without any changes.
Thoughts, questions.

But announce and notification will be triggered on local wiki, I hope, not on meta?

But announce and notification will be triggered on local wiki, I hope, not on meta?

We just discussed this and it seems like notifying a user on a specific wiki is possible. We have to think:

  • Should a user be able to configure, say by Special:Preferences on which wiki to get notified
  • Always notify on the $wiki the newsletter is connected to. (we have this information already)

Change 446590 abandoned by Addshore:
WIP DNM TODOs & pointers for global newsletter stuff?

https://gerrit.wikimedia.org/r/446590

So long since we talked about this, but @Addshore: I know you did some works on this, but it would be great if you can mention here why the we could not get it all the way to the end. I know there were some build fails, but I see some fresh conscience on this (from private emails), so I am wondering if we can resume the work you already started.

Adding @Lea_Lacroix_WMDE, who has some use-cases in taking this forward. If the problem was simply the build fails, we should get this going.

9 years later, we see that this is one of the part of the puzzle that would get the extension out of its current state. I hope to take a look at reviving that old PR: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Newsletter/+/446590 to see if we can get somewhere.

Answering to the wikitech letter. As a heavy user of the extension over the years I can add my five cents.

  1. I really hope there will be local notifications, so the readers can see their content immediately, without a need to go to Meta.
  2. The expected Flow shutdown will kill the Newsletter extension, FMHO, if there will not be done adjustments in the extension code.
  3. The Newsletter extension is a great thing, and it should continue and evolve.

Thank you @IKhitron

The expected Flow shutdown will kill the Newsletter extension, FMHO, if there will not be done adjustments in the extension code.

This is news for me (was away for a long time). From what I know, we are not directly dependant on Flow. Or, do you see something which I dont, @IKhitron?

The expected Flow shutdown will kill the Newsletter extension, FMHO, if there will not be done adjustments in the extension code.

This is news for me (was away for a long time). From what I know, we are not directly dependant on Flow. Or, do you see something which I dont, @IKhitron?

As fas as I know, the extension, as for now, is unable to use page sections as new issues. Only new pages. Creating a new page for every issue will leave us with tons of redundant pages with almost zero content. So, when creating my Newsletter, many years ago, I was recommended to create a new Flow topic on the Newsletter Talk page, with very soft redirect to the relevant section, and it definitely helped, every time. Now, I can see two ways: 1) a lot of almost empty pages that noone needs. 2) finding another way to recognize sections by the extension.

Hmm. Good point. I think we will have to change how we only store the issuePageId right now to a bit more elegant structure where we can store #fragments from pages, and maybe interwiki properties as well. There are these couple of properties on the Title object, but I am not so sure how to save it in a nice way into the database. Asked in #wikimedia-dev. Lets see if someone answers.

Change #1056971 had a related patch set uploaded (by 01tonythomas; author: 01tonythomas):

[mediawiki/extensions/Newsletter@master] [WIP] Inter-wiki support for newsletter extension

https://gerrit.wikimedia.org/r/1056971

Regarding cross wiki reads and central table. You can use virtual domains now. Here is more details on it: https://www.mediawiki.org/wiki/Manual:$wgVirtualDomainsMapping

You need to set it to something like this in production:

[ 'virtual-newsletter' => [ 'db' => 'metawiki' ] ]

(Even better, you can create the table in x1 instead?)

Here are some examples of adoption of virtual domain: T348573: All Wikimedia extensions that store their data outside the main database should use a virtual database domain

Change #1057213 had a related patch set uploaded (by 01tonythomas; author: 01tonythomas):

[mediawiki/extensions/Newsletter@master] Support interwiki links in Newsletter extension

https://gerrit.wikimedia.org/r/1057213

Thank you @Ladsgroup; I also had to do some work arounds to setup 2 wikis locally to test this whole inter-wiki setup, and it kind of is looking good in shape.

Current progress (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Newsletter/+/1057213):

  • I could setup two wikis sharing the same nl_newsletters table with their own locally unique main_pages.
  • Special:Newsletters show newsletters that are either marked as global (right now I have to do it manually from the db) or are created in their own realm.

TODO:

  • Ask from a user to mark their newsletter as global.
  • Filters on Special:Newsletters to filter out local and global newsletters

Some questions:

  • Do we want names of global newsletters to be unique? For example, meta could now list two newsletters with the same name.
  • Do we want to show where the newsletter is from on the Special:Newsletters page somehow? Right now the ink points to the right place (is an interwiki link) but you wont know it unless you click.

in the screenshot below, we have two wikis localhost:8080 and localhost:9090 wikis, with 8080 acting like meta. The A global newsletter on 9090 wiki is a global wiki from 9090.

Screenshot 2024-07-28 at 14.29.22.jpg (790×2 px, 240 KB)

@Pppery fixed. Also, added an early screen video here: https://commons.wikimedia.org/wiki/File:Newsletter_inter-wiki_early_demo.webm

Things to do:

  • Setup an option to declare a newsletter as global during creation/edit. (Do we want to support this while editing?)
  • ... tests?

Here is a new demo video of the progress, after:

  • Added an option to [x] Make this newsletter globally available on all wikis:

Hoping to get someone to review the Gerrit change: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Newsletter/+/1057213 specially given Wikimania-Hackathon-2024 is happening as well :)