Page MenuHomePhabricator

Write an extension for displaying Discourse feeds on mediawiki.org
Open, MediumPublic

Description

Summary

When viewing a wiki page users can see a feed of content from Wikimedia Space. Users should be able to interact with the feed to be directed to the content on Wikimedia Space. Wiki editors should be able to customize and embed these feeds within wiki content in an easy, familiar manner.

Requirements

Discourse can provide a feed of activity based upon criteria specified by calling the API. This can include lists such as the most recent topics, most active, and recent replies. This extension would allow for user-configurable display of feeds on-wiki from Wikimedia Space.
Most likely by extending MediaWiki with new markup tag (For example, <SpaceFeed>...</SpaceFeed> or parser function.

These feeds fall into the following classifications:

  • Topics - a feed of topics
    • Latest - newest topics by creation date
    • Top - most recently active by a given timeframe
  • Posts - Latest posts across topics
  • Categories - Latest or top topics in a given category
  • Tags - Latest or top topics with a given tag
  • Groups - a feed of groups
    • A list of available groups
    • a list of members within a given public group

Feature

Feeds from Wikimedia Space should display the following information:

  • Title of topic
  • Title of post
  • Categories and tags posts are in
  • Group names
  • User names
  • Date
  • Excerpts from the content

Feeds from Wikimedia Space should be configurable upon the following attributes:

  • Items restricted to a particular category or tag
  • Items restricted to those created by a particular user
  • Maximum number of items to list
  • Period of time to query (daily, weekly, monthly, etc.)
  • An excerpt from the content
  • Show or hide user names
  • Show or hide the date attribute

Feeds should be dynamic where their content is refreshed automatically. This can be done with an API call on page load or by using the MediaWiki job queue.

Feeds should provide clean HTML and element classes so they can be styled by on-wiki CSS.

Feeds should be programmatic. This is so feeds can be included in templates to be extended by wiki variables.

Changes to feed contents should update the Watchlist for the page where they are embedded.

Examples

Pseudocode

<SpaceFeed>
category = Sri Lanka
count = 5
author = UserName
class = fooClass

</SpaceFeed>

or

{{#discourse: latest | category=cat-slug | count=5 | order=ascending }}

Benefit

Affiliate groups that wish to provide a feed of discussions happening in topics related to their work
Event organizers that wish to provide a feed of discussions happening related to the event.
Topic working groups that wish to see a list of recent news related to their area of interest.
Users who wish to share a list of posts they have created or are participating in.

Acceptance Criteria

A MVP is one where an editor can add specific markup to a wiki page, save, and a list of topics would be presented from Wikimedia Space given a time frame and maximum number. This list could be static at the time of save.

A complete, stable extension can do the above as described with compatibility with all major browsers, functionality with javascript disbled, aria compliant, and with support for RTL languages.

Content would be refreshed dynamically. Pages using the extension would not bust the caching of pages.

See also

T185075: A template to show discourse-mediawiki.wmflabs.org feeds)

Details

Related Gerrit Patches:
mediawiki/extensions/Discourse : masterFirst draft
integration/config : masterAdd new Discourse extension

Event Timeline

Tgr created this task.Feb 1 2019, 5:48 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 1 2019, 5:48 AM

Would a parser function like this be what we're looking for?—

{{#discourse: latest }}
{{#discourse: latest | tag=tag-slug }}
{{#discourse: latest | category=cat-slug }}

I thought the latest might be good so one day we could add e.g. profile or whatever, to display other stuff.

And for display, would a table that matches the Discourse layout be good? e.g. (but maybe without the user icon column):

Tgr added a comment.Feb 18 2019, 2:05 AM

Something like that, yes. Or it could use a wiki template and just provide the parameters from the feed - that's a little more effort to use, but also more flexible. (Or it could be a Lua module that just provides the data in a structured form and it's up to the caller to format it.) Whichever is least effort, I'd say. (Probably the HTML one?)

Oh yes, I hadn't thought about customizing the display. It's easy enough to do that by modifying the extension though; let's start with extension-built HTML and see how it goes (it's easier, as you say). :)

Next question: should it support multiple Discourse sites? We could whitelist Discourse URLs (and set a default) in config, and add an optional site=<site-code> parameter to the parser function.

Tgr added a comment.Feb 18 2019, 3:50 AM

Sounds like something to worry about once we do have multiple (non-testing) Discourse sites :)

Qgil awarded a token.Feb 18 2019, 7:28 AM
Qgil added a comment.Mar 22 2019, 11:00 PM

All these options you are discussing would display RSS feeds but they would not change the actual content of a wiki page, and therefore they wouldn't create any watchlist changes, right? If so, this is not a problem problem. Displaying RSS feeds on wiki pages without touching the wiki page content is useful for several use cases.

We could consider cases where changes are made in the page based on an RSS feed. I am thinking of the use cases of new topics created with a specific tag (which has a specific RSS feed) would automatically create a new section in a Village Pump, a Talk page... Something like this:

Issue with unrelated recommendation as part of Related Pages for an article about machines
The 3rd recommendation as part of Related Pages at the bottom of the below page in Tamil doesn’t seem...
Read the full post and potential replies at the Wikimedia Developer Support forum. --DiscourseBot (talk) 8:26, 14 March 2019 (UTC)

And the equivalent for Structured Discussions. I guess I should create a new task for this.

Tgr added a comment.Mar 22 2019, 11:04 PM

I theory it's not hard to inject remote changes into the watchlist, Wikidata does that for example. Watchlists are fine-tuned for policing changes to the content maintained by the community though, they are not a generic notification interface (we have Echo for that), so I'd tread with care.

We could do it Listeriabot-style, and have a bot update e.g. a template's subpage every hour with the latest from a Discourse tag. That way people could watchlist it if they wanted to, and it could also be transcluded wherever required.

I've created a repo request at https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests

Other than latest posts (optionally by tag or category), the other thing I want is coming-events (optionally by tag), so we can display coming events on the Wikimedia Australia wiki (I'm happy for that wiki to be a test place for this extension).

Change 534905 had a related patch set uploaded (by Samwilson; owner: Samwilson):
[mediawiki/extensions/Discourse@master] First draft

https://gerrit.wikimedia.org/r/534905

Change 534906 had a related patch set uploaded (by Samwilson; owner: Samwilson):
[integration/config@master] Add new Discourse extension

https://gerrit.wikimedia.org/r/534906

Samwilson claimed this task.Sep 8 2019, 1:40 AM

There seems to be an issue with filtering by date with the calendar API. For example, this is supposed to work: https://discuss-space.wmflabs.org/calendar.json?start=2019-08-01&end=2019-08-30 but events outside of August are still returned. I'm not sure it matters for the first draft, because we probably most commonly want to be able to list events by tag rather than date range. It's probably just me misunderstanding the API.

TheDJ added a subscriber: TheDJ.Sep 8 2019, 6:40 PM

Idea, how about exposing the data as a lua module ?
Might make using it in different visual forms easier and more flexible than a parserfunction.

Change 534906 merged by jenkins-bot:
[integration/config@master] Add new Discourse extension

https://gerrit.wikimedia.org/r/534906

Trizek-WMF added a subscriber: Trizek-WMF.
Ainali added a subscriber: Ainali.Sep 17 2019, 4:42 PM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptSep 17 2019, 10:13 PM

Hey @Samwilson and others. I've been working on creating a more better description for this task. Something we could workshop and discuss and eventually provide guidance to a contract MediaWiki developer. I hope I'm not stepping on toes and happy to discuss!

Samwilson added a comment.EditedSep 17 2019, 10:40 PM

That looks great @CKoerner_WMF thanks! (And you're not stepping on any toes.)

I've currently got a nearly-ready first take on an extension that provides a Lua module that can retrieve arbitrary data from Discourse, and also has some convenience methods for displaying coming events (optionally filtered by tag and dates) and recent topics (optionally filtered by tag and category). Hopefully will get the patch up in the next day or so.

The earlier patch in this thread implements a parser function to provide the same two convenience methods, but as @TheDJ points out, a Lua module is going to be much more flexible. I think it should still provide ready-formatted outputs as well, so every wiki doesn't have to recreate things if they are happy with the defaults.

The extension adds a $wgDiscourseSites config var for listing permitted site URLs (one of which can be made default).

The main part of the Lua module is two methods: discourse.getBaseUrl(site) (which does what it sounds like) and discourse.getData(urlPath, site) (which takes an API URL path e.g. calendar.json?tags=foo and returns whatever the API returns. This may be too free, and we'll want to lock things down to only particular queries, I'm not sure.

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Discourse/+/534905 is ready for review. There's masses of stuff missing, but I'm sure there are bugs with what's there. I'll work on it more tomorrow, but if anyone wants to have a look I'd love some feedback.

Qgil triaged this task as Medium priority.Oct 2 2019, 9:07 AM
Qgil edited projects, added Space (Oct-Dec-2019); removed Space.
Aklapper removed a subscriber: Aklapper.Oct 31 2019, 10:16 AM
Qgil added a comment.Nov 4 2019, 6:33 AM

Could this functionality be used for T228862: One feed to pull all the announcements published by the Foundation, where a Space feed would update a page in English Wikipedia?

Yes, I think so!

  • It wouldn't modify the wiki page, but it could be used to display the latest announcements. So it wouldn't make things appear on watchlists, for instance.
  • And are those announcements distinguished by anything on Space? The extension can pull in a category or a tag (or combination thereof) — I think a tag might make most sense, but it doesn't look like there's a "wmf-announce" tag or similar yet.

The patch I've been working on for the Discourse extension lets you consume any part of the Discourse API in Lua, so there's lots of freedom. We haven't talked yet about whether this is too much freedom though! (e.g. API queries that are slow might want to be prevented.)

Qgil added a comment.EditedNov 4 2019, 2:47 PM

If it doesn't appear on watchlists, how do users know there are updates?

And yes, these announcements would be identified by a tag, most likely.

Notes from this afternoon's discussion about this extension at Wikimedia-Technical-Conference-2019 (not wonderfully cleaned up, so do ask for clarification about anything, and don't take things as certain):

  • First extension patch has been written and is ready for review.
  • Links in talk pages being exposed in MediaWiki? No, not yet.
  • Bring in data from any API endpoint of Discourse, to Lua.
  • Hard dependency on Scributo. No parser function.
  • Discourse easier than MediaWiki for 3rd party partners (GLAMs etc.), so encourage them to use Discourse. Those discussions would then be visible from MediaWiki.
  • Will search find things on wiki? Yes; it'll be normal. Other search might be able to be done via ElasticSearch but that's for later.
  • Nofication via Echo? Mentions on either platform sent to the other? Probably down the track a bit.
  • Other feature: iframe embed on talk pages? Discourse supports this mode.
  • Site config in LocalSettings.php shortname => base URL.
  • Could this be replced by interwiki links? Probably. If so, any interwiki link could be requested with lua.discourse(), but non-Discourse ones just wouldn't return data.
  • Search Discourse from MediaWiki? ElasticSearch might be able to configure this, but it's not part of the extension.
  • How to discourage wiki users from adding material to Discourse that should really be on the wiki? e.g. some images.
  • Could a rendered wikitext blob be able to be injected into Discourse?
  • Discourse isn't replacing talk pages.
  • Discourse events are good because they have metadata, queriable.
  • Think more about mailing lists, than talk pages. (Not announcemt lists, but discussion ones).
  • User groups on wiki?
  • If watchlist isn't to be modified when the wiki page Discourse list changes, Echo will be even more important.
  • If there's a firm metadata connection between a wiki page and a Discourse tag, then we can maybe notify watchers of that page when there's a new thread.
  • Threads could then be associated with (and displayed on) multiple talk pages.
  • Could page titles be used as tags on Discourse? To cement the relationship between a set of threads and a page.
  • Next step: merge generic Discourse-lookup for Lua.
  • Before that, switch config to interwiki links, with checks in place to determine if the remote site is a Discourse site.
  • Is there a response header or something that can tell us if it's Discourse? This wouldn't be super security, because the remote site could always change their response.