Make Recent Changes available via XMPP
Closed, DeclinedPublic

Description

Daniel Kinzler wrote an excellent piece on why it makes sense to make RC available over IRC. In his blog post, he also describes some ideas, and why he cannot be the one to code it up.

Just so that this does not only gather dust on his blog, I'm adding this feature request here.

While we're at it, I guess that Watchlist over XMPP would also be interesting...


Version: unspecified
Severity: enhancement
URL: http://brightbyte.de/page/RecentChanges_via_Jabber

bzimport added a subscriber: Unknown Object (MLST).
bzimport set Reference to bz17450.
siebrand created this task.Via LegacyFeb 11 2009, 8:04 PM
daniel added a comment.Via ConduitFeb 12 2009, 12:03 AM

marking the more general bug 14045 as dependant on this.

daniel added a comment.Via ConduitFeb 12 2009, 12:04 AM

To carify: I wrote about why it does NOT make sense to provide the feed via IRC, and why XMPP would be much better. http://brightbyte.de/page/RecentChanges_via_Jabber

brion added a comment.Via ConduitFeb 12 2009, 12:13 AM

nom nom nom :)

To summarize for those too lazy to go read the awesome blog post:

  • Provide for Jabber output embedding XML payload in format of API recentchanges feed
  • Some internal interface making it easy for MW to ship those out to a Jabber server to send to the outside world (some handy UDP-packet-stream-to-XMPP-bridge so we can use off-the-shelf Jabber server for that part?)
  • in future, consider options for pubsub/filtering fancy stuff
bzimport added a comment.Via ConduitFeb 12 2009, 12:16 AM

mike.lifeguard+bugs wrote:

(In reply to comment #1)

marking the more general bug 14045 as dependant on this.

So the intent is definitely to provide an easy-to-parse/machine-readable feed specifically intended for this use? *drools*

daniel added a comment.Via ConduitApr 15 2009, 7:33 AM

This was discussed at the developer meet-up in Berlin. Notes here: http://www.mediawiki.org/wiki/Project:Developer_meet-up_2009/Notes/XMPP

Catrope and Leafnode said they would work on it.

TheDJ added a comment.Via ConduitMay 22 2009, 2:15 PM

This would also be wonderful for SUL talkpage changes. I'd love to be able to receive notifications of stuff like that in my Adium client for instance.

daniel added a comment.Via ConduitNov 8 2009, 10:11 AM

Code for generating the XML notification messages, and some rudiments of XMPP integration, are in SVN at http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions/XMLRC/. The code is somewhat incomplete and largely untested though, pending a test setup with a working XMPP server (poke poke brion wink wink).

bzimport added a comment.Via ConduitFeb 28 2010, 12:25 PM

matthew.britton wrote:

Is this still being worked on? I've been told not to make changes to the IRC recent changes feed because it's going to be replaced by this, yet all I see is some old discussion. Meanwhile core MediaWiki functionality is completely unsupported in the feed making client applications far more inefficient than they should be. I don't have the necessary access/knowledge to progress this issue further (apparently some kind of test server needs to be set up) but I am more than capable of modifying the feed code.

daniel added a comment.Via ConduitFeb 28 2010, 9:06 PM

@Gurch this has been sitting on my todo-list for some time now, but I plan to make serious progress on this in march and April. The part for emitting XML via UDP is pretty much done, the bit I have to do is a udp-to-xmpp-bridge that stuffs that XML info a MUC or, better, pubsub channel.

Poke me again in a month or so.

bzimport added a comment.Via ConduitJun 5 2010, 1:50 PM

guandalug wrote:

Just a 2-Months-later-poke....

bzimport added a comment.Via ConduitJun 5 2010, 5:08 PM

tisane2718 wrote:

Is there any possibility of, in addition to summary, user, etc., also distributing the new article contents via XMPP when edits are made? That could take some load off the servers by eliminating the need for bots to query the API for article contents, e.g. when they suspect vandalism and need to check it out.

brion added a comment.Via ConduitJun 6 2010, 8:46 PM

Best way to include text is probably to have distinct feeds you could subscribe to for metadata-only and metadata-plus-full text. That'll keep the main feeds lightweight, but let's those who need the text avoid an extra web hit.

bzimport added a comment.Via ConduitJul 16 2010, 3:53 PM

tisane2718 wrote:

I notice that XMLRC_XMPP.class.php has this line: include( "XMPP.php" );

But, there is no XMPP.php in the XMLRC directory in SVN. What PHP XMPP library should be used with this? There seem to be several available on the Internet.

daniel added a comment.Via ConduitJul 17 2010, 9:41 AM

@tisane: that class is work in progress, i never got around to finishing it. i think i was using the xmpphp library, but i'm not sure any more. writing the xmpp connector (direct, and more imprtantly a standalone udp-to-xmpp bridge) is still on my todo list.

bzimport added a comment.Via ConduitJul 17 2010, 3:26 PM

tisane2718 wrote:

*** Bug 23001 has been marked as a duplicate of this bug. ***

bzimport added a comment.Via ConduitJul 19 2010, 1:02 AM

soxred93 wrote:

Perhaps an alternate method while this is being implemented is to set up an alternate IRC server that sends JSON-encoded messages to make it easier to program bots?

Nikerabbit added a comment.Via ConduitJul 19 2010, 5:35 AM

You would hit the size limit of one message quite quickly.

bzimport added a comment.Via ConduitJul 22 2010, 11:44 PM

tisane2718 wrote:

What XMPP size limit are you referring to? I thought one of the advantages of XMPP is that it does not specify a hardcoded limit on the size of XMPP stanzas. http://xmpp.org/extensions/inbox/stanzalimits.html

MZMcBride added a comment.Via ConduitJul 22 2010, 11:55 PM

(In reply to comment #18)

What XMPP size limit are you referring to? I thought one of the advantages of
XMPP is that it does not specify a hardcoded limit on the size of XMPP stanzas.
http://xmpp.org/extensions/inbox/stanzalimits.html

I think comment 17 was referring to the IRC-related idea in comment 16.

daniel added a comment.Via ConduitAug 17 2010, 3:31 PM

I finished XMLRC, an extension implementing this: http://www.mediawiki.org/wiki/Extension:XMLRC. I have set up a demo on the toolserver (fed by polling the API). You can join it at enwiki@conference.jabber.toolserver.org using any jabber client.

The human readable messages are similar to the ones send to the RC channels on IRC, but there's extra XML payload in the messages, like this:

<message xmlns="jabber:client" to="john@jabber.org/mars"

       from="enwiki@conference.jabber.toolserver.org/xmlrc"
       id="38" type="groupchat">
<body>[[List of Knight's Cross recipients: Z]]  
      http://en.wikipedia.org/w/index.php?diff=378715833&amp;rcid=389791968&amp;oldid=378714319&amp;title=List+of+Knight%27s+Cross+recipients%3A+Z * 
      MisterBee1966 * (+1203) /* Recipients */</body>
<rc comment="/* Recipients */ " newlen="26554" rcid="389791968"
    pageid="8089657" title="List of Knight's Cross recipients: Z"
    timestamp="2010-08-13T14:08:49Z" wikiid="enwiki" 
    revid="378715833" old_revid="378714319" user="MisterBee1966"
    ns="0" type="edit" oldlen="25351">
  <tags />
</rc>

</message>

The extension includes a demo client in python.

bzimport added a comment.Via ConduitAug 17 2010, 3:46 PM

soxred93 wrote:

You may want to set the xml:space="preserve" attribute in the <body> tag, so XML parsers will leave whitespace in the content alone.

daniel added a comment.Via ConduitAug 17 2010, 3:58 PM

(In reply to comment #21)

You may want to set the xml:space="preserve" attribute in the <body> tag, so
XML parsers will leave whitespace in the content alone.

Uh, i could, but... there's no text, all information is in attributes. So i see no point...

daniel added a comment.Via ConduitAug 17 2010, 3:59 PM

(In reply to comment #21)
or do you mean in the regular xmpp message body? that'S up the xmpp library, i won't mess with that. anyway, it's all on one line.

bzimport added a comment.Via ConduitAug 17 2010, 5:36 PM

soxred93 wrote:

All right then, don't bother with the library. I would LOVE to see this, at least on test.wikipedia.org for testing.

bzimport added a comment.Via ConduitAug 17 2010, 11:33 PM

soxred93 wrote:

Quick query, why use just XML? It would be nice to be able to get an alternate format on request, such as JSON or serialized PHP.

daniel added a comment.Via ConduitAug 18 2010, 7:29 AM

(In reply to comment #25)

Quick query, why use just XML? It would be nice to be able to get an alternate
format on request, such as JSON or serialized PHP.

Two reasons:

  1. XMPP supports XML payload - that means I can embed XML natively in XMPP stanzas. No need for wrapping, escaping or encoding. It will also get parsed automatically, and is accessible directly in the xmpp client framework.
  1. there are no requests. Everyone gets the same data, this is basically a multicast setup. Broadcasting everything in all formats would be silly.

That being said: I'm thinking about making an XMPP interface to the API, so people can send queries and receive answers via XMPP. In that case, supporting different formats would be an option, though it still seems silly: with XMPP, XML parsing comes for free, handling any other format adds work and cost.

daniel added a comment.Via ConduitAug 18 2010, 8:30 AM

I have described the prototype setup at http://meta.wikimedia.org/wiki/Recentchanges_via_XMPP

Ireas added a comment.Via ConduitOct 11 2010, 4:11 PM

I would appreciate having XMLRC enabled on the Wikimedia wiki family soon as it is *really* helpful in writing RC appliations. Especially on writing new programs and scripts (as I am doing at the moment) it is suboptimal if you have to implement IRC support knowing that a. it will be outdated soon and b. it is very complicated to extract the real information from the mixture of color codes and special characters. So I want to confirm the previous wishes for quick activation.

bzimport added a comment.Via ConduitOct 11 2010, 5:11 PM

soxred93 wrote:

Just realized that this would not be able to be placed in core as it stands (which is what I would like to see eventually), due to the Python prerequisite.

daniel added a comment.Via ConduitOct 11 2010, 9:29 PM

perhaps the bit that emits the udp packets could be split of and put into core - that's the part that is actually a proper mediawiki extension. the rest is utilities, really...

MZMcBride added a comment.Via ConduitOct 11 2010, 10:54 PM

(In reply to comment #24)

All right then, don't bother with the library. I would LOVE to see this, at
least on test.wikipedia.org for testing.

This is bug 24836.

(In reply to comment #29)

Just realized that this would not be able to be placed in core as it stands
(which is what I would like to see eventually), due to the Python prerequisite.

Well, there's nothing to say that core has to be pure PHP. In fact, parts currently aren't (like math support). It should certainly be discouraged, though.

bzimport added a comment.Via ConduitOct 12 2010, 12:16 AM

soxred93 wrote:

math is in there for historical reasons. However, I do know that extensions like timeline, etc have been denied merging with core for non-PHP-ness.

Nemo_bis added a comment.Via ConduitFeb 19 2012, 10:03 AM

This feature would be the long-term solution to problems like 34508 (not marking as blocker since a near-term solution will be pursued perhaps).

gerritbot added a comment.Via ConduitMay 5 2014, 1:17 AM

Change 131040 had a related patch set uploaded by Ori.livneh:
Add 'rcstream' module for broadcasting recent changes over WebSockets

https://gerrit.wikimedia.org/r/131040

gerritbot added a comment.Via ConduitMay 9 2014, 12:21 PM

Change 131040 merged by Faidon Liambotis:
Add 'rcstream' module for broadcasting recent changes over WebSockets

https://gerrit.wikimedia.org/r/131040

siebrand added a comment.Via ConduitMay 9 2014, 12:38 PM

Sometimes it takes more than 5 years, but it's great to see that stuff is happening!

Krinkle added a comment.Via ConduitMay 13 2014, 4:59 PM

Closing this bug in MediaWiki core.

  • MediaWiki now provides the wgRCFeeds and MachineReadableRCFeedFormatter interfaces. These don't set up their own subscribable web service, but that seems outside of the scope for MediaWiki. The interface to broadcast to such a service (set up by the server admin) exists, and that seems enough.
  • While this bug specifically requested XMPP, the overall use case for Wikimedia wikis is bug 14045. It focusses on setting up a public service that will make use of this feed for Wikimedia wikis (though JSON-based, not XML like this bug suggested). If we really want XMPP specifically, we should file a new bug for that I think,
edsu added a comment.Via ConduitMay 13 2014, 6:12 PM

Thanks for this work, much appreciated!

Add Comment