Hi @elukey, and thanks for the ping. Our SUL watcher bots do certainly use this channel to report account creations to #cvn-unifications and deal with account creation vandalism if needed. The bot code (rTSTW SULWatcher/SULWatcher.py) parses the feed from that channel and filters it. We're having difficulties to maintain our bots due to lack of stewards with knowledge of Python. So if this channel goes, I'm not sure we'll be able to craft a replacement sadly. Best regards.

@MarcoAurelio thanks a lot for the feedback! I am trying to find use cases for the IRC channel, we don't want to cause disruptions to important bots. Do you have more info about how the account creation report is used by consumers of #cvn-unifications? I am wondering if the info is actively used or not to fight counter-vandalism, as you were saying some bots are old and they may not be up to date now (the info could also come from other places etc..)

#cvn-unifications is intensively used by stewards to detect abusive user names. It's the easiest (if not only useful) source of global account creations.

As a steward, I very much rely on #cvn-unifications.
The SUL watcher bots report all new account creations there, with a link to https://meta.wikimedia.org/wiki/Special:CentralAuth/username. That's very useful to lock LTAs very quickly or at least examine suspicious accounts. Also, there are some regexes to highlight potentially unsuitable user names. And of course you could use your IRC client to set "private" pings for names/name patterns that are of special interest for you.
Some LTAs troll in a specific wiki with a specific name pattern, and when their IP/IP range is blocked locally, switch to another wiki to continue. Without #cvn-unifications it's next to impossible to follow them, because you couldn't watch ~ 950 account creation logs manually. (Okay, you could watch Special:Log/newusers @ Loginwiki, but you need to reload that page permanently ...)

("LTA" is a Wikimedia abbreviation for "long-term abuser", often involving multiple IPs and/or accounts.)

@elukey do you know if SULWatcher is currently the only consumer of #central? If so, I think I can help port it to eventstreams once the current Python 3 migration happens (T216020).

For the records: I don't use cvn-unification that often, but I always get pinged by someone who uses it whenever the bot is down (within 12 hours, usually).

STRONG OPPOSE for now - until we migrate SULWatcher bot's system to use EventStreams/Kafka. #cvn-unification channel is highly valuable for us, SWMT members. I second Schniggendiller's comments in T242712#5821559.

Pywikibot has a powerfull eventstreams handler. Probably we can use it or its code parts for this proposal.

until we migrate SULWatcher bot's system to use EventStreams/Kafka

... Can we do that?

In T242712#5824196, @Ottomata wrote:

until we migrate SULWatcher bot's system to use EventStreams/Kafka

... Can we do that?

@Ottomata Sure, I guess that's posible but we have a problem: None of us (current stewards) either have the time or the knowledge about how to do this. Now I see @Legoktm very kind offer at T242712#5821880 to help us, which I accept of course. The code used by SULWatcher and StewardBot were both written by users that are no longer around, and we've just been able to keep it running until now.

In T242712#5823758, @Masumrezarock100 wrote:

STRONG OPPOSE for now <snip>

Comments like these aren't productive on Phabricator, thanks. Bold and all caps doesn't help make your point.

In T242712#5824196, @Ottomata wrote:

until we migrate SULWatcher bot's system to use EventStreams/Kafka

... Can we do that?

Yes. We just need there to be a public EventStreams feed to consume first :) Which I suppose is T240182: Create EventStream's equivalent to irc.wikimedia.org's #central channel.

Ah right, ok. I think we were hoping we wouldn't need the centralauth/account-add data as a stream but it seems like we do, even if we are able to deprecate #central in irc.wikimedia.org. Hm. Once the new event stream exists though, it'll be trivial to include it to be used by T234234: Port architecture of irc-recentchanges to Kafka in the same way that other streams will, so we might as well keep the #central channel around anyway.

Porting SULWatcher to EventStreams will still be nice though, but I guess it won't really let us deprecate #central.

In T242712#5821559, @Schniggendiller wrote:

[…]
(Okay, you could watch Special:Log/newusers @ Loginwiki, but you need to reload that page permanently ...)

The #central feed is currently powered by CentralAuth in MediaWiki directly. The same code that is called after recent changes and log events, is called by CentralAuth directly from a hook to power #central.

The mention of loginwiki is interesting here. Note that loginwiki did not exist until relatively recently, it certainly is much newer than the #central feed.

It is worth checking whether the recent changes feed for loginwiki/logging/newusers contains enough information to reproduce the essence of #central. If so, then T240182 could be implemented as a light transformation step by consuming the existing machine-readable recentchanges feed, without needing support from MediaWiki PHP and CentralAuth to provide it explicitly. If the basic information is equal or a superset, then I suppose the two feeds could be compared side-by-side for a window of say 24 hours to uncover any discrepancies (e.g. missing or unexpected items).

@Legoktm sorry for the late reply, thanks a lot for the help!

@Krinkle thanks a lot for the info, can you add a bit more detail about:

It is worth checking whether the recent changes feed for loginwiki/logging/newusers contains enough information to reproduce the essence of #central.

Namely, what concrete things do I need to check? The eventstream's recentchanges feed for wiki: login or similar? Or other feeds etc..? (Just to understand how to do the comparison, this test seems promising but it should be done asap since kraz is running Jessie and needs to be moved to Buster soon).

Get a sample from #central on IRC and from wiki=login.wikimedia on EventStreams, then in the latter look for log/newusers entries that correlate with the lines in #central and see if all meta data (numbers, user names, other strings, etc.) are present there.

I tried a couple of things:

• kafkacat -b kafka-jumbo1001.eqiad.wmnet:9092 -t eqiad.mediawiki.recentchange -C | grep login
• curl https://stream.wikimedia.org/v2/stream/recentchange | grep login

In both cases I don't see any event related to wiki=login, are we sure that we have it in recent changes?

Also I see some differences between the info in https://login.wikimedia.org/wiki/Special:Log and https://login.wikimedia.org/wiki/Special:RecentChanges, for example the latter doesn't show info about user creation.

In T242712#5856672, @elukey wrote:

[…] I don't see any event related to wiki=login, are we sure that we have it in recent changes?

Also I see some differences between the info in https://login.wikimedia.org/wiki/Special:Log and https://login.wikimedia.org/wiki/Special:RecentChanges, for example the latter doesn't show info about user creation.

Hm.. you're right. It is one of a few rare LogEntry objects that we insert() on (for the logging table behind Special:Log), but not publish() (for the recentchanges table behind Special:RecentChanges, and RCFeed/EventStreams).

For new account registrations from any wiki, source code calls insert() and publish().

However when visiting a (different) wiki for the first time and auto-creating/attaching a local account, we only call insert() (source code).

As example:

• If you create a new account on say en.wikipedia.org or www.mediawiki.org, then that is a newusers/create log entry and RCFeed event for that specific wiki.
• Immediately after sign up, via async JS requests, the same account is also auto-created locally on login.wikimedia.org, Commons, Meta-Wiki and a handful of others. That creates newusers/autocreate log entries, but no RCFeed events. Also during the autocreation procedure, the CentralAuth extension emits a hardcoded message directly to RC2IRC #central channel (source code).
• At any point in in the weeks/months/years after this, whenever you first visit a public WMF wiki you haven't visited before, the same autocreation procedure happens (after handshaking and bouncing in the background via login.wikimedia.org).

So as-is we can't simply filter the existing recentchanges feed to get this information. Some options:

• Chat with stewards of MediaWiki's "Authentication and Authorization" (CPT and @Tgr) and determine based on product wishes, scalability, and community/UX aspects whether it's okay to have these go to recent changes as well. This means adding a call to $logEntry->publish(). Then from there, the above idea could still work.
• As lighter and less controversial option, would be to add a call to $logEntry->publish('udp'). Passing the string 'udp' here is confusing way of telling it to only notify RCFeeds (such as irc.wikimedia.org and EventStreams) without adding database entries to recentchanges. This means we don't have any database or UX considerations.
• Alternatively, we could go from a hardcoded optional "IRC" plug in CentralAuth, to a hardcoded optional "EventBus" plug in CentralAuth. Some minimal conditional statement that then calls EventBus to emit the event in some form directly into some kind of centralauth related topic.

@Ottomata any opinion about the last two ideas from Timo?

I think any of these ideas would work. For the third option (the EventBus option), if we do that, maybe a more generic account creation stream is more useful? Is there a hook that would allow us to fire an event every time an account is created? The #central IRC channel could could then just filter and transform that stream.

Autocreation would probably be too spammy for recentchanges. (It's less annoying than wikibase / category changes in that it's a log event and so it gets collapsed into a single item per day in the modern RC interface, but not everyone uses that.) Only publishing to RCFeed sounds fine.

In general, using loginwiki autocreations seems unnecessarily fragile. They are not a necessary byproduct of account creation, we just have a job that autocreates there after registration since the antivandal people said having all accounts searchable there helps their work. If that job breaks, reporting of account creations should ideally not break.
(OTOH, I guess it would at least make it very obvious when it breaks.)

Wrt EventStreams, it's not clear to me whether that's production infrastructure or analytics infrastructure (with weaker commitments on how long it is acceptable to delay fixing it if it breaks)?

In T242712#5861741, @Tgr wrote:

Autocreation would probably be too spammy for recentchanges. (It's less annoying than wikibase / category changes in that it's a log event and so it gets collapsed into a single item per day in the modern RC interface, but not everyone uses that.) Only publishing to RCFeed sounds fine.

In general, using loginwiki autocreations seems unnecessarily fragile. They are not a necessary byproduct of account creation, we just have a job that autocreates there after registration since the antivandal people said having all accounts searchable there helps their work. If that job breaks, reporting of account creations should ideally not break.
(OTOH, I guess it would at least make it very obvious when it breaks.)

Thanks a lot for the feedback!

Wrt EventStreams, it's not clear to me whether that's production infrastructure or analytics infrastructure (with weaker commitments on how long it is acceptable to delay fixing it if it breaks)?

EventStreams will be migrated to Kubernetes this quarter, and it is a service with shared responsibility between Analytics and SRE basically. It can be considered production in my opinion.

Quick summary for everybody about the current status:

irc.wikimedia.org runs on kraz, a Debian Jessie host that needs to be migrated to Buster very soon (by the end of quarter would be ideal). The current architecture has been revamped by Faidon (see parent tasks, especially T234234) and he wrote a Python prototype of a new irc.wikimedia.org backend that uses eventstreams rather than the UDP recent changes feed (that hopefully will be decommed at the end of this migration). In order to test this properly, we'd need to have the #central equivalent channel/feed to be populated from eventstreams, that is what we are discussing in this task. So it would be really important in my opinion that we'd reach an agreement about how to proceed, in order to unblock testing of the new stack :)

Any help is really appreciated, thanks all for the collaboration given the small notice!

@Tgr do you have any preference among the solutions proposed by Timo? If not, is there anything that would recommend to explore/think-about ?

he wrote a Python prototype of a new irc.wikimedia.org backend that uses eventstreams rather than the UDP recent changes feed (that hopefully will be decommed at the end of this migration). In order to test this properly, we'd need to have the #central equivalent channel/feed to be populated from eventstreams

To be precise, the new irc.wm.org backend uses Kafka (not EventStreams), and we need the #central equivalent channel/feed to be produced to Kafka, likely via EventBus either as a new stream (sounds like this would be better) or in RCFeed (which EventBus itself uses to produce the recentchange event stream to Kafka).

https://wikitech.wikimedia.org/wiki/Event* :p

In T242712#5864925, @Ottomata wrote:

he wrote a Python prototype of a new irc.wikimedia.org backend that uses eventstreams rather than the UDP recent changes feed (that hopefully will be decommed at the end of this migration). In order to test this properly, we'd need to have the #central equivalent channel/feed to be populated from eventstreams

To be precise, the new irc.wm.org backend uses Kafka (not EventStreams), and we need the #central equivalent channel/feed to be produced to Kafka, likely via EventBus either as a new stream (sounds like this would be better) or in RCFeed (which EventBus itself uses to produce the recentchange event stream to Kafka).

https://wikitech.wikimedia.org/wiki/Event* :p

My impression from @faidon was that he wanted to use EventStreams, not Kafka directly, so the final version of the tool may change. I think that we should assume that the #central feed replacement should be available in ES, not only kafka, but I might be wrong..

Hm. I didn't realize that. Moving to other ticket to discuss...

Untagging stewardbots, SULWatcher no longer uses this channel

From what I can tell, SULWatcher was the only reason stated so far (did I miss something?) for keeping the #central channel. Is this correct?

Reviving this old task, if anybody could follow up I'd be grateful :)

My understanding now is that #central is not used anymore by the SULWatcher, and that #cvn-unifications is not part of irc.wikimedia.org. If so I Think that we could attempt to send an email to Wikitech-l asking for people using #central, and deprecate it if nobody claims it. Maybe a soft deprecation, like removing the channel from irc.wikimedia.org for a while and wait for feedback.

In light of T348388, I'm checking to make sure SULWatcher does not rely on creations happening on login.wikimedia.org. In reviewing its code, I find that it is using stream.wikimedia.org and looking across all wikis for newusers log events. The reason this doesn't double report auto creations is that autocreation skips the RC publish step (as detailed in T242712#5858564). Even if we did RC_publish them, SULWatcher coudl be easily fixed by checking for newusers/create specifically since we do differiate that subtype from newusers/autocreate already.

What happens today is that account creation happens locally on whichever wiki the action was initiated, and then an autocreate is triggered immediately after that on loginwiki and metawiki.

For example, last 3 samples from SULWatcher at #wikimedia-unifications:

• Skyrunner10 https://meta.wikimedia.org/wiki/Special:CentralAuth/Skyrunner10
  • Skyrunner10 on de.wikipedia.org
• Niamkedebora https://meta.wikimedia.org/wiki/Special:CentralAuth/Niamkedebora
  • Created on fr.wikipedia.org
• Sjrjoel https://meta.wikimedia.org/wiki/Special:CentralAuth/Sjrjoel
  • Created on es.wikipedia.org

In SUL3 this will stay exactly the same (it'll happen via a central domain, but not a central wiki, it's application-wise and backend-side the same).

So yeah, looks like we won't be needing this anymore. I'm not aware of other consumers. Are there any in the irc.wikimedia.org #central channel in the past 24 hours? Do they have contact info?

SULWatcher does not rely on creations happening on login.wikimedia.org. In reviewing its code, I find that it is using stream.wikimedia.org and looking across all wikis for newusers log events

Really!? Interesting. Link to code?

In T242712#9919028, @Ottomata wrote:

SULWatcher does not rely on creations happening on login.wikimedia.org. In reviewing its code, I find that it is using stream.wikimedia.org and looking across all wikis for newusers log events

Really!? Interesting. Link to code?

T242712 (this task) -> T244542 -> https://gerrit.wikimedia.org/r/c/labs/tools/stewardbots/+/670326/.

Latest version (unchanged): https://gerrit.wikimedia.org/g/labs/tools/stewardbots/+/687d2d13a6/SULWatcher/SULWatcher.py#810

Subscribes to recentchanges and processes newuser event from any wiki. No mention of loginwiki, or login.wikimedia. It works directly from the local wiki events, as it should :)

Thanks!

The only users that I can see are in #central:

[ BanBot] [ masti] [ op-56495]

No contact info under /whois afaics :(