[RFC] Compat deprecation
Open, Needs TriagePublic

Description

Background

Currently, the Pywikibot framework is split in two branches: 'core' (formerly 'rewrite') has full API support and is much more up-to-date with new MediaWiki features, while 'compat' (formerly 'trunk') is old and largely based on screen-scraping.

'compat' currently gets no new features, and bug fixes are implemented very slowly. This means compat effectively is deprecated already, but this has not been explicitly pointed out to the community.

Problem

There are several issues with this status quo:

  • Documentation is partially still compat-based, confusing newcomers
  • Even worse, some newcomers start with compat because of that
  • Bug fixes are expected when compat breaks

All in all, many wasted man hours that could have been used more effectively.

Goal

Defining the long-term deprecation status for compat, and communicating this to the community.

What do we mean with 'deprecation'?

  • Code:
    • Do 'we' no longer provide new features?
    • No general bug fixes, but still some important ones (e.g. api access broken)?
    • No bug fixes at all?
    • Do we still accept 3rd party bug fixes?
  • Documentation:
    • Just mark documentation as deprecated?
    • Move it?
    • Remove it?
  • Warning for users they should migrate?

Deprecation does *not* mean:

  • Actively preventing compat users from using compat

Blocking issues

Before we can deprecate compat, these issues *must* be solved:

  • ...

Open questions

  • How do we define deprecation? (see above)
  • What to do with the large set of scripts that have not been migrated.
  • How to communicate the deprecation to the communities
  • Do we need more extensive 'porting to core' documentation?

Blocking issues

There is a list of potentially useful things at T57880: Functionality existing in compat but missing from core (tracking), but none of them look blocking to me.

  • The open questions above
  • Providing good getting started documentation {T???}

Timeframe

  • April 2014: original RFC
  • May 2015: moved to Phabricator
  • mid May 2015: Hackathon discussions on the following subjects:
    • ...

Related discussions

Related Objects

StatusAssignedTask
OpenNone
OpenNone
ResolvedXqt
OpenNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
valhallasw updated the task description. (Show Details)
valhallasw raised the priority of this task from to Needs Triage.
valhallasw added subscribers: valhallasw, XZise, Ladsgroup and 4 others.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 16 2015, 2:02 PM
valhallasw updated the task description. (Show Details)May 16 2015, 2:18 PM
valhallasw set Security to None.
valhallasw updated the task description. (Show Details)May 16 2015, 2:32 PM
Ladsgroup added a comment.EditedMay 16 2015, 2:44 PM

I wrote this, please check out and feel free to change or add anything you want:
https://www.mediawiki.org/wiki/User:Ladsgroup/Sandbox

We will use this as the notice for deprecation.

Steps we should take in order to let users know about deprecation of compat:
1- Add a notice in compat code and link them the page I created. Something like this https://www.mediawiki.org/wiki/Special:Code/pywikipedia/9971
2- Send a mass message to all village pumps and send the notice
3- Send an email to pywikipedia-l pywikipedia-announces, and wikitech-l

What do you think?

  1. not sure if we should. See e.g. http://thread.gmane.org/gmane.comp.python.pywikipediabot.general/13901 . If people want to keep using compat, that's fine. Just don't ask us to fix bugs ;-)
  1. Maybe. I think just adding it to Tech News might be more reasonable.
  1. pywikipedia-announce: yes. wikitech-l: maybe, although I don't think that will reach any audience that's not already reached with pwb-announce.
XZise added a comment.May 16 2015, 7:06 PM

To be honest I'm not sure how important this is. Compat's source code is for everyone available and for people who still want to use and support it they may use it. I personally are not interested in the compat development. So if I don't see anything where I am of help and is compat specific I let others do it.

The compat people will say core is not usable and core people will say compat is outdated and the only outcome (I see) of this is that the operators of the compat version will be upset. You could argue that we could remove the compat documentation and made it clearer as it doesn't need to apply to both versions, but it should be separated anyway more clearly.

Compat has been de facto deprecated for quite some time. There's nothing left to do in order to deprecate compat.
What we should do is to help 'casual' users make the move, and of course to separate documentation more accurately.

jayvdb added a subscriber: jayvdb.May 18 2015, 8:32 AM

I think the wiki documentation should only help users use core. Compat documentation should be moved to extra pages with big warning at the top of the page.

JAnD added a subscriber: JAnD.May 18 2015, 12:00 PM

There are still issues in core which make it lower usable than compat for interwiki (on wiktionary):

Major:
T74674

Normal:
T91906
T87222
T75934

Minor:
T92822
T86493
T74943

and maybe others

Statistics for compat usage: there's about 20 unique IPs using it. We're working on more detailed stats, among others T100111: Update user-agent in compat to include username

@valhallasw how is it possible to track the IPs that are using compat?

@Alex_brollo your migration is a success story, isn't it?

@Candalua your bot is on the blacklist :-P

ArchCom trusts the pywikibot developers to make the right decision, so removing the MediaWiki-RfCs tag :)

I just run some banal "interactive" scripts but yes, migration has been not so painful (a matter of minutes) and recent , few contributions of Alebot come from core.

Consider that I'm far from a true programmer/developer..... my main reference dealing with Python doc is its tutorial.

@valhallasw how is it possible to track the IPs that are using compat?

The WMF statistics team has access to server logs, and is able to provide statistics such as these. My statement was broader than it should have been, however, so please allow me to rephrase: there were about 20 unique IPs using compat on WMF wikis in a one-hour time frame during the Lyon Hackathon.

binbot added a subscriber: binbot.Aug 12 2015, 6:12 AM

I'm here just to tell you I'm finally moving to core, after checking that it (finally) works for me (in my last try, I couldn't get HTTP responses from servers). I may have been missed from your stats as I usually run my bot in batch once or twice a month.

I just met https://phabricator.wikimedia.org/T57689 and was surprised to see that some existing funcionalities of compat are not inplemented to core, but may be requested as extra feature.

Isn't it a basic requirement to port full funcionality to new branch before deprecating old one?

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptJan 24 2016, 5:05 PM

I just met https://phabricator.wikimedia.org/T57689 and was surprised to see that some existing funcionalities of compat are not inplemented to core, but may be requested as extra feature.
Isn't it a basic requirement to port full funcionality to new branch before deprecating old one?

Pywikibot is an open source project, so there are no 'requirements', just 'things that active developers think are important'. 'Providing support for compat' does not fall in that category, nor does 'providing every feature that was in compat'. Instead, we are focusing on making Pywikibot better, and while adding some features from compat fall into that category, not all of them do. Hence the 'feature request' method.

To answer your second question, I'm going to reply with a question. What does it mean to deprecate something?

I'm asking this because compat effectively is already deprecated. No-one is required to provide support, and deprecation simply means "if it breaks, don't expect us to fix it". At the moment, @Xqt is still running some compat bots, and keeps compat running because of that. This might no longer be the case if he's no longer interested, or if the required changes become too extensive. If that's a risk you're willing to take: keep using compat. If it's not, either step up to take on that maintenance task, or move to core, even if it doesn't provide you all the features you would like.

Ideally try to use core for everything. The vast majority of pywiki bots are using core; it has the 'compat' features most people need, and many many more which were never in compat. If you find there is missing functionality that you need, which existed in compat, let us know about them. Prioritisation can help; i.e. add a comment here with the top 5 missing features, so we can fix them first.

binbot added a comment.EditedAug 29 2016, 8:03 AM

First of all: I implemented many new parameters to replace.py. -save, -savenew, -savexc, -savexcnew and edit counters which I remember now. These are missing from core, and I would really appreciate if someone ported them.

(Requirements belong to programming, not to open/closed source. Backward compatibility is a general expectation and is broken in very rare cases.)

solve_disambiguation.py is also missing -first parameter that was implemented in compat.

binbot added a comment.EditedSep 3 2016, 6:23 AM

I also implemented the 'include' key to 'exceptions' dictionary in replace.py. That is very much missing from core, too. Lack of these features (with the above mentioned) makes core totally useless for me as I would face all the difficulties with that I began to work back in 2008. All these should be just ported from compat.

@binbot, can you check Pywikibot-replace.py and Pywikibot-solve-disambiguation.py for the missing functionality, and create new bugs for any that do not have an existing bug.

binbot added a comment.EditedSep 4 2016, 1:19 PM

@jayvdb, done, you see the mentionings above. This is my response to your "add a comment here with the top 5 missing features, so we can fix them first" sentence.

Of course, other people may have other preferences. I always read that interwiki.py users still use compat. For me, these are the most important, as I use replace.py in 90% of my time, and disambig in the second place. Thanky you in advance.

XXN added a subscriber: XXN.Nov 13 2016, 4:16 PM

The last person maintaining compat ( @Xqt ) proposes at https://lists.wikimedia.org/pipermail/pywikibot/2016-November/009635.html to finally close compat down. I agree. Probably next steps:

  • Close all the open compat bugs with a link to this task
  • Close the compat boards here in phabricator
  • Some final commit so the compat library always outputs a big warning? (or do we already do that?)
  • Update all documentation to remove references to compat and to archive old compat pages (https://www.mediawiki.org/wiki/Manual:Pywikibot )
  • .....

Regarding Phabricator, https://phabricator.wikimedia.org/tag/pywikibot-compat/ has a "Unported (still maintained)" column and I'm not sure what that means.
There are 29 open tasks that are in Pywikibot-compat && not in Pywikibot-core (and one of them, T57056, which is in Pywikibot-compat-to-core). In case you plan to close them as declined an explanatory comment should get added.

Change 322502 had a related patch set uploaded (by Xqt):
Decommission compat

https://gerrit.wikimedia.org/r/322502

XXN added a comment.Nov 20 2016, 4:29 PM

Are there some statistics on the compat usage in last 3 months?

In T99365#2809160, @XXN wrote:

Are there some statistics on the compat usage in last 3 months?

Given the fact that it's broken on WIkimedia wiki's, I assume no edits.

XXN added a comment.EditedNov 20 2016, 5:23 PM
In T99365#2809160, @XXN wrote:

Are there some statistics on the compat usage in last 3 months?

Given the fact that it's broken on WIkimedia wiki's, I assume no edits.

Nope. At least I used it a lot, until T150613: Compat login broken. KeyError: 'lgtoken'. But besides me, there should be many other compat users, especially on wiktionaries and other WMF non-Wikipedia wikis.

XXN added a comment.Nov 20 2016, 9:56 PM

If tickets related to compat are not allowed anymore on phabricator.wikimedia.org, maybe it's time to reopen, revive https://sourceforge.net/projects/pywikipediabot/ and to make it 'home' for compat branch only? Not every MediaWiki wiki in this world uses a fresh, recent version of MW. WMF wikis are an exception, I'd say.

Xqt added a comment.Nov 20 2016, 11:51 PM

Please don't. Core does and should support MediaWiki up from version 1.14 which has the earliest API version. It is not forbidden to use compat anymore but I think the developer team will not support it. I wouldn't say you must not. But imho we should concentrate to improving the newer core branch now.

Compat is dead, long live core. Compat has been in a zombie state for too long. Now it's time to bury it.
1.14 is ancient, see https://www.mediawiki.org/wiki/Release_notes/1.14 . If people are still running that version or an older one, it's time to upgrade. See https://www.mediawiki.org/wiki/Version_lifecycle for a full overview of versions.

Change 322502 merged by jenkins-bot:
Decommission compat

https://gerrit.wikimedia.org/r/322502

Base added a subscriber: Base.Dec 14 2016, 2:04 AM