Page MenuHomePhabricator

Archive variables being changed onwiki and saved as such
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • Run archivebot

What happens?:
It converts %d to %s on wiki and leaves it that way

What should have happened instead?:
It should leave it as %d or it breaks backwards compatibility. This is what one of our bc’s said
“ This has become a bit more complicated now, because the latest patch explicitly states the %(...)d format string convention to be a valid one (lines 55 to 67). It is not acceptable for the bot to be modifying the on-wiki configuration from %d to %s. I believe the root cause for this behavior is at lines 598 to 600, where the bot internally performs a regex substitution to transparently account for non-latin digits. The issue is that this modified format string should not be saved back to the page on-wiki. Until this has been clarified on the developer's side, please hold off on any announcement for the d to s change, because it will be a breaking (and potentially backward-incompatible) change. Chenzw [he]“

Software version (skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):

Event Timeline

Also, from the trial run it had converted my %d to %s so that needs to be reverted as well.

Mentioned in SAL (#wikimedia-cloud) [2022-07-26T09:11:12Z] <wm-bot> <maurelio> Temporarily suspend archivebot operations refs. T313785

It should leave it as %d or it breaks backwards compatibility.

Having %s instead of %d does not break anything because %s can work with any python types whereas %d accepts real numbers only.

Having %d instead of %s lead archivebot to fail on a few of small wikis like bn, chk, fa, hi and others for seven years but it would work with %s. With Python 7.5.0 the textlib.to_local_number function was fixed but unfortunately this broke archivebot when having %d variable fields. Therefore a patch was made fix this issue by converting %d to %s because all fields are strings not numbers after textlib.to_local_number changes.

With rPWBC5178710 the 7 years old patch was dropped and few new fields were intoduced. This patch is part of the current master and will be ported to stable release in about 4 weeks. The new fields must use %s formatting whereas the old field can use %d formatting for the number fields. The following table shows what happens for counter field and the new localcounter field as a sample.

Small wikis

Language code: bn, chb fa, gu, hi, km, kn, lo, lo, ml, or, pa, te
All numbers are localized with Pywikibot 7.5.1 and below. With Pywikibot 7.6 the local... fields must be used instead.

Pywikibot release%(counter)s%(counter)d%(localcounter)s%(localcounter)d
7.6.0 master
7.5.2 stable
7.5.1
7.5.0
7.4.0 and below

Other wikis

Pywikibot release%(counter)s%(counter)d%(localcounter)s%(localcounter)d
7.6.0 master
7.5.2 stable
7.5.1
7.5.0
7.4.0 and below

Legend

iconbehavior
Bot works
Bot works but templates are changed
Bot fails
WARNING: The fields monthname and monthnameshort must always have %s format.

In short: Using %s is always safe.

matej_suchanek,

I ran a test on my talk page and it again converted the "d" to "s" on wiki as you can see in this diff https://simple.wikipedia.org/w/index.php?title=User_talk:PotsdamLamb&type=revision&diff=8362575&oldid=8362568&diffmode=source.

Before I ran it, I did clone using git clone --recursive --branch stable "https://gerrit.wikimedia.org/r/pywikibot/core" $HOME/pywikibot

Now what I did was on my local install I ran git clone --recursive --branch master "https://gerrit.wikimedia.org/r/pywikibot/core" $HOME/pywikibot

This did not change the variable from "%d" to "%s" on-wiki as you can see in this diff https://simple.wikipedia.org/w/index.php?title=User_talk:PotsdamLamb&type=revision&diff=8362590&oldid=8362580&diffmode=source. Prior to running this I had changed the %s back to %d; this has the desired effect.

Is it possible the master branch did not replicate to the stable version?

Is it possible the master branch did not replicate to the stable version?

You can also run the master branch on toolforge. There is no impact to convert fields from %d to %s. The master branch is the development fork. It is merged to the stable branch roughly every month. The next merge is planned at the end of August.

Ok, so what will be the impact going forward? Will the stable version change the variable on wiki when released, or will it leave it as %d? I just need the solid answer as our BC wants to know before I can do more trials as it changed everyone's from d to s.

Yes, it will change all "d" to "s" on-wiki in the future due to.........

No, it will not change the "d" to "s" on the wiki, so there will be no issues, including backward compatibility.

Thanks.

Ok, so what will be the impact going forward? Will the stable version change the variable on wiki when released, or will it leave it as %d? I just need the solid answer as our BC wants to know before I can do more trials as it changed everyone's from d to s.

Yes, it will change all "d" to "s" on-wiki in the future due to.........

No, it will not change the "d" to "s" on the wiki, so there will be no issues, including backward compatibility.

Thanks.

See the table above.

  • Running the current master (7.6) will do no changed from %d to %s when find a %d field except for the new fields
  • Running the current stable (7.5.1) will change from %d to %s when find a %d field
  • Running older release 7.5 will always fail when find a %d field
  • Running older release 7.4 and below will always fail on several wikis when find a %d field
  • Running any release will never fail and no template changes are made when find a %s field
  • %s fields are backward compatible and upward compatible
  • new local fields are not backward compatible
  • The current master will become stable in August I guess

What and who is BC btw?

Hello @Xqt. By reading the above I see it's safe to continue to run the script, right? I'm asking because my bot (which runs on Toolforge's /shared/pywikipedia/core/scripts) did change on July 25th %(counter)d to %(counter)s and I just wanted to inquire if that'll cause any issues. My bot is not running on any of the following languages: bn, chb fa, gu, hi, km, kn, lo, lo, ml, or, pa, te. Thank you.

@MarcoAurelio To get it to change back I had to delete my .pywikibot folder and then clone the master that left it at d, however, I have to run AWB to get all of the changes made from d to s back to d.

Hello @Xqt. By reading the above I see it's safe to continue to run the script, right?

This depends on whether you are using the stable or master release.
Just run pwb.py version. For me it gives:

tools.xqtools@tools-sgebastion-10:~$ core/pwb.py version
Pywikibot: [https] r-pywikibot-core.git (e2a09f9, g1, 2022/07/25, 21:51:25, OUTDATED)
Release version: 7.6.0.dev0

for the master. The timestamp must be 2022/07/25 or newer and the hash must be 5178710 or e2a09f9 or 2cfba54.

For the stable release I get

Pywikibot: [https] r-pywikibot-core.git (3f39d0c, g1, 2022/07/25, 13:20:35, stable)
Release version: 7.5.1

It has to be 7.5.1 or newer.

I think I will backport the last few archivebot changes to stable soon. It is idle to debate whether a change of a bot's template is an impact or not even it is not as shown above. What comes with the new release?

  • localized field variables are supported for those small wikis listed above. This can break the archives if bot owner had used %s format strings.
  • fasten up the bot and double processing speed
  • catch KeyboardInterrupt
  • Look & feel of a CurrentPageBot and showing the execution time.

I also made an experimental implemetation which makes the bot 7 times faster but this will not be included with this backport.

Change 817363 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@stable] [backport] backport archivebot.py from master

https://gerrit.wikimedia.org/r/817363

Xqt changed the task status from Open to In Progress.Jul 26 2022, 7:35 PM
Xqt claimed this task.
Xqt triaged this task as Low priority.

Change 817363 merged by Xqt:

[pywikibot/core@stable] [backport] backport archivebot.py from master

https://gerrit.wikimedia.org/r/817363

Stable release 7.5.2 is published, I updated the table above. Use pwb.py version to verify the release. It should be either 7.6.0.dev2 or 7.5.2.

Thanks @Xqt - This is what I get:

tools.mabot@tools-sgebastion-10:/data/project/shared/pywikipedia/core$ python3 pwb.py version
Pywikibot: [https] r-pywikibot-core.git (3cec5b0, g1, 2022/07/26, 21:40:46, master)
Release version: 7.6.0.dev2
setuptools version: 40.8.0
mwparserfromhell version: 0.6.3
wikitextparser version: n/a
requests version: 2.21.0
  cacerts: /etc/ssl/certs/ca-certificates.crt
    certificate test: ok
Python: 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0]