Page MenuHomePhabricator

Pages starting with "ß" (german sharp s) are saved as "SS..."
Closed, ResolvedPublic

Description

Steps to reproduce

  1. Find a page starting with "ß" on cswiki
  2. Run pwb.py touch (just and only) on that page

Expected behavior
Page is touched. Nothing special happens.

Current behavior
Bot saves the page content under a new title "SS..." (e.g. "SSwhatever" for original title "ßwhatever") with summary "Pywikibot touch edit". For example bot tried to rewrite page called "SS" with the content of page "ß" or created a new page "Talk:SS-YbAlB4" with content of a page "Talk:ß-YbAlB4" on cswiki.

Configuration
Python 3.6.2, Pywikibot core master last commit

Event Timeline

Dvorapa created this task.Oct 26 2017, 8:19 PM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptOct 26 2017, 8:19 PM
Dvorapa updated the task description. (Show Details)Oct 26 2017, 8:23 PM

A page starting with ß (in German) doesn't exist. Maybe a page 'ß' (the letter).

A page starting with ß (in German) doesn't exist. Maybe a page 'ß' (the letter).

Yes, just page ß and also some users have got usernames starting with this letter. But bot is broken on these sites

Xqt added a subscriber: Xqt.Oct 29 2017, 7:49 AM

I cannot reproduce that problem neither with py 3.6.1 nor with 2.7.13.

I took the following commands:

py -2 pwb.py touch -simulate -lang:cs -page:ß
py -3 pwb.py touch -simulate -lang:cs -page:ß

$ python pwb.py cosmetic_changes -page:"Portál:ß"
ATTENTION: You can run this script as a stand-alone for testing purposes.
However, the changes that are made are only minor, and other users
might get angry if you fill the version histories and watchlists with such
irrelevant changes. Some wikis prohibit stand-alone running.
Do you really want to continue? ([y]es, [N]o): y
Retrieving 1 pages from wikipedia:cs.
WARNING: Page "Portál:SS" does not exist on wikipedia:cs.

1 pages read
0 pages written
Execution time: 0 seconds
Read operation time: 0 seconds
Script terminated successfully.

The same for hiding Portál:ß into -file:"list.txt", therefore it is not in my bash/terminal.

Dvorapa added a comment.EditedNov 9 2017, 12:27 PM

For your case:

$ python2 pwb.py touch -simulate -lang:cs -page:ß
Retrieving 1 pages from wikipedia:cs.
SIMULATION: edit action blocked.
Page [[ß]] saved

1 pages read
0 pages written
Execution time: 0 seconds
Read operation time: 0 seconds
Script terminated successfully.
$ python pwb.py touch -simulate -lang:cs -page:ß
Retrieving 1 pages from wikipedia:cs.
SIMULATION: edit action blocked.
Page [[SS]] saved

1 pages read
0 pages written
Execution time: 0 seconds
Read operation time: 0 seconds
Script terminated successfully.

Therefore it is broken only for Python 3, Python 2 works as expected

Dvorapa added a comment.EditedNov 9 2017, 12:34 PM

Cosmetic_changes are not causing this issue, this must be broken in python3 or maybe in pagegenerators?:

$ python2 pwb.py replace -cc -page:"Portál:ß" "." "," -simulate -summary:"Test"
NOTE: option cosmetic_changes is False

Retrieving 1 pages from wikipedia:cs.


>>> Portál:ß <<<
@@ -2,2 +2,2 @@
- Test. 
+ Test, 
- Test II.
+ Test II,

Do you want to accept these changes? ([y]es, [N]o, [e]dit original, edit [l]atest, open in [b]rowser, [a]ll, [q]uit): a
SIMULATION: edit action blocked.

1 pages changed.
$ python pwb.py replace -cc -page:"Portál:ß" "." "," -simulate -summary:"Test"
NOTE: option cosmetic_changes is False

Retrieving 1 pages from wikipedia:cs.
Page [[Portál:SS]] not found

0 pages changed.

Pywikibot shell:

$ python pwb.py shell
Welcome to the Pywikibot interactive shell!
>>> family = pywikibot.Site("cs")
>>> pywikibot.Page(family, "ß")
Page('SS')

@Xqt I think I've found the issue:

$ python2 pwb.py shell
Welcome to the Pywikibot interactive shell!
>>> print("ß".upper())
ß
$ python pwb.py shell
Welcome to the Pywikibot interactive shell!
>>> print("ß".upper())
SS

When calling pywikibot.Page(family, title) there is a method first_upper() used on page title right? And first_upper() is just a upper() applied to first letter of the string. The function behavior changed between Py2 and Py3, but Wikimedia allows to create an article with "ß" at the beginning of title

Xqt added a comment.Nov 9 2017, 7:39 PM

Oh great!

Xqt triaged this task as High priority.Nov 9 2017, 7:40 PM
Xqt edited projects, added Pywikibot-General; removed Pywikibot-cosmetic-changes.py.

Change 390582 had a related patch set uploaded (by Matěj Suchánek; owner: Matěj Suchánek):
[pywikibot/core@master] [Bugfix] Do not capitalize ß in first_upper

https://gerrit.wikimedia.org/r/390582

Change 390582 merged by jenkins-bot:
[pywikibot/core@master] [Bugfix] Do not capitalize ß in first_upper

https://gerrit.wikimedia.org/r/390582

Xqt closed this task as Resolved.Nov 11 2017, 1:02 PM