Usage of unicode_literals from future package
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	XZise
	Feb 15 2015, 11:56 AM

Description

Hi, the code should use unicode_literals throughout to avoid using u'…' which is one the reasons why Python 3 versions before 3.3 aren't supported. @jayvdb said somewhere (determine), that he'd prefer to do that in one go. I suggest to atleast do it in all new code, so that changing u'…' into '…' won't break git blame for those files when it's introduced later. It is possible to use u'…' on conjunction with unicode_literals, but that could confuse newcomers into thinking that u'…' is necessary.

One primary advantage is that the usage of ur'…' has been disallowed in Python 3 so unicode_literals is a way around it, as that does allow to use r'…' with unicode chars inside.

In case we want to revert the unicode_literals patch 1e54a7d6 the following patches are solely fixing the original patch and could be reverted with that too:

a0a194d5ca54
0c4197272821
853e6b0bdce3
4ca47e10ab24
4426e881762b
1a6d29b80c0c

Details

Subject	Repo	Branch	Lines +/-
[cleanup] cleanup docs/conf.py	pywikibot/core	master	+17 -15
[cleanup] cleanup some tests/ files	pywikibot/core	master	+34 -31
[cleanup] cleanup tests/[data_ingestion_tests.py-exceptions_tests.py]	pywikibot/core	master	+221 -163
[cleanup] cleanup scripts/welcome.py	pywikibot/core	master	+179 -176
[IMPROV] Use unicode_literals	pywikibot/core	master	+466 -98

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Invalid	None	T72936 Important tasks to be solved (tracking)
Resolved	Xqt	T60053 Pywikibot Python 3 compatibility (tracking)
Resolved	jayvdb	T94688 reflinks.py doesn't save changes to articles
Resolved	xSavitar	T89589 Usage of unicode_literals from __future__ package
Resolved	jayvdb	T95803 UnicodeDecodeError when using pywikibot delete.py
Resolved	XZise	T95671 UnicodeDecodeError when using pywikibot category.py
Declined	None	T95810 config does differ between unicode and str
Declined	None	T99366 error with httplib2 and unicode_literals: UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1872: ordinal not in range(128)
Duplicate	None	T95809 Page's repr returns invalid data causing Python to error
Resolved	jayvdb	T111615 pywikibot.__all__ is unicode, causing `TypeError: Item in ``from list'' not a string`
Resolved	jayvdb	T114487 Pywikibot master setup.py won't install
Resolved	Dalba	T114551 Pywikibot's setup.py install is not being tested

Event Timeline

XZise created this task.Feb 15 2015, 11:56 AM

XZise raised the priority of this task from to Low.

XZise updated the task description. (Show Details)

XZise added projects: Pywikibot, Pywikibot-compat-to-core.

XZise added subscribers: XZise, jayvdb.

Restricted Application added subscribers: Aklapper, Unknown Object (MLST). · View Herald TranscriptFeb 15 2015, 11:56 AM

I think it makes sense to add unicode_literals everywhere in a big bang before 2.0 RC1, and wait for the screams. This was discussed a little on https://gerrit.wikimedia.org/r/#/c/189914/2/scripts/cosmetic_changes.py

We can then remove u'...' over time, as part of regular code modifications.
I dont see that removing them particularly helps newcomers; writing a style guide for new developers is a better approach.

XZise updated the task description. (Show Details)Feb 15 2015, 12:20 PM

XZise set Security to None.

Regarding changing u'…': Obviously that doesn't need to happen independently. But (especially after unicode_literals have been imported) it's possible to change it when the line is changed anyway.

Ricordisamoa awarded a token.Feb 17 2015, 9:26 AM

Ricordisamoa subscribed.

Change 186962 had a related patch set uploaded (by XZise):
[IMPROV] Use unicode_literals in fixes

https://gerrit.wikimedia.org/r/186962

Patch-For-Review

Ladsgroup awarded a token.Mar 22 2015, 9:27 PM

Change 202373 had a related patch set uploaded (by XZise):
[IMPROV] Use unicode_literals

https://gerrit.wikimedia.org/r/202373

Change 202373 merged by jenkins-bot:
[IMPROV] Use unicode_literals

https://gerrit.wikimedia.org/r/202373

XZise mentioned this in rPWBC1e54a7d6886d: [IMPROV] Use unicode_literals.Apr 8 2015, 2:30 PM

XZise added subtasks: T95803: UnicodeDecodeError when using pywikibot delete.py, T95671: UnicodeDecodeError when using pywikibot category.py.Apr 11 2015, 11:36 AM

Hmm... it seems to be causing even more harm than benefits, doesn't it?

Well it discovered some flaws and afaik all patches after that fix the original patch work independently. So if it's really unusable we can still just revert that one patch.

But just for good measure I'll list all patches which are unicode_literals specific in the first post so that if we want to abandon that project we could revert everything of that (but leave the useful stuff).

XZise updated the task description. (Show Details)Apr 12 2015, 9:44 AM

XZise updated the task description. (Show Details)Apr 12 2015, 11:29 AM

jayvdb closed subtask T95803: UnicodeDecodeError when using pywikibot delete.py as Resolved.Apr 17 2015, 11:28 PM

XZise mentioned this in T94688: reflinks.py doesn't save changes to articles.Apr 19 2015, 10:40 AM

Rubin16 added a parent task: T94688: reflinks.py doesn't save changes to articles.Apr 19 2015, 10:44 AM

Rubin16 subscribed.

Rubin16 awarded a token.Apr 20 2015, 5:59 PM

jayvdb added a subtask: T99366: error with httplib2 and unicode_literals: UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1872: ordinal not in range(128).Jun 5 2015, 3:41 AM

zhuyifei1999 closed subtask T99366: error with httplib2 and unicode_literals: UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1872: ordinal not in range(128) as Resolved.Jun 21 2015, 6:51 AM

jayvdb added a subtask: T95809: Page's repr returns invalid data causing Python to error.Jun 23 2015, 4:42 AM

jayvdb removed projects: Patch-For-Review, Pywikibot-compat-to-core.

Xqt subscribed.Jun 23 2015, 5:28 AM

XZise mentioned this in T66958: Python 2 Page representation (repr) returns non-ASCII bytes.Jun 26 2015, 10:58 AM

XZise reopened subtask T99366: error with httplib2 and unicode_literals: UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1872: ordinal not in range(128) as Open.Jul 21 2015, 7:39 PM

jayvdb added a subtask: T111615: pywikibot.__all__ is unicode, causing `TypeError: Item in ``from list'' not a string`.Sep 5 2015, 10:13 PM

Xqt closed subtask T111615: pywikibot.__all__ is unicode, causing `TypeError: Item in ``from list'' not a string` as Resolved.Sep 6 2015, 7:17 AM

jayvdb added a subtask: T114487: Pywikibot master setup.py won't install.Oct 3 2015, 1:07 AM

XZise changed the status of subtask T114487: Pywikibot master setup.py won't install from Duplicate to Resolved.Oct 4 2015, 11:04 PM

• Peter601980 closed subtask T99366: error with httplib2 and unicode_literals: UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1872: ordinal not in range(128) as Invalid.Dec 28 2015, 8:13 AM

Aklapper reopened subtask T99366: error with httplib2 and unicode_literals: UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1872: ordinal not in range(128) as Open.Dec 28 2015, 9:28 AM

matej_suchanek added a parent task: T60053: Pywikibot Python 3 compatibility (tracking).Feb 7 2017, 1:42 PM

Xqt closed subtask T99366: error with httplib2 and unicode_literals: UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1872: ordinal not in range(128) as Declined.Jul 11 2017, 10:35 AM

Dalba reopened subtask T95810: config does differ between unicode and str as Open.May 22 2018, 2:56 PM

@Dalba: could you explain why this is open again or better what is still to do here?

This task has never been closed as far as I can tell.

There is one subtask: T95810: config does differ between unicode and str which can be investigated independently.

The only thing that seems to be remaining here is to decide about the complete removal of the u prefixed string literals. I'm personally in favor of it, but if others disagree, this task can be closed.

The only thing that seems to be remaining here is to decide about the complete removal of the u prefixed string literals. I'm personally in favor of it, but if others disagree, this task can be closed.

I aggree now. This mixed types are harder to read imho.

xSavitar subscribed.Sep 23 2018, 2:01 PM

Change 462270 had a related patch set uploaded (by D3r1ck01; owner: Alangi Derick):
[pywikibot/core@master] [cleanup] cleanup scripts/welcome.py

https://gerrit.wikimedia.org/r/462270

gerritbot added a project: Patch-For-Review.Sep 23 2018, 2:07 PM

Change 462270 merged by jenkins-bot:
[pywikibot/core@master] [cleanup] cleanup scripts/welcome.py

https://gerrit.wikimedia.org/r/462270

Dvorapa assigned this task to xSavitar.Sep 29 2018, 9:16 AM

Dvorapa removed a project: Patch-For-Review.

xSavitar awarded a token.Sep 29 2018, 11:05 AM

xSavitar added a project: User-xSavitar.

Change 463604 had a related patch set uploaded (by D3r1ck01; owner: Alangi Derick):
[pywikibot/core@master] [cleanup] cleanup tests/[data_ingestion_tests.py-exceptions_tests.py]

https://gerrit.wikimedia.org/r/463604

gerritbot added a project: Patch-For-Review.Sep 29 2018, 11:40 AM

xSavitar removed a project: Patch-For-Review.Sep 29 2018, 11:58 AM

xSavitar moved this task from Backlog to Doing [WIP] on the User-xSavitar board.Sep 29 2018, 7:39 PM

Change 463604 merged by jenkins-bot:
[pywikibot/core@master] [cleanup] cleanup tests/[data_ingestion_tests.py-exceptions_tests.py]

https://gerrit.wikimedia.org/r/463604

@Xqt, do we still have instances of u'' or u"" in the code base to remove? Or has this been complete?

In T89589#4633212, @D3r1ck01 wrote:

@Xqt, do we still have instances of u'' or u"" in the code base to remove? Or has this been complete?

I guess not, seems it's completed

In T89589#4633212, @D3r1ck01 wrote:

@Xqt, do we still have instances of u'' or u"" in the code base to remove? Or has this been complete?

There are some remaining "bad quotes": https://integration.wikimedia.org/ci/job/pywikibot-core-tox-docker/4720/console

Let me clean those up @Xqt :)

Change 463939 had a related patch set uploaded (by D3r1ck01; owner: Alangi Derick):
[pywikibot/core@master] [cleanup] cleanup some tests/ files

https://gerrit.wikimedia.org/r/463939

gerritbot added a project: Patch-For-Review.Oct 2 2018, 11:44 AM

Change 463939 merged by jenkins-bot:
[pywikibot/core@master] [cleanup] cleanup some tests/ files

https://gerrit.wikimedia.org/r/463939

@Xqt, anything left?

In T89589#4633483, @D3r1ck01 wrote:

@Xqt, anything left?

docs/conf.py is the last one I guess

Change 463954 had a related patch set uploaded (by D3r1ck01; owner: Alangi Derick):
[pywikibot/core@master] [cleanup] cleanup docs/conf.py

https://gerrit.wikimedia.org/r/463954

docs/conf.py is the last one I guess

Done! Since Pywikibot is to be added to codesearch, we could use it to also search though other repos in case we have some instances of this left? Maybe?

Change 463954 merged by jenkins-bot:
[pywikibot/core@master] [cleanup] cleanup docs/conf.py

https://gerrit.wikimedia.org/r/463954

Xqt closed this task as Resolved.Oct 2 2018, 6:55 PM

xSavitar moved this task from Doing [WIP] to Completed (To be Resolved) on the User-xSavitar board.Oct 5 2018, 9:24 AM

Xqt closed subtask T95810: config does differ between unicode and str as Declined.Jun 30 2020, 4:58 PM