In T173385#5809505, @Ruthven wrote:

@Mpaa Well, the same happens without the option ql=4.

I am not sure but in this case you are creating a new page in a different wiki, using an existing page with ql=4.
Probably it is trying to set ql=4 also in the new page, violating the rule about change of status Not Proofread->Proofread, Validated.
@Tpt, any opinion?
Maybe pywikibot could be smarter here and check?

Went back to both integers.
To me this is solved.

Seems OK to me.

All pages:

To understand the wanted behaviour, what is the wanted Site for a redirected 'code'?
Site('aa', 'wikisource') = ?
E.g. APISite("mul", "wikisource")?

In T242517#5795298, @Umherirrender wrote:

Image width and height are always integer, but it seems that some files are now uploaded with float values

I think width and height are evaluated at run time (possibly cached), so I assume when this will be deployed, things will get back to normal.

Duplicate of T188885?

Dimensions are floats in both places now.

@Legoktm, @Umherirrender could you please take a look at this commit?
Is it intentional that the computation has been changed?
https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/PdfHandler/+/560127/2/includes/PdfImage.php

It's impossible to get image description from Commons via pywikibot API.
Because it is not exposed by mediawiki API, afaik.

In T242169#5788350, @Jan.Kamenicek wrote:

But when I open the PDF document in my computer and copypaste the text into a word processor, it looks thus:

...passing safely through a country occupied by Sigismund's
troops, they arrived near Kralov6 Hradec. They called to...

This means that the original text layer of the PDF is good, only Mediawiki extracts it badly.

This is not indicative, it can depend on a lot of things (OS, browser, pdf plugin used, etc.). I tried once on Linux and two on Windows (one with AcrobatReader, one reading the pdf in Edge), and I got three different results.

@Jan.Kamenicek, I didn't mean that the text layer got improved by the djvu converting process.
Flags used with pdftotext command matters, and they are set in Mediawiki, see my comparison above.

@Aklapper , I readded MediaWiki-extensions-PdfHandler as it seems relevant to me.

This might be useful to understand pdftotext options.
https://github.com/EmpowermentZone/EdSharp/blob/master/Convert/Xpdf/pdftotext.txt

Playing with "pdftotext" options, output can be similar to djvu text layer.

Have you considered to get data from dumps, if possible?

In T238448#5668653, @Bugreporter wrote:

not all (special) projects are Wikidata clients.

In T235500#5663181, @DD063520 wrote:

@Xqt : hi, would you consider again this patch, currently I make this modification locally to have some code running

In T238404#5668434, @DD063520 wrote:

Mhmmm .... on github I cannot find the "ban" language in the wikipedia_family:

https://github.com/wikimedia/pywikibot/blob/cbf194a60124fece9355517030aaa7091dcc093a/pywikibot/families/wikipedia_family.py

So would the newest version solve my problem?

I think it was added here: 2c565d01c381a858c4d02cf1f4c3372b589d5422

Works for me, ban is already in wikipedia_family.py.
It is quite slow, though.

This is OS dependent, on Linux it is acceptable.

If you are going to submit a patch with the fix (plus possibly tests), it will be appreciated.
You can also use https://tools.wmflabs.org/gerrit-patch-uploader/ in case you do not want to go through the standard process (git/gerrit).

It also broke archiving on en.wikisource, which uses "User:Wikisource-bot/config".

Wouldn't loading config from self.page.raw_extracted_templates instead of self.page.templatesWithParams() keep the right order?

It looks like googleOCR answer is not deterministic.
An option could be to check that at least x% of chars are equal instead of full equality.
The purpose is to check that the query to googleOCR is successful, not to test google algorithm.

See T214729

I would suggest -grep and -grepnot, similar to -titleregex and -titleregexnot.

In T212076#4959217, @Xqt wrote:

Now we have a json.decoder.JSONDecodeError:

https://ci.appveyor.com/project/ladsgroup/pywikibot-g4xqx/build/job/dhuv540dipdiw9si

Wikisource is needing to touch 000000s of files across multiple languages.

That should probably have its own bug report... extensions are expected to work correctly without constant bot maintenance.

Note:
I fixed the file on Commons.
The buggy file is https://commons.wikimedia.org/w/index.php?title=File:Philosophical_Transactions_-_Volume_053.djvu&oldid=336270264

I do not agree.
There is a patch proposing to introduce -querypage, which will be valid for all special pages, in order to avoid proliferation of argument.
See T214234.

Something like this.

...
$txt = preg_replace_callback( $reg, [ $this, 'pageTextCallback' ], $txt );
$reg_failed = '/(?m)^failed$/'; 
$txt = preg_replace_callback( $reg_failed, [ $this, 'pageTextCallbackFailed' ], $txt );
txt = "<DjVuTxt>\n<HEAD></HEAD>\n<BODY>\n" . $txt . "</BODY>\n</DjVuTxt>\n";

The improvement is to be done in DjVuImage.php: function retrieveMetaData()
https://doc.wikimedia.org/mediawiki-core/master/php/DjVuImage_8php_source.html#l00246

It is the djvutxt that fails for this page 51:

I guess the problem might lie somewhere here:
https://doc.wikimedia.org/mediawiki-core/master/php/DjVuImage_8php_source.html#l00308

https://commons.wikimedia.org/wiki/Commons:File_captions#How_can_I_query_the_captions_using_the_API?

Not sure this is a bug or a must, it is more about how you want the classes to cooperate with each other.
MRO chain is not broken in the first example, the design is such that the chain of calls does not propagate all the way up (it might also be a design choice, see e.g. https://rhettinger.wordpress.com/2011/05/26/super-considered-super/#comment-86).

qpoffset is used, the issue is same as T173293.

I copy it here for convenience.
Interesting, it looks like googleOCR answer is not deterministic or some bytes are lost somewhere.

I think it is has been a temporary unavailability of googleOCR service.

I don't think it is a pywikibot issue.

OK, we worked at the same time.
Added more check in proofreadpage.py
PYSETUP_TEST_EXTRAS=1 installs bs4

It fails on:

"env": "LANGUAGE=en FAMILY=wikipedia PYWIKIBOT_TEST_PROD_ONLY=1",

and pass on:

"env": "LANGUAGE=zh FAMILY=wikisource PYSETUP_TEST_EXTRAS=1 PYWIKIBOT_TEST_PROD_ONLY=1 PYWIKIBOT_TEST_NO_RC=1",

so I guess it is related to family.

This is not correct.
proofreadpage.py uses "contentformat" in edit action, which is not deprecated. See https://en.wikisource.org/w/api.php?action=help&modules=edit

proofreadpage_tests are still waiting for upstream fixes.

The problem is that in site.preloadpages(), max_ids is computed after the pageslist is splitted in chunks of 240.

for sublist in itergroup(pagelist, groupsize):   <----------- groupsize = 240
    # Do not use p.pageid property as it will force page loading.
    pageids = [str(p._pageid) for p in sublist
               if hasattr(p, '_pageid') and p._pageid > 0]
    cache = {}
    # In case of duplicates, return the first entry.
    for priority, page in enumerate(sublist):
        try:
            cache.setdefault(page.title(with_section=False),
                             (priority, page))
        except pywikibot.InvalidTitle:
            pywikibot.exception()

Thank you.

@JJMC89. could you please elaborate about it? So I might try to look into it. Thanks

In T113450#4517887, @zhuyifei1999 wrote:

In T113450#4513357, @JAnD wrote:

I am afraid, this is not because bot is broken, but because database is broken.

File a bug against MediaWiki-libs-Rdbms. Pywikibot itself has nothing to do with this.

BTW, I could not find a way to edit a page directly in VE. So I need to switch.

Hi. Also here I switched from wikitext to VE. And then just save, no other actions.

In T202197#4516595, @matmarex wrote:

I don't think this needed to be marked as "Unbreak Now!". The issue has existed for a while, and ProofreadPage is able to deal with missing or invalid usernames given in the "user" field (it must be – existing usernames already saved in existing content can become invalid, e.g. when the user is renamed).

But, since I already investigated it, there's not a lot of work to actually make the patch…

Yes, that it is how I reproduced. To clarify: 1) I went to edit mode, wikitext by default, then w/o saving 2) I switched to VE, 3) I saved.

The user seems to be blanked after a second edit to a Proofread page.
See https://en.wikisource.org/w/api.php?action=query&prop=revisions&titles=Page:From%20Kulja,%20across%20the%20Tian%20Shan%20to%20Lob-Nor%20(1879).djvu/273&rvlimit=10&rvprop=tags%7Ctimestamp%7Cuser%7Ccomment%7Ccontent%7Cinfo

Thanks.
@Tpt, is this possible?
In my understanding missing "pagequality" rights with OAuth is the reason for T173385.

Just for the records, think the issue is with an account be logged in via OAuth or not.
Also "Mpaa" behaves differently depending on how I am logged in.

I found out that MpaaBot has no 'pagequality' rights.
Mpaa works as expected instead.
I think this is not an issue wth ProofreadPage but with permissions of users on enwikisource.

@Tpt, could you please look into this? Thanks.