Any updates on this? The task has been moved around a bit recently, but it's not clear what is happening. Is it difficult to fix?

Just a thought: perhaps the HTML dumps should be generated from the XML dumps, so that the revisions in both match (and they can both be used interchangeably without consistency problems).

Seeing that T114072 is marked as resolved, is it now possible to implement this?

Even if you don't want to change this behavior, it should probably be mentioned in the documentation of #iferror. Because of this limitation, the function is practically useless when used with Scribunto.

Thanks for fixing this so quickly, I'll wait for rc2 and re-test.

If Lua on MediaWiki can't be upgraded to 5.2 or later (T178146 is stalled, with "re-evaluation in 2024"), maybe just the GC changes could be backported to 5.1, to have at least some predictable GC behaviour?

Thanks for investigating. I'm running a Spark job to parse some CBOR files (~ 100MB) to generate a list of missing words on Wiktionary. It really should not take so much memory but I noticed in the logs that Spark seems to aggressively request a lot of memory up front as some sort of buffer space/working memory. I'll see if I can tweak this. Update: solved.

I'm running into more problems with Java on the grid: the JVM gets killed (exit code 137 = SIGKILL), but if I do a qacct -j jobid I don't see any enforced limits:

In T165935#6953017, @Urbanecm wrote:

In T165935#6952930, @jberkel wrote:

The problem seems to be that the version of Lua used by Scribunto does not run the garbage collector when a memory allocation fails, so memory is not reclaimed when it is needed most. I would consider that a bug of the the implementation. With that behaviour it's hard to tell if we're really out of memory, or if we're just out of luck, because the GC didn't run in time.

Can you tell us how did you reach to that conclusion?

The problem seems to be that the version of Lua used by Scribunto does not run the garbage collector when a memory allocation fails, so memory is not reclaimed when it is needed most. I would consider that a bug of the the implementation. With that behaviour it's hard to tell if we're really out of memory, or if we're just out of luck, because the GC didn't run in time.

In this case it's obviously not Zotero's fault, even when setting a modern user agent in the header the server/CDN sometimes returns different content.

SQLite might not be your typical production setup, but it's very handy for getting a MediaWiki instance up and running in a docker container, I'm using it to process Wiktionary data. After switching to Parsoid the performance suffered quite noticeably, related to the aggressive write locking mentioned above.

If there's no caching, perhaps the problem lies at the source: GETs on the source url also return different results:

I agree it's annoying that VE inserts empty parameters which have not been touched at all by the user. It's unnecessary baggage in the page text, and complicates the diff. Ideally VE should produce a diff similar to whatever the equivalent non-Visual ("human") edit would have produced: don't change whitespace, parameter order, or add any superfluous parameters.

You're right, this is confusing – the order of the languages listed in the subtitle circled in your screenshot should match the order on the "Wikipedia languages" page to help make it more clear what the expected display language in the widget will be. I'll make a separate ticket for us to look into this.

@jberkel Thanks for testing them! Currently, the widgets display content in the "Primary" language you select in Settings > My languages in the app. Seeing all the languages you've added in the larger widget families is an interesting idea. But I think the ideal scenario for us is to utilize what Apple refers to as customizable intents in order to make each individual widget customizable to whatever language you'd like. These widgets seem like a fine candidate for customization, especially when considering the challenge of trying to mix both LTR and RTL language results into a single widget.

Nice work on the widgets, been testing with iOS 14 beta. I have a question regarding multiple languages, in the preferences I can set several languages for the "Top read" widget, but in practice I only see items in one language (the first in the list). I'd expect to see a mix of entries for all languages, maybe the top read in each language.

Looks like this is related/a duplicate: https://phabricator.wikimedia.org/T251664

I agree this is a good UX improvement to make. Perhaps the autocapitalize attribute of the input field could be based on the case sensitivity of the underlying Mediawiki installation? Most (if not all) Wiktionaries are configured as case sensitive.

In my case the error message doesn't include a code, it just says "WMF.WikidataAPIResult.APIError error" (perhaps the view is clipped), and the error doesn't seem to get logged. Only with a debugger attached I was able to get more details ("permissiondenied").

The edit dialog is still accessible via long press on description, then "Edit" (1742)

I've just noticed this as well, good to see that it's already been reported. When you make a change in that dialog, the Wikidata item gets updated, overwriting the previous value with the version from Wikipedia. This whole Wikidata/Wikipedia description duplication thing is a mess. Are local overrides only used in the English edition?

I'm not sure if this is feasible without negative impact in performance.

thanks, with qacct -j jobid I get:

...
failed       37  : qmaster enforced h_rt, h_cpu, or h_vmem limit
maxvmem      16.558GB
category     -q task -l h_vmem=16777216k

Sorry if this isn't directly related to the reported bug, but is there any way to find out why a process got sigkilled on the grid? I have a Java process getting terminated without traces of hserr*.log, and I suspect it is related to some native libraries allocating too much off-heap memory. I'd like to confirm that it's due to the OOM killer (or perhaps quota related?)

Ok, but the Gadget's change to the body tag must still be kept, otherwise the display: none would be active again? So VE just replaces the content inside div#content I suppose?

thanks for the research and recommendation. I'm a bit unclear on what exactly happens when the VE is loaded. If there's is logic inside the gadget which hides quotes or other content, won't that get applied in VE as well?

A year has passed, and this bug hasn't even been triaged. The memory problems on the English Wiktionary are still there, and ugly workarounds are needed to get some pages to render without errors.

@Jhernandez "parsedExamples" was added in T187430, but to maintain compatibility the "examples" field was kept as-is.

Fix was deployed a long time ago; closing this now.

I pinged some editors who work with hieroglyphs on the English Wiktionary and got a reply from @Vorziblix:

I am very curious about this statement. How do you think you can convinced 100+ communities to change the way data are structured?

@Tobias1984 there's the mobile mailinglist, although there's not much activity (mostly release announcements).

what's missing? frwikt support?

Bot approval vote passes 11-0-0.

Then, do a test run (under the bot account) on some 10-50 entries until you’re certain everything goes well (bots with a single function, like interwiki, should not run more than 25 edits). If the edits are minor (as most bot-performed tasks should be), please mark the edits as minor so that they do not swamp the Recent Changes list. If all goes well, post a request for bot status at the votes page.

Sorry for the confusion, yes this is correct (I have not been through the process).

https://en.wiktionary.org/wiki/Wiktionary:Beer_parlour/2017/December#Inviting_IABot

@Cyberpower678 do you only need translations for frwikt? Is anything else needed from enwikt?

@Lea_Lacroix_WMDE my main planned use case right now is T190210. Maybe it could also be used to automate the "See also" section at the top of the page (currently maintained manually with Template:also + bot).

Looks like the storage part of RESTBase isn't conceptually a cache which can be "thrown away" (that was my assumption). Maybe this has to do with a requirement to serve older revisions for other endpoints (not applicable for definitions)?

I think I still don't understand some parts of the service architecture. Why do all 5+ million entries have to be cached in one go? If the cache gets purged, won't it automatically/lazily repopulate the most requested ones? Some entries will probably never/rarely get requested, why should they get pre-cached?

Also, if I make a normal change to a Wiktionary page, shouldn't that invalidate the cache and return the new schema? What are the typical delays for the change propagation?

Is there no way to just invalidate the entire cache? seems quite complicated to make changes to the API.

why not clear the whole cache on schema change?

yes, maybe T131092, that shouldn't take too long. it would also be good to get a sense of what else is needed from a client's perspective.

reopened this task since the API still returns duplicates

@bearND ok, thanks. if the new code is deployed, where would the old responses come from? or is the header change required for the cache invalidation? is the cache invalidated on source page change (e.g. /wiki/fazer) ? what about changes in dependent templates?

Maybe a duplicate of T131092?

@bearND Looks like the change hasn't been deployed yet? Still getting duplicate sentences

I've been a few times now in this situation where unwanted changes were recovered.

I had a look through the codebase to see how hard it would be to restore iOS9 compatibility. Sadly, iOS 10+ calls are a bit all over the shop.

It always displays "Discard my changes and switch", even without changes. Which means that you need two clicks to switch to visual editor.

Ok, thanks. For reference, T55784 seems to be the ticket to follow.

@Rua since parsoid isn't used for the normal page rendering I don't expect this to be available in Scribunto, which is unfortunate. Maybe @GWicke or @ssastry could explain?

@Halfak ah ok, that makes sense. how many edits need to be labeled? how sensitive is the whole approach to template/code level changes?

@Halfak ok, so it's separate from a user's normal patrolling activity, meaning you work on an older sample instead of labeling recent changes? I think the participation rate could be a lot higher if it is integrated into the "normal" patrolling activity (opt-in, of course).

@bearND I can handle Gerrit, just remembered that I made some contributions to the Wikipedia iOS app using GH a while back, it's just so much easier.

I have a first patch ready for review. Can I submit PRs via Github or do I have to go through gerrit 😱?

@Aklapper currently failing pages: wind, night, do

@ArielGlenn so these would include the output from RestBase, with parsoid-annotated DOM? That would be very helpful for all sorts of processing tasks.

@bearND I think it would be preferable to extract the definition from the glossary and not the linked Wiktionary definition page (which might contain some other unrelated content). But I also understand that you don't want to add extra parsing code, so it might be a good first compromise. Maybe there's something simple we could do on Wiktionary to make things easier?

@Mholloway an approximation for the primary definition could be the first gloss/sense of an entry, skipping all obsolete/archaic senses which are sometimes listed first (presumably to illustrate the semantic development). in case of several part of speech headers this is trickier, since there's often no clear ordering