In T230653#6863932, @Tacsipacsi wrote:

(there may be even bots written in compiled languages like Java or C# whose source codes have been lost).

Sounds good. Sorry about cutting the line. I misunderstood what the columns meant.

There are unwanted line breaks in the middle of this reply: https://en.wikipedia.org/w/index.php?title=Talk:Investigations_into_the_origin_of_COVID-19&diff=1008382587&oldid=1008382503

Couldn't a JS module / gadget extract the section id from the DOM and either add (a) the right edit link (b) update an edit link stub with the section id? In the HTML, all sections have section ids / editsection links. So, all the template needs to do is add a span/div/$tag tag with some attribute/class that the JS can inspect and update the link with the section id.

We are looking at DOM forests, not a selection of contiguous nodes during inorder traversal. We don't want go down that other route - to handle cases like that for templates, we expand the DOM range to span a DOM forest.

Looks like this was the 4th instance of this bug actually! So, maybe we ought to consider fixing this one of these days.

Given that we have run into this for the very first time after so many years, I am marking it low as being an edge case. But we'll fix it.

What is the bug? This is just the HTML5 tree building algorithm at work. If you create that HTML, open it in Firefox, and dump the inner HTML of the body, you get the same thing (I stripped whitespace while pasting here):

<a class="foo"><div class="bar"></div></a>
<div class="baz"><a class="foo"></a><a class="quux"></a></div>

This is still early in the process, but I believe there are plans to create Phabricator templates and have all that content live here (cc @kchapman). But as for this ticket, I can inline this information into phab at a later point unless someone else gets to it before me. No reason to move to MediaWiki right now unless the plan is for all these review decisions to live there.

This phab task exists solely for the purpose of review as part of the new Technical Decision Making Process where step 1 is to file a phab task tagged with tech-decision-forum. So, this is not a dupe of other tasks. But, I will add appropriate links.

I filed T275063.

The transclusion in question is:

{{AuxTOC|
{{c|{{xx-larger|'''সূচীপত্র'''}}}}
{{block center|width=300px}}
{{Table| title='''পরিচ্ছেদ'''| page= '''পাতা'''}}
{{Table| title=[[/১/]]| page=[[পাতা:অবরোধ বাসিনী.pdf/৪|১]]}}
...
...
{{Table| title=[[/৪৭/]]| page=[[পাতা:অবরোধ বাসিনী.pdf/২৮|২৫]]}}
}}

If I go to the API sandbox @ https://bn.wikisource.org/wiki/Special:ApiSandbox#action=expandtemplates&format=json&formatversion=2 and enter that wikitext and click submit, the output doesn't include the TOC. Since Parsoid effectively uses this API (internally via calls), Parsoid is emitting the empty table. So, something is broken in how Parsoid is using that or we are tripping over some bug ... this is just a quick report without any additional investigation. We'll take a look.

To clarify, I am not attached to id. That was just a placeholder for something more appropriate. Pick something that makes sense in the translate extension context. I don't need to be involved in that bikeshedding. :)

Otherwise, we'll have to abandon the annotation approach and hand off the wikitext string to the translate extension to preprocess it and convert to DOM. But, that in turn impacts HTML -> WT of edited content at the very least without additional tricks or massaging.

@Nikerabbit so, i ran into a glitch which is the tvar syntax which uses the <tvar|N> and </> syntax for opening and closing tag instead of say <tvar id='N'> and </tvar> syntax. The unusual syntax means the peg tokenizer treats it as plain text and then causes further breakage (ex: if used in links, the links won't parse as links).

Looks like this may be an issue with the encoding we use for the database / field. Apparently, mysql utf8 only supports the BMP (3-byte chars) and so, as in the example in the description, if emojis and non-BMP characters are involved, the inserts fail. One option is to switch the encoding to uf8mb4 or as @cscott says in option #1, maybe make it a binary field.

@Nikerabbit and @Pginer-WMF given that this bug has been around from the beginning, I take it this is not an ultra high priority issue and it is okay for it to take a few weeks to get done on the Kartographer end.

Since we were able to unbreak this by using socket path for connecting to the mysql, i am lowering priority. You guys can investigate the localhost issue separately.

I am still seeing this error:

ssastry@testreduce1001:~$ sudo service parsoid-rt status
...
Feb 08 21:38:02 testreduce1001 nodejs[14116]: Unable to connect to database, error: Error: Packets out of order. Got: 1 Expected: 0

So, something else is still broken that is preventing the node.js mysql driver from talking with the database. Not sure if there is something other config that needs an update. Nothing changed even after a sudo service mysql restart.

In T274133#6811209, @Nikerabbit wrote:

As for how to fix this:

1. Cxserver probably shouldn't be parsing mw-data thing on elements not marked as templates.

...
I suggest that we implement 1 and/or 2 in cxserver regardless of other actions.

We've purged Parsoid/JS from the codebase. So, that just means that someone (not us) needs to update the vagrant roles.

This doesn't render in Parsoid HTML either (just get spinning wheels) - https://en.wikipedia.org/api/rest_v1/page/html/COVID-19_pandemic_in_Bulgaria - I wonder if some of this is a result of changes to graph or this is the usual known issue of missing JS resource modules in Parsoid's output.

T272987: Parsoid read views for DiscussionTools has been filed

Might have been caused by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/649719. Adding @Pchelolo. and @daniel

In T265720#6778668, @dancy wrote:

Thanks for the info ssastry. I would definitely like to see these errors cleaned up since they make it harder to analyze the status of a deployment.

In the meantime I will revert the priority to Medium and remove this as train blocker.

Also, commonswiki is rolled out as part of group1 (so tomorrow), so the commonswiki error couldn't be from the train. Looking at https://logstash.wikimedia.org/app/dashboards#/view/AW4Y6bumP44edBvO7lRc?_g=h@270dedf&_a=h@4e9499a I see 100s of assertions on dewiki:Benutzer:Tschubby/Testseite2.