⚓ T262408 [TRACKING] Selser issues on talk pages

Project	Branch	Lines +/-	Subject
mediawiki/vendor	master	+813 -484	Bump wikimedia/parsoid to 0.13.0-a17
mediawiki/services/parsoid	master	+37 -0	Selser: Skip over templates while adding selser-wrapper <span>s
mediawiki/core	wmf/1.36.0-wmf.16	+1 -1	vendor: Bump wikimedia/parsoid to 0.13.0-a15
mediawiki/vendor	wmf/1.36.0-wmf.16	+44 -34	Bump wikimedia/parsoid to 0.13.0-a15
mediawiki/vendor	master	+44 -34	Bump wikimedia/parsoid to 0.13.0-a15
mediawiki/services/parsoid	master	+31 -21	Prevent selser corruption from unclosed tags in list items in some cases
mediawiki/vendor	master	+2 K -1 K	Bump wikimedia/parsoid to 0.13.0-a14
mediawiki/services/parsoid	master	+436 -27	SelSer: Preprocess DOMs to wrap text children of <li> in <span>s
mediawiki/vendor	master	+1 K -974	Bump wikimedia/parsoid to 0.13.0-a12
mediawiki/services/parsoid	master	+183 -66	Record length of trimmed whitespace in additional DSR fields

Status	Assigned	Task
Open	Ryasmeen	T262408 [TRACKING] Selser issues on talk pages
Resolved	ssastry	T262409 Space at end of another list item removed
Open	None	T262410 Space at start of parent list item (before template) removed
Open	None	T262411 <nowiki>s added to previous comment
Open	None	T262412 HTML normalised in parent tag
Open	None	T262413 Newlines added after templates to previous paragraphs when adding a comment
Declined	matmarex	T262414 {{safesubst:}} else where in doc is substituted when adding a comment
Resolved	ssastry	T262448 Unmatched closing tags are removed
Resolved	ssastry	T262449 Syntax for empty list item is removed
Open	None	T264028 Whitespace in <nowiki>…</nowiki> normalized or something
Open	None	T264027 <nowiki/> tag removed inside template parameter
Open	ssastry	T265094 Line break in a link label was changed to a space
Resolved	ppelberg	T266140 HTML entity replaced by the Unicode character in an edit
Open	None	T267157 Weird list corruption in a DiscussionTools edit (2020-09-11)
Duplicate	None	T268035 Random space inserted in a list item in a DiscussionTools edit
Duplicate	None	T268037 Template substituted in a DiscussionTools edit
Open	None	T268038 Parsoid removing unmatched closing tag causes unexpected indent-pre styling
Open	Ryasmeen	T269017 Empty list item has whitespace removed
Resolved	ssastry	T269901 Dirty diff: Single additional trailing whitespace added to several discussion tools edits in last 3+ weeks
Open	None	T270180 White space before top of page meta items is removed

Esanders created this task.Sep 9 2020, 1:22 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 9 2020, 1:22 PM

Esanders mentioned this in T261321: Inspect Reply Tool diffs for instances of corruption.Sep 9 2020, 1:22 PM

Esanders updated the task description. (Show Details)Sep 9 2020, 1:43 PM

The two whitespace issues (T262409 & T262410) appear to account for the majority of corruption we are seeing, if you want to prioritise your focus.

This below reproduces both the bugs.

[subbu@earth:~/work/wmf/parsoid] echo -e "*  a \n** b \n" > /tmp/wt
[subbu@earth:~/work/wmf/parsoid] php bin/parse.php < /tmp/wt > /tmp/old.html
[subbu@earth:~/work/wmf/parsoid] sed 's/b<\/li>/b<\/li>\n<li>b-new<\/li>/g;' < /tmp/old.html > /tmp/new.html
[subbu@earth:~/work/wmf/parsoid] php bin/parse.php --html2wt --selser --oldtextfile /tmp/wt --oldhtmlfile /tmp/old.html< /tmp/new.html > /tmp/edited.wt
[subbu@earth:~/work/wmf/parsoid] diff /tmp/wt /tmp/edited.wt
1,3c1,3
< *  a 
< ** b 
< 
---
> * a 
> ** b
> ** b-new

ppelberg added a project: Editing-team (Tracking).Sep 9 2020, 3:59 PM

ppelberg moved this task from Backlog to Blocked by others on the Editing-team (Tracking) board.

ppelberg added a subscriber: ppelberg.

ssastry triaged this task as Medium priority.Sep 10 2020, 5:26 PM

ssastry moved this task from Needs Triage to Current & Upcoming Work on the Parsoid board.

ssastry closed subtask T262409: Space at end of another list item removed as Resolved.Sep 17 2020, 6:00 PM

Change 628937 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid@master] WIP: Selser: Preprocess doms to wrap li text nodes in wrapper

https://gerrit.wikimedia.org/r/628937

gerritbot added a project: Patch-For-Review.Sep 21 2020, 8:51 PM

ssastry claimed this task.Sep 23 2020, 10:59 PM

ssastry added a project: Parsing-Active-Work.

ssastry moved this task from Incoming Work (but not yet ready) to Doing (Other Work) on the Parsing-Active-Work board.

ppelberg mentioned this in T263709: [SPIKE] Why can the Reply Tool be opened on a page with broken/incomplete table syntax..Sep 24 2020, 12:34 AM

Change 630219 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid@master] Record trimmed whitespace in additional DSR fields

https://gerrit.wikimedia.org/r/630219

matmarex added subtasks: T264028: Whitespace in <nowiki>…</nowiki> normalized or something, T264027: <nowiki/> tag removed inside template parameter.Sep 28 2020, 5:46 PM

ssastry raised the priority of this task from Medium to High.Oct 1 2020, 4:39 PM

ssastry moved this task from Doing (Other Work) to Epics / Tracking Tasks on the Parsing-Active-Work board.Oct 1 2020, 4:55 PM

Change 630219 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Record length of trimmed whitespace in additional DSR fields

https://gerrit.wikimedia.org/r/630219

Change 635100 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a12

https://gerrit.wikimedia.org/r/635100

Change 635100 merged by jenkins-bot:
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a12

https://gerrit.wikimedia.org/r/635100

matmarex added a subtask: T266140: HTML entity replaced by the Unicode character in an edit.Oct 21 2020, 2:32 PM

cscott mentioned this in T266285: Deploy 6-element DSR to prod.Oct 22 2020, 9:50 PM

Change 628937 merged by jenkins-bot:
[mediawiki/services/parsoid@master] SelSer: Preprocess DOMs to wrap text children of <li> in <span>s

https://gerrit.wikimedia.org/r/628937

Maintenance_bot removed a project: Patch-For-Review.Oct 30 2020, 11:10 PM

Change 638161 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a14

https://gerrit.wikimedia.org/r/638161

gerritbot added a project: Patch-For-Review.Nov 2 2020, 7:14 PM

Change 638161 merged by jenkins-bot:
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a14

https://gerrit.wikimedia.org/r/638161

Maintenance_bot removed a project: Patch-For-Review.Nov 2 2020, 10:10 PM

Change 638210 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid@master] Prevent selser corruption from unclosed tags in list items in some cases

https://gerrit.wikimedia.org/r/638210

gerritbot added a project: Patch-For-Review.Nov 3 2020, 12:36 AM

Change 638210 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Prevent selser corruption from unclosed tags in list items in some cases

https://gerrit.wikimedia.org/r/638210

matmarex added subtasks: T267158: Parsoid's Cleanup DOM pass incorrectly deletes empty nodes with transclusion info, T267157: Weird list corruption in a DiscussionTools edit (2020-09-11), T267156: Weird list corruption in a DiscussionTools edit (2020-10-29).Nov 3 2020, 7:31 PM

Change 638735 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a15

https://gerrit.wikimedia.org/r/638735

Change 638735 merged by jenkins-bot:
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a15

https://gerrit.wikimedia.org/r/638735

Change 638520 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/vendor@wmf/1.36.0-wmf.16] Bump wikimedia/parsoid to 0.13.0-a15

https://gerrit.wikimedia.org/r/638520

Change 638520 merged by jenkins-bot:
[mediawiki/vendor@wmf/1.36.0-wmf.16] Bump wikimedia/parsoid to 0.13.0-a15

https://gerrit.wikimedia.org/r/638520

Change 639214 had a related patch set uploaded (by Brennen Bearnes; owner: Brennen Bearnes):
[mediawiki/core@wmf/1.36.0-wmf.16] vendor: Bump wikimedia/parsoid to 0.13.0-a15

https://gerrit.wikimedia.org/r/639214

Change 639214 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.16] vendor: Bump wikimedia/parsoid to 0.13.0-a15

https://gerrit.wikimedia.org/r/639214

ssastry removed a subtask: T267156: Weird list corruption in a DiscussionTools edit (2020-10-29).Nov 4 2020, 8:50 PM

ppelberg edited projects, added Editing QA; removed Editing-team (Tracking).Nov 5 2020, 10:50 PM

ppelberg moved this task from Inbox to High Priority on the Editing QA board.

ssastry removed a subtask: T267158: Parsoid's Cleanup DOM pass incorrectly deletes empty nodes with transclusion info.Nov 9 2020, 7:23 PM

Change 640438 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid@master] Selser: Skip over templates while adding selser-wrapper <span>s

https://gerrit.wikimedia.org/r/640438

Change 640438 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Selser: Skip over templates while adding selser-wrapper <span>s

https://gerrit.wikimedia.org/r/640438

ssastry closed subtask T265094: Line break in a link label was changed to a space as Resolved.Nov 16 2020, 5:41 PM

ssastry reopened subtask T265094: Line break in a link label was changed to a space as Open.

ssastry closed subtask T265094: Line break in a link label was changed to a space as Resolved.Nov 16 2020, 5:54 PM

@ppelberg could you all evaluate which of the subtasks are fixed and resolve them? There are some tasks where the dirty diffs would be fixed for list items but not outside list items (ex: unbalanced closing tags might still be removed outside lists).

matmarex closed subtask T262414: {{safesubst:}} else where in doc is substituted when adding a comment as Declined.Nov 16 2020, 10:06 PM

In T262408#6625619, @ssastry wrote:

@ppelberg could you all evaluate which of the subtasks are fixed and resolve them? There are some tasks where the dirty diffs would be fixed for list items but not outside list items (ex: unbalanced closing tags might still be removed outside lists).

Absolutely, @ssastry. Thank you for the ping. We have plans to explicitly test this task's subtasks this week.

Assigning this over to @Ryasmeen for further testing.

Before that, I (@ppelberg) need to make sure testing instructions are added to the task description of each of this task's substasks.

ppelberg added a project: Editing-team (FY2020-21 Kanban Board).Nov 16 2020, 10:53 PM

ppelberg moved this task from Needs Discussion / Investigation to Blocked / Needs More Work on the Editing-team (FY2020-21 Kanban Board) board.

Change 641308 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a17

https://gerrit.wikimedia.org/r/641308

Change 641308 merged by jenkins-bot:
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a17

https://gerrit.wikimedia.org/r/641308

matmarex added subtasks: T268035: Random space inserted in a list item in a DiscussionTools edit, T268037: Template substituted in a DiscussionTools edit, T268038: Parsoid removing unmatched closing tag causes unexpected indent-pre styling.Nov 17 2020, 4:46 PM

ppelberg moved this task from Blocked / Needs More Work to Ready for Sign Off on the Editing-team (FY2020-21 Kanban Board) board.Nov 18 2020, 3:40 PM

Ryasmeen edited projects, added Verified; removed Editing QA.Nov 23 2020, 8:17 PM

ssastry closed subtask T262449: Syntax for empty list item is removed as Resolved.Nov 23 2020, 8:21 PM

JTannerWMF moved this task from Ready for Sign Off to Quarterly Goals on the Editing-team (FY2020-21 Kanban Board) board.Dec 2 2020, 6:30 PM

ssastry closed subtask T262448: Unmatched closing tags are removed as Resolved.Dec 9 2020, 2:38 PM

Esanders reopened subtask T262409: Space at end of another list item removed as Open.Dec 11 2020, 11:32 AM

ssastry closed subtask T262409: Space at end of another list item removed as Resolved.Dec 11 2020, 2:29 PM

ssastry added a subtask: T269901: Dirty diff: Single additional trailing whitespace added to several discussion tools edits in last 3+ weeks.Dec 11 2020, 3:21 PM