I'm not sure what to document, so here are some random thoughts. Hopefully I'm not forgetting anything important.

Most of the constraints result from the fact that unsigned comments are possible. If we ever reach the point where 100% of talk page comments are properly signed, some of the constraints will disappear.

The main limitation is that we can only reliably detect the end of the comment, where the signature is. We can't detect the start of the comment (we just assume it starts after the previous signature, or a section heading), so unsigned comments get "merged" into the comment following it.

We also can't entirely reliably detect which comments are in reply to which ones, because comments may be indented incorrectly (or intentionally "outdented"). This is less limiting than the previous problem though, even in discussion systems with proper structure users will sometimes click "Reply" on the wrong comment and nobody gets too upset at that.

We also can't mark exactly which part of the comment is a signature, when the user/talk page link is preceded by some other text (e.g. in --[[User:Matma Rex]] ..., the -- would not be marked as part of the signature). (We could experiment with some guessing to improve this, if it was useful.)

We can reliably detect the date and time (timestamp), and the author of a comment.

You can add ?dtdebug=1 at the end of the page address to visualize the internal structure we detected (this only works on wikis where DiscussionTools is installed, so only our prototype server right now, but you can copy-paste pages there from other wikis to test). This highlights the extent of comments, signatures and timestamps, and draws indications of which comment is a reply to which. For example: http://visualeditor-prototype.wmflabs.org/wiki/Talk:Reply_test?dtdebug=1

Other minor things to keep in mind:

• There may be text within a section that is not a part of any comment. (Currently only when the last comment is unsigned [https://pl.wikipedia.org/w/index.php?title=Wikipedia:Zgłoś_błąd_w_artykule&oldid=58213297#Klan_Ausran], but we will probably find some cases where this is needed for hatnotes/warnings etc. [https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/Index_of_Pakistan-related_articles])
• Comments may have no author (even if they are signed) – for example, Tech News postings and similar [https://pl.wikipedia.org/w/index.php?title=Wikipedia:Kawiarenka/Kwestie_techniczne&oldid=58211936#Tech_News:_2019-49]. The "signature" is just a timestamp. (This may not be a good idea and we might change it, in which case this will be handled like text that is not a part of any comment, I guess?)
• Each section can have multiple top-level (not indented) comments [https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(technical)&oldid=930027311#Admin_access_via_the_API?] (it's especially common in long discussions when someone wants to summarize, or reply to "everyone else"). I think of it as: section heading represents the topic of the discussion, top-level comments are "replies" to the topic, other comments are replies to top-level comments.
• There may not be a section heading on a talk page if someone forgot to add one when starting the first discussion.
• We will probably need to special-case some existing workflows (e.g. voting).

We should be careful about moving things around or adding "rigid" boxes where it could be misleading about authorship of comments. Currently, it's fairly obvious (to a human) when a piece of text is an unsigned comment. But in our internal structure, unsigned comments can get merged with signed comments by other users, and drawing a highlight around such a "comment" makes it difficult to figure out that it's partially an unsigned comment by another person. It could make for confusing conversations and very annoyed people when their words are misattributed.

There are still cool things we could do, e.g. collapsing/expanding the replies to a comment, highlighting new comments, etc.

I'm not saying that certain things are impossible, but rather that we should weigh the utility of any feature against the cost of messing up the page when we detected the structure wrong because some comments have the wrong markup or are unsigned.