Noting that we received interest from a non-WMF engineer about writing a CM6 language package for MediaWiki. I'm waiting to hear back, but when someone picks this task up, they should probably focus on porting the stream parser (much easier!) for the time being.

I've looked deeply into creating language pack for wikitext and I have some conclusions. Lezer grammar is impossible to be written for wikitext thus resulting in necessity of writing completely custom parser (like is done for Markdown). This is a lot of work and unfortunately cannot be done alone. I recommend to fully focus on stream-parser way for now.
From my observations downsides of using stream-parser are with nested styling (maybe not a case here) as e.g. in Fandom we had headings that have equal signs in blue color and this heading was having bigger font size, I couldn't achieve having that bigger font size without having text between equal signs in blue color instead of white.

I have also looked into Lezer since CM6 becomes stable, and I completely agree with MrVanosh's comment on the difficulty of writing a language pack.

In T348019#9300871, @MrVanosh wrote:

I've looked deeply into creating language pack for wikitext and I have some conclusions. Lezer grammar is impossible to be written for wikitext thus resulting in necessity of writing completely custom parser (like is done for Markdown). This is a lot of work and unfortunately cannot be done alone. I recommend to fully focus on stream-parser way for now.
From my observations downsides of using stream-parser are with nested styling (maybe not a case here) as e.g. in Fandom we had headings that have equal signs in blue color and this heading was having bigger font size, I couldn't achieve having that bigger font size without having text between equal signs in blue color instead of white.

In T348019#9306562, @Bhsd wrote:

I have also looked into Lezer since CM6 becomes stable, and I completely agree with MrVanosh's comment on the difficulty of writing a language pack.

Thanks for investigating! A pity that we'd have to write our own parser if we want a proper language pack, but I'm still interested in that effort in the long-term. For now, we will indeed be porting our existing stream parser, which I'm nearly done with.

The situation you describe with nested styling is one of the downsides for us too, but we have various hacks in place to make it work a bit better, and more can be added if there's enough demand for it. I believe our users are mostly content with the status quo, so it's not terrible that we stick with the same. However without a structured syntax tree, we may be limited in what new features CodeMirror 6 that we can take advantage of, such as code folding and autocompletion (but I could be wrong).

I'm going to rewrite this task to focus on just porting the stream parser. Writing a new language pack can be made into a separate ticket, once we are committed to that effort.

Change 972438 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/extensions/CodeMirror@master] [WIP] Implement MediaWiki stream parser for CodeMirror 6

https://gerrit.wikimedia.org/r/972438

Okay, allllmost done with this! I have discovered there is one CodeMirror 5 feature that we may have to do away with in the new version. That is, the changing of the font size and line height of level 1, 2 and 3 section headings, as seen here:

I've been trying hard to reverse-engineer the old stream parser code, and I see nothing that should make these styles apply to the whole line and not just the individual tokens. There's some magic somewhere that converts .line- CSS classes into full line styles. This is normally done with doc.addLineClass() but I see we aren't using that method.

Anyway, in CodeMIrror 6, we need to use Decorations and apply them to the EditorView, which isn't available within the scope of the stream parser code. This means while possible, it's very difficult to replicate the same behaviour in CodeMirror 6.

I'm hoping it's okay that we do without the styling, like so:

I'm not even sure the section heading styling is widely desired; to me personally, it's only a tiny bit beneficial when scanning the wikitext. It's also worth noting the 2017 wikitext editor doesn't show this styling either, so at least some users are used to section headings having the same font size and line height as everything else.

EDIT: Now tracked at T351686

Change 980541 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/extensions/CodeMirror@master] CodeMirrorWikiEditor: add bracketMatching as default extension

https://gerrit.wikimedia.org/r/980541

This is now ready for review! I tried to break up into as many smaller patches as possible. Note there are known issues which I hope to tackle separately (and I made need some help!):

• T348684: Implement mediawiki TagModes for CodeMirror 6
• T352917: CodeMirror 6 MediaWiki stream parser produces excess markup
• T351686: Implement line-level styling (i.e. section headings) in CodeMirror 6

Change 981420 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/extensions/CodeMirror@master] CodeMirrorModeMediaWii: don't detect additional { as template name

https://gerrit.wikimedia.org/r/981420

Change 980541 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] CodeMirrorWikiEditor: add bracketMatching as default extension

https://gerrit.wikimedia.org/r/980541

Change 987271 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/extensions/CodeMirror@master] mode.mediawiki: rename mnemonic to html-entity and deprecate variants

https://gerrit.wikimedia.org/r/987271

Change 972438 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] Implement core MediaWiki stream parser for CodeMirror 6

https://gerrit.wikimedia.org/r/972438

Change 981420 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] CodeMirrorModeMediaWiki: don't detect additional { as template name

https://gerrit.wikimedia.org/r/981420

Change 987271 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] mode.mediawiki: rename mnemonic to html-entity and deprecate variants

https://gerrit.wikimedia.org/r/987271

Change 1007029 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/extensions/CodeMirror@master] CodeMirrorModeMediaWikiConfig: add missing tokens for nested templates

https://gerrit.wikimedia.org/r/1007029

Change 1007029 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] CodeMirrorModeMediaWikiConfig: add missing tokens for nested templates

https://gerrit.wikimedia.org/r/1007029

Change #1014700 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/extensions/CodeMirror@master] CodeMirrorModeMediaWikiConfig: add missing TagStyle for cm-mw-link

https://gerrit.wikimedia.org/r/1014700

Change #1014700 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] CodeMirrorModeMediaWikiConfig: add missing TagStyle for cm-mw-link

https://gerrit.wikimedia.org/r/1014700