Steps to replicate the issue (include links if applicable):

1. Open a wiki that has disabled magic links via $wgEnableMagicLinks (e.g. enwiki)
2. Go to Special:ExpandTemplates
3. Paste the sample wikitext (see below) and click "OK" to render it.
4. Open a wiki that has enabled magic links via $wgEnableMagicLinks (e.g. mediawiki.org)
5. Go to its Special:ExpandTemplates
6. Paste the same sample wikitext and click "OK" to render it:

* RFC: {{anchorencode:RFC}}
* PMID: {{anchorencode:PMID}}
* ISBN: {{anchorencode:ISBN}}

* RFCAAA: {{anchorencode:RFCAAA}}
* PMIDAAA: {{anchorencode:PMIDAAA}}
* ISBNAAA: {{anchorencode:ISBNAAA}}

* RFC AAA: {{anchorencode:RFC AAA}}
* PMID AAA: {{anchorencode:PMID AAA}}
* ISBN AAA: {{anchorencode:ISBN AAA}}

What happens?:
The result (not the preview) for both contains the HTML character entities, introduced by function safeEncodeAttribute in Sanitizer.php:

* RFC: RFC
* PMID: PMID
* ISBN: ISBN

* RFCAAA: RFCAAA
* PMIDAAA: PMIDAAA
* ISBNAAA: ISBNAAA

* RFC AAA: RFC_AAA
* PMID AAA: PMID_AAA
* ISBN AAA: ISBN_AAA

What should have happened instead?:

• Function safeEncodeAttribute should skip the sanitization, when the corresponding magic links are disabled. That is, it should follow the configuration parameter $wgEnableMagicLinks, like function magicLinkCallback does in Parser.php for RFC, PMID, and ISBN.
  • See also functions getMagicISBNLinks, getMagicPMIDLinks, and getMagicRFCLinks in ParserOptions.php.
• When the sanitization is enabled, function safeEncodeAttribute should check slightly harder for the required syntax of magic links. It should only do the sanitization when the attribute value has a trailing space (e.g. 'ISBN ' => 'ISBN_', instead of 'ISBN' => 'ISBN',)
  • See also https://www.mediawiki.org/wiki/Help:Magic_links
• The result of Special:ExpandTemplates for step 3 (disabled magic links) should be:

* RFC: RFC
* PMID: PMID
* ISBN: ISBN

* RFCAAA: RFCAAA
* PMIDAAA: PMIDAAA
* ISBNAAA: ISBNAAA

* RFC AAA: RFC_AAA
* PMID AAA: PMID_AAA
* ISBN AAA: ISBN_AAA

• The result of Special:ExpandTemplates for step 6 (enabled magic links) should be:

* RFC: RFC
* PMID: PMID
* ISBN: ISBN

* RFCAAA: RFCAAA
* PMIDAAA: PMIDAAA
* ISBNAAA: ISBNAAA

* RFC AAA: RFC_AAA
* PMID AAA: PMID_AAA
* ISBN AAA: ISBN_AAA

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):

• Reported at enwiki's Village pump by Mathglot, where usage of a Lua equivalent function mw.uri.anchorEncode was causing this issue.
  • Scribunto's function uri.anchorEncode in file mw.uri.lua calls function anchorEncode in PHP class UriLibrary, which calls CoreParserFunctions::anchorencode, which calls the aforementioned Sanitizer::safeEncodeAttribute.