The problem

We have a number of different features around altering or blocking "bad" links. Existing functionality for this has the following issues:

• The SecureLinkFixer extension does not work with Parsoid (T416536)
• The SpamBlacklist extension is unmaintained and buggy (T224921, T392048, T337431)
• Current approaches (eg in AbuseFilter and SpamBlacklist) focus on preventing edits containing "bad" content, but it is possible to combine a number of "permitted" edits to create prohibited content. A proper solution would include an output filter to catch & correct "bad" content that evades the edit checks. (T15569, T402144, T402299, T394020, T381716, T350480, T332491, T216657, etc)

Features of a solution

Desired features include:

• An edit-time warning for good-faith editors (AbuseFilter likely will provide this "for free")
• An comprehensive output transformation to thwart determined vandals who attempt to bypass edit checks (this is the crux of this feature)
• The ability to rewrite, not just block, links (SecureLinkFixer converts http to https; archive links may be rewritten to use other services) (this affects the hook API)

Optional features:

• Prompt editors for corrections and/or automatically correct edits instead of "just" blocking them (ie, fix http links to https automatically)
• The ability to flexibly and quickly update block/rewrite lists in response to current events or incidents
  • Including automated updates of block/rewrite lists (eg SecureLinkFixer consults a shared db maintained by Mozilla, we update this weekly, T239005)
• Have blocks/rewrites apply to generated metadata as well (T417323: Apply SecureLinkFixer earlier to include externalinks database and Special:LinkSearch)
• In some cases editors want to be able to move or rewrite content without necessarily 'fixing' all of the bad links at once. For example, in the archive.today incident, the consensus seems to favor blocking them in the generated HTML, but the links should not be removed from the original wikitext until/unless an alternative archive source has been found. The editor should be able to make unrelated changes to the article without triggering a block or warning about the archive.today link.

Implementation proposals

There are two main technical implementation proposals:

The first is to integrate rewriting into the core parser / Sanitizer. We already do this with https://en.wikipedia.org/wiki/MediaWiki_talk:Bad_image_list, which changes images included on pages to *link to* said images, based on context.
The advantages are:

• No downstream clients are exposed to the "bad" links. All derived data products (metadata, summaries, content translation tools) are guaranteed to have "fixed" links.

The disadvantages are:

• Because the rewriting happens in the "canonical" parse for the page, updates to rewrite/block lists are very slow to propagate. Main parser cache expiration time is ~2 weeks, and purging content from the parser cache is expensive. Edits and derived data consumers (inlcuding our metadata indexes) may contain unblocked "bad" content for a considerable period of time.
  • This could be mitigated by using the externallinks database (T417323) to selectively purge pages known to contain newly-bad or newly-good links, but this is an additional feature on top of the MVP that would have to be implemented.
• Difficult to "edit around" the bad content: since it is 'corrected' (rewritten or blocked) in the primary canonical representation of the content, it is likely to be silently corrected if the content is copied or moved.
  • This can be partially mitigated with Parsoid's "selective serialization", which tries to maintain the original formatting of edited articles.
• The "original editor intent" can be more difficult to determine, since the link has been removed/replaced. If your downstream tool is looking for vandalism, it won't see it because the vandalized link is no longer present.
  • This can be partially mitigated by preserving the information about the "original" link contents, for example adding a data-mw-original-href attribute. Clients need to know to look for this, and it does mean that we could potentially ship the bad link to clients albeit in a "non-executable" form.
• Because this is a "parser" feature, certain transclusions that can insert raw HTML into the page (extensions, parser functions) open gaps where banned content may still make it onto the page.
  • In theory all such content should be sanitized, but enforcing this and keeping it uniform (eg T228810, T381716, etc) have been challenges in the past.
  • Special pages and non-wikitext content types are another possible vector (T350480).
• Because this is tightly integrated into the parser, and affects the content consumed by (eg) Visual Editor and all downstream clients, the allowable transformations are fairly constrained, are are limiting to alteration or removal of the href attribute and (via the LinkColours hook) additions to the class attribute (and maybe addition of title which isn't typically set for external links).

The second option is to implement URL rewriting/blocking as an output transform. This is the approach taken in Convert SecureLinkFixer to an OutputTransformStage (1243953).
The advantages are:

• Quicker responsiveness to changes in blocked content. Although postprocessed content is also cached, regenerating it from the canonical content is much less expensive.
• "Dirty edits" which sanitize content unrelated to the edit are less likely to occur.
• "Editor intent" is preserved by default for downstream clients.
• Because the transform is done on the complete output page, and not just parsed wikitext content, it can cover more corner cases and block content from special pages, extensions & transclusions, etc.
• More modular structure: URL blocking/rewriting is done entirely outside of the core parser. Because the transformation doesn't affect the editor and other downstream clients, alterations to "bad" content can be more extensive without potentially breaking downstream.

The disadvantages are:

• We ship the "bad" urls to downstream clients by default, who may index them, etc. Core metadata (the externallinks database) will also contain the "bad" urls; harder to implement T417323: Apply SecureLinkFixer earlier to include externalinks database and Special:LinkSearch for example.
  • Potentially mitigated by the "flavor" system we have been working on (T348255) which ought to make it easier for clients to request the "sanitized" flavor of the canonical output, in the same way they request the "language converted" flavor (T380517).

Suggested plan of work

The ability to ensure sanitized metadata is a compelling advantage to the first proposal, and the disadvantages have mitigations. In the below we'll assume that Content-Transform-Team settles on the first proposal.

We assume that AbuseFilter will be used to provide the "warning to editor" functionality, and focus on the "comprehensive output filter". We will assume this feature will be built into AbuseFilter to allow some amount of coordination between the output blocks and the editor warnings.

Although SecureLinkFixer could be integrated into AbuseFilter as well, that would require reworking the bot which currently syncs the HSTS database from mozilla into gerrit. We will keep them separate for now and just ensure that the hook used by AbuseFilter to block links can also be used by SecureLinkFixer to rewrite them.

1. A new hook will be added which is called for external links (only) which takes a string by reference, and either mutates it (rewrite, for SecureLinkFixer) or sets it to null (block, AbuseFilter).
   • The hook could pass an array of CSS classes by references as well, like the GetLinkColours hook does.
   • The hook could allow the title attribute to be modified.
   • If the hook doesn't let the handler do this directly, if the returned value is null a CSS class (mw-blocked) should be added and the title text changed to a localized parser-url-blocked message, so that a blocked link can be styled correctly as a non-clickable link with some explanation for the block reason.
   • This is deliberately a narrower API than the existing LinkerMakeExternalLinkHook to ensure downstream clients are not disrupted by the (limited) changes to links possible through the hook. (It might be possible to reuse the existing hook and deprecate some of its functionality.)
2. Parsoid will be updated to use the hook via SiteConfig for external links. If the returned value is different from the passed value, Parsoid will add a data-mw-original-href attribute with the original value of the href for downstream clients. If the returned value is null Parsoid will remove the href attribute. Parsoid will put only the altered link in its ParserOutput metadata.
3. SecureLinkFixer will have a headless handler for this hook which will implement HSTS.
4. AbuseFIlter should have UX added to manage a list of blocked domains (it may already have this) and implement the hook to compare links against this list and return null where appropriate. CSS styling and title messages would be provided by AbuseFilter.
5. We should verify that other downstream tools (VisualEditor, ContentTranslationTools) handle an <a> tag without href in a reasonable fashion.

Some optional follow-ons:

6. A tool should be written to take as input a list of "changed" domains, and invalidate the parser cache for all pages whose externallinks table includes links in those domains. This probably should be throttled and run once a day on the list of new or updated domains from SecureLinkFixer and AbuseFilter, although the exact throttle rate and interval can be adjusted to keep the infrastructure load managable; this should ensure that our metadata and renderings are kept reasonably up to date.
7. The "edit check" feature of VisualEditor could be taught about data-mw-original-href and prompt editors for a correction to the href or removal of the link.
8. These changes could be made in the core parser as well for legacy compatibility, adding the hook to the existing LinkRenderer::makeExternalLink() method in core.

I'd expect Content-Transform-Team would take responsibility for #1, #2, #3 and #5 and possibly #8, while Product Safety and Integrity would be responsible for #4 and #6. Editing-team may wish to consider #7.