Page MenuHomePhabricator

AbuseFilter: Add a wikitext Summary field to filters and allow search to operate on it
Open, Needs TriagePublicFeature

Description

Feature summary (what you would like to be able to do and where):

The AbuseFilter interface currently provides two fields for documenting a filter: a single-line Description field and a plain-text Notes field.

  1. Please add a new Summary field positioned between the description and the filter conditions, formatted as wikitext, and subject to the same access controls as the filter itself.
  1. Please allow the search at Special:AbuseFilter to search the summary (in addition to or instead of the conditions).

Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):

The existing documentation fields each have significant limitations.

  • Description: a short single-line which is only suitable as a short label. More importantly, it is always public so filters that target long-term abuse (LTA) patterns, specific sockpuppets, or other sensitive activity cannot describe their purpose accurately without tipping off bad actors. Editors working with these filters must resort to obfuscating names that convey little information to other filter editors.
  • Notes: The notes field is plain text so it isn't possible to link to relevant pages such as related project space pages, sockpuppet investigation cases, RFCs that led to filters being added, user accounts, etc. The notes field also sits below the filter conditions. That's fine for a change log, but this has led to a common anti-pattern: abuse filter managers often put a summary of the filter's purpose, false positive handling guidance, etc. into a C-style comment at the top of the filter conditions themselves. This pollutes the conditions with documentation and makes both harder to read. And the notes field itself tends to grow into a long change-log with a summary at the top that often becomes stale.
  • The search function at Special:AbuseFilter is only able to search filter conditions, which is another reason why comments are being added to filters.
  • Because there is no dedicated summary field, filter documentation ends up in inconsistent locations. Sometimes, it's a comment at the top of conditions, sometimes it's at the top of the notes field, and sometimes it's even documented separately in an index to work around the lack of search. This makes it impossible for scripts to reliably surface a summary without parsing free-form text.
  • Wikitext links such as [[Wikipedia:Sockpuppet investigations/...]] are inert inside these text fields. In a wikitext summary field they would be functional links.

Some examples from English Wikipedia:

Benefits (why should this be implemented?):

A wikitext summary field, placed above the conditions and governed by the filter's own access level, would:

  • Allow private filters to include a frank, detailed description of their purpose including links to relevant pages without exposing sensitive information publicly.
  • Give filter editors a natural place for structured documentation (purpose, false positive guidance, maintenance notes) that is cleanly separated from the conditions and from the append-style notes field.
  • Eliminate the need to embed general documentation in C-style comments inside the filter conditions, and reduce the pressure on the notes field to serve as both summary and change log.
  • Support links to related pages without requiring client-side workarounds (see https://en.wikipedia.org/wiki/User:Suffusion_of_Yellow/filterNotes for an example).
  • Allow each wiki to adopt its own documentation conventions using wikitext section headings (e.g. "False positive reports", "Maintenance notes", "Currently targeting"), without requiring any schema changes.
  • Allow better integration with Wikipedia tooling: popup scripts could display the lead section of the summary field as a hover summary when browsing a filter link on noticeboards, filter hits in the abuse log, etc.

Event Timeline

This reminds me of T406952#11547903, where I suggested separating the description into private and public labels. Either way, we’d need a new database column.

Hi @Dragoniez

I’d like to work on this, but before I start, I want to make sure I understand the expected design correctly.

A few questions:

  • Should the Summary field follow the same permissions as the filter? For example, if a filter is private, should its summary also be private, or should summaries have separate settings?
  • Is this connected to the restructuring discussed in T406952? I want to make sure the database schema matches the larger plan.
  • Would adding a single new column to abuse_filter be enough, or should abuse_filter_history also store summary changes?

I’m also happy to put together a short design proposal before making any code changes.

@Rajveer42

I think it will be the right call to have the summary field following the same permissions as the filter. If the filter is not private, there's no reason to hide it. If the filter is private, then the description will be the only public information (like it is now).

Thanks @Dragoniez. So the summary visibility follows af_hidden, that makes sense.

Two quick follow-ups before I write up the design:

Should abuse_filter_history also track summary changes? My instinct says yes (every other filter field is in history), but I want to confirm before adding the column.

Is there anything in T406952 I should align the schema with, or are these independent enough to design separately?

Sorry I missed the mention.

Should abuse_filter_history also track summary changes? My instinct says yes (every other filter field is in history), but I want to confirm before adding the column.

Yes, and in fact this would be required for the proposed change. I would call the filter description a "public summary", and this is stored in the af_public_comments column of the abuse_filter table. There is a corresponding column in the abuse_filter_history table, and I would expect the new column for a private summary to follow the same pattern.

Is there anything in T406952 I should align the schema with, or are these independent enough to design separately?

Not particularly, although it'd be worth noting that the IDs of private filters should remain hidden from users without the appropriate permissions when using the query module. Once the private summary field is added, we should probably expose it through the API (if the usual permission checks pass), but that itself wouldn't block the proposed change, AFAICT.

Also note that the schema change should precede any functionality changes per https://wikitech.wikimedia.org/wiki/Schema_changes

Thanks @Dragoniez, that clears everything up.

So the plan is:

  • Patch 1 (schema only): add af_summary to abuse_filter and afh_summary to abuse_filter_history, mirroring the af_public_comments / afh_public_comments pattern, across all three DB engines + migration patches. No code changes that read/write it yet. I'll tag this task with schema-change per the wikitech workflow.
  • Patch 2 onward: data model (Specs + FilterStore), UI (AbuseFilterViewEdit), search (Special:AbuseFilter), API exposure (QueryAbuseFilters with permission checks per your note on private filter IDs).

I'll write up the design proposal as the first patch's commit message and CC the DBA team on the schema-change subtask when we get there.

Quick follow-up on the schema before,

  • I draft the patch the task asks for "wikitext section headings" and structured multi-paragraph documentation (purpose, false-positive guidance, maintenance notes), which won't fit af_public_comments's blob(255).
  • I'm planning to use blob(65530) for af_summary and afh_summary, matching af_pattern / af_comments. Does that match what you had in mind, or do you want a different length?

cc @Dragoniez

Change #1296257 had a related patch set uploaded (by Rajveer42; author: Rajveer42):

[mediawiki/extensions/AbuseFilter@master] Add af_summary and afh_summary schema fields

https://gerrit.wikimedia.org/r/1296257

First patch is up: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/AbuseFilter/+/1296257

Schema-only : adds af_summary (blob, 65530, nullable) to abuse_filter and afh_summary to abuse_filter_history, plus per-engine patches for MySQL/SQLite/Postgres and registration in SchemaChangesHandler. No code reads or writes the columns yet; follow-up patches will cover the data model, UI, search, and API exposure.

Noted the merge conflict with the abusefilter-blocker-user change, will rebase once one of them lands.

Quick follow-up on the schema before,

  • I draft the patch the task asks for "wikitext section headings" and structured multi-paragraph documentation (purpose, false-positive guidance, maintenance notes), which won't fit af_public_comments's blob(255).
  • I'm planning to use blob(65530) for af_summary and afh_summary, matching af_pattern / af_comments. Does that match what you had in mind, or do you want a different length?

cc @Dragoniez

I was wondering how we would design the editor UI. If we allow wikitext in the new summary field, we'd at least need a field for editing the raw wikitext. How do you plan to preview the rendered summary? Would we:

  1. Require the user to submit the form once to preview the summary?
  2. Provide a dedicated real time preview section?
  3. Use some other approach?

I'm not entirely convinced that (2) is ideal since it would take up a fair amount of space in an already somewhat packed interface. And (1) would probably raise the same issue once the form is submitted for preview. What's your thinking on this?
(I was originally thinking of a shorter, non-wikitext summary, partly influeced by T406952#11547903, but I realize that would take the task in a somewhat different direction.)

Sorry if I'm jumping ahead a bit. I just think it'd be beneficial to have a clearer picture of the intended functionality and UI before settling on the schema design.

Thoughts

Preview UI

I'd lean toward the simplest approach: no preview in the edit form.

This matches how descriptions and notes work today:

  • Users save the filter.
  • The rendered summary appears on the filter view page.
  • Any tools reading af_summary can use it there as well.

This avoids adding more UI to an already busy editor and follows existing conventions.

If preview is needed later, there are existing patterns we could reuse:

  • A "Show preview" button (similar to page editing), which submits the content and renders a preview below the form.
  • A link that opens the rendered wikitext in a new tab or modal (e.g. via Special:ExpandTemplates).

Both options are more complex than shipping without preview.

Schema

Using blob(65530) doesn't force us to use wikitext. It only sets an upper storage limit.

If we later decide summaries should be plain text or have a smaller limit, that can be enforced in validation rather than in the database schema.

Because of that, I think the schema patch can be evaluated independently of the UI decision.

Scope

The task description suggests a richer solution with:

  • Wikitext support
  • Section headings
  • Structured documentation

That's the more feature-rich interpretation.

A lighter approach (shorter, non-wikitext summaries, similar to the direction discussed in T406952) would also be reasonable. That mainly affects the UI layer, not the schema.

I'm happy to adjust the schema patch if there's a preference for the lighter approach (for example, using a smaller TEXT field). Otherwise, we can keep the schema flexible and let the UI implementation determine the final constraints.

cc @Dragoniez

I'd store the summary as wikitext and render it when viewing the filter (and in history), since that's where links are useful. For editing, I wouldn't add live preview in v1, matching Description/Notes today: save, then view. That also avoids the dedicated preview pane concern; a "Show preview" button could always be added later.

I'm leaning toward wikitext because functional links (SPI pages, related filters, tasks, etc.) are the whole point of the field, and plaintext leaves them inert. Search can query the stored source either way, visibility would follow af_hidden as agreed, and T406952 is confirmed non-blocking, so this can land independently.

Since it's a blob either way, patch 1296257 isn't affected by this UI choice. That's my recommendation, but it's your call. If you'd rather keep v1 minimal, I'm also fine shipping plaintext first and doing wikitext as a follow-up.

Here's how I'd sequence T424269: [the 5-patch series above]. The data-model/storage/history piece (Patch 2) is independent of the wikitext-vs-plaintext UI question, so I can start there now while we settle the editor design. Sound good?

I’d say please feel free to have a go with what you have in mind, as I’m not really the person to “grant permission” here. That said, I personally think it might be great if implementation patches were stacked on top of the schema change patch - that could make it easier for reviewers to +2 the schema patch, as far as I’m concerned.

Thanks! That makes sense, I'll stack the implementation patches on top of the schema change (1296257) so the new columns have visible consumers. Starting with the data-model patch (read/write + history tracking) now; the editor-UI patch, where the wikitext-vs-plaintext question lives, will sit higher in the stack once that part's settled.

Change #1301485 had a related patch set uploaded (by Rajveer42; author: Rajveer42):

[mediawiki/extensions/AbuseFilter@master] Wire af_summary through the filter data model

https://gerrit.wikimedia.org/r/1301485

Change #1301486 had a related patch set uploaded (by Rajveer42; author: Rajveer42):

[mediawiki/extensions/AbuseFilter@master] Wire af_summary through the filter data model

https://gerrit.wikimedia.org/r/1301486

Change #1301486 abandoned by Rajveer42:

[mediawiki/extensions/AbuseFilter@master] Wire af_summary through the filter data model

Reason:

Duplicate created by a local git mistake; superseded by the squashed data-model change (1301485).

https://gerrit.wikimedia.org/r/1301486

Ladsgroup subscribed.

Since this is on the abuse filters themselves, the tables are tiny and won't pose any scalability or reliability issues. So we don't have any objections. That being said, I need to warn you that implementing search on databases is a mess and you either end up with not supporting many features (such case sensitivity) or making a rather complex mess. See T416172#11672972 for some of similar issues.

Thanks for the review! Good to know the tables are small enough not to be a concern, and noted on the DB-side search complexity; we're planning to match in PHP for exactly that reason.