Page MenuHomePhabricator

Present “AI tells” within suggestion mode
Closed, ResolvedPublic

Description

In T398478, we implemented a way for anyone with the technical know-how to show a suggestion when a specific span of text is matched/present within the article someone is editing.

In an effort to demonstrate one potential application of this capability, we'd like to implement a suggestion that surfaces when a community-defined "sign" of AI writing is detected within a given article.

Stories

As an experienced volunteer who is particularly interested in reducing the amount of content present on Wikipedia that is generated by AI without human intervention, I would value knowing if any "signs" of AI writing are present within the article I am reviewing/editing using the visual editor, so that I can use this information/signal to inform what – if any – moderation action I take in response.

Note: we can imagine a future wherein the suggestion this task is asking for could be combined with T402465 to further enable the above.

Requirements

  • Within the TextMatchEditCheck config, create rules that detect common language found in AI-generated content, as agreed upon in Signs of AI writing. To start, create these in en and fr.
  • Add the rules to their respective wikis, so they can be viewed live in suggestion mode.
  • To start, the only action that should accompany this particular suggestion is Dismiss

Event Timeline

@ppelberg, let me know if you'd rather this be a child of T404220.

ppelberg updated the task description. (Show Details)

@ppelberg, let me know if you'd rather this be a child of T404220.

The way you've related it works for me!

[Please review project tags/subscribers when creating subtasks - thanks!]

[Please review project tags/subscribers when creating subtasks - thanks!]

Sorry about that. Thank you!

First pass at configs for enwiki:

Configs for dewiki:

@medelius: before closing, would it be accurate for me to think that, at present, the "AI Tells" suggestion is available as an experimental Check within Suggestion Mode?

Yes, that's correct! It's built with TextMatch, which is an experimental check. So "AI tells" is available as long as the editcheck-config.json file is configured for it (which it is on enwiki).

Got it! Ok. And we'll consider whether we want to revise this as part of T402242 and T404599.

Looks great! A few hours late, but I wonder if it could be possible to integrate regular expressions to the rules? For example, we currently have:

	"stands as a testament",
	"stands as a reminder",
	"serves as a testament",
	"serves as a reminder",
	"is a testament",
	"is a reminder",

Would it be more future-proof to have a single rule "(stands as|serves as|is) a (testament|reminder)"? This would also allow us to include phrasal templates in the list, which are much more prevalent as tells in more recent models (GPT-5 for instance), while individual words/phrases were more commonplace in earlier models.

Unfortunately we don't support regex queries just yet, but we would like to in the future sometime after the beta release. It would certainly make queries like those more robust.

Great to know! Is there an open task for that?