Page MenuHomePhabricator

Special:AbuseFilter/test takes long time and sometimes ends with server error
Closed, ResolvedPublic0.25 Estimated Story PointsPRODUCTION ERROR

Description

This issue was originally reported by @Supertoff in T388124#11037278.

Steps to replicate the issue (include links if applicable):

  • Navigate to Special:AbuseFilter/test
  • Type a rule to execute (eg., added_lines rlike "", which should match every edit)
  • Click "Test" below the form

What happens?:
The page is loading for a long time and may end with 503 server error. The exact behavior seems to depend on a time when you submit the form: yesterday morning I've got the error three times in a row, while today it worked fine.

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia): plwiki

Other information (browser name/version, screenshots, etc.): I've tested this on a local wiki, running the same version of AbuseFilter as is currently installed on WMF cluster, but noticed no problems.

Data from the error page footer, for further investigation:

Request served via cp3069 cp3069, Varnish XID 622186388
Error: 503, Backend fetch failed at Mon, 28 Jul 2025 07:06:09 GMT

Request served via cp3069 cp3069, Varnish XID 616488692
Error: 503, Backend fetch failed at Mon, 28 Jul 2025 07:11:49 GMT

Request served via cp3069 cp3069, Varnish XID 631126722
Error: 503, Backend fetch failed at Mon, 28 Jul 2025 07:14:49 GMT

Acceptance criteria

  • Special:AbuseFilter/test loads at an acceptable speed
  • The fix(es) used are backported to the release versions the security patches (that caused the slower responses) was applied to

Event Timeline

Dreamy_Jazz subscribed.

Here is a Excimer for a request when following the steps on https://test.wikipedia.org/wiki/Special:AbuseFilter/test: https://performance.wikimedia.org/excimer/profile/090643dbbddee550

image.png (2,868×864 px, 297 KB)

It took around 13 seconds to compute all the variables for the 100 actions, and around 9 seconds of that was parsing content.

Adding Content-Platform-Team to see if there were any recent changes to parsing that might have caused this to run slower?

It took around 13 seconds to compute all the variables for the 100 actions, and around 9 seconds of that was parsing content.

added_lines should only involve diffing of new_wikitext w.r.t. old_wikitext, none of which requires parsing the wikitext.

I am suspicious about the dumpAllVars calls. We shouldn't need to compute (dump) all the variables that the generator knows, only those that were actually used during evaluation. The expensive variable in question is (probably) new_html (we didn't ask for it, though).

I am suspicious about the dumpAllVars calls. We shouldn't need to compute (dump) all the variables that the generator knows, only those that were actually used during evaluation. The expensive variable in question is (probably) new_html (we didn't ask for it, though).

I concur. This was added recently, in r1166844, so it lines up. Passing true to dumpAllVars causes all variables to be computed, including those that we did not use as Matěj pointed out. I think omitting the $compute argument would fix this and give you a list of variables that are already set (either because they aren't lazy-loaded, or because the filter uses them). Or even better, you don't even need VariablesManager. You can just add a new hasComputedVariable( string $varName ): bool method to VariableHolder which essentially does return $this->getVarThrow( $varName ) instanceof AFPData (like VariablesManager, but watch out for exceptions). Then, in the getUsedProtectedVariables loop in ViewTestBatch, you can call $vars->hasComputedVariable( $protectedVariable ), without needing a $varsArray.

Also, watch out that there might be more places affected by a similar bug: https://codesearch.wmcloud.org/deployed/?q=dumpAllVars%5C%28.%2Btrue&files=&excludeFiles=&repos=. For instance, CheckMatch seems affected, although the impact there is probably negligible because it checks a single action.

This may be related to the recent security patches Trust and Safety Product Team made to AbuseFilter. We called ::dumpAllVars to get the protected variables. Perhaps this call needs to be made better to avoid computing all variables.

So adding to our sprint board to be considered as maintenance to work on. Would likely mean it's not worked on until our next sprint, but can move it forward if needed.

Dreamy_Jazz changed the subtype of this task from "Bug Report" to "Production Error".
Dreamy_Jazz set the point value for this task to 1.Aug 1 2025, 12:19 PM

Thanks for the pointers.

I think omitting the $compute argument would fix this and give you a list of variables that are already set (either because they aren't lazy-loaded, or because the filter uses them).

In some places we do need to load the value of any lazily loaded protected variables, because when logging we only want to log that a value was viewed if it's not null.

Or even better, you don't even need VariablesManager. You can just add a new hasComputedVariable( string $varName ): bool method to VariableHolder which essentially does return $this->getVarThrow( $varName ) instanceof AFPData (like VariablesManager, but watch out for exceptions). Then, in the getUsedProtectedVariables loop in ViewTestBatch, you can call $vars->hasComputedVariable( $protectedVariable ), without needing a $varsArray.

In the cases where we do need the value of the variable, we can't specifically use this. In other cases I think calling VariableHolder::getVars is enough because in those places we are only interested if the variable is defined in the VariableHolder (the existing code uses array_keys so would also include lazily loaded variables that had not been loaded yet).

Also, watch out that there might be more places affected by a similar bug: https://codesearch.wmcloud.org/deployed/?q=dumpAllVars%5C%28.%2Btrue&files=&excludeFiles=&repos=. For instance, CheckMatch seems affected, although the impact there is probably negligible because it checks a single action.

Thanks. I'll update these also as part of this task.

Dreamy_Jazz changed the point value for this task from 1 to 0.25.Aug 6 2025, 7:57 PM

Change #1176312 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/AbuseFilter@master] Protected variables logging: Only compute protected variable values

https://gerrit.wikimedia.org/r/1176312

We should backport the above patch once we are happy with it and have merged it in the master branch. This is because the security fixes that cause the slowdowns will also be present on the release versions and we should avoid making the release versions slow too.

I think omitting the $compute argument would fix this and give you a list of variables that are already set (either because they aren't lazy-loaded, or because the filter uses them).

In some places we do need to load the value of any lazily loaded protected variables, because when logging we only want to log that a value was viewed if it's not null.

I'm not sure if I follow: if a (lazy-loaded) variable was not computed, doesn't that mean that the user didn't see it? As a case in point, if you're on /test and a (lazy-loaded) protected variable was not computed, you can be sure that the user didn't get any information about it. (NB: this does not mean that the tested pattern doesn't use the variable: maybe it does, but we never got to actually computing it due to shortcircuiting. At any rate, the value of the variable did not influence the outcome).

I wanted to double-check this, but then I see we aren't computing user_unnamed_ip for past edits, so I'm not sure what is being logged?

I'm not sure if I follow: if a (lazy-loaded) variable was not computed, doesn't that mean that the user didn't see it? As a case in point, if you're on /test and a (lazy-loaded) protected variable was not computed, you can be sure that the user didn't get any information about it. (NB: this does not mean that the tested pattern doesn't use the variable: maybe it does, but we never got to actually computing it due to shortcircuiting. At any rate, the value of the variable did not influence the outcome).

In the examine view, the entire VariableHolder is dumped using ::dumpAllVars( ..., true) and has been doing that since before the security patches: https://gerrit.wikimedia.org/g/mediawiki/extensions/AbuseFilter/+/ec27766a500794b86f6019bcb4a27a6b3b8d1293/includes/View/AbuseFilterViewExamine.php#416

In other places that I've left using ::dumpAllVars:

  • CheckMatch will only say a user viewed the value of a protected variable if the protected variable is included in the filter pattern being used to test against actions
  • AbuseFilterViewTestBatch does the same as CheckMatch
  • AbuseFilterViewExamine::showExaminerForLogEntry loads the VariableHolder from the afl_var_dump, so no computing is technically needed (as the values should be already stored in a non-lazily loaded way in the var dump). Therefore the logging would be restricted to what was generated and would be shown.
  • SpecialAbuseLog::showDetails, same as the above
  • AbuseFilterViewExamine::showExaminerForRC would eventually dump everything (per the existing ::dumpAllVars call I mentioned above)

I wanted to double-check this, but then I see we aren't computing user_unnamed_ip for past edits, so I'm not sure what is being logged?

We now have additional protected variables which are computed for past edits and lazily loaded. See https://www.mediawiki.org/wiki/Extension:IPReputation/AbuseFilter_variables for more detail on them.

In any case, if you want to comment on any of the places I've left with ::dumpAllVars then feel free to comment on the patch I've uploaded for this task.

I'm not sure if I follow: if a (lazy-loaded) variable was not computed, doesn't that mean that the user didn't see it? As a case in point, if you're on /test and a (lazy-loaded) protected variable was not computed, you can be sure that the user didn't get any information about it. (NB: this does not mean that the tested pattern doesn't use the variable: maybe it does, but we never got to actually computing it due to shortcircuiting. At any rate, the value of the variable did not influence the outcome).

In the examine view, the entire VariableHolder is dumped using ::dumpAllVars( ..., true) and has been doing that since before the security patches: https://gerrit.wikimedia.org/g/mediawiki/extensions/AbuseFilter/+/ec27766a500794b86f6019bcb4a27a6b3b8d1293/includes/View/AbuseFilterViewExamine.php#416

Yup, /examine is a bit different though: it lets you see the values for *all* variables, regardless of the pattern you are testing; in fact, you don't even need to test any pattern. This is OK because the view serves a different purpose, and most importantly, it shows variables for a single edit. So, in that case it is correct to include all lazy-loaded variables in the protected variables check. (Relatedly, buildVarDumpTable is where exportAllVars is called).

In other places that I've left using ::dumpAllVars:

  • CheckMatch will only say a user viewed the value of a protected variable if the protected variable is included in the filter pattern being used to test against actions
  • AbuseFilterViewTestBatch does the same as CheckMatch

This is the view I was talking about in my example. After reading your comment I also rechecked the code, but I think my original point still stands: using ViewTestBatch as an example, it calls getUsedVars to get a list of variables used by the test pattern. However, this list is obtained via checkSyntax, which disables short-circuit evaluation in the filter evaluator. So, for example, if someone tests the pattern false & user_unnamed_ip = '1.2.3.4', getUsedVars will report that the pattern uses user_unnamed_ip, but its value will never be computed, or seen by the user. My point is that if that is the case, we don't need to log the access, because the value of the variable wasn't accessed. And how do we determine if a variable, despite being used by the pattern, is not computed? For lazy-loaded variables, you can check that the variable's value is still an instance of LazyLoadedVariable. For non-lazy-loaded variables, you can't. But in any case, you don't need to forcefully compute anything, so the dumpAllVars call is unnecessary.

  • AbuseFilterViewExamine::showExaminerForLogEntry loads the VariableHolder from the afl_var_dump, so no computing is technically needed (as the values should be already stored in a non-lazily loaded way in the var dump). Therefore the logging would be restricted to what was generated and would be shown.
  • SpecialAbuseLog::showDetails, same as the above

Correct, dumpAllVars is a no-op here. These two places (and possibly others) could just use array_keys( $vars->getVars() ). Or possibly a getter that already gives you the variable names; in retrospect, I don't like my choice of exposing the entire mVars property.

  • AbuseFilterViewExamine::showExaminerForRC would eventually dump everything (per the existing ::dumpAllVars call I mentioned above)

Yep, this one can be left as-is.

I wanted to double-check this, but then I see we aren't computing user_unnamed_ip for past edits, so I'm not sure what is being logged?

We now have additional protected variables which are computed for past edits and lazily loaded. See https://www.mediawiki.org/wiki/Extension:IPReputation/AbuseFilter_variables for more detail on them.

Apologies, I wanted to end my previous comment with a huge disclaimer that I'm not up-to-date with the latest protected variables changes, but evidently I forgot to write it.

In other places that I've left using ::dumpAllVars:

  • CheckMatch will only say a user viewed the value of a protected variable if the protected variable is included in the filter pattern being used to test against actions
  • AbuseFilterViewTestBatch does the same as CheckMatch

This is the view I was talking about in my example. After reading your comment I also rechecked the code, but I think my original point still stands: using ViewTestBatch as an example, it calls getUsedVars to get a list of variables used by the test pattern. However, this list is obtained via checkSyntax, which disables short-circuit evaluation in the filter evaluator. So, for example, if someone tests the pattern false & user_unnamed_ip = '1.2.3.4', getUsedVars will report that the pattern uses user_unnamed_ip, but its value will never be computed, or seen by the user. My point is that if that is the case, we don't need to log the access, because the value of the variable wasn't accessed. And how do we determine if a variable, despite being used by the pattern, is not computed? For lazy-loaded variables, you can check that the variable's value is still an instance of LazyLoadedVariable. For non-lazy-loaded variables, you can't. But in any case, you don't need to forcefully compute anything, so the dumpAllVars call is unnecessary.

Okay. I think I get your point now. The call to ::checkConditions should come before logging. AFAICS then $vars would contain the variables that were used as part of evaluating the condition, so then the code would log if:

  • The protected variable is set in $vars
  • The value of that protected variable in $vars is not an instance of LazilyLoadedVariable
  • The value of that protected variable in $vars is not null

I think a call to ::checkSyntax is still required for permission checking because:

  • Each protected variable can optionally have additional requirements for access
  • In my opinion, the code should reject any attempt to use a protected variable the user cannot see even if it's not matched against, because getting a permission error only some of the time will be confusing (especially in an API endpoint)

Okay. I think I get your point now. The call to ::checkConditions should come before logging.

Ah, right, that's a key precondition. I didn't realize this wasn't already the case.

AFAICS then $vars would contain the variables that were used as part of evaluating the condition, so then the code would log if:

  • The protected variable is set in $vars
  • The value of that protected variable in $vars is not an instance of LazilyLoadedVariable
  • The value of that protected variable in $vars is not null

Exactly. One thing to note here is that if you have a protected variable that is NOT lazy-loaded, and if the tested pattern uses it, BUT it never actually gets used due to short-circuiting, you would still log the protected variable access. We don't have a general purpose way to tell if a variable's value has been used by a filter (lazy-loaded variables are a special-case). In practice though, this shouldn't be too common, and the worst that can happen is logging things that didn't need to be logged; which is surely better than if it were the other way around.

I think a call to ::checkSyntax is still required for permission checking because:

  • Each protected variable can optionally have additional requirements for access
  • In my opinion, the code should reject any attempt to use a protected variable the user cannot see even if it's not matched against, because getting a permission error only some of the time will be confusing (especially in an API endpoint)

Yep, I agree. Ultimately it's all about the intent: if you want to determine whether the pattern uses a protected variable (e.g. for permission checks), then use checkSyntax. If you want to determine whether the variable's value was actually used, you still need checkSyntax, but then you intersect that with checking that the variable is not an instance of LazyLoadedVariable.

Exactly. One thing to note here is that if you have a protected variable that is NOT lazy-loaded, and if the tested pattern uses it, BUT it never actually gets used due to short-circuiting, you would still log the protected variable access. We don't have a general purpose way to tell if a variable's value has been used by a filter (lazy-loaded variables are a special-case). In practice though, this shouldn't be too common, and the worst that can happen is logging things that didn't need to be logged; which is surely better than if it were the other way around.

Luckily for now we only have lazily loaded protected variables, which should avoid this for now.

I also doubt that a non-lazily loaded protected variable would ever exist because we don't want them set in all filters (as they can only be used in those which are protected). Non-lazily loaded ones would therefore always appear and violate that rule unless they were appropriately removed.

Change #1176312 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@master] Protected variables logging: Only compute protected variable values

https://gerrit.wikimedia.org/r/1176312

For QA we will need to wait until this is on production to verify that it's no longer too slow. Once we have done that, then we need to backport the fix to the supported release versions where the security fixes were applied.

Change #1178831 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/AbuseFilter@REL1_44] Protected variables logging: Only compute protected variable values

https://gerrit.wikimedia.org/r/1178831

Change #1178903 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/AbuseFilter@REL1_43] Protected variables logging: Only compute protected variable values

https://gerrit.wikimedia.org/r/1178903

Change #1178831 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@REL1_44] Protected variables logging: Only compute protected variable values

https://gerrit.wikimedia.org/r/1178831

Change #1178903 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@REL1_43] Protected variables logging: Only compute protected variable values

https://gerrit.wikimedia.org/r/1178903

The fix has been verified in production and backported. I don't think there's something remaining for QA here. Therefore, I'll mark it as resolved.