Page MenuHomePhabricator

AutoWikiBrowser memory leak when preprocessing a large list
Closed, ResolvedPublicBUG REPORT

Description

  • Prepare for a pre-parse/skip tun using specific steps (below). There are currently just over 20,000 articles in the list
  • Start

What happens?:
Process eventually throws an OOM error, with (on an x86 box) 1.5GB memory used. This happens on AWB 6.4.0.0 and 6.5.0.0.

What should have happened instead?:
Process ends without significant memory loss. This happens on AWB 6.3.1.1.

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):
Windows 7 and 11. .NET framework 4.5.2 to 4.8.1 give the same result. A simple app using ListBox with repeated single-item deletions does not leak.

Other information (browser name/version, screenshots, etc.):
Theories: it could be a true memory leak, for example with an IDisposable object that doesn't release its unmanaged resources. Or there could be another list of Articles that's not getting trucated. Investigating the checkins between 12633 (6.3.1.1) and 12927 (6.4.0.0) to look where the repro starts.

Event Timeline

Also, the memory leak occurs regardless of logging state.

Originating discussion.

Tom.Reding renamed this task from AutoWikiBrowser with preprocessing on a large list leaks memory to AutoWikiBrowser memory leak when preprocessing a large list.Jun 8 2026, 9:53 AM
Tom.Reding updated the task description. (Show Details)

Apologies: I said I would add the steps, but didn't. They are currently listed on [[:w:en:User talk:Tom.Reding]] but can be listed here if anyone else is interested.

The leak starts at revision 12852. My immediate suspicion is the closure on the LMaker local variable in FixSyntax.cs. I have no idea how C# implements the currying, and I don't see why it wouldn't get released, but I'll try a fix (there are various ways of avoiding the closure). If I'm right, the ironies are (a) LMaker is only there because it provides an object to call NormalizeTitle() with (b) the delegate is rarely called anyway.

Settings / steps to reproduce:

  1. File > Reset settings to original default settings
  2. Options > Preferences > uncheck "Enable Logging"
  3. Options > Use pre-parse mode
  4. Source > Category > Year of birth missing > Make list
  5. Options tab > uncheck "Auto tag" & uncheck "Unicodify whole page"
  6. Options tab > Auto changes skip options > check "Human category changes"
  7. Skip tab > uncheck "Only genfixes"
  8. Start tab > Start

A quick fix: make LMaker static. That works in the above testcase. A more robust solution would be to move the body of NormalizeTitle() to Tools as a static, and wrap it in the existing method in ListMaker for signature compatibility.

I still don't understand why the separate closures don't get GC'd, but don't really need to. I also noticed that the full run incrementally uses a lot of Windows handles, but that doesn't seem to cause a problem.

I realized it wasn't directly an issue with the lambda function. Processing each article creates a ListMaker object, and the constructor hooks it into provider infrastructure, meaning it doesn't get GC'ed when it goes out of scope. That static-with-wrapper solution looks better.

Checked in: SVN 13024.