Page MenuHomePhabricator
Create Task
Maniphest T281491

Reduce memory usage
Open, Needs TriagePublic
Actions

Description

Related to T203651. Until now, no particular effort was put into keeping the mem usage low. Should investigate which options are available. One might be using something other than Set/SplObjectStorage for links (e.g. plain array? custom hashmap?)

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Fix bug in preserved caused-by lines for pass-by-refDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+5 -5
Make most remaining objects immutableDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+88 -96
Avoid processing empty objects to save resourcesDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+88 -26
Make PreservedTaintedness immutableDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+76 -61
Improve shape tracking in caused-by lines for callsDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+31 -81
Deduplicate pass-by-reference objects in var linksDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+13 -12
Avoid unnecessary instantiations of MethodLinksDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+48 -26
Improve merging of caused-by linesDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+52 -15
Drop unnecessary `clone`s in CausedByLinesDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+7 -9
Make TaintednessWithError immutableDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+56 -59
Make CausedByLines immutableDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+131 -114
Drop unnecessary `clone`sDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+27 -52
Combine FunctionTaintedness flag settersDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+40 -62
Make FunctionTaintedness immutableDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+171 -168
Use Taintedness singletons whenever possibleDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+14 -14
Introduce Taintedness::safeSingleton()Daimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+49 -53
Drop Taintedness::mergeWithDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+49 -56
Remove most mutation methods from TaintednessDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+124 -157
Make MethodLinks immutableDaimona Eaytoymediawiki/tools/phan/SecurityCheckPluginmaster+10 K -124
Show related patches Customize query in gerrit

Related Objects

Event Timeline

Daimona created this task.Apr 29 2021, 3:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 29 2021, 3:05 PM
Daimona mentioned this in T269816: Investigate running taint-check with analyze-twice.Apr 29 2021, 6:55 PM
Daimona moved this task from Backlog to Plugin itself on the phan-taint-check-plugin board.Oct 15 2022, 5:21 PM
Daimona added a project: User-Daimona.Mar 3 2024, 5:10 PM
Daimona added a comment.Mar 3 2024, 6:58 PM

FTR, I had tried generating a memory profile a few months ago. From what I remember, the tools I tried (including https://github.com/BitOne/php-meminfo) would die due to the amount of stuff in memory, unless running on a smaller codebase (but then the results aren't as useful). Another option to explore might be https://github.com/arnaud-lb/php-memory-profiler.

Daimona added a comment.Oct 7 2024, 2:03 AM

Today I submitted a chain of patches that make some of the most commonly used value objects immutable. This has a few advantages:

  • Code is less prone to surprising bugs
  • Reduces the need to deep-clone objects all the time
  • Allows us to use singletons to represent the "empty" object of that type (for whatever "empty" means in the context of the type)

The chain currently consists of: r1078083, r1078091, r1078092, r1078093, r1078094, r1078095, r1078096, r1078097, r1078109, r1078112, and r1078113. I've tested patch set 4 of r1078113 on MediaWiki core and compared the results with taint-check master. Memory usage wents from 2844MB to 2567MB (-277MB); CPU time also decreased slightly from 2m52s to 2m46s. There's surely more cleanup to do, but the results are quite satisfying.

gerritbot added a comment.Oct 7 2024, 2:08 AM

Change #1078083 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Make MethodLinks immutable

https://gerrit.wikimedia.org/r/1078083

gerritbot added a project: Patch-For-Review.Oct 7 2024, 2:08 AM
gerritbot added a comment.Oct 7 2024, 2:11 AM

Change #1078091 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Remove most mutation methods from Taintedness

https://gerrit.wikimedia.org/r/1078091

gerritbot added a comment.Oct 7 2024, 2:12 AM

Change #1078092 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Drop Taintedness::mergeWith

https://gerrit.wikimedia.org/r/1078092

gerritbot added a comment.Oct 7 2024, 2:15 AM

Change #1078093 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Introduce Taintedness::safeSingleton()

https://gerrit.wikimedia.org/r/1078093

gerritbot added a comment.Oct 7 2024, 2:16 AM

Change #1078094 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Make FunctionTaintedness immutable

https://gerrit.wikimedia.org/r/1078094

gerritbot added a comment.Oct 7 2024, 2:18 AM

Change #1078095 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Use Taintedness singletons whenever possible

https://gerrit.wikimedia.org/r/1078095

gerritbot added a comment.Oct 7 2024, 2:19 AM

Change #1078096 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Combine FunctionTaintedness flag setters

https://gerrit.wikimedia.org/r/1078096

gerritbot added a comment.Oct 7 2024, 2:21 AM

Change #1078097 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Drop unnecessary `clone`s

https://gerrit.wikimedia.org/r/1078097

gerritbot added a comment.Oct 7 2024, 2:23 AM

Change #1078109 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Make CausedByLines immutable

https://gerrit.wikimedia.org/r/1078109

gerritbot added a comment.Oct 7 2024, 2:24 AM

Change #1078112 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Make TaintednessWithError immutable

https://gerrit.wikimedia.org/r/1078112

gerritbot added a comment.Oct 7 2024, 2:25 AM

Change #1078113 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Drop unnecessary `clone`s in CausedByLines

https://gerrit.wikimedia.org/r/1078113

sbassett awarded a token.Oct 10 2024, 9:22 PM
gerritbot added a comment.Oct 11 2024, 12:17 AM

Change #1079365 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Improve merging of caused-by lines

https://gerrit.wikimedia.org/r/1079365

gerritbot added a comment.Oct 11 2024, 8:11 PM

Change #1079556 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Avoid unnecessary instantiations of MethodLinks

https://gerrit.wikimedia.org/r/1079556

gerritbot added a comment.Oct 12 2024, 10:03 PM

Change #1079625 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Deduplicate pass-by-reference objects in var links

https://gerrit.wikimedia.org/r/1079625

gerritbot added a comment.Oct 16 2024, 1:33 PM

Change #1078083 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Make MethodLinks immutable

https://gerrit.wikimedia.org/r/1078083

gerritbot added a comment.Oct 16 2024, 1:41 PM

Change #1078091 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Remove most mutation methods from Taintedness

https://gerrit.wikimedia.org/r/1078091

gerritbot added a comment.Oct 16 2024, 1:41 PM

Change #1078092 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Drop Taintedness::mergeWith

https://gerrit.wikimedia.org/r/1078092

gerritbot added a comment.Oct 16 2024, 1:41 PM

Change #1078093 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Introduce Taintedness::safeSingleton()

https://gerrit.wikimedia.org/r/1078093

gerritbot added a comment.Oct 16 2024, 1:41 PM

Change #1078094 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Make FunctionTaintedness immutable

https://gerrit.wikimedia.org/r/1078094

gerritbot added a comment.Oct 16 2024, 1:41 PM

Change #1078095 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Use Taintedness singletons whenever possible

https://gerrit.wikimedia.org/r/1078095

gerritbot added a comment.Oct 16 2024, 1:44 PM

Change #1078096 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Combine FunctionTaintedness flag setters

https://gerrit.wikimedia.org/r/1078096

gerritbot added a comment.Oct 16 2024, 1:46 PM

Change #1078097 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Drop unnecessary `clone`s

https://gerrit.wikimedia.org/r/1078097

gerritbot added a comment.Oct 16 2024, 1:49 PM

Change #1078109 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Make CausedByLines immutable

https://gerrit.wikimedia.org/r/1078109

gerritbot added a comment.Oct 16 2024, 1:50 PM

Change #1078112 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Make TaintednessWithError immutable

https://gerrit.wikimedia.org/r/1078112

gerritbot added a comment.Oct 16 2024, 1:51 PM

Change #1078113 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Drop unnecessary `clone`s in CausedByLines

https://gerrit.wikimedia.org/r/1078113

gerritbot added a comment.Oct 16 2024, 1:58 PM

Change #1079365 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Improve merging of caused-by lines

https://gerrit.wikimedia.org/r/1079365

gerritbot added a comment.Oct 16 2024, 1:59 PM

Change #1079556 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Avoid unnecessary instantiations of MethodLinks

https://gerrit.wikimedia.org/r/1079556

gerritbot added a comment.Oct 16 2024, 2:06 PM

Change #1079625 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Deduplicate pass-by-reference objects in var links

https://gerrit.wikimedia.org/r/1079625

Maintenance_bot removed a project: Patch-For-Review.Oct 16 2024, 2:31 PM
gerritbot added a comment.Oct 18 2024, 1:44 AM

Change #1081302 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Improve shape tracking in caused-by lines for calls

https://gerrit.wikimedia.org/r/1081302

gerritbot added a project: Patch-For-Review.Oct 18 2024, 1:44 AM
gerritbot added a comment.Oct 21 2024, 10:15 PM

Change #1082097 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Make PreservedTaintedness immutable

https://gerrit.wikimedia.org/r/1082097

gerritbot added a comment.Oct 22 2024, 1:49 AM

Change #1082099 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Avoid processing empty objects to save resources

https://gerrit.wikimedia.org/r/1082099

gerritbot added a comment.Oct 29 2024, 1:17 PM

Change #1081302 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Improve shape tracking in caused-by lines for calls

https://gerrit.wikimedia.org/r/1081302

gerritbot added a comment.Nov 1 2024, 6:46 PM

Change #1082097 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Make PreservedTaintedness immutable

https://gerrit.wikimedia.org/r/1082097

gerritbot added a comment.Nov 1 2024, 6:50 PM

Change #1082099 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Avoid processing empty objects to save resources

https://gerrit.wikimedia.org/r/1082099

Maintenance_bot removed a project: Patch-For-Review.Nov 1 2024, 7:30 PM
gerritbot added a comment.Nov 4 2024, 12:26 AM

Change #1086572 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Make most remaining objects immutable

https://gerrit.wikimedia.org/r/1086572

gerritbot added a project: Patch-For-Review.Nov 4 2024, 12:26 AM
gerritbot added a comment.Nov 4 2024, 2:13 PM

Change #1086572 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Make most remaining objects immutable

https://gerrit.wikimedia.org/r/1086572

Maintenance_bot removed a project: Patch-For-Review.Nov 4 2024, 2:30 PM
gerritbot added a comment.Nov 5 2024, 3:11 AM

Change #1087285 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/tools/phan/SecurityCheckPlugin@master] Fix bug in preserved caused-by lines for pass-by-ref

https://gerrit.wikimedia.org/r/1087285

gerritbot added a project: Patch-For-Review.Nov 5 2024, 3:11 AM
gerritbot added a comment.Nov 27 2024, 3:34 PM

Change #1087285 merged by jenkins-bot:

[mediawiki/tools/phan/SecurityCheckPlugin@master] Fix bug in preserved caused-by lines for pass-by-ref

https://gerrit.wikimedia.org/r/1087285

Maintenance_bot removed a project: Patch-For-Review.Nov 27 2024, 4:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits