Page MenuHomePhabricator
Create Task
Maniphest T108601

Fix memory leak in LQT->Flow conversion script
Open, LowPublic
Actions

Description

Guess is that it could be in one of the in-process caches (e.g. LocalBufferedBagOStuff).

Details

SubjectRepoBranchLines +/-
Add explicit calls to the garbage collector every 100 topicsmediawiki/extensions/Flowmaster+4 -0
Change how memcache.local_buffered/buffered are handledmediawiki/extensions/Flowwmf/1.26wmf18+8 -1
Change how memcache.local_buffered/buffered are handledmediawiki/extensions/Flowmaster+8 -1
Eliminate some memory leaks affecting LQT->Flow:mediawiki/extensions/Flowwmf/1.26wmf18+103 -33
Eliminate some memory leaks affecting LQT->Flow:mediawiki/extensions/Flowmaster+103 -33
Customize query in gerrit

Related Objects

Event Timeline

• Mattflaschen-WMF created this task.Aug 10 2015, 5:36 PM
• Mattflaschen-WMF claimed this task.
• Mattflaschen-WMF raised the priority of this task from to Needs Triage.
• Mattflaschen-WMF updated the task description. (Show Details)
• Mattflaschen-WMF added a project: StructuredDiscussions.
• Mattflaschen-WMF subscribed.
Restricted Application added a project: Collaboration-Team-Triage. · View Herald TranscriptAug 10 2015, 5:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• DannyH added a project: Collaboration-Team-Archive-2015-2016.Aug 10 2015, 5:47 PM
• DannyH set Security to None.
• DannyH moved this task from Untriaged to In Development on the Collaboration-Team-Archive-2015-2016 board.
• Mattflaschen-WMF added a parent task: T106614: F9. Run LQT conversion script on mediawiki.org Project:Support desk.Aug 10 2015, 5:50 PM
gerritbot subscribed.Aug 11 2015, 1:18 AM

Change 230723 had a related patch set uploaded (by Mattflaschen):
WIP: Eliminate some memory leaks affecting LQT->Flow:

https://gerrit.wikimedia.org/r/230723

gerritbot added a project: Patch-For-Review.Aug 11 2015, 1:18 AM
• Mattflaschen-WMF moved this task from In Development to Needs Review on the Collaboration-Team-Archive-2015-2016 board.Aug 11 2015, 3:23 AM
Catrope moved this task from Needs Review to QA Review on the Collaboration-Team-Archive-2015-2016 board.Aug 11 2015, 5:58 PM
gerritbot added a comment.Aug 11 2015, 6:00 PM

Change 230723 merged by jenkins-bot:
Eliminate some memory leaks affecting LQT->Flow:

https://gerrit.wikimedia.org/r/230723

• Mattflaschen-WMF mentioned this in rEFLWd9e250d1339a: Eliminate some memory leaks affecting LQT->Flow:.Aug 11 2015, 6:00 PM
• Mattflaschen-WMF mentioned this in rMEXT149ab0f431f4: Updated mediawiki/extensions Project: mediawiki/extensions/Flow….
• DannyH triaged this task as Unbreak Now! priority.Aug 11 2015, 6:58 PM
• DannyH subscribed.
• Forrestbot added a project: WMF-deploy-2015-08-18_(1.26wmf19).Aug 11 2015, 7:00 PM
Etonkovidova subscribed.Aug 11 2015, 9:36 PM

Nothing too obvious when running convertLqtPageOnLocalWiki.php in test.wikimedia.beta.wmflabs - the process(es) properly stopped when finished and %MEM returns to normal values.

Etonkovidova moved this task from QA Review to Product Review on the Collaboration-Team-Archive-2015-2016 board.Aug 11 2015, 9:36 PM
gerritbot added a comment.Aug 11 2015, 10:18 PM

Change 230929 had a related patch set uploaded (by Mattflaschen):
Eliminate some memory leaks affecting LQT->Flow:

https://gerrit.wikimedia.org/r/230929

gerritbot added a comment.Aug 11 2015, 11:13 PM

Change 230929 merged by jenkins-bot:
Eliminate some memory leaks affecting LQT->Flow:

https://gerrit.wikimedia.org/r/230929

• Mattflaschen-WMF mentioned this in rEFLWa4bdad4b66a8: Eliminate some memory leaks affecting LQT->Flow:.Aug 11 2015, 11:13 PM
• Mattflaschen-WMF mentioned this in rMW494e30149490: Updated mediawiki/core Project: mediawiki/extensions/Flow….
• Forrestbot added a project: WMF-deploy-2015-08-11_(1.26wmf18).Aug 12 2015, 12:00 AM
• Mattflaschen-WMF added a comment.EditedAug 13 2015, 12:32 AM

See also T106614#1530532 for more background information on this.

This uses XDebug's memory profiling.

I ran:

sudo -u www-data php -d "xdebug.trace_format=1" -d "xdebug.show_mem_delta=1" -d "xdebug.auto_trace=1"  /srv/mediawiki-staging/multiversion/MWScript.php extensions/Flow/maintenance/convertLqtPageOnLocalWiki.php --wiki=mediawikiwiki --debug --srcpage "Project:Support_desk/LQT_Archive_1" --logfile "/tmp/Project:Support_desk.log" | tee --append ~/2015_08_12_LQT_Flow_Support_desk_resume.log

then ran the analyzer script with two variants (including sub-calls and not)

mattflaschen@terbium:~$ php tracefile-analyser.php /tmp/trace.1139352307.0dd119.xt memory-own 20

parsing...
 ( 0.08%)
 ...
 (100.00%)

Done.

Showing the 20 most costly calls sorted by 'memory-own'.

                                                                                  Inclusive        Own
function                                                                  #calls  time     memory  time     memory
------------------------------------------------------------------------------------------------------------------
json_encode                                                                  317  12.0353 896017832  12.0353 896017832
array_merge                                                                147232  1.4293 269969864  1.4293 269969864
ApiModuleManager->addModule                                                185234  9.0721 155236248  4.9456 155236248
preg_replace                                                               290062  3.8912 117905544  3.8912 117905544
array_keys                                                                 28509  0.3640 89114552  0.3640 89114552
str_replace                                                                244076  2.1500 79318416  2.1500 79318416
Liuggio\StatsdClient\Factory\StatsdDataFactory->produceStatsdDataEntity    63186  1.7174 70103800  1.0984 70072096
AutoLoader::autoload                                                         445  0.7737 72350848  0.2140 61333120
debug_backtrace                                                              933  0.0956 56139288  0.0956 56139288
set_error_handler                                                          97163  0.9352 54520600  0.9352 54520600
mysqli_result->fetch_object                                                23914  0.3867 49990184  0.3867 49990184
Title::makeTitle                                                            7641  1.5886 48073432  0.6851 45175272
ApiMain->getAllowedParams                                                   5495  0.1234 43879192  0.1234 43879192
func_get_args                                                              33872  0.3100 40455064  0.3100 40455064
MWNamespace::getCanonicalNamespaces                                         6582  0.1185 40900432  0.1121 39949072
array_filter                                                               103355  0.9158 39144784  0.8824 38880368
array_map                                                                  14269  0.4878 35109208  0.2360 33892208
explode                                                                    44764  0.4198 30705184  0.4198 30705184
preg_match                                                                 122175  1.3806 29076304  1.3806 29076304
str_ireplace                                                               12474  0.2096 23513432  0.2096 23513432
mattflaschen@terbium:~$ php tracefile-analyser.php /tmp/trace.1139352307.0dd119.xt memory-inclusive 20

parsing...
 ( 0.08%)
 ...
 (100.00%)

Done.

Showing the 20 most costly calls sorted by 'memory-inclusive'.

                                                                                  Inclusive        Own
function                                                                  #calls  time     memory  time     memory
------------------------------------------------------------------------------------------------------------------
json_encode                                                                  317  12.0353 896017832  12.0353 896017832
array_merge                                                                147232  1.4293 269969864  1.4293 269969864
call_user_func_array                                                       126187  222.4631 195126944  0.4437 18641248
Flow\Import\LiquidThreadsApi\LocalApiBackend->apiCall                        917  126.5677 170261944  0.0742  6310264
ApiModuleManager->addModules                                                9170  12.7245 169866432  2.3002 15455416
Hooks::run                                                                 60012  112.7617 163740768  5.4085 -205797128
ApiModuleManager->addModule                                                185234  9.0721 155236248  4.9456 155236248
Flow\Import\Postprocessor\LqtRedirector->afterTopicImported                  308  181.9841 152943504  0.0150    24640
Flow\Import\Postprocessor\ProcessorGroup->call                               618  182.0619 152482880  0.0152  -558000
Flow\Import\Postprocessor\ProcessorGroup->afterTopicImported                 616  182.0756 152480232  0.0107  -197304
Flow\Import\LiquidThreadsApi\CachedData->getMulti                           1320  113.9530 148248048  0.0328  3070424
Flow\Import\LiquidThreadsApi\CachedData->get                                1320  114.0018 147697792  0.0344 -2965240
Flow\Import\LiquidThreadsApi\CachedData->ensureLoaded                       1320  113.9201 145177624  0.0694 -42775296
preg_replace                                                               290062  3.8912 117905544  3.8912 117905544
ApiBase->getFinalParams                                                     8556  10.9517 109443024  0.2535  -715104
ApiMain->__construct                                                         917  10.0481 105398504  0.1481  5970824
ApiModuleManager->getModule                                                 1834  10.2599 102756648  0.0460   825088
Liuggio\StatsdClient\Factory\StatsdDataFactory->increment                  61846  12.5000 102693968  1.0014 14287480
ApiModuleManager->instantiateModule                                         1834  10.2139 101931560  0.0525  5552904
Flow\Import\LiquidThreadsApi\ApiBackend->retrievePageData                    310  71.5395 101308400  0.0086   355640

It makes it seem like json_encode is using a lot of memory (and there is a report about a json_encode memory leak). However, I'm not sure if this fully takes into account when things are freed/GCed.

• Mattflaschen-WMF moved this task from Product Review to In Development on the Collaboration-Team-Archive-2015-2016 board.Aug 13 2015, 12:53 AM
gerritbot added a comment.Aug 13 2015, 12:56 AM

Change 231201 had a related patch set uploaded (by Mattflaschen):
Add explicit calls to the garbage collector every 100 topics

https://gerrit.wikimedia.org/r/231201

ori subscribed.Aug 13 2015, 1:06 AM

Since this leak does not appear to occur with HHVM, I suggested temporarily depooling an HHVM app server and setting it aside for @Mattflaschen to use. I'm going to use mw1041 for this purpose.

• Mattflaschen-WMF added a comment.Aug 13 2015, 1:08 AM

I've never actually tried a long page with HHVM, but it's worth a shot.

• Mattflaschen-WMF mentioned this in T106614: F9. Run LQT conversion script on mediawiki.org Project:Support desk.Aug 13 2015, 6:53 AM
• Mattflaschen-WMF moved this task from In Development to Untriaged on the Collaboration-Team-Archive-2015-2016 board.Aug 13 2015, 5:31 PM
Catrope removed • Mattflaschen-WMF as the assignee of this task.Aug 13 2015, 5:41 PM
• Mattflaschen-WMF added a subscriber: matthiasmullie.Aug 13 2015, 7:10 PM

@matthiasmullie Looking at my change again, I'm worried about the impact of separating 'memcache.buffered' and 'memcache.local_buffered'.

The indexes and such still use 'memcache.local_buffered', and I think that means they may not be committed, leading to a cache fill.

• Mattflaschen-WMF added a comment.Aug 13 2015, 9:49 PM

Actually, I think the ones that were still using memcache.local_buffered were being committed to cache immediately since that's what defer does if it's not in a transaction:

https://git.wikimedia.org/blob/mediawiki%2Fextensions%2FFlow.git/215f4a015855667ee0ed6310715d17e6e9a90cdd/includes%2FData%2FBagOStuff%2FBufferedBagOStuff.php#L416

• Mattflaschen-WMF mentioned this in T108972: InvalidDataException on support desk.Aug 13 2015, 9:50 PM
gerritbot added a comment.Aug 13 2015, 10:29 PM

Change 231449 had a related patch set uploaded (by Mattflaschen):
Change how memcache.local_buffered/buffered are handled

https://gerrit.wikimedia.org/r/231449

gerritbot added a comment.Aug 13 2015, 10:46 PM

Change 231449 merged by jenkins-bot:
Change how memcache.local_buffered/buffered are handled

https://gerrit.wikimedia.org/r/231449

• Mattflaschen-WMF mentioned this in rEFLWdaec0fd3e1d8: Change how memcache.local_buffered/buffered are handled.Aug 13 2015, 10:46 PM
Diffusion mentioned this in rMEXT348dde29a151: Updated mediawiki/extensions Project: mediawiki/extensions/Flow….Aug 13 2015, 10:46 PM
gerritbot added a comment.Aug 13 2015, 10:47 PM

Change 231450 had a related patch set uploaded (by Mattflaschen):
Change how memcache.local_buffered/buffered are handled

https://gerrit.wikimedia.org/r/231450

gerritbot added a comment.Aug 13 2015, 11:08 PM

Change 231450 merged by jenkins-bot:
Change how memcache.local_buffered/buffered are handled

https://gerrit.wikimedia.org/r/231450

• Mattflaschen-WMF mentioned this in rEFLWa42fa42c3ac1: Change how memcache.local_buffered/buffered are handled.Aug 13 2015, 11:08 PM
• Mattflaschen-WMF mentioned this in rMW7c95573c21a2: Updated mediawiki/core Project: mediawiki/extensions/Flow….
• Mattflaschen-WMF lowered the priority of this task from Unbreak Now! to Low.Aug 14 2015, 7:33 AM
ori added a comment.Aug 14 2015, 6:25 PM

(mw1041 is now repooled.)

• Mattflaschen-WMF removed a project: Collaboration-Team-Archive-2015-2016.Aug 14 2015, 11:47 PM
• Mattflaschen-WMF removed a parent task: T106614: F9. Run LQT conversion script on mediawiki.org Project:Support desk.
gerritbot added a comment.Aug 24 2015, 5:13 PM

Change 231201 abandoned by Mattflaschen:
Add explicit calls to the garbage collector every 100 topics

Reason:
Abandoning this for now. This is a little hacky, and we don't have any huge boards to convert right now.

https://gerrit.wikimedia.org/r/231201

Aklapper edited projects, added Growth-Team; removed Collaboration-Team-Triage.Oct 12 2019, 4:17 AM
MBinder_WMF added a project: Growth-Team-Filtering.Apr 15 2021, 7:03 PM
Pppery removed a project: Patch-For-Review.Apr 1 2023, 7:58 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL