Page MenuHomePhabricator

Profile ORES code memory use, optimize pre-fork allocation
Open, MediumPublic

Description

Our Celery workers are forked after pre-initializing all models, so that the bulk of the application memory is shared between all sibling processes. However, child processes still use around ~310MB each once they "warm up". We want to know whether this is normal, unavoidable runtime stuff or if there's more pre-initialization we could do which would cause more of that memory to be shared.

See https://pypi.python.org/pypi/memory_profiler