Compared to equivalent/similar pre-PCS functions, mobileapps is significantly slower than the previous functionality in restbase, which in turn is exacerbating backlogs (T397072) - this can be seen by comparing historical execution times. It's not clear if this limitation is a result of extra complexity in the mobileapps service or if it is a resourcing problem - at the moment it seems like both.
Attempts have been made to increase resource allocation to the service, which leads to a paradoxical effect wherein latency skyrockets. Keeping memory limits at 1GB leads to frequent OOMKills at the Kubernetes level, which is not ideal, but allowing workers to consume more memory (even if they do not hit the memory limit or hit it less frequently) leads to requests taking significantly longer and either timing out, or just tying up workers.
Here we can see latency spiking after a resource bump of an increased limit of 2.7GB and recovering after a revert to the 1GB limit
Paradoxically we can see an overall significant improvement in the (admittedly less busy) codfw cluster with the same bump and no revert:
Notably also is the drop in error rate in codfw:
