Page MenuHomePhabricator

Benchmark CPU and memory usage of push notifications service
Closed, ResolvedPublic

Description

When development of the push service is substantially complete, we will need to obtain CPU and memory benchmarks (under normal operations and during usage spikes) to be incorporated into the service's Helm chart.

Note: We have used the https://github.com/thesocialdev/mediawiki-services-profiler tool for this with mobileapps and chromium-render, and it will likely be useful again here.

Results

Benchmark

Current service stabilized in 1500m core of CPU and 300MB of RAM.

  • Processing ~188 req/s in avg and nearly 0% of failed requests
+------+-----------------+---------------+---------------+----------------------+-----------------------+------------+------------+
| Type | Name            | Request Count | Failure Count | Median Response Time | Average Response Time | Requests/s | Failures/s |
+======+=================+===============+===============+======================+=======================+============+============+
| POST | /v1/message/fcm | 383600        | 51            | 163000               | 168835                | 188.56     | 0.03       |
+------+-----------------+---------------+---------------+----------------------+-----------------------+------------+------------+
| None | Aggregated      | 383600        | 51            | 163000               | 168835                | 188.56     | 0.03       |
+------+-----------------+---------------+---------------+----------------------+-----------------------+------------+------------+

CPU usage

image.png (216×654 px, 35 KB)

Memory usage

image.png (230×653 px, 36 KB)

Response times (ms)

response_times_(ms)_1591812831.png (347×1 px, 53 KB)

Total Request per second

total_requests_per_second_1591812831.png (347×1 px, 59 KB)

Event Timeline

Mholloway created this task.

Change 602390 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/deployment-charts@master] WIP: charts for push-notification service

https://gerrit.wikimedia.org/r/602390

So, I was able to get initial setup up and running for push-notifications service, I was even able to get the google-service credentials available so the service can properly work.

For more context, the push-notifications service uses a .json file to authenticate to the firebase service, when running the server I'm having an error unrelated to that which didn't appear in local environments:

Error while making request: unable to get local issuer certificate. Error code: UNABLE_TO_GET_ISSUER_CERT_LOCALLY

I'm assuming this has to do with proper TLS configuration for the service pod. Does anyone know how to work around it? cc/ @akosiaris @thcipriani @Jdforrester-WMF

Draft patch available here https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/602390

Update: ca-certificates were not being pulled in the production image, new patch to fix it

Benchmark

Current service stabilized in 1500m core of CPU and 300MB of RAM.

  • Processing ~188 req/s in avg and nearly 0% of failed requests
+------+-----------------+---------------+---------------+----------------------+-----------------------+------------+------------+
| Type | Name            | Request Count | Failure Count | Median Response Time | Average Response Time | Requests/s | Failures/s |
+======+=================+===============+===============+======================+=======================+============+============+
| POST | /v1/message/fcm | 383600        | 51            | 163000               | 168835                | 188.56     | 0.03       |
+------+-----------------+---------------+---------------+----------------------+-----------------------+------------+------------+
| None | Aggregated      | 383600        | 51            | 163000               | 168835                | 188.56     | 0.03       |
+------+-----------------+---------------+---------------+----------------------+-----------------------+------------+------------+

CPU usage

image.png (216×654 px, 35 KB)

Memory usage

image.png (230×653 px, 36 KB)

Response times (ms)

response_times_(ms)_1591812831.png (347×1 px, 53 KB)

Total Request per second

total_requests_per_second_1591812831.png (347×1 px, 59 KB)

Benchmark is done, if necessary a new benchmark is needed after review, please re-open it.

Change 602390 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/deployment-charts@master] charts for push-notification service

https://gerrit.wikimedia.org/r/602390

Change 602390 merged by jenkins-bot:
[operations/deployment-charts@master] charts for push-notification service

https://gerrit.wikimedia.org/r/602390