Page MenuHomePhabricator

Benchmark CPU and memory usage of push notifications service
Closed, ResolvedPublic

Description

When development of the push service is substantially complete, we will need to obtain CPU and memory benchmarks (under normal operations and during usage spikes) to be incorporated into the service's Helm chart.

Note: We have used the https://github.com/thesocialdev/mediawiki-services-profiler tool for this with mobileapps and chromium-render, and it will likely be useful again here.

Results

Benchmark

Current service stabilized in 1500m core of CPU and 300MB of RAM.

  • Processing ~188 req/s in avg and nearly 0% of failed requests
+------+-----------------+---------------+---------------+----------------------+-----------------------+------------+------------+
| Type | Name            | Request Count | Failure Count | Median Response Time | Average Response Time | Requests/s | Failures/s |
+======+=================+===============+===============+======================+=======================+============+============+
| POST | /v1/message/fcm | 383600        | 51            | 163000               | 168835                | 188.56     | 0.03       |
+------+-----------------+---------------+---------------+----------------------+-----------------------+------------+------------+
| None | Aggregated      | 383600        | 51            | 163000               | 168835                | 188.56     | 0.03       |
+------+-----------------+---------------+---------------+----------------------+-----------------------+------------+------------+

CPU usage

Memory usage

Response times (ms)

Total Request per second

Event Timeline

Mholloway triaged this task as Medium priority.Apr 17 2020, 2:39 PM
Mholloway created this task.
MSantos raised the priority of this task from Medium to High.Jun 2 2020, 3:06 PM
MSantos claimed this task.Jun 4 2020, 2:53 PM

Change 602390 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/deployment-charts@master] WIP: charts for push-notification service

https://gerrit.wikimedia.org/r/602390

So, I was able to get initial setup up and running for push-notifications service, I was even able to get the google-service credentials available so the service can properly work.

For more context, the push-notifications service uses a .json file to authenticate to the firebase service, when running the server I'm having an error unrelated to that which didn't appear in local environments:

Error while making request: unable to get local issuer certificate. Error code: UNABLE_TO_GET_ISSUER_CERT_LOCALLY

I'm assuming this has to do with proper TLS configuration for the service pod. Does anyone know how to work around it? cc/ @akosiaris @thcipriani @Jdforrester-WMF

Draft patch available here https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/602390

Update: ca-certificates were not being pulled in the production image, new patch to fix it

MSantos added a comment.EditedJun 10 2020, 7:47 PM

Benchmark

Current service stabilized in 1500m core of CPU and 300MB of RAM.

  • Processing ~188 req/s in avg and nearly 0% of failed requests
+------+-----------------+---------------+---------------+----------------------+-----------------------+------------+------------+
| Type | Name            | Request Count | Failure Count | Median Response Time | Average Response Time | Requests/s | Failures/s |
+======+=================+===============+===============+======================+=======================+============+============+
| POST | /v1/message/fcm | 383600        | 51            | 163000               | 168835                | 188.56     | 0.03       |
+------+-----------------+---------------+---------------+----------------------+-----------------------+------------+------------+
| None | Aggregated      | 383600        | 51            | 163000               | 168835                | 188.56     | 0.03       |
+------+-----------------+---------------+---------------+----------------------+-----------------------+------------+------------+

CPU usage

Memory usage

Response times (ms)

Total Request per second

MSantos updated the task description. (Show Details)Jun 11 2020, 6:58 PM
MSantos updated the task description. (Show Details)Jun 11 2020, 7:00 PM
MSantos closed this task as Resolved.Jul 8 2020, 12:06 PM

Benchmark is done, if necessary a new benchmark is needed after review, please re-open it.

Change 602390 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/deployment-charts@master] charts for push-notification service

https://gerrit.wikimedia.org/r/602390

Change 602390 merged by jenkins-bot:
[operations/deployment-charts@master] charts for push-notification service

https://gerrit.wikimedia.org/r/602390