We now know that we're looking at ~1 month to get the headless Chromium based render service deployed. It might take longer to get it undeployed if it isn't a viable replacement for the Electron-based render service. If we have an understanding of how the new service performs ahead of time, then we might be able to save folk (including us) a lot of time.
# Open Questions
1. What?
From the parent task:
>>! In T176627#3680183, @phuedx wrote:
> We should be providing median and 95/99th percentile timings for the service rendering S/M/L/XL/XL… pages and resource consumption on the server during those test runs. Moreover, these tests should be repeatable (i.e. there's a script that we can run by anyone [and at different times!]).
>
> We all acknowledge that we'd have to take the results [[ https://en.wikipedia.org/wiki/Grain_of_salt | with a grain of salt ]] as we'd be using a VPS-hosted instance but they'll be helpful in the interim while we're trying to get the service into production.
>>! In T176627#3684846, @GWicke wrote:
> It might be worth focusing more on robustness than simple-page latency, as that is the more critical issue with Electron. Previously, I tested with a few very large articles (see T142226#2537844). This tested timeout enforcement. Testing with a simulated overload (many concurrent requests for huge pages) could also be useful to ensure that concurrency limits and resource usage limits are thoroughly enforced.
2. Where?
Performance testing on a VPS isn't ideal but it's cheaper than testing on production hardware (the time cost of deploying the service is ~1 month). @bmansurov has already set up a VPS to test the service, which is accessible here: http://chromium-pdf.wmflabs.org.
Per T178278#3726445, we also have access to bare metal with a big pipe. @pmiazga would be required to set up the service on that server.
## List of test articles
### long articles
https://en.wikipedia.org/wiki/List_of_members_of_the_Lok_Sabha_(1952%E2%80%93present) - long list article
https://en.wikipedia.org/wiki/Battle_of_Mosul_(2016%E2%80%932017) - long article, lots of images
https://en.wikipedia.org/wiki/Hendrick_Motorsports - long article, tables, images
https://en.wikipedia.org/wiki/Panama_Papers - long article, images
https://en.wikipedia.org/wiki/List_of_compositions_by_Franz_Schubert - long list article
### top 5 printed articles
https://en.wikipedia.org/wiki/Mahatma_Gandhi
https://en.wikipedia.org/wiki/A._P._J._Abdul_Kalam
https://en.wikipedia.org/wiki/Vijayadashami
https://hi.wikipedia.org/wiki/%E0%A4%AE%E0%A4%B9%E0%A4%BE%E0%A4%A4%E0%A5%8D%E0%A4%AE%E0%A4%BE_%E0%A4%97%E0%A4%BE%E0%A4%82%E0%A4%A7%E0%A5%80
https://en.wikipedia.org/wiki/Halloween
### long right to left articles
https://ar.wikipedia.org/wiki/%D9%85%D8%B5%D8%B1
https://he.wikipedia.org/wiki/%D7%AA%D7%9C_%D7%90%D7%91%D7%99%D7%91-%D7%99%D7%A4%D7%95
### stubs
https://en.wikipedia.org/wiki/Berenice_Mu%C3%B1oz
https://en.wikipedia.org/wiki/Benita_Mehra
## Things to double check
During T178501#3767355 @pmiazga found out that some rendered PDFs are incomplete. This happens only when the queue is full and service is constantly handling at least 2-3 concurrent requests at once. During performance testing please verify that all PDF files are rendered correctly and contain all pages.
= A/C
[] Generate PDFs using the list of articles above. Verify that the PDF contents look good, i.e. the PDFs contain actual article text, and not some error message from RESTBase or somewhere else.
[] Measure and report times spent rendering articles in succession. The service logs contain this information.
[] Use [[ https://www.joedog.org/siege-home/ | siege ]] to [[ https://www.joedog.org/siege-manual/#a08 | report ]] the performance of the service. Create various scenarios where you have a combination of articles (from above) and control the number of concurrent requests. Analyze the service logs to make sure that nothing unexpected happens. For example, if the concurrency in siege is set as 5 and it's set in service as 3, then make sure that after the 3rd request, the next two are aborted immediately.
[] Monitory the system load while doing these tests. See T175853#3616304 for reference.