Change Details

We need to test, given different pod sizes (see T278220), what perfomance do we get from mediawiki on kubernetes. We can use our own mediawiki performance testing framework for this work. There are several dimensions to this problem so it will need proper testing; specifically we need to test which combination of: - N. of php workers per pod - CPU and Memory limits of the pods - Opcache/apcu size - Socket or TCP proxying to fcgi give us the best latency and throughput. In order to be able to compare results with production, we will need to reserve one kubernetes node for mediawiki only, run as many pods on it we can given their size, then run our testing framework https://gerrit.wikimedia.org/g/operations/software/benchmw on `mwdebug.discovery.wmnet` on the HTTP port (8444), and on one appserver in the active datacenter, so prepared: - should have similar hardware (esp. cpu) to the kubernetes node - depooled from traffic - php-fpm has been restarted before starting the tests Our goal is to get the same performance (within reasonable limits) at all concurrencies with the ones of the appserver. I would also suggest we get in touch with Performance to ask for other URLs they think we should benchmark. **Single request profiling** Another thing we should check is what is faster and what is slower on kubernetes; one way to test this is the following: - Disable the timer doing the automatic deployments to mediawiki on deploy1002 for the duration of the test - Deploy mediawiki to k8s using the latest mediawiki-multiversion-debug image https://docker-registry.wikimedia.org/restricted/mediawiki-multiversion-debug/tags/, that includes tideways and so it's able to send profiling data to xhgui - Run profiling request on k8s and one mwdebug server at the same time, after some warmup of the cache on both (basically request the same page twice without profiling, then grab a profile) - Check for big differences in the results - functions that require much more time on k8s in order to run Again for this test we can use the set of URLs we use in mwbench, but we should also reach out to performance to ask them if they see other stuff that should be tested.

We need to test, given different pod sizes (see T278220), what perfomance do we get from mediawiki on kubernetes. We can use our own mediawiki performance testing framework for this work. There are several dimensions to this problem so it will need proper testing; specifically we need to test which combination of: - N. of php workers per pod - CPU and Memory limits of the pods - Round 1: 4000-5000m CPU, 1000-2000Mi mem - Opcache/apcu size - Socket or TCP proxying to fcgi - Will be tested later on as we expect it to scrape a few ms from a single request. **Notes**: * Opcache/apcu sizes will be determined when we have a wider list of urls test, eg replay reads from production * Regardless of other factors, in production we will probably always observe some latency on k8s vs baremetal. In baremetal, a single request can warm up eg 96 workers on a server for a codepath, while we will need multiple requests to hit different pods to achieve the same result. Pods are more ephemeral, and we will have pods being terminated and spawned constantly. Our goal is to get the same performance (within reasonable limits) at all concurrencies with the ones of the appserver. I would also suggest we get in touch with Performance to ask for other URLs they think we should benchmark. **Single request profiling** Another thing we should check is what is faster and what is slower on kubernetes; one way to test this is the following: - Disable the timer doing the automatic deployments to mediawiki on deploy1002 for the duration of the test - Deploy mediawiki to k8s using the latest mediawiki-multiversion-debug image https://docker-registry.wikimedia.org/restricted/mediawiki-multiversion-debug/tags/, that includes tideways and so it's able to send profiling data to xhgui - Run profiling request on k8s and one mwdebug server at the same time, after some warmup of the cache on both (basically request the same page twice without profiling, then grab a profile) - Check for big differences in the results - functions that require much more time on k8s in order to run Again for this test we can use the set of URLs we use in mwbench, but we should also reach out to performance to ask them if they see other stuff that should be tested. ==Methodology and Testing== Original plan: In order to be able to compare results with production, we will need to reserve one kubernetes node for mediawiki only, run as many pods on it we can given their size, then run our testing framework https://gerrit.wikimedia.org/g/operations/software/benchmw on `mwdebug.discovery.wmnet` on the HTTP port (8444), and on one appserver in the active datacenter, so prepared: - should have similar hardware (esp. cpu) to the kubernetes node - depooled from traffic - php-fpm has been restarted before starting the tests **Round 1: mw2254 vs k8s, TLS enabled, 12 pods x 8 workers** **Tideways-xhprof:** generally increases request times, while we already know that mediawiki's performance on VMs is poorer, so in order to get a more informative profile, we selected a random production appserver where we temporarily installed tideways-xhprof. Addtionally we run ab tests with both tideways enabled and disabled. mw2254 re-parcing profile: https://performance.wikimedia.org/xhgui/run/view?id=613b33ac1d2b2a8ba4f624bf k8s re-parcing profile: https://performance.wikimedia.org/xhgui/run/view?id=613b33ea1e630124211e665e **Hardware:** Given than mw2254 is a 2016 server where almost all kubernetes nodes in codfw were purchased after 2017, we didn't put an limitations as to where a pod can be spawned (except a server having an SSD disk T288345) **TLS:**:The caching layer is talking to our app layer via TLS, for that reason, we chose to use TLS at least for this round of tests. **Pods specs:** * 96 workers (12 pods x 8 workers, mw2254 has 96 workers configured as well) * PHP: opcache.size=500M, opcache.interned_strings_buffer=50M, apc.size=400M * CPU: 4000-5000m, MEM: 1000-2000Mi

We need to test, given different pod sizes (see T278220), what perfomance do we get from mediawiki on kubernetes. We can use our own mediawiki performance testing framework for this work. There are several dimensions to this problem so it will need proper testing; specifically we need to test which combination of: - N. of php workers per pod - CPU and Memory limits of the pods - Round 1: 4000-5000m CPU, 1000-2000Mi mem - Opcache/apcu size - Socket or TCP proxying to fcgi give us the best latency and throughput. In order to be able to compare results with production, we will need to reserve one kubernetes node for mediawiki only, run as many pods on it we can given their size, then run our testing framework https://gerrit.wikimedia.org/g/operations/software/benchmw on `mwdebug.discovery.wmnet` on the HTTP port (8444), and on one appserver in the active datacenter, so prepared: - should have similar hardware (esp. cpu) to the kubernetes node - depooled from traffic - php-fpm has been restarted before starting the tests - Will be tested later on as we expect it to scrape a few ms from a single request. **Notes**: * Opcache/apcu sizes will be determined when we have a wider list of urls test, eg replay reads from production * Regardless of other factors, in production we will probably always observe some latency on k8s vs baremetal. In baremetal, a single request can warm up eg 96 workers on a server for a codepath, while we will need multiple requests to hit different pods to achieve the same result. Pods are more ephemeral, and we will have pods being terminated and spawned constantly. Our goal is to get the same performance (within reasonable limits) at all concurrencies with the ones of the appserver. I would also suggest we get in touch with Performance to ask for other URLs they think we should benchmark. **Single request profiling** Another thing we should check is what is faster and what is slower on kubernetes; one way to test this is the following: - Disable the timer doing the automatic deployments to mediawiki on deploy1002 for the duration of the test - Deploy mediawiki to k8s using the latest mediawiki-multiversion-debug image https://docker-registry.wikimedia.org/restricted/mediawiki-multiversion-debug/tags/, that includes tideways and so it's able to send profiling data to xhgui - Run profiling request on k8s and one mwdebug server at the same time, after some warmup of the cache on both (basically request the same page twice without profiling, then grab a profile) - Check for big differences in the results - functions that require much more time on k8s in order to run Again for this test we can use the set of URLs we use in mwbench, but we should also reach out to performance to ask them if they see other stuff that should be tested. ==Methodology and Testing== Original plan: In order to be able to compare results with production, we will need to reserve one kubernetes node for mediawiki only, run as many pods on it we can given their size, then run our testing framework https://gerrit.wikimedia.org/g/operations/software/benchmw on `mwdebug.discovery.wmnet` on the HTTP port (8444), and on one appserver in the active datacenter, so prepared: - should have similar hardware (esp. cpu) to the kubernetes node - depooled from traffic - php-fpm has been restarted before starting the tests **Round 1: mw2254 vs k8s, TLS enabled, 12 pods x 8 workers** **Tideways-xhprof:** generally increases request times, while we already know that mediawiki's performance on VMs is poorer, so in order to get a more informative profile, we selected a random production appserver where we temporarily installed tideways-xhprof. Addtionally we run ab tests with both tideways enabled and disabled. mw2254 re-parcing profile: https://performance.wikimedia.org/xhgui/run/view?id=613b33ac1d2b2a8ba4f624bf k8s re-parcing profile: https://performance.wikimedia.org/xhgui/run/view?id=613b33ea1e630124211e665e **Hardware:** Given than mw2254 is a 2016 server where almost all kubernetes nodes in codfw were purchased after 2017, we didn't put an limitations as to where a pod can be spawned (except a server having an SSD disk T288345) **TLS:**:The caching layer is talking to our app layer via TLS, for that reason, we chose to use TLS at least for this round of tests. **Pods specs:** * 96 workers (12 pods x 8 workers, mw2254 has 96 workers configured as well) * PHP: opcache.size=500M, opcache.interned_strings_buffer=50M, apc.size=400M * CPU: 4000-5000m, MEM: 1000-2000Mi