Make change-prop tests independent of Kafka and Redis
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Pchelolo
	Mar 15 2019, 12:47 PM

Description

Currently, ChangeProp has only a very limited set of unit tests. The vast majority of tests are integration tests and require installation of Kafka and Redis. For the deployment pipeline, we need to make change-prop more unit-testable. One idea is to create a mock Kafka client and allow running the tests against it in the service-pipeline-test stage.

The full integration tests will run in service-pipeline-test-and-publish stage. @akosiaris is it possible to make the stage run an npm or bash command instead of relying on the service checker? Which config will the service have when running in the pipeline?

Details

	Subject	Repo	Branch	Lines +/-
	Change enable_blacklist to disable_blacklist.	mediawiki/services/change-propagation/deploy	master	+3 -4
	Disable ratelimiter and blacklisting.	mediawiki/services/change-propagation/jobqueue-deploy	master	+2 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T198901 Migrate production services to kubernetes using the pipeline
Invalid	None	T212801 TEC3:O3:O3.1:Q3 Goal - Move cxserver, citoid, changeprop, eventgate (new service) and ORES (partially) through the production CD Pipeline
Resolved	• holger.knust	T213193 Migrate changeprop to kubernetes
Resolved	• Pchelolo	T218396 Make change-prop tests independent of Kafka and Redis

Event Timeline

• Pchelolo renamed this task from Make change-prop tests undefended of Kafka and Redis to Make change-prop tests independent of Kafka and Redis.Mar 15 2019, 12:47 PM

• Pchelolo created this task.

is it possible to make the stage run an npm or bash command instead of relying on the service checker? Which config will the service have when running in the pipeline?

Moving my comment from https://gerrit.wikimedia.org/r/#/c/integration/config/+/496387/ here, let's continue the discussion in the task

Regarding the integration tests - change-prop is special in a sense that it has no REST API, so service-checker doesn't do anything for it. Is it possible to run an NPM command for integration tests?

Yes. The integration tests are specified per chart as a hook that implements the command helm test. Take a look at https://github.com/wikimedia/operations-deployment-charts/blob/master/charts/mathoid/templates/tests/test-service-checker.yaml for an example.

Obviously we can do whatever we want with that.

The config will be based on the defaults values.yaml supplied with the chart currently, but we can probably alter the pipeline to support overriding that.

The PR has been merged, resolving

Oh, no, not resolving yet. Next step - mock redis.

Change 497826 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/change-propagation/jobqueue-deploy@master] Disable ratelimiter and blacklisting.

https://gerrit.wikimedia.org/r/497826

gerritbot added a project: Patch-For-Review.Mar 20 2019, 6:27 PM

Change 497831 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/change-propagation/deploy@master] Change enable_blacklist to disable_blacklist.

https://gerrit.wikimedia.org/r/497831

• Pchelolo mentioned this in rMSCDd2ec6849cbfc: Change enable_blacklist to disable_blacklist..Mar 21 2019, 12:18 AM

Now it's ready - CP tests are independent of both Kafka and Redis.

Change 497826 merged by Ppchelko:
[mediawiki/services/change-propagation/jobqueue-deploy@master] Disable ratelimiter and blacklisting.