Change Details

The `ImportConstraintStatements.php` maintenance script in `WikibaseQualityConstraints` currently imports constraint statements for all properties in a simple loop, with no offset, limit, batching, or sleeping. This can potentially lead to a high DB load, and high replication lag. **Example:** We could group the imports into batches (with some default batch size, e. g. 100) and sleep between each batch (default e. g. 3 s). Batch size and sleep time should be configurable via command line arguments. Some offset/limit arguments could also be added, so that a deployer can first import constraint statements for the first 1000 properties, then proceed later with the next 1000, and so on. **Acceptance criteria:** - [ ] The rate of action of the maintenance script is configurable - [ ] The script includes a wait for replication between batches **Open questions:** * Is batch+wait the best approach? Do we need offset+limit? * What should the default batch size and wait time be (in case they are implemented)? **//A: @addshore reckons it should be about 100 batch size and 1 second wait.//** **Original Comment:** >As a deployer, I want to run the `extensions/WikibaseQualityConstraints/maintenance/ImportConstraintStatements.php` maintenance script when necessary in order to resolve problems with the constraint definitions, without having to worry about its impact on the wiki. >This only takes a few minutes – less than 10 ms for most properties, not much more for some larger properties, and we’re still well below 10000 properties – but it was enough to cause a [visible traffic spike on the s8 master db host](https://grafana.wikimedia.org/d/000000273/mysql?panelId=5&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1071&var-port=9104&from=1561554000000&to=1561555200000) during the deployment for T223372 (around 13:17 UTC). As Wikidata keeps getting more properties, I believe this will gradually become more important.

The `ImportConstraintStatements.php` maintenance script in `WikibaseQualityConstraints` currently imports constraint statements for all properties in a simple loop, with no offset, limit, batching, or sleeping. This can potentially lead to a high DB load, and high replication lag. **Example:** We could group the imports into batches (with some default batch size, e. g. 100) and sleep between each batch (default e. g. 3 s). Batch size and sleep time should be configurable via command line arguments. Some offset/limit arguments could also be added, so that a deployer can first import constraint statements for the first 1000 properties, then proceed later with the next 1000, and so on. **Acceptance criteria:** - [ ] The rate of action of the maintenance script is configurable - [ ] The script includes a wait for replication between batches **Open questions:** * Is batch+wait the best approach? Do we need offset+limit? **Original Comment:** >As a deployer, I want to run the `extensions/WikibaseQualityConstraints/maintenance/ImportConstraintStatements.php` maintenance script when necessary in order to resolve problems with the constraint definitions, without having to worry about its impact on the wiki. >This only takes a few minutes – less than 10 ms for most properties, not much more for some larger properties, and we’re still well below 10000 properties – but it was enough to cause a [visible traffic spike on the s8 master db host](https://grafana.wikimedia.org/d/000000273/mysql?panelId=5&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1071&var-port=9104&from=1561554000000&to=1561555200000) during the deployment for T223372 (around 13:17 UTC). As Wikidata keeps getting more properties, I believe this will gradually become more important.

The `ImportConstraintStatements.php` maintenance script in `WikibaseQualityConstraints` currently imports constraint statements for all properties in a simple loop, with no offset, limit, batching, or sleeping. This can potentially lead to a high DB load, and high replication lag. **Example:** We could group the imports into batches (with some default batch size, e. g. 100) and sleep between each batch (default e. g. 3 s). Batch size and sleep time should be configurable via command line arguments. Some offset/limit arguments could also be added, so that a deployer can first import constraint statements for the first 1000 properties, then proceed later with the next 1000, and so on. **Acceptance criteria:** - [ ] The rate of action of the maintenance script is configurable - [ ] The script includes a wait for replication between batches **Open questions:** * Is batch+wait the best approach? Do we need offset+limit? * What should the default batch size and wait time be (in case they are implemented)? **//A: @addshore reckons it should be about 100 batch size and 1 second wait.//** **Original Comment:** >As a deployer, I want to run the `extensions/WikibaseQualityConstraints/maintenance/ImportConstraintStatements.php` maintenance script when necessary in order to resolve problems with the constraint definitions, without having to worry about its impact on the wiki. >This only takes a few minutes – less than 10 ms for most properties, not much more for some larger properties, and we’re still well below 10000 properties – but it was enough to cause a [visible traffic spike on the s8 master db host](https://grafana.wikimedia.org/d/000000273/mysql?panelId=5&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1071&var-port=9104&from=1561554000000&to=1561555200000) during the deployment for T223372 (around 13:17 UTC). As Wikidata keeps getting more properties, I believe this will gradually become more important.