It looks like generally the loading took around 2 seconds, EditFilterHookRunner took around 11 seconds and the actual saveEntity took 10 seconds.

Wikibase\EntityContent::collectValues doesn't look like it is that big of an issue, and it contributes 3 of the EditFilterHookRunner seconds.

I think there is an improvement to be made in Wikibase\EntityContent::collectValues.
it currently has a hard coded list of keys to ignore:

		//XXX: $ignore contains knowledge about the Entity's internal representation.
		//     This list should therefore rather be maintained in the Entity class.
		static $ignore = [
			'language',
			'site',
			'type',
		];

This should be provided by each entity type as they can be different.
For regular items and properties there are keys missing that should be added.
For lexeme etc there are other keys that possible need to be added / thought about that have no way of getting into that list.

Ignoring more keys here will result in less data for AbuseFilter to then run through which would increase performance...
Right now for example the AbuseFilter will get handed all of the reference hashes and snak types etc.

After some experimenting I managed to add a bunch of keys to the ignore which are probably fine to ignore:

$ignore = array(
	'language',
	'site',
	'type',
	'hash',
	'id',
	'rank',
	'entity-type',
	'datatype',
	'snaktype',
	'qualifiers-order',
	'property',
);

For Q56596767 on wikidata.org running collectValues:
Before:
Total execution time: 1.4714670181274
Number of strings: 30426
After:
Total execution time: 0.12493205070496
Number of strings: 4673

I have no doubt that this will have knock on effects in Abusefilter, probably reducing the run time by ~60%?

/me should file some tickets.....

Change 461697 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Improve the performance of EntityContent::collectValues

https://gerrit.wikimedia.org/r/461697

Change 461755 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Track entity save times in detail in statsd

https://gerrit.wikimedia.org/r/461755

Change 473790 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] EditEntity, Track EditFilterHook run times in statsd

https://gerrit.wikimedia.org/r/473790

Change 473791 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] EditEntity, Track EntityStore save times in statsd

https://gerrit.wikimedia.org/r/473791

Change 473802 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] EntityContent::collectValues, add some basic test coverage

https://gerrit.wikimedia.org/r/473802

For reviewers, i made a topic with all of the commits in:

• https://gerrit.wikimedia.org/r/#/q/topic:%22EntityContent::collectValues%22+(status:open+OR+status:merged)
• The end of the chain is https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/461697/, but the rest of the chain needs to be merged and on the way to deployment before we merge this one.

Change 461755 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] EditEntity, Track entity save times in statsd

https://gerrit.wikimedia.org/r/461755

Change 473790 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] EditEntity, Track EditFilterHook run times in statsd

https://gerrit.wikimedia.org/r/473790

Change 473791 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] EditEntity, Track EntityStore save times in statsd

https://gerrit.wikimedia.org/r/473791

Change 473802 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] EntityContent::collectValues, add some basic test coverage

https://gerrit.wikimedia.org/r/473802

• The end of the chain is https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/461697/, but the rest of the chain needs to be merged and on the way to deployment before we merge this one.

The rest of the chain has been deployed with wmf.6, so I tried putting the data into a Grafana dashboard: Wikidata EditEntity.

If my interpretation is correct, then the “Item” panel tells us that the total time of EditEntity.attemptSave is almost entirely taken up by EditFilterHookRunner.run and EntityStore.saveEntity, with everything else being almost negligible; and the “Item, store vs. hooks” panel shows more clearly that the majority of that time, somewhere around 60%, is taken up by EditFilterHookRunner.run, not EntityStore.saveEntity.

When we merge and deploy the end of the chain, we should see that ratio drop drastically, since EditFilterHookRunner.run should now be much more efficient, and the overall timings in the “Item” panel should decrease accordingly, and now probably be dominated by EntityStore.saveEntity alone.

Does that sound right?

Looking forward to the results :)

Change 461697 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] ArrayValueCollector, Improve the performance

https://gerrit.wikimedia.org/r/461697

As far as I can tell, all the changes relevant to this task are merged now (there are still some open for the subtask T205252). Moving to the verification column – hopefully we should see the Grafana stats improve once this goes out with the train.

But because there’s no train this week (releng offsite), that would be next week, on December 12th. Do we wait until then or do we want to backport the “improve performance” patch?