• I have locally benchmarked the linter code, looping up to 1000 - 100000 rows the logic of checking is not the issue here.
• I tried to limit the loops to from 1000 to 5 rows and exiting ms gains in margin of error.
• Still as shown from past month p99 https://grafana.wikimedia.org/d/000000048/parsoid-timing-wt2html?orgId=1&from=now-1M&to=now&viewPanel=44&kiosk&refresh=1m and git history it is the reason.
• The issue as I see it here is the amount of table data and tables being processed since introducing the rule, the number of database read/writes and the size of the data introduced.

I would suggest:

• Disabling the linting rule suspected large-tables on next train to make sure it is the cause.
• As data size and query numbers are suspected if no p99 timing improvements seen we might reenable it and check disabling missing-end-tag-in-heading.
• Once we are sure which rule is the cause we can modify the rule parameters.
• If non of these rules are the cause we need a check in parsing timing, as it might not be a linting performance issue

@Sbailey please advise

Change 915788 had a related patch set uploaded (by Mabualruz; author: Mabualruz):

[mediawiki/services/parsoid@master] Linter: P99 performance of Linting went from 1-2s to 8-11 seconds

https://gerrit.wikimedia.org/r/915788

Change 927783 had a related patch set uploaded (by Mabualruz; author: Mabualruz):

[mediawiki/services/parsoid@master] Linter: Stop calling Large table linting rule

https://gerrit.wikimedia.org/r/927783

Change 927783 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Linter: Stop calling Large table linting rule

https://gerrit.wikimedia.org/r/927783

Change 929160 had a related patch set uploaded (by Isabelle Hurbain-Palatin; author: Isabelle Hurbain-Palatin):

[mediawiki/vendor@master] Bump parsoid to 0.18.0-a14

https://gerrit.wikimedia.org/r/929160

Change 929160 merged by jenkins-bot:

[mediawiki/vendor@master] Bump parsoid to 0.18.0-a14

https://gerrit.wikimedia.org/r/929160

I am not sure if @Sbailey responded elsewhere, but my comments on this below:

In T336528#8912411, @Mabualruz wrote:

..

• The issue as I see it here is the amount of table data and tables being processed since introducing the rule, the number of database read/writes and the size of the data introduced.

Note that the linting perf numbers reported in the grafana charts are *only* Parsoid numbers where there is no database access / read / write issues involved. It is only the linter dom analysis performance numbers. Wikis do have some pages with really large tables ... editors / bot authors sometimes do tend to use wiki pages as "databases" in a way and hence can be really massive.

I would suggest:

• Disabling the linting rule suspected large-tables on next train to make sure it is the cause.

Looks like you proceeded with the approach, but I would be really really surprised if this is not the case. I think I had run local linting analysis numbers before and flagged the loop as a potential perf issue then, so my money is still on this.

• As data size and query numbers are suspected if no p99 timing improvements seen we might reenable it and check disabling missing-end-tag-in-heading.

Is there a reason you believe there is a problem with that linting code in Parsoid? We can go look at the code, but while reviewing your large-tables linter patch, I had run profiling locally using the bin/parse.php --profiling option and had seen linting latency double or worse on some large tables I had created locally. So, my strong hunch is still that it is the loop used for large tabes.

@Mabualruz https://grafana.wikimedia.org/d/000000048/parsoid-timing-wt2html?orgId=1&refresh=1m&viewPanel=44&from=now-24h&to=now shows that the linting p99 time went back to old levels.

@ssastry That is a good sign for the new patch we have a baseline to measure reenabling it with the new code

Change 937146 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/services/parsoid@master] Linter: Performance Large Table Optimisation

https://gerrit.wikimedia.org/r/937146

Change 915788 abandoned by Mabualruz:

[mediawiki/services/parsoid@master] Linter: Performance Large Table Optimisation

Reason:

https://gerrit.wikimedia.org/r/915788

Change 937146 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Linter: Fix performance of large-tables lint and re-enable it

https://gerrit.wikimedia.org/r/937146

Change 946615 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/vendor@master] Bump parsoid to 0.18.0-a21

https://gerrit.wikimedia.org/r/946615

Change 946615 merged by jenkins-bot:

[mediawiki/vendor@master] Bump parsoid to 0.18.0-a21

https://gerrit.wikimedia.org/r/946615