Page MenuHomePhabricator

Experiment generating selser change assignments using permutations
Open, LowPublic

Description

  • Create selser change assignments dynamically. Currently we rely on an external file that needs to be updated manually. The generation is actually already deterministic with a seeded PRNG (seed is the test title), and the overhead of dynamic generation was around 1 second for a full 60-second test run IIRC. So drop the external file and always generate assignments on the fly.
  • Speed up selser change assignments. Currently we generate & test for duplicates. We are really generating permutations, which can be done much quicker.
  • Remember the output of failing (blacklisted) tests and fail if that output changes. We have many tests where our output is actually correct, but due to limitations in the test setup the test is still failing. This can be a difference to the PHP parser output or something like comparing to wt2wt output in selser testing which expects normalization of attribute quoting etc. By failing on changing blacklisted test output we can still catch regressions in our behavior for these tests. We'll also see improvements that are not quite enough to make the tests pass yet. Rewriting the blacklist is easy enough and documents the changes in failing test output along with the commit.

Version: unspecified
Severity: enhancement

Details

Reference
bz50316

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 1:42 AM
bzimport added a project: Parsoid.
bzimport set Reference to bz50316.

It would probably be worth fixing bug 50982 first, while you can easily see the empty selser changes in the output file. Subbu thinks these might be the tests "without wt2wt parsoid option ... would be good to atleast verify/confirm that hypothesis."

Re: comment 1. Unfortunately, that doesn't appear to be the case. A counter example is "Parsoid only: Quote balancing context should be ..." which has the options "parsoid=wt2html,wt2wt".

Change 76870 had a related patch set uploaded by Arlolra:
Generate selser change assignments dynamically.

https://gerrit.wikimedia.org/r/76870

Change 76870 merged by jenkins-bot:
Generate selser change assignments dynamically.

https://gerrit.wikimedia.org/r/76870

3rd bullet point in bug description is actually bug 51718 -- need to figure out best approach for this (work through what is best -- technique as outlined in #3 here or something else).

With generate & test our assignments are not guaranteed to be exhaustive, which might be relevant for bug 52139. It might be worth moving to direct permutation generation instead, as that should also make change generation faster.

gwicke: In what way are these permutations? From the blacklist,

add("selser", "Non-word characters don't terminate tag names (bug 17663, 40670, 52022) [[3],3,[3],3,4,3,4,4,4,2,3]");

this just looks like combinations with replacements. Given that there are 11 numbers between 2 and 4 inclusive, you'd have to generate 3^11 changes, rather than 20 random ones. That doesn't seem faster.

Deterministic generation will be faster than random generate & test, as the latter will often result in duplicates which are then filtered out. Keep in mind that we try to generate a random assignment up to 1000 times, even if there are only a handful possible permutations in a small test. The extra attempts to generate permutations will just generate duplicates once the few possible permutations have been found.

I agree that we'll need to limit the number of permutations we generate for large test cases. That means that generating all permutations with the current assignments won't be possible. On the bright side, there is a chance that we can get away with less permutations without really losing test coverage. As an example, case 2 (node insertion before current node) and case 4 (child node insertion) can result in the same actual change, so should probably be collapsed when that happens. Similarly, new node insertion is very similar to attribute changes for selser processing: the full 'outerwikitext' needs to be serialized in both cases. Lets discuss the possible cases and think about which cases need to be handled.

Change 85952 had a related patch set uploaded by Arlolra:
WIP: Remember the output of failing (blacklisted) tests

https://gerrit.wikimedia.org/r/85952

  • Bug 49222 has been marked as a duplicate of this bug. ***

Change 85952 merged by jenkins-bot:
Remember the output of failing (blacklisted) tests

https://gerrit.wikimedia.org/r/85952

Gabriel: All patches merged months ago - is there more work left here, or can you close this ticket as RESOLVED FIXED?

We have a good workable solution for now, but I think Gabriel had the enhancement idea of generating selser tests by going through permutations. Gabriel: do you want to create a different enhancement ticket for it and close this one?

(In reply to ssastry from comment #13)

We have a good workable solution for now, but I think Gabriel had the
enhancement idea of generating selser tests by going through permutations.
Gabriel: do you want to create a different enhancement ticket for it and
close this one?

Gabriel: ping?

Lets keep using this bug, but reclassify it as an enhancement.

Our selser test coverage can be improved further. Generating permutations systematically still seems to be a promising candidate solution for doing so.

Arlolra removed GWicke as the assignee of this task.Nov 25 2014, 8:08 PM
Arlolra set Security to None.
ssastry moved this task from Needs Triage to Testing on the Parsoid board.Dec 20 2014, 1:12 AM
ssastry renamed this task from Generate selser change assignments dynamically to Experiment generating selser change assignments using permutations.Feb 2 2015, 11:51 PM
ssastry lowered the priority of this task from Medium to Low.
marcoil moved this task from Testing to Needs Triage on the Parsoid board.Feb 13 2015, 12:50 PM
ssastry moved this task from Needs Triage to Testing on the Parsoid board.Dec 18 2017, 10:08 PM