Page MenuHomePhabricator

WebPageReplay tests gets "Error: module already implemented"
Closed, ResolvedPublic

Description

In T203060#7721160 I saw something strange: Suddenly the standard deviation for desktop tests on the bare metal server increased. I could see when it happened and thought it was because of activating a new feature but when I digged into it I saw that this has happened before and also happened for some URLs also on the tests running on the cloud servers. The problem only happened when we run tests with WebPageReplay:I could see that sometimes (not every time) we get an error on the page where we where testing that looks like "Error: module already implemented: ..."

For example if we did 11 runs against the WebPageReplay server, 2-3 of the times we got that error. Different URLs got different modules. The Obama page got "Error: module already implemented: ext.growthExperiments.SuggestedEditSession\".

Screenshot 2022-03-06 at 21.24.46.png (332×2 px, 119 KB)

The times the page got that error, the first visual change was a couple of 100 ms faster so the standard deviation become higher. I had a hard time understand why it happened, I couldn't see any pattern: it could happened the first run, the fifth etc. When I tried it locally on my machine I couldn't reproduce ... but I missed using the exact same configuration. Using the same configuration I could reproduce it so I started to try out different configurations.

The problem was the profile that was re-used between runs. It worked like this:
We start Chrome with an empty profile, access the WebPageReplay recorder and records the Wikipedia page, access the WebPageReplay replay server and measure the page. Using the same profile but clearing the cache between runs using the Chrome Devtools Protocol. However it seems it didn't clear everything. I made a fix so between every run we completely removes everything in the profile folder and it seems like the errors are gone but I will know for sure tomorrow.

I don't fully understand the error or exactly why it happened, do you follow @Krinkle ? What I feel extra strange is that it doesn't happen all the time and only for some of X runs.

Event Timeline

Change 768741 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[performance/synthetic-monitoring-tests@master] Update to Chrome 99 and cleaning Chrome profile dir.

https://gerrit.wikimedia.org/r/768741

Change 768741 merged by jenkins-bot:

[performance/synthetic-monitoring-tests@master] Update to Chrome 99 and cleaning Chrome profile dir.

https://gerrit.wikimedia.org/r/768741

Peter added a subscriber: Krinkle.