Page MenuHomePhabricator

Sitelink browser test sometimes fails with firefox because of rate limit?
Closed, ResolvedPublic

Description

Edit sitelinks/Remove multiple sitelinks unreliably fails on sauce and locally in firefox:

And I remove all sitelinks                   # features/step_definitions/sitelinks_steps.rb:51
  unable to locate element, using {:css=>"div[data-wb-sitelinks-group='wikipedia'] .wikibase-toolbar-container span.wikibase-toolbar-button-edit:not(.wikibase-toolbarbutton-disabled) > a"} (Watir::Exception::UnknownObjectException)
  (eval):1:in `process_watir_call'
  ./features/support/modules/sitelink_module.rb:71:in `remove_all_sitelinks'
  ./features/step_definitions/sitelinks_steps.rb:52:in `/^I remove all sitelinks$/'
  features/sitelinks.feature:204:in `And I remove all sitelinks'

Event Timeline

adrianheine raised the priority of this task from to Medium.
adrianheine updated the task description. (Show Details)
adrianheine added a subscriber: adrianheine.
adrianheine renamed this task from Sitelink browser test fails with firefox to Sitelink browser test sometimes fails with firefox.Feb 11 2016, 9:24 AM
adrianheine updated the task description. (Show Details)
adrianheine set Security to None.

This is a rate limiting problem. Whenever the test fails, I have the following in my debug log:

[ratelimit] User '0:0:0:0:0:0:0:1' (IP ::1) tripped my_wiki:limiter:edit:ip:%3A%3A1 at 8 (limit 8 in 60s)

@Tobi_WMDE_SW @Addshore @JanZerebecki Rate limiting settings for beta need to be adapted, either by IP or in general.

do we need more than saucelabs ip range for this? e.g. our office IP range if we run these also from the office?

Change 269944 had a related patch set uploaded (by Aude):
Exclude Sauce Labs IP ranges from rate limits on beta

https://gerrit.wikimedia.org/r/269944

Change 269944 merged by jenkins-bot:
Exclude Sauce Labs IP ranges from rate limits on beta

https://gerrit.wikimedia.org/r/269944

Change deployed, please verify.

The failures seem unrelated to this. It seems that the some of the articles used for testing do not really exist on the corresponding test wikis. Either we need to change the test data or ideally make the test cases more robust.

Locally this failed because of rate limiting and stopped failing after I changed config. I'll try again locally.

I checked again: This works locally.

JanZerebecki renamed this task from Sitelink browser test sometimes fails with firefox to Sitelink browser test sometimes fails with firefox because of rate limit?.May 7 2016, 8:41 PM
JanZerebecki removed a project: Patch-For-Review.

Looking at the actual screenshots on Saucelabs, e.g. https://assets.saucelabs.com/jobs/e6cd0f782ac54c04bfaa75923bff8f00/0022screenshot.png suggested that it is a problem with the corresponding article on beta Wikipedia. I've just checked that, and indeed the article "Test" was not existing on German beta Wikipedia. I've re-created it, and I assume this test failure is fixed now.
However, as I've stated before, we need to make the test more robust, e.g. by ensuring the used test data does actually exist. We could do this in an extra Given-step.

adrianheine claimed this task.

Nice catch. When I tried it locally it probably ran against enwp, not beta.