Run fails in jenkins but is show as successful in raita, the later seems to be correct.
Example:
http://raita.wmflabs.org/#projects/https%3A%2F%2Fgithub.com%2Fwmde%2FWikidataBrowserTests.git/builds/HcBqYuGqTmqTYpl-KYIktA
https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/409/console
Probably because of this:
ERROR: Publisher 'Publish JUnit test result report' failed: No test report files were found. Configuration error?
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T108946 [Epic] Improve the development infrastructure | |||
| Resolved | zeljkofilipin | T94150 Fix scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily | |||
| Duplicate | None | T105985 Run Wikidata browser tests on testwikidata via Jenkins | |||
| Resolved | • Jonas | T88541 [DO NOT USE] Wikidata Browsertests (tracking) [superseded by #Browser-Tests] | |||
| Resolved | Tobi_WMDE_SW | T101497 [Story] Run browsertests regularly on test.wikidata.org via Jenkins | |||
| Resolved | JanZerebecki | T101498 [Task] Create a user with necessary privilegs for browsertests on test.wikidata.org | |||
| Resolved | Tobi_WMDE_SW | T101499 [Task] Setup a Jenkins job to run Wikidata browsertests on test.wikidata.org | |||
| Resolved | Tobi_WMDE_SW | T92619 make Wikidata browsertests non-flaky (tracking) | |||
| Declined | None | T116164 No test report files were found: job fails in jenkins but is shown as successful in raita |
When executing the cucumber command locally, the result directory is filled correctly. Last successful run was on Aug, 6. Failing runs have no build artifacts at all, so maybe this is not junit export related at all (Since dashBoard_*.xml should be present in any case and is not in the log/junit directory). @JanZerebecki added an ls to the job which asserted that log/junit is present but empty:
12:04:23 + ls -la /mnt/jenkins-workspace/workspace/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/log/junit 12:04:23 total 8 12:04:23 drwxr-xr-x 2 jenkins-deploy wikidev 4096 Oct 30 12:04 . 12:04:23 drwxr-xr-x 3 jenkins-deploy wikidev 4096 Oct 30 12:04 ..
From an IRC discussion on 2015-10-27 in Wikidata:
<Adrian_WMDE> hashar: Can you help us/me fixing https://integration.wikimedia.org/ci/view/BrowserTests/view/Wikidata/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/415/consoleFull? <Adrian_WMDE> It currently fails with `ERROR: Publisher 'Publish JUnit test result report' failed: No test report files were found. Configuration error?` <hashar> seems some Junit xml file is not generated <hashar> Adrian_WMDE: looks like no scenario matches the cucumber tags <hashar> cucumber is invoked with --tags @performance_testing --tags @firefox <hashar> and since no tests are run, no JUnit result is written and the Jenkins plugin that deal with JUnit files complains and fail the build <Adrian_WMDE> But features/performance_test.feature matches that, and the output looks to me like it is recognized <Adrian_WMDE> Also, I just ran `bundle exec cucumber --backtrace --color --verbose --format junit --out junit --tags @performance_testing --tags @firefox` locally and it ran
The job does CUCUMBER_TAGS=@performance_testing and cucumber is executed with:
bundle exec cucumber --backtrace --color --verbose \ --format Cucumber::Formatter::Sauce --out log/junit \ --tags @performance_testing --tags @firefox
It apparently only match a single feature:
Features:
- features/performance_test.feature
# Wikidata item tests # # License:: GNU GPL v2+ # # testing loading time of huge entities @chrome @firefox @internet_explorer_10 @internet_explorer_11 @local_config @performance_testing Feature: High performance Background: Given Entity Italy defined in data/q38.json exists Scenario Outline: Loading a huge entity When I load the huge item <item> Then Header edit button should be there And Statement add button should be there Examples: | item | | Italy |
It runs just fine on my local machine. An oddity is whenever RAITA_URL is set, cucumber output is not shown. I suspect there is a background error related to Raita that is not being shown somehow :-(
I ran the job manually directly on the slave reusing the env variables and I apparently I can not reproduce :-(
I suppose you mean you cannot reproduce the issue, right? Fyi, it's indeed only one scenario that should be run, so that's alright.
I just used --expand in my cucumber call to get more/all the meaningful data. I don't suppose that would help here, but who knows. I'm running cucumber 2.0.0.
@zeljkofilipin is there any status update on this? To me it looks like the performance plugin for Jenkins is somewhat broken..
@Tobi_WMDE_SW sorry, I was traveling, still catching up. I will take a look probably next week.