Page MenuHomePhabricator

No test report files were found: job fails in jenkins but is shown as successful in raita
Closed, DeclinedPublic

Description

Run fails in jenkins but is show as successful in raita, the later seems to be correct.
Example:
http://raita.wmflabs.org/#projects/https%3A%2F%2Fgithub.com%2Fwmde%2FWikidataBrowserTests.git/builds/HcBqYuGqTmqTYpl-KYIktA
https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/409/console
Probably because of this:

ERROR: Publisher 'Publish JUnit test result report' failed: No test report files were found. Configuration error?

Event Timeline

JanZerebecki raised the priority of this task from to Needs Triage.
JanZerebecki updated the task description. (Show Details)
JanZerebecki added a subscriber: JanZerebecki.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 21 2015, 1:38 PM
zeljkofilipin moved this task from Ruby to CI on the Browser-Tests-Infrastructure board.
zeljkofilipin triaged this task as Normal priority.Nov 2 2015, 3:56 PM
zeljkofilipin set Security to None.

When executing the cucumber command locally, the result directory is filled correctly. Last successful run was on Aug, 6. Failing runs have no build artifacts at all, so maybe this is not junit export related at all (Since dashBoard_*.xml should be present in any case and is not in the log/junit directory). @JanZerebecki added an ls to the job which asserted that log/junit is present but empty:

12:04:23 + ls -la /mnt/jenkins-workspace/workspace/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/log/junit
12:04:23 total 8
12:04:23 drwxr-xr-x 2 jenkins-deploy wikidev 4096 Oct 30 12:04 .
12:04:23 drwxr-xr-x 3 jenkins-deploy wikidev 4096 Oct 30 12:04 ..
hashar added a subscriber: hashar.Nov 3 2015, 9:37 AM

From an IRC discussion on 2015-10-27 in Wikidata:

<Adrian_WMDE> hashar: Can you help us/me fixing https://integration.wikimedia.org/ci/view/BrowserTests/view/Wikidata/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/415/consoleFull?
<Adrian_WMDE> It currently fails with `ERROR: Publisher 'Publish JUnit test result report' failed:
              No test report files were found. Configuration error?`

<hashar> seems some Junit xml file is not generated
<hashar> Adrian_WMDE: looks like no scenario matches the cucumber tags
<hashar> cucumber is invoked with --tags @performance_testing --tags @firefox
<hashar> and since no tests are run, no JUnit result is written and the Jenkins plugin that deal with JUnit files complains and fail the build

<Adrian_WMDE> But features/performance_test.feature matches that, and the output looks to me like it is recognized
<Adrian_WMDE> Also, I just ran `bundle exec cucumber --backtrace --color --verbose --format junit --out junit --tags @performance_testing --tags @firefox` locally and it ran

The job does CUCUMBER_TAGS=@performance_testing and cucumber is executed with:

bundle exec cucumber --backtrace --color --verbose \
   --format Cucumber::Formatter::Sauce --out log/junit \
   --tags @performance_testing --tags @firefox

It apparently only match a single feature:

Features:

  • features/performance_test.feature
# Wikidata item tests
#
# License:: GNU GPL v2+
#
# testing loading time of huge entities

@chrome @firefox @internet_explorer_10 @internet_explorer_11 @local_config @performance_testing
Feature: High performance

  Background:
    Given Entity Italy defined in data/q38.json exists

  Scenario Outline: Loading a huge entity
    When I load the huge item <item>
    Then Header edit button should be there
      And Statement add button should be there

    Examples:
      | item          |
      | Italy         |

It runs just fine on my local machine. An oddity is whenever RAITA_URL is set, cucumber output is not shown. I suspect there is a background error related to Raita that is not being shown somehow :-(

hashar added a comment.Nov 3 2015, 9:43 AM

I ran the job manually directly on the slave reusing the env variables and I apparently I can not reproduce :-(

I suppose you mean you cannot reproduce the issue, right? Fyi, it's indeed only one scenario that should be run, so that's alright.

zeljkofilipin moved this task from CI to Next on the Browser-Tests-Infrastructure board.

I just used --expand in my cucumber call to get more/all the meaningful data. I don't suppose that would help here, but who knows. I'm running cucumber 2.0.0.

@zeljkofilipin is there any status update on this? To me it looks like the performance plugin for Jenkins is somewhat broken..

@Tobi_WMDE_SW sorry, I was traveling, still catching up. I will take a look probably next week.

zeljkofilipin removed zeljkofilipin as the assignee of this task.Feb 16 2016, 10:02 AM

Can this be closed?