Page MenuHomePhabricator

Run language screenshots script for VisualEditor in Jenkins
Closed, ResolvedPublic8 Story Points

Description

We used to have a matrix jenkins job that would run the script for multiple languages. The job was deleted because it was broken. It should be recreated. Existing Ruby script and credentials (User:LanguageScreenshotBot) for uploading to Commons could be reused. The uploading script could be rewritten in JavaScript later.

Relevant pages:

Details

Related Gerrit Patches:
integration/config : masterRun language screenshots script for VisualEditor in Jenkins
mediawiki/extensions/VisualEditor : masterIncrease Mocha and Selenium timeout for language screenshots to 40 s
mediawiki/extensions/VisualEditor : masterWIP Run language screenshots for 10 languages
mediawiki/extensions/VisualEditor : masterWIP Run grunt and upload separately for two languages
mediawiki/extensions/VisualEditor : masterWIP Test not failing the script on Mocha failure
mediawiki/extensions/VisualEditor : masterFix failing Selenium tests
mediawiki/extensions/VisualEditor : masterThe latest version of commons_upload Ruby gem is needed for language screenshots
mediawiki/extensions/VisualEditor : masterFix language screenshots timeout error
mediawiki/extensions/VisualEditor : masterWIP Dots separate language code in language screenshot file name
mediawiki/extensions/VisualEditor : masterWIP Run all language screenshots script for VisualEditor in Jenkins
mediawiki/extensions/VisualEditor : masterWIP Use languages from build/screenshots.yml instead of build/screenshotLangs.json
mediawiki/extensions/VisualEditor : masterRun language screenshots from Jenkins
mediawiki/extensions/VisualEditor : masterWIP use chromedriver from npm package

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 7 2016, 2:05 PM
zeljkofilipin renamed this task from Run language screenshots for VisualEditor in Jenkins to Run language screenshots script for VisualEditor in Jenkins.Jul 7 2016, 2:06 PM
zeljkofilipin moved this task from Backlog 🔙 to Next 🔜 on the User-zeljkofilipin board.
zeljkofilipin moved this task from Next 🔜 to In Progress 🔨 on the User-zeljkofilipin board.
zeljkofilipin edited subscribers, added: Esanders, Amire80, Elitre; removed: edsu.
zeljkofilipin updated the task description. (Show Details)Jul 7 2016, 2:16 PM
greg added a subscriber: greg.Jul 7 2016, 3:59 PM

Change 298013 had a related patch set uploaded (by Zfilipin):
WIP use chromedriver from npm package

https://gerrit.wikimedia.org/r/298013

zeljkofilipin added a comment.EditedJul 9 2016, 10:34 AM

Running the job at ci-jessie-wikimedia. Unfortunatelly, looks like there is a problem, or the number of instances is really low. :(

Parent job:

https://integration.wikimedia.org/ci/view/Selenium/job/selenium-VisualEditor-T139613/12/console

10:04:39 Triggering selenium-VisualEditor-T139613 » firefox,beta,Linux,ci-jessie-wikimedia

Child job (at 10:33):

https://integration.wikimedia.org/ci/view/Selenium/job/selenium-VisualEditor-T139613/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=ci-jessie-wikimedia/

#1​2 (pending—All nodes of label ‘ci-jessie-wikimedia’ are offline)

Change 298013 abandoned by Zfilipin:
WIP use chromedriver from npm package

Reason:
Looks like chromedriver is already present at ci-jessie-wikimedia nodes, so there is no need for this.

00:00:17.027 /usr/lib/chromium/chromedriver -v
00:00:17.093 ChromeDriver 2.21 (undefined)

https://gerrit.wikimedia.org/r/298013

I am doing something wrong. :(

...
Xvfb :94 &
export DISPLAY=:94
/usr/lib/chromium/chromium https://www.wikipedia.org/
...

causes (selenium-VisualEditor-T139613...7)

...
00:00:56.105 + export DISPLAY=:94
00:00:56.106 + DISPLAY=:94
00:00:56.106 + /usr/lib/chromium/chromium https://www.wikipedia.org/
00:00:56.106 + Xvfb :94
00:00:56.108 (EE) 
00:00:56.108 Fatal server error:
00:00:56.109 (EE) Server is already active for display 94
00:00:56.109 	If this server is no longer running, remove /tmp/.X94-lock
00:00:56.109 	and start again.
00:00:56.110 (EE) 
00:00:56.309 Xlib:  extension "RANDR" missing on display ":94".
00:00:56.583 [1273:1281:0715/132057:ERROR:bus.cc(432)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
00:00:56.685 Xlib:  extension "RANDR" missing on display ":94".
00:00:57.132 [1310:1310:0715/132058:ERROR:gl_surface_glx.cc(381)] glxQueryVersion failed
00:00:57.132 [1310:1310:0715/132058:ERROR:gl_surface_x11.cc(62)] GLSurfaceGLX::InitializeOneOff failed.
00:00:57.146 [1310:1310:0715/132058:ERROR:gpu_child_thread.cc(397)] Exiting GPU process due to errors during initialization
00:00:57.239 [1273:1292:0715/132058:ERROR:browser_gpu_channel_host_factory.cc(119)] Failed to launch GPU process.
00:02:54.628 Build was aborted
00:02:54.628 Aborted by Zeljko Filipin
00:02:54.727 Finished: ABORTED

If I remove Xvfb :94 & and leave only

...
export DISPLAY=:94
/usr/lib/chromium/chromium https://www.wikipedia.org/
...

I get (selenium-VisualEditor-T139613...8)

...
00:00:12.201 + export DISPLAY=:94
00:00:12.201 + DISPLAY=:94
00:00:12.201 + /usr/lib/chromium/chromium https://www.wikipedia.org/
00:00:12.455 Xlib:  extension "RANDR" missing on display ":94".
00:00:12.632 [1267:1275:0715/132310:ERROR:bus.cc(432)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
00:00:12.712 Xlib:  extension "RANDR" missing on display ":94".
00:00:12.906 [1303:1303:0715/132310:ERROR:gl_surface_glx.cc(381)] glxQueryVersion failed
00:00:12.906 [1303:1303:0715/132310:ERROR:gl_surface_x11.cc(62)] GLSurfaceGLX::InitializeOneOff failed.
00:00:12.932 [1303:1303:0715/132310:ERROR:gpu_child_thread.cc(397)] Exiting GPU process due to errors during initialization
00:00:13.072 [1267:1286:0715/132311:ERROR:browser_gpu_channel_host_factory.cc(119)] Failed to launch GPU process.
00:00:13.081 [1267:1286:0715/132311:ERROR:browser_gpu_channel_host_factory.cc(119)] Failed to launch GPU process.
00:00:38.424 Build was aborted
00:00:38.424 Aborted by Zeljko Filipin
00:00:38.522 Finished: ABORTED

Change 299159 had a related patch set uploaded (by Zfilipin):
WIP Run language screenshots from Jenkins

https://gerrit.wikimedia.org/r/299159

Gosh, awesome work. :-)

Problems installing ruby gems on ci-jessie-wikimedia instance (cc @hashar), details at selenium-VisualEditor-299159...8/console

00:00:12.134 + bundle install --verbose
...
00:00:16.926 We trust you have received the usual lecture from the local System
00:00:16.926 Administrator. It usually boils down to these three things:
00:00:16.926 
00:00:16.926     #1) Respect the privacy of others.
00:00:16.926     #2) Think before you type.
00:00:16.926     #3) With great power comes great responsibility.
00:00:16.926 
00:00:16.927 sudo: no tty present and no askpass program specified
00:00:16.930 
00:00:16.930 Bundler::GemspecError: Could not read gem at /var/lib/gems/2.1.0/cache/rake-10.4.2.gem. It may be corrupted.
00:00:16.930 /usr/lib/ruby/vendor_ruby/bundler/rubygems_integration.rb:177:in `rescue in spec_from_gem'
00:00:16.930 /usr/lib/ruby/vendor_ruby/bundler/rubygems_integration.rb:174:in `spec_from_gem'
00:00:16.930 /usr/lib/ruby/vendor_ruby/bundler/source/rubygems.rb:92:in `install'
00:00:16.930 /usr/lib/ruby/vendor_ruby/bundler/installer.rb:104:in `block in install_gem_from_spec'
00:00:16.930 /usr/lib/ruby/vendor_ruby/bundler/rubygems_integration.rb:162:in `with_build_args'
00:00:16.931 /usr/lib/ruby/vendor_ruby/bundler/installer.rb:103:in `install_gem_from_spec'
00:00:16.931 /usr/lib/ruby/vendor_ruby/bundler/installer.rb:264:in `block in install_sequentially'
00:00:16.931 /usr/lib/ruby/2.1.0/forwardable.rb:183:in `each'
00:00:16.931 /usr/lib/ruby/2.1.0/forwardable.rb:183:in `each'
00:00:16.931 /usr/lib/ruby/vendor_ruby/bundler/installer.rb:263:in `install_sequentially'
00:00:16.931 /usr/lib/ruby/vendor_ruby/bundler/installer.rb:90:in `run'
00:00:16.931 /usr/lib/ruby/vendor_ruby/bundler/installer.rb:18:in `install'
00:00:16.931 /usr/lib/ruby/vendor_ruby/bundler/cli/install.rb:79:in `run'
00:00:16.932 /usr/lib/ruby/vendor_ruby/bundler/cli.rb:145:in `install'
00:00:16.932 /usr/lib/ruby/vendor_ruby/thor/command.rb:27:in `run'
00:00:16.932 /usr/lib/ruby/vendor_ruby/thor/invocation.rb:126:in `invoke_command'
00:00:16.932 /usr/lib/ruby/vendor_ruby/thor.rb:359:in `dispatch'
00:00:16.932 /usr/lib/ruby/vendor_ruby/thor/base.rb:440:in `start'
00:00:16.932 /usr/lib/ruby/vendor_ruby/bundler/cli.rb:9:in `start'
00:00:16.932 /usr/bin/bundle:20:in `block in <main>'
00:00:16.933 /usr/lib/ruby/vendor_ruby/bundler/friendly_errors.rb:5:in `with_friendly_errors'
00:00:16.933 /usr/bin/bundle:18:in `<main>'
00:00:16.939 An error occurred while installing rake (10.4.2), and Bundler cannot continue.
00:00:16.939 Make sure that `gem install rake -v '10.4.2'` succeeds before bundling.
00:00:16.976 Build step 'Execute shell' marked build as failure
00:00:17.155 Finished: FAILURE

bundler invokes gem which, on Debian, default to /usr/ which is not writable.

What we did on Nodepool instance is to set BUNDLE_PATH to have gems installed under the jenkins homedir, that is then saved/restored by the package manager cache system that load/save for builds (castor).

In integration/config zuul/parameter_functions.py the Nodepool jobs have:

params['BUNDLE_PATH'] = '/home/jenkins/workspace/vendor/bundle'

Make your job template to use BUNDLE_PATH='/home/jenkins/workspace/vendor/bundle' and you will be fine.

Bonus point for using the builder castor-load and publisher castor-save. That speeds up the build by loading dependencies from the central cache.

zeljkofilipin added a comment.EditedJul 19 2016, 4:35 PM

Thanks @hashar, Ruby part is now working fine.

The first job that managed to upload images to commons.wikimedia.beta.wmflabs.org is selenium-VisualEditor-299159...16.

Example upload is VisualEditor_references_list-en.png.

Files uploaded to commons.wikimedia.beta.wmflabs.org by Selenium user.

Change 300035 had a related patch set uploaded (by Zfilipin):
WIP Run language screenshots script for VisualEditor in Jenkins

https://gerrit.wikimedia.org/r/300035

The first job created using Jenkins Job Builder: language-screenshots-VisualEditor

Looks great! Well done, can polish it up after vacations :}

Change 300882 had a related patch set uploaded (by Zfilipin):
WIP Run all language screenshots script for VisualEditor in Jenkins

https://gerrit.wikimedia.org/r/300882

Change 300882 abandoned by Zfilipin:
WIP Run all language screenshots script for VisualEditor in Jenkins

https://gerrit.wikimedia.org/r/300882

zeljkofilipin removed zeljkofilipin as the assignee of this task.Jul 26 2016, 11:08 AM
zeljkofilipin moved this task from In Progress 🔨 to Next 🔜 on the User-zeljkofilipin board.
Jdforrester-WMF set the point value for this task to 8.Aug 12 2016, 8:24 PM

Change 305966 had a related patch set uploaded (by Zfilipin):
WIP Fix Error: timeout of 20000ms exceeded

https://gerrit.wikimedia.org/r/305966

Change 305979 had a related patch set uploaded (by Zfilipin):
WIP Dots separate language code in language screenshot file name

https://gerrit.wikimedia.org/r/305979

Change 305979 abandoned by Zfilipin:
WIP Dots separate language code in language screenshot file name

Reason:
This change would not be backwards compatible with existing file names.

https://commons.wikimedia.org/wiki/File:VisualEditor_Insert_Menu-en.png

https://gerrit.wikimedia.org/r/305979

Change 305966 merged by jenkins-bot:
Fix language screenshots timeout error

https://gerrit.wikimedia.org/r/305966

Change 306455 had a related patch set uploaded (by Zfilipin):
The latest version of commons_upload Ruby gem is needed for language screenshots

https://gerrit.wikimedia.org/r/306455

Change 306455 merged by jenkins-bot:
The latest version of commons_upload Ruby gem is needed for language screenshots

https://gerrit.wikimedia.org/r/306455

Change 306459 had a related patch set uploaded (by Zfilipin):
WIP Fix failing Selenium tests

https://gerrit.wikimedia.org/r/306459

Change 306459 merged by jenkins-bot:
Fix failing Selenium tests

https://gerrit.wikimedia.org/r/306459

Change 306908 had a related patch set uploaded (by Zfilipin):
WIP Run language screenshots for 10 languages

https://gerrit.wikimedia.org/r/306908

Change 307280 had a related patch set uploaded (by Zfilipin):
WIP Run language screenshots script for VisualEditor in Jenkins

https://gerrit.wikimedia.org/r/307280

Change 307313 had a related patch set uploaded (by Zfilipin):
WIP Test not failing the script on Mocha failure

https://gerrit.wikimedia.org/r/307313

Change 307313 abandoned by Zfilipin:
WIP Test not failing the script on Mocha failure

https://gerrit.wikimedia.org/r/307313

Change 307280 abandoned by Zfilipin:
WIP Run grunt and upload separately for two languages

https://gerrit.wikimedia.org/r/307280

Change 306908 abandoned by Zfilipin:
WIP Run language screenshots for 10 languages

https://gerrit.wikimedia.org/r/306908

Change 307318 had a related patch set uploaded (by Zfilipin):
WIP Increase Mocha and Selenium timeout to 40 seconds

https://gerrit.wikimedia.org/r/307318

Change 307318 merged by jenkins-bot:
Increase Mocha and Selenium timeout for language screenshots to 40 s

https://gerrit.wikimedia.org/r/307318

zeljkofilipin closed this task as Resolved.Sep 2 2016, 9:53 AM

Change 300035 merged by jenkins-bot:
Run language screenshots script for VisualEditor in Jenkins

https://gerrit.wikimedia.org/r/300035