Page MenuHomePhabricator

Store CI builds output outside Jenkins (e.g. static storage)
Open, Stalled, LowPublic

Description

Openstack is already doing this. There's probably a plugin for Zuul or Jenkins that can do this.

When we activate it we will need to make sure we update the links jenkins-bot outputs from Zuul to Gerrit to point to these static files instead of the urls in Jenkins.

And when update the Jenkins configuration to store no or a very limited build history.

Details

Reference
bz51447

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 2:03 AM
bzimport set Reference to bz51447.
bzimport added a subscriber: Unknown Object (MLST).
Krinkle created this task.Jul 16 2013, 4:29 PM

Change 75637 had a related patch set uploaded by Hashar:
publish-console.py script

https://gerrit.wikimedia.org/r/75637

Change 75632 had a related patch set uploaded by Hashar:
contint: python dependency for publish-console.py

https://gerrit.wikimedia.org/r/75632

Change 74424 had a related patch set uploaded by Hashar:
experimental HTML console publishing

https://gerrit.wikimedia.org/r/74424

I have sprinted a python script to let us fetch the HTML console and publish it under /srv/org/wikimedia/integration/logs

The scheme is looking like:

/12/74312,5/mediawiki-core-regression-phpcs-HEAD-1447/console.html

12 is the last two bytes of the Gerrit change number

URL vary by patchset number with a comma, might want a sub directory though to make it easier to browse all patchsets of a change.

The directory can later host debug log files and tests results. We can do that with Jenkins artifact publishing.

Change 75632 merged by Dzahn:
contint: python dependency for publish-console.py

https://gerrit.wikimedia.org/r/75632

I compressed the old console log via gzip (Jenkins supports reading compressed logs). That give out some space.

I am busy with release/QA things right now, so unassigning bug from myself.

Change 74424 abandoned by Hashar:
experimental HTML console publishing

Reason:
no use for now.

https://gerrit.wikimedia.org/r/74424

Change 75637 abandoned by Hashar:
publish-console.py script

Reason:
no use for now.

https://gerrit.wikimedia.org/r/75637

Zuul upstream now supports passing Swift related informations to the jobs: http://ci.openstack.org/zuul/zuul.html#swift

Needs a wrapper / post build task in the jobs though.

demon removed a subscriber: demon.Dec 16 2014, 6:05 PM
Krinkle renamed this task from Jenkins: Store build output in a static storage and wipe from Jenkins build history to Store Jenkins build output outside Jenkins (e.g. static storage).Apr 10 2015, 12:47 PM
Krinkle lowered the priority of this task from Medium to Low.
Krinkle moved this task from Untriaged to Backlog on the Continuous-Integration-Infrastructure board.
Krinkle updated the task description. (Show Details)
Krinkle set Security to None.
Krinkle removed a subscriber: Unknown Object (MLST).

OpenStack does do this, but they're working on phasing out Jenkins entirely in favour of letting Zuul communicate with Gearman directly.

It may be interesting to wait until they've finished that before we invest in building a solution to store Jenkins output, integrated with JJB etc.

Potentially, we can already write a wrapper that would send the log to a central storage. OpenStack uses swift with the settings being held directly in Zuul configuration ( http://ci.openstack.org/zuul/zuul.html#swift ).

They have a script to fetch the Jenkins job console output. There is then two macros in their JJB configuration: zuul-swift-upload-console-log and zuul-swift-upload-logs-with-console which are run in a postbuildscripts regardless of the build status. Ultimately that invokes zuul_swift_upload.py which crafts a basic web layout and push the changes to swift based on SWIFT_* env variable that have been passed by Zuul to the Jenkins job.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 9 2015, 1:21 PM
hashar changed the task status from Open to Stalled.Nov 9 2015, 1:21 PM
Krinkle removed a subscriber: Krinkle.Mar 27 2017, 10:32 PM
hashar renamed this task from Store Jenkins build output outside Jenkins (e.g. static storage) to Store CI builds output outside Jenkins (e.g. static storage).Feb 5 2019, 10:43 AM
hashar updated the task description. (Show Details)

Zuul has support to reach a Swift cluster and establish everything that is needed to then be able to push files. We would need a container in Swift and write some code to push artifacts there.

To be determined is whether we should reuse the current Swift cluster used for SRE-swift-storage or whether we need a dedicated cluster. After all we don't need much redundancy.

Another way is to just use rsync but we would have to deal with the garbage collection and the trouble of scaling the storage.