Page MenuHomePhabricator

Automated Wikidata build process
Closed, ResolvedPublic

Description

New Wikidata builds (that get deployed to beta) are currently made manually. It would be really helpful to have an automated process for this, e.g. a jenkins job that makes a Wikidata build on a daily basis.

Related Objects

Event Timeline

Tobi_WMDE_SW raised the priority of this task from to Needs Triage.
Tobi_WMDE_SW updated the task description. (Show Details)
Tobi_WMDE_SW added a project: Wikidata.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 24 2015, 12:02 PM
JanZerebecki updated the task description. (Show Details)Feb 24 2015, 1:10 PM
JanZerebecki set Security to None.

There might already be a way for jenkins jobs to get gerrit access, which is that? If not how can we add one?

aude added a comment.Feb 24 2015, 1:23 PM

what we had before, with the WikidataBuilder instance worked ok.

https://github.com/wmde/puppet-builder

it's just that the WikidataBuilder grunt tool didn't work for us anymore but the bash scripts probably can easily be modified to work with current process.

aude added a comment.Feb 24 2015, 1:24 PM

having daily jenkins jobs that do this is another option. jenkins can really do any type of task, not just run tests.

There is something that pushes to gerrit from jenkins@gallium, which is no option for Wikidata as we don't want to run composer there. Quick greping revealed nothing else.

@hashar @Krinkle @chad Is pushing the build output as a new gerrit review from a jenkins job something that would be ok? Any ideas how to best do it?

Lydia_Pintscher triaged this task as Normal priority.Mar 2 2015, 2:04 PM
Tobi_WMDE_SW moved this task from Backlog to Doing on the § Wikidata-Sprint-2015-02-25 board.

My first idea was to use the same way as what should replace ssh agent forwarding for deployment (an ssh agent that the deployer, here jenkins job user, has access to but which is running as a different user) and generate that key by puppet and register the public key manually in gerrit. But this will not work on isolated jenkins jobs where the vm is created just for one job. Now those still need some way to store the build artifacts, but if found no mention of how that will work at https://www.mediawiki.org/w/index.php?title=Continuous_integration/Architecture/Isolation .

While I would prefer using Jenkins (i.e. our CI/build infrastructure to do builds), there are still two other ways: 1) a vm running for this purpose 2) a tool in tools-lab that is regularly run via cron.

I'm going for (1) as that is probably the least work and this is only needed until deployment to beta includes running composer. (After beta supports composer, builds are only needed for deployment branches which should be done manually before a submodule update.)

Change 195567 had a related patch set uploaded (by JanZerebecki):
Wikidata builder

https://gerrit.wikimedia.org/r/195567

Alternatively to https://github.com/wmde/WikidataBuilder/issues/29 we could make the build procedure simpler.

The gerrit patch now works with WikidataBuildResources instead. https://github.com/wmde/WikidataBuilder/pull/33 officially abandons WikidataBuilder .

Change 195567 merged by Dzahn:
Wikidata builder

https://gerrit.wikimedia.org/r/195567

Tobi_WMDE_SW closed this task as Resolved.Apr 23 2015, 8:31 AM
Tobi_WMDE_SW moved this task from Backlog to Done on the Wikidata-Sprint-2015-04-21 board.