In T114421#1697181, @BGerstle-WMF wrote:

FYI, we have a Mac Mini on an internal network for building, testing, & deploying merges to master. Adding it to CI infrastructure is a possibility, but—as we've discussed previously—not trivial to maintain. That said, using our own Mac Mini does enable us to (eventually) run different kinds of tests which aren't always practical to run on Travis due to VM performance (e.g. UI tests). We could even run them on physical devices, but it's probably preferable to use a provider like Sauce Labs for that. Maintaining OS X machines is one thing, having someone maintain test devices plugged into them is an order of magnitude more labor-intensive—not to mention requiring them to be co-located with the machines.

I think there are parts of Wikimedia CI that are already using SauceLabs, e.g. for running browser tests on IE. Would that fulfill those needs?

@JanZerebecki Although I mentioned SL, I'm not sure it would be our preferred option since they only run iOS tests that are written in Appium AFAIK. Also, they only run tests on simulators, though they're working on real devices. Perhaps we could reach out to them and see if we could be early adopters of their "real devices" platform and/or run our tests using something besides Appium (e.g. Xcode 7 UI tests).

That said, maintaining a single Jenkins box that listens to GitHub/Gerrit/whatever and runs secure builds and/or integration tests isn't insurmountable. The iOS team is already investing some time maintaining it, but would appreciate help with integrating it with the rest of our infrastructure when appropriate. There are only a couple simple things we'd like help with atm:

• IRC notifications
• Email notifications
• Occasional support from office IT if we ever want to plug in physical devices

This gives us the security we want and the flexibility to choose whatever UI test solution we like best, as opposed to what another platform supports (e.g. Appium). Specifically, we're keen on trying out Xcode's new UI testing capabilities. As long as our needs remain relatively modest, this should get us pretty far.

However, using Travis for testing during code review is still beneficial because:

• It separates "secure" CI from public-facing/insecure CI
• Low maintenance
• Upgrades pretty quickly to new versions of Xcode & other tools

Some other options:

If our needs grow more sophisticated or we need to scale up our capabilities, we could evaluate some other options:

GitLab CI

As I mentioned on the RFC, GitLab CI can run on OSX and could be integrated with a GitLab repo. I'm not sure if it's something we could easily mix and match, and might still require us to maintain a setup on our own hardware, but at least the orchestration piece would be taken care of. This assessment is only based on a cursory read of their documentation, a proof of concept spike would probably provide more insight.

Travis Enterprise

Running Travis-CI (either integrated w/ our GitHub repo or triggered via npm-travis) on our own hardware could give us the security we want while still leveraging tech implemented by the Travis team. Not sure if the ROI would be significant here due to increased cost and time to configure the setup.