The task is vague, can you give specific example scenarios or something? I'm assuming services would run as regular users and not have root privileges, and we'd be applying updates in a timely fashion to make the likelihood of root escalation low. It's really two exploits that have to be possible and exploited together in on attack, one in embedded in the other: first breaking the app code, then utilizing the (often limited) access to that account to exploit a root privilege escalation on the host.

Also, the language at the top seems to sometimes talk about the "sensitive services with weak built-in security" (what are those, and how are they weak?) as platforms for launching attacks, and then the same as targets of said attack? If so, then putting all the services together in one "isolated network environment" doesn't fix that as they're all in that pool together and one can still be used to attack the other. Are you asking for a separate "isolated network environment" for each individual service?

At some other level of decoding this, what does "isolated network environment" even mean here? Isolated from being able to route traffic to each other (reach each other for cross-service API calls) at all? Firewall rules to only allow specific service ports (which we have in puppet at the host/role level)?

The task is vague, can you give specific example scenarios or something?

Anybody controlling one of these services has at least code execution privileges as the service user. This is already sufficient to attack many other services in the production network. Given the attack surface on bare hardware, motivated attackers also have a non-zero chance for elevating their privileges to root, which would let them circumvent purely IP-based restrictions.

Are you asking for a separate "isolated network environment" for each individual service?

Yes, that would provide the best protection, and I think is warranted especially for semi-trusted services developed by third-party volunteers. But, even one extra network segment with fairly harmless stateless services would improve security by isolating those from stateful and more sensitive services in the production environment.

At some other level of decoding this, what does "isolated network environment" even mean here?

It means that the level of isolation is sufficient to prevent an owned host from circumventing network protections & attacking other services. If we want to cover local root exploits or support handing out per-service root to developers, this means that the enforcement happens at the ethernet / switch port level. A way to implement this I'm familiar with would be VLans + explicitly whitelisted routes between those VLans on an as-needed basis. This was met by ridicule on IRC earlier for unclear reasons, so I removed explicit mentions since I don't care too much whether VLans or some other tech is used.

Do we actually care whether a service is developed by volunteers or not?

Do we actually care whether a service is developed by volunteers or not?

@Krenair: I agree with you that expertise and responsiveness of maintainers as well as the amount of scrutiny a service is receiving should matter more than the maintainer's professional affiliation. I used it as an example for services that might be experimental & do not receive as much ongoing security scrutiny. Ideally, most services (both internally & externally developed) would fall into this "semi-trusted" category, but this means that we need to enforce the needed security properties at a different level.

Removing myself and adding the other Tomasz instead :-)

I've had this conversation with ops a few times, but I'll document it here for reference. I think we do need to find a way to segment our network more, to confine compromises. Starting with an "untrusted services" segment might be a good first step, although I'll leave that to ops to prioritize.

I've had a number of requests for security reviews of services where I would prefer that they were run on an isolated network. WDQS was an example-- a service where the blazegraph developers were saying it wasn't typically run exposed to the internet, but we wanted to do that. Citoid/zotero being another one I can think of. We compromised in rolling those out having them confined with firejail, although I think that still leaves us with more risk than I like.

@BBlack, addressing the multiple exploits specifically. For a serious attacker, using multiple exploits is pretty standard. Also, mediawiki completely trusts its caching mechanisms, so the ability to connect to memcache allows the attacker to escalate their mediawiki privileges.