"Real" service discovery is something that we should get from a PaaS layer or other orchestration system for containers. In reality all that will do however is manage some well known service names via a DNS proxy specific to each deployment. Adding a side car container to act as a transparent proxy for every external service needed is not a very scalable pattern.

At an application level, deployment specific configuration in the form of environment variables (12-factor style) or a config file should be used even if there is a service discovery layer.

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

In T151002#9188509, @dcaro wrote:

We have a static floating name redis.svc.tools.eqiad1.wikimedia.cloud for the shared redis, I think that should address the concerns here […]

Resolving it but feel free to reopen if that does not address the issue.

tools-redis existed previously, which seems equally static and equally Toolforge-specific. I might be missing something, but it seems all that happened is that there is now another alias called redis.svc.tools.eqiad1.wikimedia.cloud for the same thing. That doesn't make any difference to individual tools or the issue in this task, I think? It's not more static or more discoverable than the previous name. I don't mind the rename, but I don't see how it relates to this task.

I've worked around this since 2016 and maybe I'm the only one trying to develop tools locally with minimal changes. Indeed I can invert the issue and instead hardcode the Toolforge production value in the tool, and use ENV for local dev. This assumes the name will never change of course. But I guess we use DNS as an API within Toolforge in the same way as one would e.g the name of an ENV variable. Although I must say, the canonical life span of DNS service names appears significantly shorter than the life span of the services they represent (e.g. the hypotheetical ENV key and config varialbe name). The labs replicas for example, have changed 3+ times so far, and now tools-redis has a new preferred service name as well.

The cost of these isn't high to me, but it's a small itch I reported in 2016. Feel free to re-open if the team is interested in providing ENV-based discovery built-in to the Toolforge platform, or otherwise close as declined.

In T151002#9188686, @Krinkle wrote:

In T151002#9188509, @dcaro wrote:

We have a static floating name redis.svc.tools.eqiad1.wikimedia.cloud for the shared redis, I think that should address the concerns here […]

Resolving it but feel free to reopen if that does not address the issue.

tools-redis existed previously, which seems equally static and equally Toolforge-specific. I might be missing something, but it seems all that happened is that there is now another alias called redis.svc.tools.eqiad1.wikimedia.cloud for the same thing. That doesn't make any difference to individual tools or the issue in this task, I think? It's not more static or more discoverable than the previous name. I don't mind the rename, but I don't see how it relates to this task.

I've worked around this since 2016 and maybe I'm the only one trying to develop tools locally with minimal changes. Indeed I can invert the issue and instead hardcode the Toolforge production value in the tool, and use ENV for local dev. This assumes the name will never change of course. But I guess we use DNS as an API within Toolforge in the same way as one would e.g the name of an ENV variable. Although I must say, the canonical life span of DNS service names appears significantly shorter than the life span of the services they represent (e.g. the hypotheetical ENV key and config varialbe name). The labs replicas for example, have changed 3+ times so far, and now tools-redis has a new preferred service name as well.

The cost of these isn't high to me, but it's a small itch I reported in 2016. Feel free to re-open if the team is interested in providing ENV-based discovery built-in to the Toolforge platform, or otherwise close as declined.

I think setting it in an ENV variable is a good solution, we are currently setting the replicas and toolsdb user and password like that (since last week).

I think it would be good to set the service names/ips that way too, so your app does not depend on hardcoded values (as you say), I'll create a subtask to add that to the envvars service :)

But until then, it'd be good to be able to adopt services like Redis in a tool while still being able to easily run them locally (without having to set up a hostname called "tools-redis" on my local machine etc.)

This sounds like an application specific configuration issue to me really. I write lots of tools myself that use some form of https://12factor.net/config to configure things via environment variables. I tend to setup this config such that the defaults are what is needed for running on Toolforge, but CI and local testing can set envvars to override the defaults.

These things end up looking slightly different in based on implementation language, but an example from a recent Python project can be seen at https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/blob/main/src/glaab/settings.py. When running tests, I inject dummy variables for required envvar secrets. When testing more completely locally I can create a .env file that is read to populate the environment. When running on Toolforge the toolforge envvars system can be used to configure the code when running under the jobs framework or as a webservice.

I worry that if we try to expose some set of known services with globally provisioned envvars this will actually only be of use to a small number of tools that were explicitly written to look for a particular envvar name and content format. We have a wiki to document services and their connection information. It feels like a more scalable solution to rely on that documentation as a source of truth for tool maintainers and let them figure out how to configure their applications as a result.

I worry that if we try to expose some set of known services with globally provisioned envvars this will actually only be of use to a small number of tools that were explicitly written to look for a particular envvar name and content format

Can you elaborate?

I think that a very interesting use-case would be for us to be able to change the redis url/port without having to rely on the tool maintainer by just changing the envvar and restarting the service. It also paves the path to a very well supported and best practice pattern that we want to promote (using envvars for config).

I'm not sure I see a downside (worst case scenario, we end up in the same situation we have, best case scenario, the tools that use the envvars will not need maintainer intervention when/if we change the host or port of the internal services we provide).

In T151002#9584308, @dcaro wrote:

I worry that if we try to expose some set of known services with globally provisioned envvars this will actually only be of use to a small number of tools that were explicitly written to look for a particular envvar name and content format

Can you elaborate?

Sure. My point is that there is no universal standard for configuration variables or connection strings. There are various language/framework specific pseudo standards, often based around the semantics of an URL. The TOOL_REDIS_URI="redis://redis.svc.tools.eqiad1.wikimedia.cloud:6379" setting you theorized in T347141: [envvars,maintain-kubeusers] create and populate envvars for common service names is a URL example. As far as I can tell the "official" python redis client does not support that type of connection declaration which would mean additional custom parsing logic would be needed to use the envvar. This is not horrible to imagine, but it is also difficult for me to see the value vs publishing the service name and well known port on wiki as we do today.

I think that a very interesting use-case would be for us to be able to change the redis url/port without having to rely on the tool maintainer by just changing the envvar and restarting the service. It also paves the path to a very well supported and best practice pattern that we want to promote (using envvars for config).

I think this hypothetical highlights exactly what I'm worried about. This use-case is sounds to me like a one-size-fits-all mandate that all tools using redis in Toolforge must be adapted to get connection information from a fixed format envvar or risk arbitrarily losing connectivity due to unannounced platform changes. There is a lot of extrapolation from your exact wording in that fear, and I am not stating that this interpretation at all matches your intent, but it is my fear none the less.

This use-case is sounds to me like a one-size-fits-all mandate that all tools using redis in Toolforge must be adapted to get connection information from a fixed format envvar or risk arbitrarily losing connectivity due to unannounced platform changes.

There's no mandate (I'm not even sure if that would be possible), it's up to the tool maintainers to decide if they prefer hard-codding those values in their code/config, and have to keep an eye to the wiki/list/etc. for updates and manually change it, or use the envvars and not have to care. This is not removing any option, it's adding a new one.

About the format, urls are pretty well documented and most languages have already libraries to parse them correctly in most cases (for sure this one), but even then, we can expose it as TOOL_*_{PROTOCOL,USER,PASSWORD,PORT,HOST} and let the user build up the url as/if they need instead of having to parse it (that was my first approach, but someone told me to use the URI instead and I did not see a big differences so I complied).