The discussion in T125069: Create a service location / discovery system for locating local/master resources easily across all WMF applications is highly relevant here.

I still think that DNS has served us well so far, and provides a reliable and extremely simple solution for node services. While DNS caching *can* be broken in some services, I haven't really seen concrete information on whether this is the case for HHVM, and if so, whether an OS-level configuration change could fix it. Lets at least check these things before rewriting a custom DNS look-alike at the app level.

This is not limited to a discovery system, and is focused on MediaWiki (as it's the most common offender in terms of having tons of reference to cluster state in its configuration). Most of what is needed by MediaWiki (and potentially, any other service needing more than just one url, but - say- lists of servers in a pool) can't be served satisfactorily just using DNS (I know you can with SRV records, but that's really suboptimal).

Which can be still a good alternative whenever we don't need complex logic: it was cited both in the linked ticket and in the linked article, I wasn't discarding it. I just think that's just one part of the story and one that doesn't apply well to the MediaWiki use case, and that I think is suboptimal whenever we want to have fast changes - we really don't want to set the DNS TTL too low.

In any case, whatever is the mean to receive discovery information, a global mechanism should be able to account for relatively complex cases as the mediawiki-config one.

Most of what is needed by MediaWiki (and potentially, any other service needing more than just one url, but - say- lists of servers in a pool) can't be served satisfactorily just using DNS (I know you can with SRV records, but that's really suboptimal).

There is a long tradition of using multiple A records for such lists. Most clients will round-robin between those contacts, but others like Cassandra also use the entire list, in cassandra's case to initialize its list of contact points.

PHP seems to support the retrieval of all records quite well with [dns_get_record()](http://php.net/manual/en/function.dns-get-record.php). That said, I haven't used this implementation in practice, so can't vouch for quality or performance. There is similar built-in support in nodejs. Python requires a separate package (dnspython): https://c0deman.wordpress.com/2014/06/17/find-nameservers-of-domain-name-python/

Edit: It seems that Python's built-in socket.gethostbyname_ex() can return all A records as well, so no external library is needed there either: http://stackoverflow.com/questions/3837744/how-to-resolve-dns-in-python

I’m interested in this topic, particularly because I try to develop a wiki farm and I wrote and released a configuration management extension for MediaWiki. For now it is only focused on MediaWiki config in the context of a farm (hierarchical configuration similar to InitialiseSettings.php with multiversion management) and a next step would be exactly what is described in this task: integrate MediaWiki in its environment and manage inter-dependencies.

If we place MediaWiki in the core of the environment, there are "input dependencies" (e.g. MediaWiki depends on PHP) and "output dependencies" (e.g. Parsoid is called by MediaWiki/VisualEditor). The difficulty is to avoid re-developing what Puppet/Ansible/etc does (e.g. install PHP, MediaWiki, Parsoid, etc), but in the same time give some retroaction to these tools (e.g. given we want to install the VisualEditor feature on a MediaWiki 1.28, we have to install Parsoid 0.6.1 and say to MediaWiki that Parsoid is located on this host). Hence, somewhere, there should be a database with a map (tool:version <-> tool:version), and a map (service <-> configuration), and this second map could depend on context parameters like the datacenter or the realm (prod/labs) (although in my mind, in the MediaWikiFarm extension, something like the wmflabs would be only another farm derived from the prod farm, just with some overloaded config parameters like URL and database config).

+1 for the etc/confd/json + APC approach for MediaWiki, at least starting with DB configuration.

What were the APC caveats?

In T149617#2832697, @aaron wrote:

Task description:

• Either parse this output from wmf-config/CommonSettings.php where we currently include those files, or have a hook that stores that data into HHVM's APC (some caveats apply)

+1 for the etc/confd/json + APC approach for MediaWiki, at least starting with DB configuration.

What were the APC caveats?

So this means a separate background process that is triggered by etc/confd, which takes the data and stores it as an associative array in APC - which we read at run-time in wmf-config PHP code.

The benefit would be that it can do a hot swap (because it's re-read every request) without any notification or on-disk file for HHVM (with or without RepoAuth enabled). The downside is that APC is considered ephemeral and storing this there seems a bit fragile.

Aside from potentially unpredictable evictions (which are actually rare in HHVM, and a problem in itself that helps us in this case) - we'd have to very carefully orchestrate HHVM reboots.

We currently cache the expansion/extraction process of SiteConfig from InitialiseSettings.php in a temporary file containing serialised php (source). This is invalidated and regenerated at run-time based on a newer filemtime. This could perhaps be moved to APC as a proof of concept.

If it works well, we can do the same with db configs. E.g. have etcd/confd produce a json file, that is parsed and cached in APC lazily by wmf-config. That way it will fall back gracefully. We could of course populate that APC key directly from the script that creates the json file (e.g. right before the json file is written to wmf-config). That way it's only a fallback.

The background process would write to the JSON file. APC caching would be done by wmf-config PHP code I assume, checking the mtime of the JSON.

In T149617#2832982, @aaron wrote:

The background process would write to the JSON file. APC caching would be done by wmf-config PHP code I assume, checking the mtime of the JSON.

Sounds good.

@aaron another possibility is to have the process call a special url on HHVM to push the APC refresh.

In T149617#2832764, @Krinkle wrote:

We currently cache the expansion/extraction process of SiteConfig from InitialiseSettings.php in a temporary file containing serialised php (source). This is invalidated and regenerated at run-time based on a newer filemtime. This could perhaps be moved to APC as a proof of concept.

If it works well, we can do the same with db configs. E.g. have etcd/confd produce a json file, that is parsed and cached in APC lazily by wmf-config. That way it will fall back gracefully. We could of course populate that APC key directly from the script that creates the json file (e.g. right before the json file is written to wmf-config). That way it's only a fallback.

This is about what is done in Extension:MediaWikiFarm: the config (originally in YAML, JSON, or PHP arrays) is cached in a PHP file (return array( 'config1' => 'value1', … ); so that this opcode can be cached directly by PHP (APC or Opcache).

Note that Parsoid also loads a bunch of configuration from mediawiki at startup and caches it. Some changes to mediawiki configuration require a Parsoid restart.
,
It would be great to have a built in/standard mechanism for propagating configuration changes. At the very least, a monotonic counter associated with the configuration, so third-party services like Parsoid could easily determine whether they need to reload siteinfo.

In T149617#2918215, @cscott wrote:

Note that Parsoid also loads a bunch of configuration from mediawiki at startup and caches it. Some changes to mediawiki configuration require a Parsoid restart.
,
It would be great to have a built in/standard mechanism for propagating configuration changes. At the very least, a monotonic counter associated with the configuration, so third-party services like Parsoid could easily determine whether they need to reload siteinfo.

I am completely unaware of this; can you please elaborate?

Also, given we have no way of signalling parsoid that the mediawiki config has changed, this would look as a very very bad design decision; but I'm sure I misunderstood.

To the owner of this session: Here is the link to the session guidelines page: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Session_Guidelines. We encourage you to recruit Note-taker(s) 2(min) and 3(max), Remote Moderator, and Advocate (optional) on the spot before the beginning of your session. Instructions about each role player's task are outlined in the guidelines. The physical version of the role cards will be made available in all the session rooms. Good luck prepping, see you at the summit! :)

Note-taker(s) of this session: Follow the instructions here: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Session_Guidelines#NOTE-TAKER.28S.29 After the session, DO NOT FORGET to copy the relevant notes and summary into a new wiki page following the template here: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Your_Session and also link this from the All Session Notes page: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/All_Session_Notes. The EtherPad links are also now linked from the Schedule page (https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Schedule) for you!

The summary of the session with all the relevant links (detailed notes, slides and recording) is available on MediaWiki: Integrating MediaWiki (and other services) with dynamic configuration.

Extracting from the session outcomes:

What we want to do:

• Stop relying in configuration repository to store the live state of clusters, it needs to be treated separately.
• We already have a tool to store such state reliably, and that is conftool. We have no reason not to expand it to be able to cover such cases to.
• Use DNS for service location/discovery, and simple data structures, but has some limitations.
  • We want to integrate conftool with gdnsd, as we see it as the most reliable and flexible option instead of coredns/skydns, which can read from etcd, but only from their own somewhat limited format
• Use Confd with templates and scripts to manage more complex data structures.
• For the MediaWiki specific case there are a bunch of options:
  • generate a JSON file that is read by the application, parsed by MediaWiki and cached into APC, via APC (preferred)
  • generate a PHP file that will be included directly by the application.
  • Add safe measures to ensure that the configuration is consistent across the cluster and no hosts have a stale one.

Since this is at least partially interesting for the TechOps Goal for the current quarter, I'll add subtasks for the actual implementation of the system.

@Joe could you also upload the slides from https://docs.google.com/presentation/d/1r4g_3yZxfPzzEEvOgj4ZbbTBrNshBylDAb5KYdnzke4/edit?usp=sharing to commons? :)

In T149617#2977429, @Legoktm wrote:

@Joe could you also upload the slides from https://docs.google.com/presentation/d/1r4g_3yZxfPzzEEvOgj4ZbbTBrNshBylDAb5KYdnzke4/edit?usp=sharing to commons? :)

Yes, I am just unsure how / to who I can attribute the template design. That's what is blocking me at the moment.

In T149617#2980969, @Joe wrote:

In T149617#2977429, @Legoktm wrote:

@Joe could you also upload the slides from https://docs.google.com/presentation/d/1r4g_3yZxfPzzEEvOgj4ZbbTBrNshBylDAb5KYdnzke4/edit?usp=sharing to commons? :)

Yes, I am just unsure how / to who I can attribute the template design. That's what is blocking me at the moment.

While we wait for a response from communications, I decided I'd rather ask forgiveness than wait for permission and license everything under CC-by-SA 3.0

https://commons.wikimedia.org/wiki/File:Dynamic_configuration_in_MediaWiki_and_other_applications.pdf

Change 339673 had a related patch set uploaded (by Giuseppe Lavagetto):
profile::conftool::client: add default schema

https://gerrit.wikimedia.org/r/339673

Change 339674 had a related patch set uploaded (by Giuseppe Lavagetto):
conftool-data: add first discovery objects

https://gerrit.wikimedia.org/r/339674

Mentioned in SAL (#wikimedia-operations) [2017-02-27T10:11:46Z] <_joe_> upgrading conftool to 0.4.0 across the cluster T149617

Change 339673 merged by Giuseppe Lavagetto:
profile::conftool::client: add default schema

https://gerrit.wikimedia.org/r/339673

Change 339674 merged by Giuseppe Lavagetto:
conftool-data: add first discovery objects

https://gerrit.wikimedia.org/r/339674

Change 340538 had a related patch set uploaded (by Giuseppe Lavagetto):
[operations/puppet] profile::discovery::client: create confd-generate files for discovery

https://gerrit.wikimedia.org/r/340538

Change 340538 merged by Giuseppe Lavagetto:
[operations/puppet] profile::discovery::client: create confd-generate files for discovery

https://gerrit.wikimedia.org/r/340538

Change 340935 had a related patch set uploaded (by oblivian):
[operations/puppet] profile::discovery::client: expose services as well

https://gerrit.wikimedia.org/r/340935

So, I just found out that the dns cache feature we were supposedly using in HHVM has been removed from it for some time, so while we have the ini setting in our setup, it's not doing much of an effect.

We can therefore probably use the DNS discovery for service discovery instead of needing to use confd for that.

This will still be useful for other uses, like lists of servers or databases

Change 340935 merged by Giuseppe Lavagetto:
[operations/puppet] profile::discovery::client: expose services as well

https://gerrit.wikimedia.org/r/340935

Status as of now:

• DNS based discovery is live and functioning for most things, it's just pending a couple of merges in MediaWiki.
• Etcd-based mediawiki configuration is being worked on

Change 411296 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Enable EtcdConfig on the debug hosts

https://gerrit.wikimedia.org/r/411296

Change 411296 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable EtcdConfig on the debug hosts

https://gerrit.wikimedia.org/r/411296