Just a note that for today's promotion of group1 to 1.33.0-wmf.13 (T206667), I segmented the group1 error-log dashboard to have a view of just these timeout errors and a view excluding them. It was very helpful in keeping on both the rise in timeouts and side effects or unrelated errors. I plan on saving the dashboards and adding links in the train docs.

Cutting the branch.

In T210267#4873881, @thcipriani wrote:

What new problems does this create?

• Potential that one merge can use a lot of CI executors (also currently the case, but RelEng has some ability to mitigate currently)
• Others?

Today's all-wiki deployment went much like yesterday's: We saw an increase in the MediaWiki error rate due to a flood of "timed out" errors and what seemed like a side effect of 500s caused by nginx/apache/hhvm socket timeouts. The rate increases lasted from approximately 20:11 until 20:34 UTC at which point they subsided to pre-deployment levels.

In T212427#4862694, @Addshore wrote:

@greg @dduvall train is unblocked now, verified on beta.

In T212251#4834349, @akosiaris wrote:

In T212251#4834325, @hashar wrote:

In T212251#4833906, @akosiaris wrote:

As SREs I don't think we would like to expose a public service that is not going to receive any traffic any time soon, mostly for operational and complexity reasons. I guess we don't really have that need yet, do we?

But that is exactly the point of this task? Open the service to the outside world so we can start using it. Instances on WMCS can not reach production internal network (10.0.0.0/8), they get out via PAT/NAT and are considered to be just like any internet traffic (== untrusted).

Is it? Cause I gather that it is specifically for CI. In which case an instance in CI is architecturally a better construct.

@thcipriani +1 to the proposal in general. I think it adds clarity to artifact definition and to the purpose of copies. Just some notes on config structs and implementation of sane defaults.

Latest image for use with initial chart implementation is docker-registry.wikimedia.org/wikimedia/blubber:20181212233039-production.

One refactoring option I can think of would be to make the config for copying of project/application files explicit and use a sane default.

I just merged the final patch related to T210030: RedisBagOStuff is broken on beta that removed the stale redis server entries from the labs related mediawiki-config, and the instances are terminated.

I'm also seeing MW fatals resulting from the inoperable redis servers and the mediawiki-config for redis_lock that still references them.

AFAICT, this and related errors are no longer occurring. It's not yet clear exactly what the underlying issue might have been, but in troubleshooting with @thcipriani yesterday we noticed a few things that might have had an impact. Since @thcipriani graciously jumped in to take care of the actual fixes, he might want/need to correct some of this information. :)

Looking at @Krenair 's log of replication lag, there's no indication of lag at any given time since the first entry ("Thu Nov 29 20:13:01 UTC 2018").

Sorry, I posted the wrong server's output. :) Editing...

I'm also wondering how MediaWiki detects replication lag. Does it look at all DB hosts? And which MaridDB variables does it inspect? Is it possible that it's looking at deployment-db03 and errantly detecting it as a lagged replica?

Looking at SHOW SLAVE STATUS\G on deployment-db04 shows no current lag according to seconds_behind_master, but I'm looking into where we might see its historical values. I thought there was a prometheus collector set up for beta's maria DBs, but I'm not whether it monitors master/slave replication status.

@Legoktm could you clarify whether this task tracks progress of programatic extension installation or of programatic enablement/disablement? The title says (and comments seem to relate to) the latter but the description links to a section of the feedback article for the former.

Another example of how to implement a decent YAML to JSON layer might be the Golang OpenAPI library's YAML parser.

@greg, we noticed this in the triage meeting you weren't at. Would you care to adjust priority? ;-)

@hashar, would you please triage priority and placement on the RelEng Kanban board? Thanks!

@hashar, this is marked as "Done" in RelEng Kanban. Any update?

@hashar, any update? Moving to backlog until we have one.

Two integration/config patches were deployed to fix the issue. The first refactored all Docker based jobs to invoke docker run via a common builder. The second adds to the common docker-run builder the --init option which ensures a PID 1 within the container that will properly reap child processes and forward signals, and --label options which tag the running containers with the job name and build number; it also implements an additional safety measure, a docker-reap-containers post-build script that kills any running containers that could be errantly running at the end of the build (using the added labels to filter for only the build's containers).

More long-running containers seen today: