Thanks for filing this!

This is what would be helpful for us in debugging registry access issues.

• nginx access logs and error logs on the registry hosts (registry{1003,1004}.eqiad.wmnet, registry{2003,2004}.codfw.wmnet)
• docker-registry service logs and access logs on the registry hosts
• jwt-authorizer systemd logs on the registry hosts

@dancy, @jnuche ^ does this seem exhaustive enough for us?

I can imagine a scenario where there might be issues with the swift backend, but I don't think anything at that level wouldn't be surfaced by the other logs somehow and probably noticed and fixed by SRE very quickly.

In T322579#8377279, @dduvall wrote:

Thanks for filing this!

This is what would be helpful for us in debugging registry access issues.

• nginx access logs and error logs on the registry hosts (registry{1003,1004}.eqiad.wmnet, registry{2003,2004}.codfw.wmnet)
• docker-registry service logs and access logs on the registry hosts
• jwt-authorizer systemd logs on the registry hosts

@dancy, @jnuche ^ does this seem exhaustive enough for us?

I can imagine a scenario where there might be issues with the swift backend, but I don't think anything at that level wouldn't be surfaced by the other logs somehow and probably noticed and fixed by SRE very quickly.

I would be ok with granting access to logs via logstash, I'm quite opposed to widen access to the registry hosts, it seems not really useful.

Also: both the registry and nginx keep access logs, so I guess it's enough to export one of the two.

In T322579#8378397, @Joe wrote:

[...]
Also: both the registry and nginx keep access logs, so I guess it's enough to export one of the two.

I'd say export those from docker-registry. It writes access log plus additional stuff that might be useful

I just added this small section to the Wikitech Logstash page how I got logs from "misc" systems into logstash with 2 changes.

One to get them to local syslog via rsyslog and then another to tell rsyslog to ship to kafka. And then I could see them in logstash filtering by "program". Example patches linked below are what I would copy to get nginx logs there.

https://wikitech.wikimedia.org/wiki/Logstash#Getting_logs_from_misc_systems_into_logstash

In T322579#8378513, @JMeybohm wrote:

In T322579#8378397, @Joe wrote:

[...]
Also: both the registry and nginx keep access logs, so I guess it's enough to export one of the two.

I'd say export those from docker-registry. It writes access log plus additional stuff that might be useful

There will be nginx access log entries that are not reflected in the docker-registry access logs that would be useful to us as well, most notably anything related to auth using the GitLab JWT tokens, subrequests between nginx and jwt-authorizer and the related nginx responses to the client. These do not hit docker-registry at all.

In T322579#8378397, @Joe wrote:

I would be ok with granting access to logs via logstash, I'm quite opposed to widen access to the registry hosts, it seems not really useful.

Also: both the registry and nginx keep access logs, so I guess it's enough to export one of the two.

FWIW we're not looking for anything close to root. We just need visibility into jwt-authorizer (via journalctl -u jwt-authorizer would be plenty), the communication between nginx and jwt-authorizer (so subrequest entries in the access logs), and the nginx responses from auth (that never hit docker-registry).

In T322579#8377279, @dduvall wrote:

Thanks for filing this!

This is what would be helpful for us in debugging registry access issues.

• nginx access logs and error logs on the registry hosts (registry{1003,1004}.eqiad.wmnet, registry{2003,2004}.codfw.wmnet)
• docker-registry service logs and access logs on the registry hosts
• jwt-authorizer systemd logs on the registry hosts

@dancy, @jnuche ^ does this seem exhaustive enough for us?

That list looks good to me.

Friendly ping. :)

Change 919350 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] Add nginx logs for docker-registry host to rsyslog

https://gerrit.wikimedia.org/r/919350

Change 919351 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] Send nginx and docker-registry logs to kafka

https://gerrit.wikimedia.org/r/919351

@dduvall I put up some changes to move the nginx (/var/log/nginx/{error,access}.log) and docker-registry (journalctl -u docker-registry) logs to logstash. The logs for jwt-authorizer seem to be empty though (from looking at journalctl -u jwt-authorizer), is there another place they might be?

Change 919350 merged by EoghanGaffney:

[operations/puppet@production] Add nginx logs for docker-registry host to rsyslog

https://gerrit.wikimedia.org/r/919350

Change 919351 merged by EoghanGaffney:

[operations/puppet@production] Send nginx and docker-registry logs to kafka

https://gerrit.wikimedia.org/r/919351

Change 930719 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] registry: Add nginx logs to rsyslog

https://gerrit.wikimedia.org/r/930719

Change 930719 merged by EoghanGaffney:

[operations/puppet@production] registry: Add nginx logs to rsyslog

https://gerrit.wikimedia.org/r/930719