Ingest webrequest sampled 1000 into logstash
Open, MediumPublic
Actions

Assigned To

None

Authored By

	fgiunchedi
	Feb 7 2022, 11:15 AM

Description

We have 5xx.json available in logstash (and on file on centrallog hosts in /srv/weblog/webrequest, which is useful to debug/investigate errors.

With ELK7 and in general more capacity (and headroom) I (Filippo) think we should be ingesting the sampled (1/1000) webrequest stream, specifically for:

Access to dashboards to debug/investigate abuse incidents
Sharing dashboards and findings during incidents

The data has PII, however I don't think it is at a greater risk than PII already in kafka/logstash (e.g. ip addresses and user agent)

Implementation wise, we currently funnel 5xx as such:

kafkatee -> grep/jq/logger -> rsyslog -> kafka -> logstash

The easy (not necessarily simple) thing to do is to replicate the same with sampled-1000 kafkatee output, (i.e. an additional load of max ~200 logs/s)

Related Objects
Search...

Status	Assigned	Task
Resolved	fgiunchedi	T213157 Increase utilization of application logging pipeline (FY2018-2019 Q3 TEC6)
Resolved	fgiunchedi	T220103 TEC6: Logging infrastructure (Q4 2018/19 goal)
Open	colewhite	T213902 Implement sensitive logstash access control
Open	None	T301110 Ingest webrequest sampled 1000 into logstash

Event Timeline

fgiunchedi created this task.Feb 7 2022, 11:15 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 7 2022, 11:15 AM

fgiunchedi added a project: User-fgiunchedi.Feb 7 2022, 12:59 PM

akosiaris subscribed.Feb 7 2022, 5:22 PM

My (perhaps dated or incorrect) understanding is that:

We currently have no RBAC in Logstash;
Everyone in the "NDA" group have access to all data stored in Logstash;
Access to access logs in general is more restricted, to a subset of NDA users, to the analytics-privatedata group (membership managed by the D/E team);
sampled-1000 is a subset of access logs, available in the centrallog hosts, where only ops/roots have access to (so even more restricted)

Are any of these assumptions incorrect at this time? If not, does that mean that this task is effectively proposing to expand access to (a 1:1000 sample of) access logs to a wider group of individuals? Not saying (yet) whether this is a problem per se, but I think it'd be good to establish shared understanding of what is being proposed and discussed here -- specifically whether this is a technical change, or an access control or PII-sharing change. Thanks!

In T301110#7707705, @faidon wrote:

My (perhaps dated or incorrect) understanding is that:

We currently have no RBAC in Logstash;

Everyone in the "NDA" group have access to all data stored in Logstash;

Access to access logs in general is more restricted, to a subset of NDA users, to the analytics-privatedata group (membership managed by the D/E team);

sampled-1000 is a subset of access logs, available in the centrallog hosts, where only ops/roots have access to (so even more restricted)

Are any of these assumptions incorrect at this time? If not, does that mean that this task is effectively proposing to expand access to (a 1:1000 sample of) access logs to a wider group of individuals? Not saying (yet) whether this is a problem per se, but I think it'd be good to establish shared understanding of what is being proposed and discussed here -- specifically whether this is a technical change, or an access control or PII-sharing change. Thanks!

Thank you @faidon for the questions -- assumptions 1, 2 and 4 are correct TTBOMK. Whereas for 3 (in my opinion) things are a little fuzzier since cn=nda has access to webrequest_sampled_128 via turnilo (though the raw data isn't available for download).

Hope that helps!

jbond triaged this task as Medium priority.Feb 16 2022, 4:58 PM

fgiunchedi moved this task from Backlog to Up next on the User-fgiunchedi board.Mar 1 2022, 10:58 AM

colewhite added a parent task: T213902: Implement sensitive logstash access control.Jul 26 2022, 10:26 PM

colewhite moved this task from Inbox to Blocked on the Observability-Logging board.Sep 21 2022, 10:54 AM

fgiunchedi removed a project: User-fgiunchedi.Nov 25 2022, 8:51 AM

This is still valid, though nowadays the implementation will be much simpler: we can ingest webrequest_sampled directly from Kafka!