Page MenuHomePhabricator

Grant sbassett and aranyap expanded logstash access
Open, Needs TriagePublic

Description

Hey Logstash folks -

Aranya and I on Product Safety and Integrity would like to get expanded logstash download access. The default 10k CSV download limit for the Discover reports won't work for our CSP analysis work where we'll be dealing with potentially many millions of rows of data. I don't know what the maximum limit is for this kind of access, but that's what Aranya and I would require for the next couple of months or so. Please let me know if you have any additional questions or concerns. Thanks.

Event Timeline

Dzahn subscribed.

Hi, collaboration-services is not particularly related to logstash. I added Observability team which I think is a better fit. Cheers

Hi, collaboration-services is not particularly related to logstash. I added Observability team which I think is a better fit. Cheers

Ok, thanks. Do you know if they have a triage clinic or queue for these kinds of requests?

@sbassett Let's just add the global SRE-access-requests tag (in addition to the team tag). That has a rotating clinic duty / triaging.

Ok, thanks. Do you know if they have a triage clinic or queue for these kinds of requests?

Generally I have this: https://wikitech.wikimedia.org/wiki/SRE/Observability#Phabricator_Workflow

And access requests are normally handled by SRE clinic duty.

I think you might be in luck since by coincidence @tappof is on both groups this week.

We’ll discuss your request with the o11y team, and I’ll let you know.
In the meantime, I’m going to remove some tags, since the correct one to reach out to the team (observability-logging) is already present.

We’ll discuss your request with the o11y team, and I’ll let you know.
In the meantime, I’m going to remove some tags, since the correct one to reach out to the team (observability-logging) is already present.

Ok, thanks. I don't want to be too pushy, but this is currently blocking hypothesis-driven research that we had scheduled for this quarter.

Thank you for the response @tappof! Let us know if there's any further information we can give or a meeting we can join to clarify our needs.
We want to move aggressively through the pilot to broader deployment and the data will be essential to mitigate any unintended breakage.

We can provide you access through the API (please take a look at https://wikitech.wikimedia.org/wiki/Logstash#API and https://docs.opensearch.org/latest/api-reference/search-apis/search/) to help automate the process of downloading log entries.

We will share temporary credentials via 1Password, since the actual authorization process is intended more for apps than for regular users.

I sent you a 1Password link to share the credentials. You’ve just been added, so please wait a while for the Puppet agent to run.
Please keep in mind that you can download a maximum of 10,000 entries per batch, but this way you can easily script your searches.
Please feel free to reach out if you need any help.