Hi, collaboration-services is not particularly related to logstash. I added Observability team which I think is a better fit. Cheers

In T416501#11591852, @Dzahn wrote:

Hi, collaboration-services is not particularly related to logstash. I added Observability team which I think is a better fit. Cheers

Ok, thanks. Do you know if they have a triage clinic or queue for these kinds of requests?

@sbassett Let's just add the global SRE-access-requests tag (in addition to the team tag). That has a rotating clinic duty / triaging.

In T416501#11592078, @sbassett wrote:

Ok, thanks. Do you know if they have a triage clinic or queue for these kinds of requests?

Generally I have this: https://wikitech.wikimedia.org/wiki/SRE/Observability#Phabricator_Workflow

And access requests are normally handled by SRE clinic duty.

I think you might be in luck since by coincidence @tappof is on both groups this week.

We’ll discuss your request with the o11y team, and I’ll let you know.
In the meantime, I’m going to remove some tags, since the correct one to reach out to the team (observability-logging) is already present.

In T416501#11597079, @tappof wrote:

We’ll discuss your request with the o11y team, and I’ll let you know.
In the meantime, I’m going to remove some tags, since the correct one to reach out to the team (observability-logging) is already present.

Ok, thanks. I don't want to be too pushy, but this is currently blocking hypothesis-driven research that we had scheduled for this quarter.

Thank you for the response @tappof! Let us know if there's any further information we can give or a meeting we can join to clarify our needs.
We want to move aggressively through the pilot to broader deployment and the data will be essential to mitigate any unintended breakage.

We can provide you access through the API (please take a look at https://wikitech.wikimedia.org/wiki/Logstash#API and https://docs.opensearch.org/latest/api-reference/search-apis/search/) to help automate the process of downloading log entries.

We will share temporary credentials via 1Password, since the actual authorization process is intended more for apps than for regular users.

@tappof - sounds great, thanks!

I sent you a 1Password link to share the credentials. You’ve just been added, so please wait a while for the Puppet agent to run.
Please keep in mind that you can download a maximum of 10,000 entries per batch, but this way you can easily script your searches.
Please feel free to reach out if you need any help.

Hi there @tappof , we're able to authenticate to https://logs-api.svc.eqiad.wmnet/ (GET / works) using the credentials you gave us, but all search endpoints (/_search, /_msearch, _cat/*) return 401 Unauthorized.

Just wanted to check if we have permissions to execute read/search to run search queries or if there's another way we need to access this data.

Thank you!

Hey @aranyap, I'm checking...

Hi everyone, we'd also like to add Alex Sanford (shell username is alexsanford) to this request as well. I updated the original ticket to include him. Thanks!

In T416501#11623551, @aranyap wrote:

Hi there @tappof , we're able to authenticate to https://logs-api.svc.eqiad.wmnet/ (GET / works) using the credentials you gave us, but all search endpoints (/_search, /_msearch, _cat/*) return 401 Unauthorized.

Just wanted to check if we have permissions to execute read/search to run search queries or if there's another way we need to access this data.

Hello! Looks like this likely was a knock on effect of T417934 which prevented the new accounts from actually being deployed. Could you please try again after about 30 minutes and let us know if the 401s persist?

@herron @ssingh @tappof - this access seems to work for me, testing on deployment with curl. The page size still maxes out at 10k (there's a max_result_window var we could bump up if we wanted to) but I can also just quickly page through results via the scroll and scroll_id params. I was working on some bash (that'll probably turn into python) yesterday to do this; I'll post a finished version of that on this bug today, for posterity.

Hey @tappof as @aranyap mentioned above I'll be working on this too. Could you please send me creds as well?

Just adding that I've been successfully using this bash script to connect to and pull data from opensearch, based upon this documentation:

https://gitlab.wikimedia.org/-/snippets/278

I plan to add some scroll-based pagination support to it soon.

Just sent @ASanford-WMF a link on Slack with his credentials.
You’ve just been added, so please wait a while for the Puppet agent to run.

@sbassett A note regarding the above script: I believe the blocked-url, document-uri, and referrer should each be prefixed with csp-report. in order to include their values in the API output.

Confirming that I can access the API and run search queries. Thank you so much for your help @tappof !

Hey there - just a thought that came up in team discussion yesterday: mwlog hosts also have the full csp-report-only flow in files on disk with gzipped history also kept in the archive directory. This might be closer to what you're looking for if you're doing bulk analysis and streaming live entries isn't what you're looking for. These logs do get rotated out over time so beware of that.

Thanks, @hnowlan. We might pivot to using the raw logs for certain types of data analysis. Though I think, so far, the extended logstash access has been working at least alright for @aranyap, @ASanford-WMF and myself.