A few things come to mind immediately.

1. We don't want to introduce an external (to us) dependency. Logstash has no internet access and it shouldn't have it. (I am guessing the workaround is to use IPoid?)
2. We don't want to introduce additional latency to the overall pipeline. The desired stream ought to be isolated so that delays induced by the external data provider do not add latency for other tenants.
3. Logstash handles thousands of events per second which translates into requests against the data source. Whatever handles the data should be able to absorb this extra load.
4. The Logstash SLO may need to be tuned to handle the introduction of an external dependency.

In T407219#11274660, @colewhite wrote:

I am guessing the workaround is to use IPoid?

Yeah. IPoid serves up a database created from the daily IP data dumps Spur gives us, so theoretically that dump could be provided directly to the Logstash hosts in some different format that can be queried efficiently locally (e.g. sqlite) but I imagine that would result in an infeasible amount of maintenance overhead.

2. We don't want to introduce additional latency to the overall pipeline. The desired stream ought to be isolated so that delays induced by the external data provider do not add latency for other tenants.
3. Logstash handles thousands of events per second which translates into requests against the data source. Whatever handles the data should be able to absorb this extra load.

That seems problematic even with IPoid. I guess this is a no-go then?

Is there maybe a way to do such post-processing inside the OpenSearch index?
I see @kostajh wrote about how to import the Spur data to OpenSearch, so maybe there's an intent to do that anyway and we'd only have to connect the two indexes somehow?

In T407219#11309704, @Tgr wrote:

Yeah. IPoid serves up a database created from the daily IP data dumps Spur gives us, so theoretically that dump could be provided directly to the Logstash hosts in some different format that can be queried efficiently locally (e.g. sqlite) but I imagine that would result in an infeasible amount of maintenance overhead.

Possibly not an infeasible amount of maintenance overhead. Depends on how stable the data source is and how performant it can be made. Performance testing is needed.

That seems problematic even with IPoid. I guess this is a no-go then?

I wouldn't say it's no-go because of that. Isolating the stream is possible and so is a separate data enrichment step. Isolating the stream would give us a more concrete picture of the load this feature will induce on the Spur data provider (whatever form it takes).

Is there maybe a way to do such post-processing inside the OpenSearch index?

Post-processing indexed events is less preferable as it introduces tombstoning overhead. We'd like to avoid that. It is far better to inject the data into the event before it reaches OpenSearch for storage. Knowing the volume of the isolated stream would help us asses the impact.

I see @kostajh wrote about how to import the Spur data to OpenSearch, so maybe there's an intent to do that anyway and we'd only have to connect the two indexes somehow?

OpenSearch has no JOIN clause. Depending on the size of the data Spur gives us, I do see the possibility of the separate data enrichment pipeline pulling data backed by an OpenSearch index, though. Performance testing is needed.

A somewhat messy way to do this would be to implement GetSecurityLogContext in Extension:IPReputation, and set a very low timeout (50ms?) to fetch the data and add relevant fields. Generally, the lookup should be fast, and after an IP is requested once, the result is cached in Memcache and should be < 10ms. We'll probably miss adding ipoid data to some log entries but we should have it for subsequent logs associated with an IP.

T410878: wmfGetPrivilegedGroups is slow is a similar problem (although more easily fixable via caching), which I guess is a point in favor of handling this in MediaWiki. (Although wmfGetPrivilegedGroups() is used for things other than logging, so that problem will have to be solved anyway, but it does show that the problem with IPReputation isn't that unique.)

Change #1227758 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/IPReputation@master] GetSecurityLogContext: Add IP reputation metadata

https://gerrit.wikimedia.org/r/1227758

Change #1227758 merged by jenkins-bot:

[mediawiki/extensions/IPReputation@master] GetSecurityLogContext: Add IP reputation metadata

https://gerrit.wikimedia.org/r/1227758