Thanks, sounds like a classic PHP can't tell the difference between and empty object and an empty list bug.

I have a strong suspicion that this bug also affects mw.eventLog.submit() because:

• mw.eventLog.streamInSample('test.event') returns false
• mw.eventLog.streamConfig('test.event') returns null
• mw.eventLog.streamConfig('eventlogging_SearchSatisfaction') returns { "destination_event_service": "eventgate-analytics-external" }
• mw.eventLog.streamInSample('eventlogging_SearchSatisfaction') returns true

streamInSample() doesn't use streamConfig() (although there's an argument to be made that it should) and checks config directly via config.streamConfigs[ streamName ] === undefined, so "test.event" stream should be in-sample because it is in the config and so we would expect config.streamConfigs['test.event'] to be {}, but it must be evaluating to undefined which causes that stream to be marked as not in sample

In T259917#6379915, @mpopov wrote:

I have a strong suspicion that this bug also affects mw.eventLog.submit() because:

• mw.eventLog.streamConfig('test.event') returns null

This is not possible if the value of config.streamConfigs[ 'test.event' ] is {} or []. These are truthy in JS. See code. The only possibility is if $.extend( true, {}, {} ) or $.extend( true, {}, [] ) evaluate to null, which I don't think that they do.

• mw.eventLog.streamInSample('test.event') returns false

Again, this does not seem possible unless there is, in fact, no key 'test.event' in config.streamConfigs.

Are you sure that 'test.event' is really defined?

Nevermind, test.event is a stream in wgEventStreams but is not registered with EventLogging in wgEventLoggingStreamNames. Okay, so we're good on MediaWiki! Sorry for the false alarm.

This is still a blocker for apps clients, though.

I don't think it is; it is not listed in wgEventLoggingStreamNames. BTW, I am removing 'test.event' as a stream; we are moving towards eventgate service specific test event streams, e.g. 'eventgate-analytics-external.test.event'.

In T259917#6380167, @Ottomata wrote:

BTW, I am removing 'test.event' as a stream; we are moving towards eventgate service specific test event streams, e.g. 'eventgate-analytics-external.test.event'.

Question about data lifecycle… After you remove that stream, if we configure a "test.event" stream (as part of T259714) using an analytics test schema (with client_dt, session_id, device_id, pageview_id fields), what happens to the existing test_event table in the database? If a stream name is re-used with a totally different schema, what happens?

There's nothing special about the test event streams, other than eventgate services know how to produce them. You can certainly make an analytics specific test schema and stream.

But what happens to the existing table in the database that was created from the old stream under a different schema? Does it need to be manually dropped via Hive CLI before data produced to the replacement stream gets stored? Can anyone drop it from the events database (and events_sanitized, for that matter) or just your team?

I guess we can drop it, but we don't really care about the data in test.event. It's just test data. The table will stick around, but no more data will be written to it.

under a different schema?

BTW, the schema is the same one (test/event), they are just new stream names.

I was just using test.event as an immediate example, but the question I'm asking is about what happens in general when a stream name is re-used with a different, potentially incompatible schema. A team expecting their instrument's events to show up in the database would be surprised to learn that none of that data is making it in because they just happened to name their stream the same name as a previously configured-and-then-deleted stream and the old table/data was never dropped when the stream was removed from the config.

what happens in general when a stream name is re-used with a different, potentially incompatible schema.

Bad stuff, it won't work, unless we do the work to totally purge any previous artifacts of that stream, like its Hive table.

But I see your point, a collision due to removing the stream config could be a problem. I guess this is true of legacy EventLogging metawiki schemas too; someone could delete a schema page, and then later someone could create a new one with the same name.

In T259917#6380598, @Ottomata wrote:

what happens in general when a stream name is re-used with a different, potentially incompatible schema.

Bad stuff, it won't work, unless we do the work to totally purge any previous artifacts of that stream, like its Hive table.

Thoughts https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To#Decommissioning including a bit on filing a Phab request for Analytics to delete the associated tables from both databases when removing a stream from the config?

Should streams specify owners in their configs, especially for cases when the person/team who created the stream is not the same as the person who removes it? Or, say Analytics removes a stream because the system is getting flooded. Would y'all use git blame to figure out who added that stream and then check with them how they want to handle retention/dropping of any collected data? Seems we should coordinate on a policy for that stuff.

Also this conversation has gone way off-topic from the actual phab task and we should continue it elsewhere.

In T259917#6380640, @mpopov wrote:

Should streams specify owners in their configs, especially for cases when the person/team who created the stream is not the same as the person who removes it? Or, say Analytics removes a stream because the system is getting flooded. Would y'all use git blame to figure out who added that stream and then check with them how they want to handle retention/dropping of any collected data? Seems we should coordinate on a policy for that stuff.

This is a larger issue and an important one but we should create a new ticket about tracking/identifying data responsibility that can capture this as well as other issues around this.

This is one of the user stories listed in T205319: Modern Event Platform: Stream Configuration, so indeed stream config is probably the place to do this. However, we should probably coordinate with some bigger data governance projects that both PA and AE have mentioned wanting to work on. E.g. where does this type of information fit in if we were to use something like Apache Atlas? Will we build some kind of EventStreamConfig + Atlas (or whatever) integration?

Change 622881 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/extensions/EventStreamConfig@master] Recursively set array type 'assoc' on ApiStreamConfig results

https://gerrit.wikimedia.org/r/622881

Change 622881 merged by Ottomata:
[mediawiki/extensions/EventStreamConfig@master] Recursively set array type 'assoc' on ApiStreamConfig results

https://gerrit.wikimedia.org/r/622881