Summary of data QA for the data collected by Jan 19, 2024.

QAed Schemas:
Old: mediawiki_web_ui_scroll
New: mediawiki_web_ui_scroll_migrated

Questions to confirm with engineers

1. The number of events, sessions and pages are slightly higher in the new schema. Is it expected?
2. Which field is to capture Spider user agent?
3. Is access_method captured in agent.client_platform_family in the new schema?
4. Please review the field mapping table below and confirm whether all entries are as expected.

QA details

What has been checked	Status	Note	Snapshot of the result from the old schema	Snapshot of the result from the new schema
Pick one session_id, compare the result	✅Pass	Same number of events are captured
By date	❓	The events are available since 2024-01-08. The number of events, sessions and pages are slightly higher in new schema. In average, 0.18% more events daily in new schema; 0.19% more sessions daily in new schema.
By action	✅Pass	Both schemas only captured scroll-to-top action
By wiki	✅Pass	Both schemas captured events from 518 wikis on 2024-01-18
By skin name	✅Pass	Both schemas only captured events in vector-2022 skin
By user type	✅Pass	Old schema captures the status of being an anonymous user, while the new schema captures the status of being a logged in users.
By access method	❓To check with engineer	Is agent.client_platform_family the field in the new schema?
By agent type	❓To check with engineer	Not captured in the new schema.

The above summary is also documented in https://docs.google.com/spreadsheets/d/1cR2R2K54tzkHiIdUoQta53YbQapVm3YVN5KIemgmjzA/edit?usp=sharing .

In T352342#9474097, @jwang wrote:

Questions to confirm with engineers

1. The number of events, sessions and pages are slightly higher in the new schema. Is it expected?

I see the following discrepancies in counts:

I would expect both streams to have very nearly almost the same number of events, sessions, and pages because the original and migrated instruments execute on the same pageviews, one after the other. I would expect a small number of events to be dropped because variations in network conditions out of our control. Would that explain all of the 0.2% discrepancy? I'm not sure. I'd be happy to discuss this further.

2. Which field is to capture Spider user agent?

When I checked, both streams have the same number of events with lower(user_agent_map['device_family']) = 'spider' or lower(user_agent_map['device_family'] = 'bot' [4][5].

3. Is access_method captured in agent.client_platform_family in the new schema?

Yes. When agent.client_platform=mediawiki_js, agent.client_platform_family will be either desktop_browser or mobile_browser.

4. Please review the field mapping table below and confirm whether all entries are as expected.

👍

select
  count(*) as n,
  count(distinct web_session_id) as n_distinct_sessions
from
  event.mediawiki_web_ui_scroll
where
  year = 2024
  and month = 1
  and day > 16 and day < 24
;

|n|n_distinct_sessions|
|---|---|
|11447909|6047393|

```sql
select
  count(*) as n,
  count(distinct performer.session_id) as n_distinct_sessions
from
  event.mediawiki_web_ui_scroll_migrated
where
  year = 2024
  and month = 1
  and day > 16 and day < 24
;

|n|n_distinct_sessions|
|---|---|
|11471797|6060790|

select
  count(distinct page_id) as n_distinct_pages
from
  event.mediawiki_web_ui_scroll
where
  year = 2024
  and month = 1
  and day > 16 and day < 24
;

|n_distinct_pages|
|---|
|2303725|

select
  count(distinct page.id) as n_distinct_pages
from
  event.mediawiki_web_ui_scroll_migrated
where
  year = 2024
  and month = 1
  and day > 16 and day < 24
;

|n_distinct_pages|
|---|
|2306791|

select
  count(*) as n
from
  event.mediawiki_web_ui_scroll
where
  year = 2024
  and month = 1
  and day > 16 and day < 24
  
  and (
    lower(user_agent_map['device_family']) = 'spider'
    or lower(user_agent_map['device_family']) = 'bot'
  )
;

|n|
|---|
|13|

select
  count(1) as n
from
  event.mediawiki_web_ui_scroll_migrated
where
  year = 2024
  and month = 1
  and day > 16 and day < 24
  
  and (
    lower(user_agent_map['device_family']) = 'spider'
    or lower(user_agent_map['device_family']) = 'bot'
  )
;

|n|
|---|
|13|

@phuedx, Thanks for resolving all the questions. I will further investigate the remaining question of why the numbers of events, sessions and pages are slightly higher in the new schema. Will bring it up to you when I have more data.

@phuedx, Here are some findings from my investigation.

Takeaways

• The data discrepancy is mainly due to the difference in number of unique sessions. Some sessions are captured in the new instrument but not captured in the old one. For example, session '46261717108c4df8c582'. See the query in the cell L3 and M3 at tab by_wiki in [1].
• By browser family, the data discrepancy in unique sessions are mainly from Firefox browser family. See the result and query at tab by_browser_family in [1].
• By OS family, the data discrepancy in unique sessions are mainly from windows, which is the prevalent OS. While other OS, like Linux and Ubuntu, also exhibit a higher different rate, these OS variations are observed from smaller groups of sessions. See the result and query at tab by_os_family in [1]
• By wiki, did not observe a pattern by wiki/wiki family. See the result and query at tab by_wiki in [1]

[1] https://docs.google.com/spreadsheets/d/18-ZNWXwiIanAZrCFVLaORzNEhfLdHUexqIpo70hpiKE/edit?usp=sharing

Question/hypothesis for discussion:
I remember in this schema only the scroll actions that meet specific conditions are counted as 'scroll-to-top' events to avoid capturing random scroll actions. Is it possible that these events are more easily triggered in the new instrument compared to the old one?
Is it possible that the new instrument is correct as it captured more sessions?

I am happy to discuss and explore further on it.

Thanks for checking on it. Regarding 0.2% discrepancy, it can be marked as PASS given 1) it's within variance range , 2.5% variance for daily events across all wikis, that we defined in Metrics Platform Instrument Migration Data QA Process Description ; 2) the new instrumentation is capturing more unique sessions than old instrumentation.

Hi, @KSarabia-WMF , Can you confirm if below sample rate captured in the new schema is correct?

Query

SELECT distinct sample, performer.is_logged_in
FROM  mediawiki_web_ui_scroll_migrated
WHERE year=2024 and month=3 and day=1

Hi @jwang
Sorry for the delay.
Yes, this is correct, but I don't believe we decided as a group if we wanted to maintain the sampling rates that were set in the old web_ui_scroll schema or not.
Question for you-->What would best support your work? Make the new migrated event's sampling consistent with the old event or not?
If we want the new migrated scroll event to match the sample rates of the old events, I can make a ticket to make that adjustment. Just let me know.
Currently the new migrated scroll schema is set to 100%.
The old sampling rate (non-migrated) is currently:

• 'default' configuration: the sampling rate is set to 0.1/10% of anonymous users will be captured.
• 'enwiki' configuration: the sampling rate is set to 0.01/ only 1% of events

@phuedx does this sound right? Or should the config vars have applied to both instruments (current + migrated)?

@KSarabia-WMF, thanks for checking. Can you also clarify what's the sample rate for logged-in users?

Based on the sample rates of anonymous users mentioned above, it appears something is not right. The two instruments have different sample rate, but captured a similar amount of data.

	Old	New
sample rate	10% of anonymous users	100% sample rate
result

query_old <- 
"
SELECT is_anon, COUNT(1) AS events,
COUNT(DISTINCT web_session_id) AS sessions,
COUNT(DISTINCT page_id) AS pages
FROM event.mediawiki_web_ui_scroll
WHERE year=2024 AND month=1 AND day=18
GROUP BY is_anon
"
query_new <- 
"
SELECT performer.is_logged_in, COUNT(1) AS events,
COUNT(DISTINCT performer.session_id) AS sessions,
COUNT(DISTINCT page.id) AS pages
FROM event.mediawiki_web_ui_scroll_migrated
WHERE year=2024 AND month=1 AND day=18
GROUP BY  performer.is_logged_in
"

In T352342#9614206, @jwang wrote:

Based on the sample rates of anonymous users mentioned above, it appears something is not right. The two instruments have different sample rate, but captured a similar amount of data.

That appears to be a bug or a shortcoming in the Metrics Platform JS client. The stream is currently figured to sample at a rate of 100% of all pageviews but the instrument is configured to execute on 10% of all pageviews.

This is a fairly common pattern – not executing an instrument for various performance reasons – and so could/should be handled by the Metrics Platform client. I'll write up a task…

Actually… We can save some time and just remove the sampling config from the stream config (per https://gitlab.wikimedia.org/repos/data-engineering/metrics-platform/-/blob/main/js/src/SamplingController.js?ref_type=heads#L27):

Change 1009718 had a related patch set uploaded (by Phuedx; author: Phuedx):

[operations/mediawiki-config@master] ext-EventStreamConfig: Remove mediawiki.web_ui_scroll_migrated sampling config

https://gerrit.wikimedia.org/r/1009718

@phuedx Thank you!

Change #1009718 merged by jenkins-bot:

[operations/mediawiki-config@master] ext-EventStreamConfig: Remove mediawiki.web_ui_scroll_migrated sampling config

https://gerrit.wikimedia.org/r/1009718

Mentioned in SAL (#wikimedia-operations) [2024-03-21T21:06:18Z] <cjming@deploy1002> Started scap: Backport for [[gerrit:1009718|ext-EventStreamConfig: Remove mediawiki.web_ui_scroll_migrated sampling config (T352342)]]

Mentioned in SAL (#wikimedia-operations) [2024-03-21T21:08:45Z] <cjming@deploy1002> cjming and phuedx: Backport for [[gerrit:1009718|ext-EventStreamConfig: Remove mediawiki.web_ui_scroll_migrated sampling config (T352342)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-03-21T21:20:42Z] <cjming@deploy1002> Finished scap: Backport for [[gerrit:1009718|ext-EventStreamConfig: Remove mediawiki.web_ui_scroll_migrated sampling config (T352342)]] (duration: 14m 24s)

As a followup, I have documented sample rate at data hub.

QA code has been cleaned and uploaded to gitlab. gitlab link