Page MenuHomePhabricator

[Mediawiki History] Default boolean fields to False instead of NULL where relevant
Open, Needs TriagePublic

Description

A number of the boolean fields default to null instead of false in hard-to-predict ways. I give the example below of the different user types which should always be true or false for a given edit, but sometimes are null when it's an IP editor (or a bug and no user type is assigned in a smaller number of cases). But related to T433511: [Mediawiki History] make array fields empty as opposed to null when relevant but no values present, it can complicate analyses and break assumptions when evaluating WHERE clauses to have these null values.

Example of the different user types that shows inconsistency of false vs. null:

spark.sql("""
SELECT
  event_user_is_anonymous,
  event_user_is_permanent,
  event_user_is_temporary,
  COUNT(1) AS num_instances
FROM wmf.mediawiki_history
WHERE
  snapshot = '2026-06'
  AND event_entity = 'revision'
GROUP BY
  event_user_is_anonymous,
  event_user_is_permanent,
  event_user_is_temporary
""").show(500, False)

+-----------------------+-----------------------+-----------------------+-------------+
|event_user_is_anonymous|event_user_is_permanent|event_user_is_temporary|num_instances|
+-----------------------+-----------------------+-----------------------+-------------+
|null                   |null                   |null                   |1099181      |
|false                  |null                   |null                   |1763187      |
|false                  |true                   |false                  |7603998755   |
|false                  |false                  |true                   |12249847     |
|true                   |null                   |null                   |588195546    |
+-----------------------+-----------------------+-----------------------+-------------+