Page MenuHomePhabricator

AQS pageviews/v3/top_pages_per_editor not returning data
Closed, ResolvedPublic

Description

At T414020#11922156, it was reported that the pageviews/v3/top_pages_per_editor does not return any data on or after 2025-12-01.

E.g.
https://wikimedia.org/api/rest_v1/metrics/pageviews/v3/top_pages_per_editor/12345/monthly/20251201/20260501

{
  "detail": "The date(s) you used are valid, but we either do not have data for those date(s), or the user_central_id you asked for does not have data for those dates. Please check documentation for more information",
  "method": "get",
  "status": 404,
  "title": "Not Found",
  "type": "about:blank",
  "uri": "/metrics/pageviews/v3/top_pages_per_editor/12345/monthly/20251201/20260501"
}

Contrast this to https://wikimedia.org/api/rest_v1/metrics/pageviews/v3/top_pages_per_editor/12345/monthly/20251101/20260501 which returns data for 2025-11. Months 2025-02 to 2025-11 return data, but nothing after.

Everything looks fine in Airflow. However, examining the Cassandra loading Spark job logs shows these errors:

sudo -u analytics      yarn logs -appOwner analytics -applicationId application_1773845446826_818239
[...]
26/05/01 02:29:28 INFO Native: Unable to load JNR native implementation. This could be normal if JNR is excluded from the classpath
java.lang.NoClassDefFoundError: jnr/posix/POSIXHandler
	at com.datastax.oss.driver.internal.core.os.Native$LibcLoader.load(Native.java:42)
[...]
Caused by: java.lang.ClassNotFoundException: jnr.posix.POSIXHandler
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
[...]
26/05/01 02:39:03 INFO Executor: Finished task 2.0 in stage 0.0 (TID 4). 1786 bytes result sent to driver
26/05/01 02:40:07 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

I can't tell if that error is benign or actually something wrong.

See also this Slack thread.

Event Timeline

I gave up my root access a few months ago, so I can't query cassandra directly anymore to see if the data is there. I can't find any page-analytics AQS service logs anywhere (neither in logstash nor via k8s logs).

We need @Eevans or someone with cassandra access to debug further.

Eric, the (Hive referenced) AQS cassandra table name is aqs.analytics.pageviews_top_pages_per_editor. The query that loads data looks something like:

INSERT INTO aqs.analytics.pageviews_top_pages_per_editor
SELECT 
    user_central_id,
    granularity,
    wiki_id,
    page_id,
    rank,
    top_k,
    view_count,
    dt
FROM
    wmf_readership.pageviews_top_pages_per_editor
WHERE
    dt = CAST(TO_DATE('2026-04-01', 'yyyy-MM-dd') AS TIMESTAMP)
    AND granularity = 'monthly'
;

Can you tell if there is any data in that table after 2026-11?

Oh, Eric commented in Slack. Investigating more...

So, the data is not in Cassandra. That means the cassandra_load_pageview_top_pages_per_editor_monthly.load_cassandra Airflow task is failing silently.

Will need more investigation.

Ahoelzl triaged this task as High priority.
Ahoelzl updated Other Assignee, added: amastilovic.
Ahoelzl updated Other Assignee, removed: amastilovic.

Data is available now in Cassandra / API

% curl -X GET "https://wikimedia.org/api/rest_v1/metrics/pageviews/v3/top_pages_per_editor/12345/monthly/20241201/20260601" -H "accept: application/json" | jq '.items[].timestamp'
"2025-02-01T00:00:00.000Z"
"2025-03-01T00:00:00.000Z"
"2025-04-01T00:00:00.000Z"
"2025-05-01T00:00:00.000Z"
"2025-06-01T00:00:00.000Z"
"2025-07-01T00:00:00.000Z"
"2025-08-01T00:00:00.000Z"
"2025-09-01T00:00:00.000Z"
"2025-10-01T00:00:00.000Z"
"2025-11-01T00:00:00.000Z"
"2025-12-01T00:00:00.000Z"
"2026-01-01T00:00:00.000Z"
"2026-02-01T00:00:00.000Z"
"2026-03-01T00:00:00.000Z"
"2026-04-01T00:00:00.000Z"