SystemdUnitFailed - wmde-analytics-daily-early.service
Open, Needs TriagePublicBUG REPORT
Actions

Assigned To

None

Authored By

	dr0ptp4kt
	Mar 17 2025, 11:48 AM

Description

What kind of problem are you reporting?

Access related problem
Service related problem
Data related problem

For a service related problem:

Discussion on this was opened in Slack. It has a relation to T387514: Allow analytics-wmde-users journalctl access to their systemd timers running on stat1011 as well.

What is the nature of the issue?

There are several things going on here.

The SRE Observability and Data Platform Alerts mailing lists have been receiving messages with subject:"SystemdUnitFailed data-platform (stat1011:9100 wmde-analytics-daily-early.service ops critical eqiad prometheus)".

Here's the related journalctl output:

P74237 SystemdUnitFailed - wmde-analytics-daily-early.service March 2025

1	stat1011:~$ sudo journalctl -u wmde-analytics-daily-early.service --no-pager
2	-- Journal begins at Thu 2025-03-13 18:39:07 UTC, ends at Fri 2025-03-14 10:52:27 UTC. --
3	Mar 13 23:00:08 stat1011 systemd[1]: wmde-analytics-daily-early.service: start operation timed out. Terminating.
4	Mar 13 23:00:08 stat1011 systemd[1]: wmde-analytics-daily-early.service: Main process exited, code=killed, status=15/TERM
5	Mar 13 23:00:08 stat1011 systemd[1]: wmde-analytics-daily-early.service: Failed with result 'timeout'.
6	Mar 13 23:00:08 stat1011 systemd[1]: Failed to start Daily jobs for wmde analytics infrastructure.
7	Mar 13 23:00:08 stat1011 systemd[1]: wmde-analytics-daily-early.service: Consumed 5min 56.980s CPU time.
8	Mar 14 03:00:07 stat1011 systemd[1]: Starting Daily jobs for wmde analytics infrastructure...
9	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131582]: + '[' -z /srv/analytics-wmde/graphite/src/scripts ']'
10	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131582]: + date '+%F %T daily.03.sh Started!'
11	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131591]: 2025-03-14 03:00:07 daily.03.sh Started!
12	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131592]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/datamodel/sitelinks_per_item.php
13	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131592]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/datamodel/sitelinks_per_item.php
14	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131593]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/datamodel/statements_per_entity.php
15	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131593]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/datamodel/statements_per_entity.php
16	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131594]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/datamodel/sitelinks_per_site.php
17	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131594]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/datamodel/sitelinks_per_site.php
18	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131596]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/datamodel/properties_by_datatype.php
19	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131596]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/datamodel/properties_by_datatype.php
20	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131598]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/ranks.php
21	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131598]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/ranks.php
22	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131600]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/constraintsByType.php
23	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131600]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/constraintsByType.php
24	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131602]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/lexemes.php
25	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131602]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/lexemes.php
26	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131605]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/dumpScanProcessing.php
27	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131605]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/dumpScanProcessing.php
28	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131607]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/entityUsage.php
29	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131607]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/entityUsage.php
30	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131608]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/facebook.php
31	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131608]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/facebook.php
32	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131610]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/identica.php
33	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131610]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/identica.php
34	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131612]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/irc.php
35	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131612]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/irc.php
36	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131614]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/mail.php
37	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131614]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/mail.php
38	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131617]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/newsletter.php
39	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131617]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/newsletter.php
40	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131618]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/techmail.php
41	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131618]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/techmail.php
42	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131620]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/twitter.php
43	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131620]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/twitter.php
44	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131623]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/good_articles.php
45	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131623]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/good_articles.php
46	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131625]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/total_edits.php
47	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131625]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/total_edits.php
48	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131626]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/total_pages.php
49	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131626]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/total_pages.php
50	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131629]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/active_users.php
51	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131629]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/active_users.php
52	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131631]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/active_users_by_namespace.php
53	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131631]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/active_users_by_namespace.php
54	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131632]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/active_items.php
55	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131632]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/active_items.php
56	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131635]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/users.php
57	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131635]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/users.php
58	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131637]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/user_groups.php
59	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131637]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/user_groups.php
60	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131638]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/rolling_rc.php
61	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131638]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/rolling_rc.php
62	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131641]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/pages_by_namespace.php
63	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131642]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/page_size.php
64	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131641]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/pages_by_namespace.php
65	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131642]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/page_size.php
66	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131644]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/user_languages.php
67	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131644]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/user_languages.php
68	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131647]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/lexemes.php
69	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131647]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/lexemes.php
70	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131649]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/recent_changes_by_namespace.php
71	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131649]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/recent_changes_by_namespace.php
72	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131651]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/recent_changes_by_tags.php
73	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131651]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/recent_changes_by_tags.php
74	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131653]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/pagelinks_to_namespaces.php
75	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131653]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/pagelinks_to_namespaces.php
76	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131655]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikibase/dockerStats.php
77	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131655]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikibase/dockerStats.php
78	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131656]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/phabricatorTasks.php
79	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131656]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/phabricatorTasks.php
80	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131658]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/showcaseItems.php
81	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131658]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/showcaseItems.php
82	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131582]: + date '+%F %T daily.03.sh Waiting!'
83	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131661]: + eval /srv/analytics-wmde/graphite/src/scripts/src/wikidata/dumpDownloads.php
84	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131661]: ++ /srv/analytics-wmde/graphite/src/scripts/src/wikidata/dumpDownloads.php
85	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131663]: 2025-03-14 03:00:07 daily.03.sh Waiting!
86	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131582]: + wait
87	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131609]: 2025-03-14 03:00:07 wikidata-dumpScanProcessing Script Started!
88	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131634]: 2025-03-14 03:00:07 wikidata-site_stats-active_users_by_namespace Script Started!
89	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131619]: 2025-03-14 03:00:07 wikidata-social-mail Script Started!
90	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131603]: 2025-03-14 03:00:07 wikidata-sparql-ranks Script Started!
91	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131601]: 2025-03-14 03:00:07 wikidata-datamodel-properties_by_datatype Script Started!
92	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131599]: 2025-03-14 03:00:07 wikidata-datamodel-sitelinks_per_site Script Started!
93	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131595]: 2025-03-14 03:00:07 wikidata-datamodel-sitelinks_per_item Script Started!
94	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131606]: 2025-03-14 03:00:07 wikidata-sparql-lexemes Script Started!
95	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131639]: 2025-03-14 03:00:07 wikidata-site_stats-users Script Started!
96	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131646]: 2025-03-14 03:00:07 wikidata-site_stats-pages_by_namespace Script Started!
97	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131609]: 2025-03-14 03:00:07 wikidata-dumpScanProcessing Not many output dirs found!
98	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131609]: 2025-03-14 03:00:07 wikidata-dumpScanProcessing Script Finished!
99	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131613]: 2025-03-14 03:00:07 wikidata-social-facebook Script Started!
100	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131597]: 2025-03-14 03:00:07 wikidata-datamodel-statements_per_entity Script Started!
101	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131611]: 2025-03-14 03:00:07 wikidata-entityUsage Script Started!
102	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131615]: 2025-03-14 03:00:07 wikidata-social-identica Script Started!
103	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131621]: 2025-03-14 03:00:07 wikidata-social-newsletter Script Started!
104	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131604]: 2025-03-14 03:00:07 wikidata-sparql-constraint-snaks Script Started!
105	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131659]: 2025-03-14 03:00:07 wikidata-phabricatorTasks Script Started!
106	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131627]: 2025-03-14 03:00:07 wikidata-site_stats-good_articles Script Started!
107	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131633]: 2025-03-14 03:00:07 wikidata-site_stats-active_users Script Started!
108	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131652]: 2025-03-14 03:00:07 wikidata-site_stats-recent_changes_by_namespace Script Started!
109	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131648]: 2025-03-14 03:00:07 wikidata-site_stats-user_languages Script Started!
110	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131630]: 2025-03-14 03:00:07 wikidata-site_stats-total_pages Script Started!
111	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131643]: 2025-03-14 03:00:07 wikidata-site_stats-rolling_rc Script Started!
112	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131654]: 2025-03-14 03:00:07 wikidata-site_stats-recent_changes_by_tags Script Started!
113	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131657]: 2025-03-14 03:00:07 wikidata-site_stats-pagelinks_to_namespaces Script Started!
114	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131645]: 2025-03-14 03:00:07 wikidata-site_stats-page_size Script Started!
115	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131664]: 2025-03-14 03:00:07 wikidata-dumpDownloads Script Started!
116	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131664]: 2025-03-14 03:00:07 wikidata-dumpDownloads Targeting date: 10/Mar/2025
117	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131660]: 2025-03-14 03:00:07 wikibase-dockerStats Script Started!
118	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131660]: 2025-03-14 03:00:07 wikibase-dockerStats Running for: wikibase/wikibase
119	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131628]: 2025-03-14 03:00:07 wikidata-site_stats-total_edits Script Started!
120	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131662]: 2025-03-14 03:00:07 wikidata-showcaseItems Script Started!
121	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131622]: 2025-03-14 03:00:07 wikidata-social-techmail Script Started!
122	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131650]: 2025-03-14 03:00:07 wikidata-site_stats-lexemes Script Started!
123	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131636]: 2025-03-14 03:00:07 wikidata-site_stats-active_items Script Started!
124	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131624]: 2025-03-14 03:00:07 wikidata-social-twitter Script Started!
125	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131640]: 2025-03-14 03:00:07 wikidata-site_stats-user_groups Script Started!
126	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131616]: 2025-03-14 03:00:07 wikidata-social-irc Script Started!
127	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131622]: 2025-03-14 03:00:07 wikidata-social-techmail Script Finished!
128	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131616]: 2025-03-14 03:00:07 wikidata-social-irc Script Finished!
129	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131621]: 2025-03-14 03:00:07 wikidata-social-newsletter Script Finished!
130	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131662]: 2025-03-14 03:00:07 wikidata-showcaseItems Script Finished!
131	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131660]: 2025-03-14 03:00:07 wikibase-dockerStats Sending data for: wikibase/wikibase
132	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131660]: 2025-03-14 03:00:07 wikibase-dockerStats Running for: wikibase/wdqs
133	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131660]: 2025-03-14 03:00:07 wikibase-dockerStats Sending data for: wikibase/wdqs
134	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131660]: 2025-03-14 03:00:07 wikibase-dockerStats Running for: wikibase/wdqs-frontend
135	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131660]: 2025-03-14 03:00:07 wikibase-dockerStats Sending data for: wikibase/wdqs-frontend
136	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131624]: 2025-03-14 03:00:07 wikidata-social-twitter Script Finished!
137	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131657]: 2025-03-14 03:00:07 wikidata-site_stats-pagelinks_to_namespaces DB query failed:
138	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131657]: 2025-03-14 03:00:07 wikidata-site_stats-pagelinks_to_namespaces array (
139	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131657]: 0 => '42S22',
140	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131657]: 1 => 1054,
141	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131657]: 2 => 'Unknown column \'pl_namespace\' in \'SELECT\'',
142	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131657]: )
143	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131657]: 2025-03-14 03:00:07 wikidata-site_stats-pagelinks_to_namespaces Skipping!
144	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131657]: 2025-03-14 03:00:07 wikidata-site_stats-pagelinks_to_namespaces Script Finished!
145	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131627]: 2025-03-14 03:00:07 wikidata-site_stats-good_articles Script Finished!
146	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131660]: 2025-03-14 03:00:07 wikibase-dockerStats Running for: wikibase/wdqs-proxy
147	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131630]: 2025-03-14 03:00:07 wikidata-site_stats-total_pages Script Finished!
148	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131640]: 2025-03-14 03:00:07 wikidata-site_stats-user_groups Running query for admins group
149	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131639]: 2025-03-14 03:00:07 wikidata-site_stats-users Script Finished!
150	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131611]: 2025-03-14 03:00:07 wikidata-entityUsage EntityUsage DB query failed for # NOTE: This file is automatically generated. Do not edit it directly, run 'composer manage-dblist' instead., Skipping!!
151	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131611]: 2025-03-14 03:00:07 wikidata-entityUsage EntityUsage page DB query failed for # NOTE: This file is automatically generated. Do not edit it directly, run 'composer manage-dblist' instead., Skipping!!
152	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131640]: 2025-03-14 03:00:07 wikidata-site_stats-user_groups Running query for bureaucrats group
153	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131660]: 2025-03-14 03:00:07 wikibase-dockerStats Sending data for: wikibase/wdqs-proxy
154	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131613]: PHP Notice: Undefined offset: 1 in /srv/analytics-wmde/graphite/src/scripts/src/wikidata/social/facebook.php on line 32
155	Mar 14 03:00:07 stat1011 wmde-analytics-daily-early[4131613]: 2025-03-14 03:00:07 wikidata-social-facebook Script Finished!
156	Mar 14 03:00:08 stat1011 wmde-analytics-daily-early[4131628]: 2025-03-14 03:00:08 wikidata-site_stats-total_edits Script Finished!
157	Mar 14 03:00:08 stat1011 wmde-analytics-daily-early[4131603]: 2025-03-14 03:00:08 wikidata-sparql-ranks Script Finished!
158	Mar 14 03:00:08 stat1011 wmde-analytics-daily-early[4131619]: 2025-03-14 03:00:08 wikidata-social-mail Script Finished!
159	Mar 14 03:00:08 stat1011 wmde-analytics-daily-early[4131640]: 2025-03-14 03:00:08 wikidata-site_stats-user_groups Running query for bots group
160	Mar 14 03:00:08 stat1011 wmde-analytics-daily-early[4131640]: 2025-03-14 03:00:08 wikidata-site_stats-user_groups Running query for propertycreators group
161	Mar 14 03:00:08 stat1011 wmde-analytics-daily-early[4131640]: 2025-03-14 03:00:08 wikidata-site_stats-user_groups Script Finished!
162	Mar 14 03:00:08 stat1011 wmde-analytics-daily-early[4131660]: 2025-03-14 03:00:08 wikibase-dockerStats Script Finished!
163	Mar 14 03:00:11 stat1011 wmde-analytics-daily-early[4131664]: 2025-03-14 03:00:11 wikidata-dumpDownloads Script Finished!
164	Mar 14 03:00:11 stat1011 wmde-analytics-daily-early[4131615]: 2025-03-14 03:00:11 wikidata-social-identica Script Finished!
165	Mar 14 03:00:13 stat1011 wmde-analytics-daily-early[4131601]: 2025-03-14 03:00:13 wikidata-datamodel-properties_by_datatype Script Finished!
166	Mar 14 03:00:13 stat1011 wmde-analytics-daily-early[4131659]: 2025-03-14 03:00:13 wikidata-phabricatorTasks Script Finished!
167	Mar 14 03:00:13 stat1011 wmde-analytics-daily-early[4131650]: 2025-03-14 03:00:13 wikidata-site_stats-lexemes Script Finished!
168	Mar 14 03:00:14 stat1011 wmde-analytics-daily-early[4131604]: 2025-03-14 03:00:14 wikidata-sparql-constraint-snaks Script Finished!
169	Mar 14 03:00:14 stat1011 wmde-analytics-daily-early[4131652]: 2025-03-14 03:00:14 wikidata-site_stats-recent_changes_by_namespace Script Finished!
170	Mar 14 03:00:16 stat1011 wmde-analytics-daily-early[4131643]: 2025-03-14 03:00:16 wikidata-site_stats-rolling_rc Script Finished!
171	Mar 14 03:00:52 stat1011 wmde-analytics-daily-early[4131636]: 2025-03-14 03:00:52 wikidata-site_stats-active_items Script Finished!
172	Mar 14 03:01:08 stat1011 wmde-analytics-daily-early[4131633]: 2025-03-14 03:01:08 wikidata-site_stats-active_users Script Finished!
173	Mar 14 03:01:18 stat1011 wmde-analytics-daily-early[4131646]: 2025-03-14 03:01:18 wikidata-site_stats-pages_by_namespace Script Finished!
174	Mar 14 03:01:34 stat1011 wmde-analytics-daily-early[4131634]: 2025-03-14 03:01:34 wikidata-site_stats-active_users_by_namespace Script Finished!
175	Mar 14 03:03:55 stat1011 wmde-analytics-daily-early[4131595]: 2025-03-14 03:03:55 wikidata-datamodel-sitelinks_per_item Script Finished!
176	Mar 14 03:03:57 stat1011 wmde-analytics-daily-early[4131654]: 2025-03-14 03:03:57 wikidata-site_stats-recent_changes_by_tags Script Finished!
177	Mar 14 03:05:58 stat1011 wmde-analytics-daily-early[4131645]: 2025-03-14 03:05:58 wikidata-site_stats-page_size Script Finished!
178	Mar 14 03:06:02 stat1011 wmde-analytics-daily-early[4131648]: 2025-03-14 03:06:02 wikidata-site_stats-user_languages Script Finished!
179	Mar 14 03:07:13 stat1011 wmde-analytics-daily-early[4131599]: 2025-03-14 03:07:13 wikidata-datamodel-sitelinks_per_site Script Finished!
180	Mar 14 03:10:52 stat1011 wmde-analytics-daily-early[4131606]: PHP Fatal error: Uncaught TypeError: Return value of WikimediaSparql::query() must be of the type array, null returned in /srv/analytics-wmde/graphite/src/scripts/lib/WikimediaSparql.php:44
181	Mar 14 03:10:52 stat1011 wmde-analytics-daily-early[4131606]: Stack trace:
182	Mar 14 03:10:52 stat1011 wmde-analytics-daily-early[4131606]: #0 /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/lexemes.php(172): WikimediaSparql::query()
183	Mar 14 03:10:52 stat1011 wmde-analytics-daily-early[4131606]: #1 /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/lexemes.php(145): WikidataLexemes->queryCountsAndSendToGraphite()
184	Mar 14 03:10:52 stat1011 wmde-analytics-daily-early[4131606]: #2 /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/lexemes.php(35): WikidataLexemes->countRepresentationsByLanguageCode()
185	Mar 14 03:10:52 stat1011 wmde-analytics-daily-early[4131606]: #3 /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/lexemes.php(13): WikidataLexemes->execute()
186	Mar 14 03:10:52 stat1011 wmde-analytics-daily-early[4131606]: #4 {main}
187	Mar 14 03:10:52 stat1011 wmde-analytics-daily-early[4131606]: thrown in /srv/analytics-wmde/graphite/src/scripts/lib/WikimediaSparql.php on line 44
188	Mar 14 03:35:11 stat1011 wmde-analytics-daily-early[4131597]: 2025-03-14 03:35:11 wikidata-datamodel-statements_per_entity Script Finished!

At the top of it notice it says Failed to start Daily jobs for wmde analytics infrastructure., which seems to possibly have to do with a timeout.
Additionally, toward the bottom there's Unknown column \'pl_namespace\' in \'SELECT\''
Moreover, toward the bottom there's PHP Fatal error: Uncaught TypeError: Return value of WikimediaSparql::query() must be of the type array, null returned in /srv/analytics-wmde/graphite/src/scripts/lib/WikimediaSparql.php:44

Regarding pl_namespace, @Lucas_Werkmeister_WMDE noted that https://gerrit.wikimedia.org/g/analytics/wmde/scripts/+/bb7be5bff2/src/wikidata/site_stats/sql/pagelinks_to_namespaces.sql should be updated in alignment with https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/KWWEWNFVXDXOHZBJO32NQLBKC5LUTEAB/ .

@BTullis noted that metrics should be migrated away from graphite. @karapayneWMDE noted there's an intent to migrate.

Continued troubleshooting in T387514: Allow analytics-wmde-users journalctl access to their systemd timers running on stat1011 ( @BTullis and @Lucas_Werkmeister_WMDE ) for access. Probably configuration of team-level destination for the job should be incorporated there as well?
Task to address next steps for pagelinks migration (T389341: @AndrewTavis_WMDE)
Task to migrate pertinent pieces away from Graphite by the 15-April-2025 deadline as noted in T228380#5519657 (T389344: @AndrewTavis_WMDE)
Task to treat PHP fatal for WikimediaSparql::query() (T389342: @karapayneWMDE ?)

Related Objects
Search...

Status	Subtype	Assigned	Task
Open	BUG REPORT	None	T389061 SystemdUnitFailed - wmde-analytics-daily-early.service
Open		None	T389341 pagelinks_to_namespaces.sql migration
Open		None	T389342 Treat PHP fatal for WikimediaSparql::query()
Open		HasanAkgun_WMDE	T389344 analytics/wmde/scripts Graphite to Prometheus migration
Resolved		fgiunchedi	T392599 Make Prometheus statsd-exporter available to analytics/wmde/scripts running on stat1011

Event Timeline

dr0ptp4kt created this task.Mar 17 2025, 11:48 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 17 2025, 11:48 AM

BTullis edited projects, added Data-Platform-SRE (2025.03.01 - 2025.03.21), WMDE-Analytics-Engineering; removed Data-Platform.Mar 17 2025, 12:10 PM

@dr0ptp4kt: Hi, which URL and/or form was used to create this issue? Asking as it's setting an archived unused project tag.

In T389061#10641342, @Aklapper wrote:

@dr0ptp4kt: Hi, which URL and/or form was used to create this issue? Asking as it's setting an archived unused project tag.

I think it may have been this one: https://phabricator.wikimedia.org/maniphest/task/edit/form/110/
...which is linked from the menu bar for the Data-Engineering project: https://phabricator.wikimedia.org/project/view/5462/

Sorry about that @Aklapper - over in T388861#10641349 I'm asking product management and engineering management if the form's subscribers and tags can be updated.

Lucas_Werkmeister_WMDE updated the task description. (Show Details)Mar 17 2025, 2:22 PM

Lucas_Werkmeister_WMDE removed a subscriber: LucasWerkmeister.

AndrewTavis_WMDE added a project: Wikidata Analytics (Kanban).Mar 18 2025, 1:20 PM

Maintenance_bot added a project: Wikidata.Mar 18 2025, 1:30 PM

karapayneWMDE added a project: Wikidata-Omega.Mar 18 2025, 4:14 PM

karapayneWMDE moved this task from Incoming to Ready for Task Time (Story Writing) on the Wikidata-Omega board.Mar 18 2025, 4:21 PM

Lucas_Werkmeister_WMDE updated the task description. (Show Details)Mar 19 2025, 10:24 AM

AndrewTavis_WMDE mentioned this in T389341: pagelinks_to_namespaces.sql migration.Mar 19 2025, 11:06 AM

AndrewTavis_WMDE updated the task description. (Show Details)

karapayneWMDE added a subtask: T389341: pagelinks_to_namespaces.sql migration.Mar 19 2025, 11:08 AM

AndrewTavis_WMDE mentioned this in T389342: Treat PHP fatal for WikimediaSparql::query().Mar 19 2025, 11:11 AM

AndrewTavis_WMDE added a subtask: T389342: Treat PHP fatal for WikimediaSparql::query().

AndrewTavis_WMDE updated the task description. (Show Details)

Are we able to resolve this ticket now, as the systemd unit is no longer in a failed state?

btullis@stat1011:~$ systemctl --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.

Or is there still an error with the underlying function? I see this PHP error in the unit status.

btullis@stat1011:~$ systemctl status wmde-analytics-daily-early
● wmde-analytics-daily-early.service - Daily jobs for wmde analytics infrastructure
     Loaded: loaded (/lib/systemd/system/wmde-analytics-daily-early.service; static)
     Active: activating (start) since Wed 2025-03-19 03:00:01 UTC; 8h ago
TriggeredBy: ● wmde-analytics-daily-early.timer
       Docs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
   Main PID: 3370907 (time)
      Tasks: 6 (limit: 153864)
     Memory: 13.5M
        CPU: 1min 49.645s
     CGroup: /system.slice/wmde-analytics-daily-early.service
             ├─3370907 /usr/bin/time /srv/analytics-wmde/graphite/src/scripts/cron/daily.03.sh /srv/analytics-wmde/graphite/src/scripts
             ├─3370923 /bin/bash -x /srv/analytics-wmde/graphite/src/scripts/cron/daily.03.sh /srv/analytics-wmde/graphite/src/scripts
             ├─3370939 /bin/bash -x /srv/analytics-wmde/graphite/src/scripts/cron/daily.03.sh /srv/analytics-wmde/graphite/src/scripts
             ├─3370943 /usr/bin/php /srv/analytics-wmde/graphite/src/scripts/src/wikidata/entityUsage.php
             ├─3771129 sh -c echo "daily.wikidata.entity_usage_per_site.dtpwiki 11515 `date -d "2025-03-19T03:00:01+00:00" +%s`" | nc -q0 graphite-in.eqiad.wmnet 2003
             └─3771131 nc -q0 graphite-in.eqiad.wmnet 2003

Mar 19 03:08:04 stat1011 wmde-analytics-daily-early[3370932]: 2025-03-19 03:08:04 wikidata-datamodel-sitelinks_per_site Script Finished!
Mar 19 03:09:30 stat1011 wmde-analytics-daily-early[3370940]: PHP Fatal error:  Uncaught TypeError: Return value of WikimediaSparql::query() must be of the type array, null returned in /srv/analytics-wmde/graph>
Mar 19 03:09:30 stat1011 wmde-analytics-daily-early[3370940]: Stack trace:
Mar 19 03:09:30 stat1011 wmde-analytics-daily-early[3370940]: #0 /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/lexemes.php(172): WikimediaSparql::query()
Mar 19 03:09:30 stat1011 wmde-analytics-daily-early[3370940]: #1 /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/lexemes.php(145): WikidataLexemes->queryCountsAndSendToGraphite()
Mar 19 03:09:30 stat1011 wmde-analytics-daily-early[3370940]: #2 /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/lexemes.php(35): WikidataLexemes->countRepresentationsByLanguageCode()
Mar 19 03:09:30 stat1011 wmde-analytics-daily-early[3370940]: #3 /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/lexemes.php(13): WikidataLexemes->execute()
Mar 19 03:09:30 stat1011 wmde-analytics-daily-early[3370940]: #4 {main}
Mar 19 03:09:30 stat1011 wmde-analytics-daily-early[3370940]:   thrown in /srv/analytics-wmde/graphite/src/scripts/lib/WikimediaSparql.php on line 44
Mar 19 03:39:56 stat1011 wmde-analytics-daily-early[3370930]: 2025-03-19 03:39:56 wikidata-datamodel-statements_per_entity Script Finished!

BTullis moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.Mar 19 2025, 11:14 AM

Lucas_Werkmeister_WMDE mentioned this in T387514: Allow analytics-wmde-users journalctl access to their systemd timers running on stat1011.Mar 19 2025, 11:22 AM

AndrewTavis_WMDE mentioned this in T389344: analytics/wmde/scripts Graphite to Prometheus migration.Mar 19 2025, 11:26 AM

AndrewTavis_WMDE added a subtask: T389344: analytics/wmde/scripts Graphite to Prometheus migration.

AndrewTavis_WMDE updated the task description. (Show Details)

Gehel edited projects, added Data-Platform-SRE (2025.03.22 - 2025.04.11); removed Data-Platform-SRE (2025.03.01 - 2025.03.21).Mar 21 2025, 9:56 AM

Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.

karapayneWMDE removed a project: Wikidata-Omega.Wed, Apr 2, 8:31 AM

Removing DPE SRE from this task as it seems that the remaining work is done by WMDE-Analytics-Engineering. Please let us know if you need us!

karapayneWMDE added a project: Wikidata-Omega.Tue, Apr 8, 7:22 AM

karapayneWMDE moved this task from Ready for Task Time (Story Writing) to to be reused on the Wikidata-Omega board.Thu, Apr 10, 2:56 PM

karapayneWMDE moved this task from to be reused to Radar/Epics/Stalled on the Wikidata-Omega board.Thu, Apr 10, 3:24 PM