Systemd services rely on cron to restart
Open, Needs TriagePublic5 Estimated Story Points
Actions

Assigned To

None

Authored By

	MusikAnimal
	Aug 24 2023, 4:36 PM

Description

We're finding the systemd services, specifically ww_events_stream.service and ww_events_stream_deletion.service, may periodically get "stuck" and never recover. All along the while WikiWho will miss every edit it's supposed to be processing, and those articles will return the "Requested data is not currently available in WikiWho database. It will be available soon." error. You can check the status of the services and it reports that all is fine, only that it noticeably it isn't actually processing any edits.

We're now relying on a cronjob to restart the services daily. This works for now, but ideally we'd figure out the root of the problem.

Related Objects

Mentioned In: T351172: WikiWho is down

Event Timeline

MusikAnimal created this task.Aug 24 2023, 4:36 PM

Restricted Application added a project: Community-Tech. · View Herald TranscriptAug 24 2023, 4:36 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

MusikAnimal moved this task from New & TBD Tickets to Needs Discussion on the Community-Tech board.Aug 24 2023, 4:42 PM

MusikAnimal triaged this task as High priority.Aug 24 2023, 4:44 PM

dmaza set the point value for this task to 5.Aug 24 2023, 5:32 PM

MusikAnimal moved this task from Needs Discussion to Up Next (June 3-21) on the Community-Tech board.Aug 24 2023, 5:37 PM

TheresNoTime claimed this task.Aug 29 2023, 10:03 AM

TheresNoTime edited projects, added Community-Tech (CommTech-Kanban); removed Community-Tech.

TheresNoTime moved this task from Ready 🎬 to In Development 💻 on the Community-Tech (CommTech-Kanban) board.Aug 29 2023, 10:05 AM

tstarling renamed this task from Systemd services abrutly failing and not restarting to Systemd services abruptly failing and not restarting.Aug 30 2023, 10:41 AM

nb. whilst debugging the following changes were made:

[celery] Set default_task_soft_time_limit to 120s
[celery] (Re)set user_task_soft_time_limit to 3600s
[flower] Set persistent to True
[flower] Add exception column

I've noticed that worker_long has failed at least once without recovering

TheresNoTime moved this task from In Development 💻 to Review/Feedback 💬 on the Community-Tech (CommTech-Kanban) board.Sep 13 2023, 9:33 AM

Hm, if worker_default/_long/_user were independent systemd services (instead of brought up together via ww_celery) we could be a little more refined in how we (potentially) restart/check services..

Samwilson moved this task from Review/Feedback 💬 to Needs Attention 👀 on the Community-Tech (CommTech-Kanban) board.Sep 20 2023, 10:37 AM

We received several complaints of WikiWho clients not working. Sure enough, the eventstream service and celery itself went down about a week ago. I've restarted all, and as a quick stopgap measure, I'm going add a cronjob to restart them daily.

Ah thanks @MusikAnimal — I've lowered the priority of this task, now that something is ensuring it restarts the service.
Shall we leave this open for investigation?

In T344936#9272352, @TheresNoTime wrote:

Ah thanks @MusikAnimal — I've lowered the priority of this task, now that something is ensuring it restarts the service.
Shall we leave this open for investigation?

I'd love to figure out what's actually going wrong, but I feel we've put enough time into this.

Let's leave it open on WikiWho but take it off the Kanban. I'll also retitle accordingly.

MusikAnimal renamed this task from Systemd services abruptly failing and not restarting to Systemd services rely on cron to restart.Oct 23 2023, 9:15 PM

MusikAnimal updated the task description. (Show Details)

TheresNoTime removed TheresNoTime as the assignee of this task.Oct 24 2023, 10:32 AM

TheresNoTime subscribed.

TheresNoTime mentioned this in T351172: WikiWho is down.Nov 14 2023, 10:29 AM

Systemd services rely on cron to restartOpen, Needs TriagePublic5 Estimated Story PointsActions

Description

Related Objects

Event Timeline

Systemd services rely on cron to restart
Open, Needs TriagePublic5 Estimated Story Points
Actions