Setup periodic crawling
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• bd808
	Mar 3 2021, 11:50 PM

Description

We have a cli tool to perform a crawl, but we do not currently have anything setup to actually run that command periodically.

This also implies figuring out what we will use to schedule tasks. Local cron is not a realistic option in a k8s deployed system, but we could do:

cron on a metal server + web API
celery + celery beat
k8s scheduled jobs? (Not sure if prod k8s supports that object type or not)

Celery is probably the "best" thing for parity between dev and prod. That would in theory give us another way to scale too (horizontal worker scaling).

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	toolhub: Add CronJob for crawler	operations/deployment-charts	master	+58 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T288685 Establish active/active multi-dc support for Toolhub
Resolved	• bd808	T115650 Create an authoritative and well promoted catalog of Wikimedia tools
Resolved	• bd808	T271483 Complete and announce initial production deployment of Toolhub
Resolved	• bd808	T276405 Setup periodic crawling

Event Timeline

• bd808 created this task.Mar 3 2021, 11:50 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 3 2021, 11:50 PM

• bd808 added a parent task: T271483: Complete and announce initial production deployment of Toolhub.Jul 14 2021, 6:32 PM

Celery is probably the "best" thing for parity between dev and prod. That would in theory give us another way to scale too (horizontal worker scaling).

The more I think about this, the more it feels like a Kubernetes CronJob would actually be the simplest thing to use initially if that is supported in the prod k8s cluster.