Page MenuHomePhabricator

[Hypothesis] WE6.3.2 Create "standard" tool (Sample Complex Tool, SCT) to measure the number of steps for a deployment
Closed, ResolvedPublic

Description

This task is to track progress on the hypothesis 6.3.2:

By creating a "standard" tool to measure the number of steps for a deployment we will be able to assess the maximal improvement in the deployment process.

Tool to be used: https://toolsadmin.wikimedia.org/tools/id/sample-complex-app

Starting set of features

The application should show a webpage with a summary of the status:

  • Db connection working
  • Db schema upgraded to latest schema
  • Backend api up and running
  • Backend jobs running
  • Last deploy
  • Number of deploy steps needed (thigh might need to be manually entered somehow as it can't deploy itself yet)
  • Scheduled job

All components should be built with buildservice, and configured through environment variables/secrets.

Backend

Repo: https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend)

Python API e.g. FastAPI, built using build service, should:

  • Listen on port 8080
  • Use a database (trove)
    • Have some setup/db upgrade script one-offs
  • Start background tasks through celery using Redis as transport (starting with the shared instance, we might move to dedicated job later)
Worker

Repo: https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend)

Celery worker executing background tasks, it should have at least one task triggered by the API that contacts the database.

Frontend

Repo: https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend

Vue.js simple fronted, built using build service, contacts the backend for queries.

Other scheduled/maintenance jobs
  • One scheduled job rotating the logs

Future features

  • Object storage
  • Non-shared Redis (redis as a component)

Event Timeline

dcaro triaged this task as High priority.
dcaro added a subscriber: Slst2020.

Some notes from brainstorming session with @dcaro:

Sample app brainstorming

  • For now split in two repos (might enable monorepo at some point, needs work on buildpacks side)

Components

  • Web application:
    • Python API e.g. FastAPI
    • Node.js frontend library, e.g. Vue
    • Separate repos for backend and frontend to start with
  • Database
    • db management scripts (one-off jobs)
  • Worker process: e.g. Celery
  • Redis (in a container or using the shared instance?)
  • Scheduled jobs: e.g. log rotation, db updates from external source
  • Config as envvars

A later iteration might include object storage.

dcaro changed the task status from Open to In Progress.Jul 15 2024, 12:41 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 12) board.

I copied and re-ordered the points to the task so we can start working on it, will start creating subtasks

dcaro renamed this task from [Hypothesis] WE6.3.2 Create "standard" tool to measure the number of steps for a deployment to [Hypothesis] WE6.3.2 Create "standard" tool (Sample Complex Tool, SCT) to measure the number of steps for a deployment.Jul 16 2024, 4:00 PM