Some topics to cover in the SHDT meeting that I feel we should seek to answer before building any dashboards:

• Do we understand the existing dashboard?
• What are the problems/limitations the existing dashboard?
• Do we have a shared understanding of what a performance budget means?
• How do we enforce a performance budget within our team?
• How do we enforce a performance budget outside our team?
• Are dashboards useful for a performance budget?
• Are alerts useful for a performance budget?

In T345997#9161967, @Jdlrobson wrote:

Some topics to cover in the SHDT meeting that I feel we should seek to answer before building any dashboards:

• Do we understand the existing dashboard?

Yes But we need to document the current state https://phabricator.wikimedia.org/T345990

• What are the problems/limitations the existing dashboard?

Mostly alerts rules, and organisation, and might need a couple of extra charts or visuals depending on the documentation and analysis.
Check subtask of https://phabricator.wikimedia.org/T337404

• Do we have a shared understanding of what a performance budget means?

• A performance budget is an estimate of expected (Goal) future performance metric values for a set period of time.
• Like a normal budget these needs recurring updated over a set periods of time, or can be mathematically auto update and correct itself over time.
• A performance budget can be set as a max value to reach, or can be a percentage over or under a mathematical calculation over previous data set.
• A performance budget has 1 alert that you exceeded the budget assigned.
• Performance regression alerts is not part of the budget alert, but surely work in toward the same goal.
• There is 2 sets of performance budget that we need to define:
  • General longterm goal budget: this is the target that we aim to get to even if it is somewhat not possible short term. Maybe a yearly or twice a year check and revision. This should have a lower level alert if any.
  • Specific short term budget: this is a target that we aim for each release or month, it should be revised on a schedule or automated. This should have a higher level alert.

• How do we enforce a performance budget within our team?

Enforcement can be achieved by:

• Communicating the budget to everyone within our team to align on goals.
• CI pipe lines for our team's projects should include more robust checks for the budgets. ( we should try to improve and experiment with the size based ones as they are easier to run with each commit, maybe more CI automated testing can be done pre or post submit)

• How do we enforce a performance budget outside our team?

• Same as we do within the team for our projects.
• For more general ones we can maybe elevate or integrate the CI to run on a core level and take into account all loaded js and css not just per skin/extension budgets.

• Are dashboards useful for a performance budget?

• grafana Dashboards and charts in general are good to visualise a situation, they do not stop us or allow us to define and implement a budget.

• Are alerts useful for a performance budget?

• grafana Alerts can be useful to check regressions in performance and a deviation from a budget.
• A budget should guide the alert behaviour and alerts should be updated or be smart enough to follow the budget guide lines.

After a discussion within the web team in the SHDT, we concluded that:

• The current Dashboard needs some organisation
• No need for extra dashboards.
• New charts could be added to the current board https://grafana-rw.wikimedia.org/d/000000205/reading-web-performance?orgId=1

This task is a sanity check over the dashboard and is not blocking to any other tasks related to performance budgeting.