Page MenuHomePhabricator

Automate SQL queries for annual end-of-year Phabricator statistics/metrics on wikitech-l@
Closed, ResolvedPublic

Description

...as proposed in T334628#8851175, for Phabricator and Gerrit end-of-year stats sent to wikitech-l@ (see T326562 for 2024-01-01 edition).

Event Timeline

Aklapper created this task.

Change 923367 had a related patch set uploaded (by Aklapper; author: Aklapper):

[operations/puppet@production] Automate yearly Phabricator metrics for wikitech-l

https://gerrit.wikimedia.org/r/923367

Change 928680 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: add support for month parameter in logmail class

https://gerrit.wikimedia.org/r/928680

Change 928680 merged by Dzahn:

[operations/puppet@production] phabricator: add support for month parameter in logmail class

https://gerrit.wikimedia.org/r/928680

Change 923367 merged by Dzahn:

[operations/puppet@production] Automate yearly Phabricator metrics for wikitech-l

https://gerrit.wikimedia.org/r/923367

  • added $month parameter to phabricator::logmail class
  • deployed change above

on phab2002, it added the new script and nothing else

on phab1004, it added the script, and a timer and a service, as it should be, based on which server is the active phabricator server:

[phab1004:~] $ sudo systemctl list-timers | grep yearly_metrics
Mon 2024-01-01 00:00:00 UTC  6 months 22 days left n/a                          n/a                phabricator_stats_job_yearly_metrics.timer        phabricator_stats_job_yearly_metrics.service
[phab1004:~] $ sudo systemctl status phabricator_stats_job_yearly_metrics.service
● phabricator_stats_job_yearly_metrics.service - phabricator statistics mail - yearly_metrics
   Loaded: loaded (/lib/systemd/system/phabricator_stats_job_yearly_metrics.service; static; vendor preset: enabled)
   Active: inactive (dead)
     Docs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
[phab1004:~] $ sudo systemctl status phabricator_stats_job_yearly_metrics.timer
● phabricator_stats_job_yearly_metrics.timer - Periodic execution of phabricator_stats_job_yearly_metrics.service
   Loaded: loaded (/lib/systemd/system/phabricator_stats_job_yearly_metrics.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Fri 2023-06-09 18:40:15 UTC; 1min 27s ago
  Trigger: Mon 2024-01-01 00:00:00 UTC; 6 months 22 days left

Jun 09 18:40:15 phab1004 systemd[1]: Started Periodic execution of phabricator_stats_job_yearly_metrics.service.
[phab1004:~] $ cat /lib/systemd/system/phabricator_stats_job_yearly_metrics.timer
[Unit]
Description=Periodic execution of phabricator_stats_job_yearly_metrics.service

[Timer]
Unit=phabricator_stats_job_yearly_metrics.service
# Accuracy sets the maximum time interval around the execution time we want to allow
AccuracySec=15sec
OnCalendar=*-1-1 0:0:00
RandomizedDelaySec=0

[Install]
WantedBy=multi-user.target
[phab1004:~] $ cat /lib/systemd/system/phabricator_stats_job_yearly_metrics.service
[Unit]
Description=phabricator statistics mail - yearly_metrics
Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state

[Service]
Type=oneshot
User=root
ExecStart=/usr/local/bin/yearly_metrics.sh

Change 928897 had a related patch set uploaded (by Aklapper; author: Aklapper):

[operations/puppet@production] Followup to "Automate yearly Phabricator metrics for wikitech-l"

https://gerrit.wikimedia.org/r/928897

Change 928897 merged by Dzahn:

[operations/puppet@production] Followup to "Automate yearly Phabricator metrics for wikitech-l"

https://gerrit.wikimedia.org/r/928897

We had a small issue with the query syntax which is fixed now.

Unrelatedly some numbers still seem to be missing in the mail body.

=== Phabricator 2022 ===

*  tasks got created.
*  tasks got closed.
*  people were active in Phabricator.
* 2194 people created tasks.
* 850 people closed tasks.
* The 20 people who created the most tasks:

Change 928955 had a related patch set uploaded (by Aklapper; author: Aklapper):

[operations/puppet@production] Followfollowup to "Automate yearly Phabricator metrics for wikitech-l"

https://gerrit.wikimedia.org/r/928955

Change 928955 merged by Dzahn:

[operations/puppet@production] Followfollowup to "Automate yearly Phabricator metrics for wikitech-l"

https://gerrit.wikimedia.org/r/928955

Grmpf, still one line left to fix in the bash land of phun:

* people were active in Phabricator.

works now after follow-up fixes.:)

check your mail one more time

Grmpf, still one line left to fix in the bash land of phun:

* people were active in Phabricator.

let's just replace all "cut" with the sed pattern across the board

Change 928993 had a related patch set uploaded (by Aklapper; author: Aklapper):

[operations/puppet@production] "Automate yearly Phabricator metrics for wikitech-l": Fix var typo

https://gerrit.wikimedia.org/r/928993

Change 928993 merged by Dzahn:

[operations/puppet@production] phabricator: "Automate yearly metrics for wikitech-l": Fix var typo

https://gerrit.wikimedia.org/r/928993

Tested and looks good to me now. Also see your inbox.

I believe this is resolved.

Thanks for all the work and help so far! There's one more bug to squash for me here: "The 20 people who created the most tasks" is empty.

Aklapper changed the task status from Open to In Progress.Jun 12 2023, 7:31 PM

Change 929402 had a related patch set uploaded (by Aklapper; author: Aklapper):

[operations/puppet@production] phabricator: "Automate yearly metrics for wikitech-l": Fix var typo

https://gerrit.wikimedia.org/r/929402

Change 929402 merged by Dzahn:

[operations/puppet@production] phabricator: "Automate yearly metrics for wikitech-l": Fix var typo

https://gerrit.wikimedia.org/r/929402

check mail one more time now :)

Awesome! Looks as good and automated as it can get at this stage. @Dzahn: Huge thanks for all the help!

Reminder to myself: Create a followup task to send this once a year to more than one single person because bus factor

@Aklapper In the scripts in the puppet repo we now have your personal email address (as sender and/or recipient) in multiple places. How about this: We move it to a single location at the start of the puppet class.. and use a variable in all other places.. you get a mailman list .. we replace your personal address with the list address in that one central puppet location.. later you just follow-up with "add people to the list".

Was there another ticket to resolve for the quarterly stuff?

In the scripts in the puppet repo we now have your personal email address (as sender and/or recipient) in multiple places.

Let's split that into T338955

Was there another ticket to resolve for the quarterly stuff?

That was T337387.