zz
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Jan 24 2024, 1:57 PM

Description

When looking at All-Users.git I have found a bunch of obsolete references under refs/starred-changes/xx/yyyyxx/zz which were to track which files got reviewed. The last change having such entry is 709572 made in August 2021. That seems to match our upgrade to Gerrit 3.3 ( T262241#7256043 ).

My guess is tracking reviewed files got moved out of the git repository. We should be able to garbage collect any reference pointing to an object having reviewed/.

git fetch origin --prune +refs/starred-changes/*:refs/remotes/origin/starred-changes/*
git ls-remote . refs/remotes/origin/starred-changes/*|sort -g -t/ -k6|egrep -v '^(8485e986e458a566e6f6160f71d704edc10c57fc|ce7b81997cf51342dedaeccb071ce4ba3ed0cf52)'|less|wc -l
1568

8485e986e458a566e6f6160f71d704edc10c57fc	`ignore`
ce7b81997cf51342dedaeccb071ce4ba3ed0cf52	`star`

The checksum are for git blobs:

star

$ echo -en 'blob 4\0star'|shasum 
ce7b81997cf51342dedaeccb071ce4ba3ed0cf52  -

ignore

$ echo -en 'blob 6\0ignore'|shasum 
8485e986e458a566e6f6160f71d704edc10c57fc  -

Though some entries have both star AND reviewed/XXX so it is not that straight forward :]

Related Objects

Mentioned In: T355529: Investigate Gerrit email delay following 3.7.6 upgrade
Mentioned Here: P55739 clean-starred-changes.py
T262241: Upgrade Gerrit to 3.3

Event Timeline

hashar created this task.Jan 24 2024, 1:57 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2024, 1:57 PM

The star set to ignore is I think removed entirely in Gerrit 3.7 via Change 343774 and the release note:

Delete ignored state of changes and ‘star:’ queries
The query predicates star:ignored, is:ignored and star:star are not supported anymore. The latter is identical to is:starred or has:star.

Thus I guess they can be dropped as well.

I think a way to achieve it is to:

crawl through the reference
show the pointed object then
- if it contains star then
  - update the reference to point to ce7b81997cf51342dedaeccb071ce4ba3ed0cf52 (star)
- else delete it

I also note @thcipriani mentioned the starred changes issue in a September 2020 blog post https://tylercipriani.com/blog/2020/09/22/migrating-git-data-with-rsync/#notedb-and-things-to-know which stated we had 22 000 references there.

hashar updated the task description. (Show Details)Jan 24 2024, 2:15 PM

Mentioned in SAL (#wikimedia-releng) [2024-01-26T13:44:54Z] <hashar> gerrit: deleting obsolete references in All-Users.git under refs/starred-changes/* to only keep the one having a star label # T355794

I have used a script to process the references which keeps the references pointing to star and update the refs that contain a star to only have a star label.

I tested that locally with a local copy of All-Users.git and fetching the refs with:

git fetch -pP origin  +refs/starred-changes/*:refs/starred-changes/*

The script emits git update-ref commands:

P55739 clean-starred-changes.py

1	#!/usr/bin/env python3
2
3	import subprocess
4
5	# replace with:
6	# git ls-remote ssh://gerrit.wikimedia.org:29418/All-Users.git refs/starred-changes///*
7	p = subprocess.run(
8	['git', 'for-each-ref',
9	'--format=%(objectname)%09%(refname)',
10	'refs/starred-changes///*',
11	],
12	check=True,
13	text=True,
14	capture_output=True,
15	)
16
17	known_for_deletion = set([
18	# `ignore` blob
19	'8485e986e458a566e6f6160f71d704edc10c57fc'
20	])
21
22	for line in p.stdout.rstrip().split("\n"):
23
24	# <blob sha> <refs/starred-changes///*>
25	(objectname, refname) = line.split("\t")
26
27	if objectname == 'ce7b81997cf51342dedaeccb071ce4ba3ed0cf52':
28	# We keep `star`
29	continue
30
31	if objectname in known_for_deletion:
32	print(f"git update-ref -d {refname}")
33	continue
34
35	p = subprocess.run(
36	['git', 'show', objectname],
37	check=True, text=True, capture_output=True
38	)
39	stars = p.stdout.rstrip().split("\n")
40
41	if 'star' in stars:
42	# convert it
43	print(f"git update-ref {refname} ce7b81997cf51342dedaeccb071ce4ba3ed0cf52")
44	continue
45	else:
46	known_for_deletion.add(objectname)
47	print(f"git update-ref -d {refname}")
48	continue

Eventually we are left with solely the star labels:

$ git for-each-ref refs/starred-changes/*/*/*|cut -d\  -f1|sort|uniq -c
  27656 ce7b81997cf51342dedaeccb071ce4ba3ed0cf52

We are left with solely star, which surely should be removed once a change got merged.

Change 997270 had a related patch set uploaded (by Hashar; author: Hashar):

[All-Users@refs/meta/config] Grant Administrators delete right on refs/starred-changes

https://gerrit.wikimedia.org/r/997270

Change 997270 merged by Hashar: