Change 828660 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[labs/striker@master] feat: migrate Diffusion repos to GitLab projects

https://gerrit.wikimedia.org/r/828660

Change 828660 merged by jenkins-bot:

[labs/striker@master] feat: migrate Diffusion repos to GitLab projects

https://gerrit.wikimedia.org/r/828660

Change 830252 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/puppet@production] striker: Bump deployed version to 2022-09-04-055313-production

https://gerrit.wikimedia.org/r/830252

Change 830252 merged by Andrew Bogott:

[operations/puppet@production] striker: Bump deployed version to 2022-09-04-055313-production

https://gerrit.wikimedia.org/r/830252

Problems seen attempting to migrate first repo in prod (https://phabricator.wikimedia.org/source/tool-replag/):

• If an http:// url is picked as the Diffusion origin, GitLab will reject it as an invalid git repo. I think this is actually caused by a 301 redirect to the same url with https://.
• The logic to trim the "tool-" prefix from the name trimmed one more character than desired resulting in "eplag" as the new repo name.
• AttributeError: 'dict' object has no attribute 'sync_maintainers_with_gitlab' from line 313 of striker/tools/models.py.
  • This is an obvious PEBKAC in the code where I attempt to call a method from GitlabRepo on a dict returned from the gitlab API client.

Change 830269 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[labs/striker@master] fix: correct errors in gitlab migration code

https://gerrit.wikimedia.org/r/830269

Change 830269 merged by jenkins-bot:

[labs/striker@master] fix: correct errors in gitlab migration code

https://gerrit.wikimedia.org/r/830269

Change 830275 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/puppet@production] striker: bump container version to 2022-09-06-213820-production

https://gerrit.wikimedia.org/r/830275

Change 830275 merged by Andrew Bogott:

[operations/puppet@production] striker: bump container version to 2022-09-06-213820-production

https://gerrit.wikimedia.org/r/830275

Change 830283 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[labs/striker@master] fix: call GitlabRepo._tool() as a function

https://gerrit.wikimedia.org/r/830283

@Matthewrbowker asked a great question on the mailing list about repos like https://phabricator.wikimedia.org/source/tool-xtools/ which were created via Striker but have been manually converted into mirrors of git repos hosted elsewhere (github in this particular case). The migration process should check to see if there are any URIs with "Observe" I/O before attempting a migration. Pretty much all we can do is skip migrating these repos and drop Striker's tracking of them. There is currently no active mirroring solution in GitLab to allow us to insert another repo in the chain.

Change 830283 merged by jenkins-bot:

[labs/striker@master] fix: call GitlabRepo._tool() as a function

https://gerrit.wikimedia.org/r/830283

Change 830321 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[labs/striker@master] fix: Improve GitlabRepo error handling

https://gerrit.wikimedia.org/r/830321

Change 830322 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[labs/striker@master] feat: phabricator.repository_is_mirror()

https://gerrit.wikimedia.org/r/830322

Change 830321 merged by jenkins-bot:

[labs/striker@master] fix: Improve GitlabRepo error handling

https://gerrit.wikimedia.org/r/830321

Change 830322 merged by jenkins-bot:

[labs/striker@master] feat: phabricator.repository_is_mirror()

https://gerrit.wikimedia.org/r/830322

Change 830688 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/puppet@production] striker: bump container version to 2022-09-07-191738-production

https://gerrit.wikimedia.org/r/830688

Change 830688 merged by Andrew Bogott:

[operations/puppet@production] striker: bump container version to 2022-09-07-191738-production

https://gerrit.wikimedia.org/r/830688

Change 830698 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[labs/striker@master] fix: revert to checking 'default' Diffusion io field for readwrite

https://gerrit.wikimedia.org/r/830698

Change 830698 merged by jenkins-bot:

[labs/striker@master] fix: revert to checking 'default' Diffusion io field for readwrite

https://gerrit.wikimedia.org/r/830698

Change 830699 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/puppet@production] striker: bump container version to 2022-09-07-203936-production

https://gerrit.wikimedia.org/r/830699

Change 830699 merged by Andrew Bogott:

[operations/puppet@production] striker: bump container version to 2022-09-07-203936-production

https://gerrit.wikimedia.org/r/830699

In T315706#8216251, @bd808 wrote:

@Matthewrbowker asked a great question on the mailing list about repos like https://phabricator.wikimedia.org/source/tool-xtools/ which were created via Striker but have been manually converted into mirrors of git repos hosted elsewhere (github in this particular case). The migration process should check to see if there are any URIs with "Observe" I/O before attempting a migration. Pretty much all we can do is skip migrating these repos and drop Striker's tracking of them. There is currently no active mirroring solution in GitLab to allow us to insert another repo in the chain.

This turned out to be a more widely configured option than I had initially imagined. I thought that there would be maybe 5-10 instances, but it turns out that there were 75 repos with an existing 'observe' URI configured.

While computing the results for T315706#8219116 I noticed a number of broken records/deleted repos. Here is a full report of DiffusionRepo objects that do not have a PHID matching repository here on Phabricator:

>>> for r in DiffusionRepo.objects.filter(is_mirror=False):
...     try:
...         _ = phab.get_repository_by_phid(r.phid)
...     except KeyError:
...         print(
...             r.pk, "|",
...             r.created_date.isoformat(), "|",
...             r.tool, "|",
...             r.name, "|",
...             r.phid, "|",
...             r.created_by.ldapname, "|",
...         )
...
25 | 2016-09-05T21:40:13+00:00 | extreg-wos | labs-tools-extreg-wos | PHID-SPCE-6l6g5p53yi3mypnlpxjw | Legoktm |
42 | 2016-12-08T20:20:07+00:00 | bd808-test2 | tool-bd808-test2-!T151422! | 1234 | BryanDavis |
105 | 2017-10-19T04:33:55+00:00 | recoin-sample | tool-recoin-sample | PHID-REPO-zdemhk76docpphaxdvsg | Vvekbv |
115 | 2018-01-06T16:37:46+00:00 | tenst | tool-tenst | PHID-REPO-g5mfn2lr3xanodglxsu2 | Freakencies |
121 | 2018-02-08T22:00:18+00:00 | cdnjs-beta | tool-cdnjs-beta | PHID-REPO-crrnmwfzyjr4bkx6jvsf | Bstorm |
125 | 2018-02-21T13:32:41+00:00 | alvaro | tool-alvaro | PHID-REPO-g3ublaxv6gwll6pv63tb | AlvaroMolina |
159 | 2018-05-16T03:41:04+00:00 | wikiintent | tool-wikiintent | PHID-REPO-ljdzd4hx3mwqvlrx5zrk | Kearabilwe |
193 | 2018-10-23T15:32:08+00:00 | basyounybot | tool-basyounybot-test | PHID-REPO-75as6it5xvoz5mzlr3hb | AhmedTest2019 |
285 | 2019-09-22T09:30:44+00:00 | swviewer | tool-swviewer | PHID-REPO-sjzreg7vfk65ae5766i3 | Iluvatar |
355 | 2020-06-26T06:53:14+00:00 | ket-bot | tool-ket-bot-v3 | PHID-REPO-bxqomtpkyghwexh7ivps | Evrifaessa |
397 | 2021-03-04T12:32:46+00:00 | mikasa-bot | tool-mikasa-bot | PHID-REPO-c4rotjvqkvt3a2bjuxz2 | LClightcat |
413 | 2021-05-24T06:49:06+00:00 | ameisenbot | tool-ameisenbot | PHID-REPO-zabb3hk6phnpwp3ymp2z | Ameisenigel |
420 | 2021-07-13T21:48:36+00:00 | technischewuensche | tool-technischewuensche | PHID-REPO-h5dnngogjji7bggpzfs7 | MichaelSchoenitzer |
445 | 2022-01-02T16:50:31+00:00 | kiranbot2 | tool-kiranbot2fnr | PHID-REPO-z5slwgpsfrmnvfupcjip | Usernamekiran |
446 | 2022-01-10T08:01:18+00:00 | kiranbot3 | tool-kiranbot3sample | PHID-REPO-sh4byz7k7ux7ysnjahep | Usernamekiran |
448 | 2022-01-12T17:17:43+00:00 | majavah-test | tool-majavah-test | PHID-REPO-vdxaaszn4umqztokbdpx | Majavah |
>>>

These records have been deleted from Striker's database to simplify the final migration.

Mentioned in SAL (#wikimedia-operations) [2022-09-07T22:12:56Z] <bd808> Attempting to migrate all remaining Striker managed git repos from Diffusion to GitLab (T315706)

403 repos migrated so far. There are 51 repos remaining with various issues that need to be figured out.

25 of the repos that failed to migrate failed because the existing Diffusion repos have been marked as "inactive":

>>> from striker import phabricator
>>> phab = phabricator.Client.default_client()
>>>
>>> for r in DiffusionRepo.objects.filter(is_mirror=False):
...     repo = phab.get_repository_by_phid(r.phid)
...     if repo["fields"]["status"] == "inactive":
...         print(r.pk, "|", r.tool, "|", r.name)
...
19 | dynbot-srv2 | tool-dynbot-srv2
36 | perflogbot | tool-perflogbot
70 | panderine | tool-panderine
71 | stang | tool-stang
94 | joaquinito01 | tool-joaquinito01
98 | t1943bot | tool-t1943bot
106 | dawiki | tool-dawiki
119 | macfanbot | tool-macfanbot
130 | tombot3 | tool-tombot3
139 | zoranzoki21bot | tool-zoranzoki21bot
149 | filesearch | tool-filesearch
170 | harvesting-data-rafinery | tool-harvesting-data-rafinery
177 | listunpatrolledpages | tool-listunpatrolledpages
190 | yemen | tool-yemen-uu
204 | zoranzoki21wiki | tool-zoranzoki21wiki
244 | dibyadutta | tool-dibyaduttabook
261 | vip-space-media | tool-vip-space-media
293 | germancon-mobile | tool-germancon-mobile-xml
300 | dimastbkbot | tool-dimastbkbot
301 | dutchbot | tool-dutchbot
321 | squirrelnest-upf | tool-squirrelnest-upf
354 | ket-bot | tool-ket-bot
367 | sourceror | tool-sourceror
459 | terabot | tool-terabot
473 | pinyin-api | tool-pinyin-api

These records have been deleted from Striker's database to simplify the final migration.

Change 830708 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[labs/striker@master] fix: Use get_repository_by_phid() in GitlabRepo.create_from_diffusion()

https://gerrit.wikimedia.org/r/830708

7 repos are connected to tools which no longer exist:

>>> for r in DiffusionRepo.objects.filter(is_mirror=False):
...     try:
...         _ = Tool.objects.get(cn="tools.{}".format(r.tool))
...     except Tool.DoesNotExist:
...         print(r.pk, "|", r.tool, "|", r.name)
...
104 | www-portal-builder | tool-www-portal-builder
175 | messenger-chatbot | tool-messenger-chatbot
188 | typo-categorizer | tool-typo-categorizer
194 | basyounybot | tool-basyounybot-test1
229 | rhinosf1-afdclose | tool-rhinosf1-afdclose
230 | draft-uncategorize-script | tool-draft-uncategorize-script
308 | brandonstools | tool-brandonstools

These records have been deleted from Striker's database to simplify the final migration.

Change 830708 merged by jenkins-bot:

[labs/striker@master] fix: Use get_repository_by_phid() in GitlabRepo.create_from_diffusion()

https://gerrit.wikimedia.org/r/830708

In T315706#8219573, @gerritbot wrote:

Change 830708 merged by jenkins-bot:

[labs/striker@master] fix: Use get_repository_by_phid() in GitlabRepo.create_from_diffusion()

https://gerrit.wikimedia.org/r/830708

This allowed another 9 repos to migrate.

For posterity, here is a dump of the etherpad I used when deploying and executing this migration:

Deployment plan for Striker changes making GitLab the git repo hosting solution

• {{Done}} disable puppet on cloudweb100{3,4}
• {{Done}} Ask Andrew to merge hiera change
• {{Done}} manually run database migrations
  • $ python3 manage.py showmigrations tools
  • $ python3 manage.py migrate tools
• {{Done}} enable puppet and force run on cloudweb1003
• {{Done}} enable puppet and force run on cloudweb1004
• {{Done}} verify basic Striker functionality via browser interaction
• {{Done}} manually run license fixture load
  • $ python3 manage.py loaddata software_license.json
• {{Done}} migrate one git repo
  • $ python3 manage.py shell
  • {{Done}} ...repeat until confident that things are working...
• {{Done}} mark existing mirrors as such
• {{Done}} migrate all remaining git repos
  • $ python3 manage.py shell

Migrate one repo

from striker.tools.models import DiffusionRepo
from striker.tools.models import GitlabRepo

old_repo = DiffusionRepo.objects.get(tool="phab-ban", name="tool-phab-ban")
new_repo = GitlabRepo.create_from_diffusion(old_repo)
old_repo.convert_to_mirror()
old_repo.delete()

Mark repos that are already mirrors of a remote as such

from striker.tools.models import DiffusionRepo
from striker import phabricator

phab = phabricator.Client.default_client()
for r in DiffusionRepo.objects.exclude(gitlab__isnull=False, is_mirror=True):
    try:
        if phab.repository_is_mirror(r.phid):
            r.is_mirror = True
            r.save()
            print("[MIRROR]", r.tool, r.name)
        else:
            print("[NORMAL]", r.tool, r.name)
    except KeyError:
        print("[ERROR] Repo {} of {} not found".format(r.phid, r.tool))

Migrate all repos

from striker.tools.models import DiffusionRepo
from striker.tools.models import GitlabRepo
from striker.tools.models import Tool
from striker import gitlab
from striker import phabricator

needs_gitlab = DiffusionRepo.objects.filter(gitlab__isnull=True)
needs_gitlab = needs_gitlab.exclude(is_mirror=True)
for r in needs_gitlab:
    try:
        _ = GitlabRepo.create_from_diffusion(r)
    except gitlab.APIError as e:
        print("gitlab.APIError", e)
    except KeyError as ke:
        print("KeyError", ke)
    except Tool.DoesNotExist:
        print("Tool {} not found".format(r.tool))
    finally:
        print("REPO:", r.tool, r.name, "\n")

ready_to_mirror = DiffusionRepo.objects.exclude(gitlab__isnull=True)
ready_to_mirror = ready_to_mirror.exclude(is_mirror=True)
for r in ready_to_mirror:
    try:
        _ = r.convert_to_mirror()
    except phabricator.APIError as e:
        print("phabricator.APIError", e)
    finally:
        print("REPO:", r.tool, r.name, "\n")

DiffusionRepo.objects.filter(is_mirror=True).delete()

Mentioned in SAL (#wikimedia-operations) [2022-09-12T22:23:11Z] <mutante> phabricator - disabling repositories: tool-xh-bot, tool-editor-contribution-dashboard, tool-ranker, tool-editor-contribution, tool-mikasa-bot-1, tool-maintun, tool-add-text, tool-wikibookassamese-book.php (none of them had commits) T296022 - T315706

Mentioned in SAL (#wikimedia-operations) [2022-09-12T22:53:29Z] <mutante> phabricator - disabling MediaWiki extension repositories in Diffusion that have 0 commits - T296022 - T315706