Page MenuHomePhabricator

De-symlink our repos
Open, Needs TriagePublic

Description

Just adding this as another task in the sprint as our symlinks create both daily complexity & docker related complexity - this task spin-off came our of tech talk & it makes sense

Note this is actually important for our civiproxy docker specifically.

The directories involved are

  • modules (this is a convenience symlink & is the low hanging fruit)
  • civicrm
  • sites

in order to fix the last of these we need to convert the drupal directory from a submodule to a normal folder. Note that we don't care much about the history of the existing submodule because we basically alternate 2 commits in it

  1. drupal stock commit
  2. re-apply wmff patches - eg https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/drupal/+/661276

(some of those patches might be unnecessary without the symlinks)

@Cstone I updated ^^

Details

Show related patches Customize query in gerrit

Event Timeline

Change 661480 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] One symlink down

https://gerrit.wikimedia.org/r/661480

Change 661509 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Remove civicrm symlink

https://gerrit.wikimedia.org/r/661509

Change 661538 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Convert drupal submodule to being 'just a folder'

https://gerrit.wikimedia.org/r/661538

Change 661541 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Convert drupal submodule to being 'just a folder'

https://gerrit.wikimedia.org/r/661541

Change 661538 merged by Eileen:
[wikimedia/fundraising/crm@master] Convert drupal submodule to being 'just a folder'

https://gerrit.wikimedia.org/r/661538

Change 661541 abandoned by Eileen:
[wikimedia/fundraising/crm@master] Convert drupal submodule to being 'just a folder'

Reason:
otherwise merged

https://gerrit.wikimedia.org/r/661541

Change 661826 had a related patch set uploaded (by Eileen; owner: Eileen):
[integration/config@master] Remove drupal submodule from docker zuul coner

https://gerrit.wikimedia.org/r/661826

Change 661828 had a related patch set uploaded (by Eileen; owner: Eileen):
[integration/config@master] Attempt to switch to recurse_submodules

https://gerrit.wikimedia.org/r/661828

Change 661829 had a related patch set uploaded (by Eileen; owner: Eileen):
[integration/config@master] Switch fundraising tech setup to use recurse_submodules

https://gerrit.wikimedia.org/r/661829

@Cstone & I forced the change to the drupal repo through just now in the hope the jenkins problem was just trying to find it's way around applying the changes (it appeared to have issues with the remove & re-add).

However, that was not joyous and I believe that we need this merged https://gerrit.wikimedia.org/r/c/integration/config/+/661829 before it will work. I'm going to revert the change in the meantime & try to get someone in releng to +2

Change 661829 abandoned by Eileen:
[integration/config@master] Switch fundraising tech setup to use recurse_submodules

Reason:
https://gerrit.wikimedia.org/r/c/integration/config/ /661829 instead

https://gerrit.wikimedia.org/r/661829

Change 661826 abandoned by Eileen:
[integration/config@master] Remove drupal submodule from docker zuul coner

Reason:
https://gerrit.wikimedia.org/r/c/integration/config/ /661829 instead

https://gerrit.wikimedia.org/r/661826

Change 661829 restored by Eileen:
[integration/config@master] Switch fundraising tech setup to use recurse_submodules

https://gerrit.wikimedia.org/r/661829

Change 661828 abandoned by Eileen:
[integration/config@master] Attempt to switch to recurse_submodules

Reason:
https://gerrit.wikimedia.org/r/c/integration/config/ /661829/2/jjb/wm-fundraising.yaml

https://gerrit.wikimedia.org/r/661828

Change 661480 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] One symlink down

https://gerrit.wikimedia.org/r/661480

Change 661509 abandoned by Eileen:
[wikimedia/fundraising/crm@master] Remove civicrm symlink

Reason:
Will redo once preliminary steps are done (reduce review noise)

https://gerrit.wikimedia.org/r/661509

Change 664359 had a related patch set uploaded (by Eileen; owner: Eileen):
[integration/config@master] Remove reference to submodule

https://gerrit.wikimedia.org/r/664359

Change 664360 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Convert drupal submodule to being 'just a folder'

https://gerrit.wikimedia.org/r/664360

I put up https://gerrit.wikimedia.org/r/c/integration/config/+/664359 to remove reference to the drupal modules - that needs releng (@hashar ) to merge it - once merged we need to merge https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/664360 and then we will no longer need the drupal repo (which can then be fully removed from the ci config & 'closed' whatever that means)

I'm going to dig into the other parts of this once that is merged - it's a trivial change but it requires a change outside our control to be co-ordinated with one in our control.

Note this change supercedes my efforts to switch to using submodules. The main reason for that approach was that it would make the co-ordination simpler but it turned out to be fatally flawed

Mentioned in SAL (#wikimedia-releng) [2021-02-16T19:58:22Z] <hashar> Updating Jenkins job wikimedia-fundraising-civicrm-docker to stop cloning the drupal repository # T273822

Change 664359 merged by jenkins-bot:
[integration/config@master] Remove reference to submodule

https://gerrit.wikimedia.org/r/664359

Change 664360 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] Convert drupal submodule to being 'just a folder'

https://gerrit.wikimedia.org/r/664360

Change 664929 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] De-symlink sites folder

https://gerrit.wikimedia.org/r/664929

Change 664946 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Commit harvey-dent with correct executable file mode

https://gerrit.wikimedia.org/r/664946

Change 664948 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm/civicrm-buildkit@master] Remove hardver dent chmod in wmff specific

https://gerrit.wikimedia.org/r/664948

Change 664946 merged by Eileen:
[wikimedia/fundraising/crm@master] Commit harvey-dent with correct executable file mode

https://gerrit.wikimedia.org/r/664946

Change 664948 merged by Eileen:
[wikimedia/fundraising/crm/civicrm-buildkit@master] Remove hardver dent chmod in wmff specific

https://gerrit.wikimedia.org/r/664948

Change 664960 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Use full path to drupal in preparation for desymlinking

https://gerrit.wikimedia.org/r/664960

Change 661829 abandoned by Eileen:
[integration/config@master] Switch fundraising tech setup to use recurse_submodules

Reason:
alternate approach used

https://gerrit.wikimedia.org/r/661829

Change 664960 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] Use full path to drupal in preparation for desymlinking

https://gerrit.wikimedia.org/r/664960

Change 667727 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] De-symlink sites folder

https://gerrit.wikimedia.org/r/667727

Change 667728 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Update gitignore for sites move

https://gerrit.wikimedia.org/r/667728

Change 667966 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Make path relative

https://gerrit.wikimedia.org/r/667966

Change 667728 abandoned by Eileen:
[wikimedia/fundraising/crm@master] Update gitignore for sites move

Reason:

https://gerrit.wikimedia.org/r/667728

Change 667966 abandoned by Eileen:
[wikimedia/fundraising/crm@master] Make path relative

Reason:

https://gerrit.wikimedia.org/r/667966

Change 667727 merged by Eileen:
[wikimedia/fundraising/crm@master] De-symlink sites folder

https://gerrit.wikimedia.org/r/667727

I just added this into the sprint as we had un-reviewed work from when it was in the sprint that we got merged - some challenges around deployment & we can't deploy anything else unit resolved

Notes of things to do when we push to live

  • (can be done in preparation ) usedrush vget & drush vset to update exchange_rates_google_account_file_path
  • move/copy civicrm.settings.php & settings.php to the new location
  • recreate files symlink
  • drush cc all
  • drush cvapi System.flush

I wound up getting it going again on staging by

  1. fully removing sites folder
  2. it then worked but later stopped -so as a stop gap I put back a symlink but going the reverse way - this seems OK as a transitional measure to me but I'm not sure how we manage in terms of deployment
  3. another cache clear in the UI
  4. removed a dashlet that seemed broken - I think that might be unrelated

@Ejegg @Dwisehaupt

Change 669973 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Update gitignore for sites move

https://gerrit.wikimedia.org/r/669973

Change 670018 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Update directory path

https://gerrit.wikimedia.org/r/670018

Change 669973 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] Update gitignore for sites move

https://gerrit.wikimedia.org/r/669973

Change 670018 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] Update directory path and geocoder

https://gerrit.wikimedia.org/r/670018

Change 672513 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Move CiviCRM folder to the 'real' location

https://gerrit.wikimedia.org/r/672513

Change 664929 abandoned by Eileen:
[wikimedia/fundraising/crm@master] De-symlink sites folder

Reason:

https://gerrit.wikimedia.org/r/664929

Change 673177 had a related patch set uploaded (by Cstone; owner: Cstone):
[mediawiki/vagrant@master] Update paths for symlink removal.

https://gerrit.wikimedia.org/r/673177

Change 673177 merged by jenkins-bot:
[mediawiki/vagrant@master] Update paths for symlink removal.

https://gerrit.wikimedia.org/r/673177

Change 672513 abandoned by Eileen:

[wikimedia/fundraising/crm@master] Move CiviCRM folder to the 'real' location

Reason:

Need to re-do 'at the time'

https://gerrit.wikimedia.org/r/672513