While deploying cxserver I found that submodule for cxserver is not updated.
On tin:
kartik@tin:/srv/deployment/cxserver/deploy/src$ git show HEAD commit 84ad47245b278a520c77422e7aec66f84cecc557
On sca1001/sca1002 (ie target):
kartik@sca1001:/srv/deployment/cxserver/deploy/src$ git show HEAD commit a3c21d2b5c1363ade428beda2de889d4e10821e1
While main git repo (cxserver/deploy) is updated on tin and targets.
Trebuchet and submodules are known to be problematic. One thing to double-check is the setup described in https://wikitech.wikimedia.org/wiki/Labs_node_setup#Trebuchet_submodule_checkouts
Historically it has often been necessary to have a root fix checkouts on individual servers, especially on first deploy.
'checkout_submodules' => true, is set for cxserver/deploy in manifests/role/deployment.pp
Another note:
kartik@sca1002:/srv/deployment/cxserver/deploy$ git status
HEAD detached at cxserver/deploy-sync-20150211-164151
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: .gitmodules
modified: src (new commits)
no changes added to commit (use "git add" and/or "git commit -a")
kartik@sca1002:/srv/deployment/cxserver/deploy$ git diff
diff --git a/.gitmodules b/.gitmodules
index c827f8c..a5fcb08 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,3 +1,3 @@
[submodule "src"]
path = src
- url = https://gerrit.wikimedia.org/r/p/mediawiki/services/cxserver
+ url = http://tin.eqiad.wmnet/cxserver/deploy/.git/modules/src
diff --git a/src b/src
index 84ad472..a3c21d2 160000
--- a/src
+++ b/src
@@ -1 +1 @@
-Subproject commit 84ad47245b278a520c77422e7aec66f84cecc557
+Subproject commit a3c21d2b5c1363ade428beda2de889d4e10821e1And,
On tin, .git/config says
[deploy] repo-name = cxserver/deploy checkout-submodules = false
While we've set it true as per: https://git.wikimedia.org/blob/operations%2Fpuppet.git/29d75a6f5aeb8a4a442efb60ea6e21bce63cffde/manifests%2Frole%2Fdeployment.pp
For now, I think I fixed this, although Alex says there is more to it.
If this happens again, try the following on tin:
# Force puppet to reclone: cd /srv/deployment/cxserver mv deploy deploy.bad sudo puppet agent -t # or wait for puppet to run # update submodules and fix server info cd deploy git submodule update --init cd .git/modules/src && git update-server-info && cd ../../.. git deploy start git deploy sync
This seems like it may have been a problem with git deploy not knowing how to do submodules properly if the submodule is added to the repository after the repo is cloned on tin for the first time.
That seems very likely. Trebuchet is more than a bit touchy about getting origin repos set up. Historically errors like this have been fixed manually on Tin and then possibly a patch attempted to keep it from happening again but more likely just a bug filed and a workaround posted on wikitech.