Page MenuHomePhabricator

Puppetize setup on upgrader-06.wmflabs
Open, Needs TriagePublic

Description

I've manually set up upgrader-06.eqiad.wmflabs for now, we should puppetize it.

Event Timeline

Legoktm created this task.Aug 17 2017, 12:32 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 17 2017, 12:32 AM

Change 372213 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] [WIP] Add libraryupgrader puppet module

https://gerrit.wikimedia.org/r/372213

Change 372213 merged by Rush:
[operations/puppet@production] wmcs tenant: add libraryupgrader puppet configuration

https://gerrit.wikimedia.org/r/372213

Finally got time to test this out...

/var/log/puppet.log
Error: /usr/bin/git  clone -b master https://gerrit.wikimedia.org/r/labs/libraryupgrader /srv/libraryupgrader returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Wmcs::Tenants::Libraryupgrader/Git::Clone[labs/libraryupgrader]/Exec[git_clone_labs/libraryupgrader]/returns: change from notrun to 0 failed: /usr/bin/git  clone -b master https://gerrit.wikimedia.org/r/labs/libraryupgrader /srv/libraryupgrader returned 1 instead of one of [0]

Not sure how to see the actual failure message though?

This is on a fresh instance, upgrader-03.wmflabs.eqiad.

bd808 added a subscriber: bd808.Dec 21 2017, 12:57 AM
$ sudo -i puppet agent -tv --debug
...
Debug: Exec[git_pull_labs/libraryupgrader](provider=shell): Executing check '["/bin/sh", "-c", "/usr/bin/git fetch && /usr/bin/git diff --quiet remotes/origin/master"]'
Debug: Executing with uid=libraryupgrader gid=libraryupgrader: '/bin/sh -c /usr/bin/git fetch && /usr/bin/git diff --quiet remotes/origin/master'
Debug: Exec[git_pull_labs/libraryupgrader](provider=shell): Executing '["/bin/sh", "-c", "/usr/bin/git  pull --quiet"]'
Debug: Executing with uid=libraryupgrader gid=libraryupgrader: '/bin/sh -c /usr/bin/git  pull --quiet'
Error: /usr/bin/git  pull --quiet returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Wmcs::Tenants::Libraryupgrader/Git::Clone[labs/libraryupgrader]/Exec[git_pull_labs/libraryupgrader]/returns: change from notrun to 0 failed: /usr/bin/git  pull --quiet returned 1 instead of one of [0]

If I wipe out the manual git clone in /srv/libraryupgrader then it fails even earlier on git operations:

Debug: Exec[git_clone_labs/libraryupgrader](provider=shell): Executing '["/bin/sh", "-c", "/usr/bin/git  clone -b master https://gerrit.wikimedia.org/r/labs/libraryupgrader /srv/libraryupgrader"]'
Debug: Executing with uid=libraryupgrader gid=libraryupgrader: '/bin/sh -c /usr/bin/git  clone -b master https://gerrit.wikimedia.org/r/labs/libraryupgrader /srv/libraryupgrader'
Error: /usr/bin/git  clone -b master https://gerrit.wikimedia.org/r/labs/libraryupgrader /srv/libraryupgrader returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Wmcs::Tenants::Libraryupgrader/Git::Clone[labs/libraryupgrader]/Exec[git_clone_labs/libraryupgrader]/returns: change from notrun to 0 failed: /usr/bin/git  clone -b master https://gerrit.wikimedia.org/r/labs/libraryupgrader /srv/libraryupgrader returned 1 instead of one of [0]

A manual run of this appears to work:

$ sudo -iu libraryupgrader /usr/bin/git  clone -b master https://gerrit.wikimedia.org/r/labs/libraryupgrader /srv/libraryupgrader
Cloning into '/srv/libraryupgrader'...
remote: Counting objects: 52, done
remote: Finding sources: 100% (29/29)
remote: Getting sizes: 100% (12/12)
remote: Compressing objects: 100% (19294/19294)
remote: Total 315 (delta 12), reused 306 (delta 9)
Receiving objects: 100% (315/315), 71.03 KiB | 0 bytes/s, done.
Resolving deltas: 100% (182/182), done.
Checking connectivity... done.

But following that we are right back to:

Debug: Exec[git_pull_labs/libraryupgrader](provider=shell): Executing check '["/bin/sh", "-c", "/usr/bin/git fetch && /usr/bin/git diff --quiet remotes/origin/master"]'
Debug: Executing with uid=libraryupgrader gid=libraryupgrader: '/bin/sh -c /usr/bin/git fetch && /usr/bin/git diff --quiet remotes/origin/master'
Debug: Exec[git_pull_labs/libraryupgrader](provider=shell): Executing '["/bin/sh", "-c", "/usr/bin/git  pull --quiet"]'
Debug: Executing with uid=libraryupgrader gid=libraryupgrader: '/bin/sh -c /usr/bin/git  pull --quiet'
Error: /usr/bin/git  pull --quiet returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Wmcs::Tenants::Libraryupgrader/Git::Clone[labs/libraryupgrader]/Exec[git_pull_labs/libraryupgrader]/returns: change from notrun to 0 failed: /usr/bin/git  pull --quiet returned 1 instead of one of [0]

The /bin/sh -c /usr/bin/git fetch && /usr/bin/git diff --quiet remotes/origin/master guard here should be exiting with status 0, so something is fundamentally wrong with the Puppet managed git operations as the libraryupgrader user. What exactly is wrong however is not obvious to me yet.

Legoktm renamed this task from Puppetize setup on upgrader-02.wmflabs to Puppetize setup on upgrader-05.wmflabs.Jul 17 2019, 3:09 AM
Legoktm removed a project: Patch-For-Review.
Legoktm updated the task description. (Show Details)
Legoktm renamed this task from Puppetize setup on upgrader-05.wmflabs to Puppetize setup on upgrader-06.wmflabs.Feb 5 2020, 12:05 AM
Legoktm updated the task description. (Show Details)

Change 570169 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] Remove initial attempt at libraryupgrader puppetization

https://gerrit.wikimedia.org/r/570169

Change 570169 merged by Dzahn:
[operations/puppet@production] Remove initial attempt at libraryupgrader puppetization

https://gerrit.wikimedia.org/r/570169

Legoktm removed Legoktm as the assignee of this task.May 21 2020, 11:33 PM
Legoktm claimed this task.Jun 24 2020, 9:09 AM
Legoktm added a subscriber: Dzahn.

Going to take a stab at this again.

@Dzahn: I'll be basing this off the codesearch puppetization we did. I added you to the library-upgrader project as well in case you can help out again :)

Change 607452 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] Add initial puppetization for libraryupgrader

https://gerrit.wikimedia.org/r/607452

Once ^ is merged, I created an upgrader-07 instance to test with.

After that patch, the next step is to migrate over the systemd units.

Change 607919 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] libraryupgrader: Add systemd units

https://gerrit.wikimedia.org/r/607919

Change 607919 merged by Dzahn:
[operations/puppet@production] libraryupgrader: Add systemd units

https://gerrit.wikimedia.org/r/c/operations/puppet/ /607919