Page MenuHomePhabricator

ORES deployments blocked by mysterious tin.eqiad.wmnet error
Closed, ResolvedPublic

Description

See the log here,

/srv/deployment/ores/deploy/scap/log/scap-sync-2018-06-06-0003-1-g65ce165.log

While trying to deploy a trivial submodule bump, I encountered stale pointers to tin, which caused the deployment to fail.

Event Timeline

awight created this task.Jun 7 2018, 5:28 PM
awight triaged this task as High priority.
thcipriani raised the priority of this task from High to Unbreak Now!.Jun 7 2018, 5:49 PM
thcipriani added a subscriber: thcipriani.

This is due to running git fetch (without --no-recurse-submodules) in the cache directory before remapping submodules to the deployment host.

That is, what we want is to:

  1. git fetch
  2. git submodule remap
  3. git submodule fetch/update

What we're doing is:

  1. git fetch recursing into submodules
  2. git submodule remap
  3. git submodule fetch/update

Patch incoming.

Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptJun 7 2018, 5:49 PM

Can we manually fix this to unbreak prior to a new scap version release?

New scap package was built and the version bump merged in puppet. The new version should be on all targets after the net puppet run which should resolve this issue.

@thcipriani Thanks! I'll retry our deployment in tonight's window and will report back either way.

awight closed this task as Resolved.Jun 12 2018, 9:03 AM
awight moved this task from Active to Done on the Scoring-platform-team (Current) board.
awight claimed this task.

It works now, thank you! Also, we're getting our LFS file loud and clear.

Change 440109 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: eqiad1: bootstrap keystone

https://gerrit.wikimedia.org/r/440109

Vvjjkkii renamed this task from ORES deployments blocked by mysterious tin.eqiad.wmnet error to ggbaaaaaaa.
Vvjjkkii reopened this task as Open.
Vvjjkkii lowered the priority of this task from Unbreak Now! to High.
Vvjjkkii removed awight as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: Aklapper, gerritbot.
CommunityTechBot assigned this task to awight.
CommunityTechBot raised the priority of this task from High to Unbreak Now!.
CommunityTechBot closed this task as Resolved.
CommunityTechBot renamed this task from ggbaaaaaaa to ORES deployments blocked by mysterious tin.eqiad.wmnet error.
CommunityTechBot added subscribers: Aklapper, gerritbot.