Page MenuHomePhabricator

ORES deployments blocked by mysterious tin.eqiad.wmnet error
Closed, ResolvedPublic

Description

See the log here,

/srv/deployment/ores/deploy/scap/log/scap-sync-2018-06-06-0003-1-g65ce165.log

While trying to deploy a trivial submodule bump, I encountered stale pointers to tin, which caused the deployment to fail.

Event Timeline

awight triaged this task as High priority.Jun 7 2018, 5:28 PM
awight created this task.
thcipriani raised the priority of this task from High to Unbreak Now!.Jun 7 2018, 5:49 PM
thcipriani added a subscriber: thcipriani.

This is due to running git fetch (without --no-recurse-submodules) in the cache directory before remapping submodules to the deployment host.

That is, what we want is to:

  1. git fetch
  2. git submodule remap
  3. git submodule fetch/update

What we're doing is:

  1. git fetch recursing into submodules
  2. git submodule remap
  3. git submodule fetch/update

Patch incoming.

Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptJun 7 2018, 5:49 PM

Can we manually fix this to unbreak prior to a new scap version release?

New scap package was built and the version bump merged in puppet. The new version should be on all targets after the net puppet run which should resolve this issue.

@thcipriani Thanks! I'll retry our deployment in tonight's window and will report back either way.

awight closed this task as Resolved.Jun 12 2018, 9:03 AM
awight claimed this task.
awight moved this task from Active to Done on the Scoring-platform-team (Current) board.

It works now, thank you! Also, we're getting our LFS file loud and clear.

Change 440109 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: eqiad1: bootstrap keystone

https://gerrit.wikimedia.org/r/440109

Vvjjkkii renamed this task from ORES deployments blocked by mysterious tin.eqiad.wmnet error to ggbaaaaaaa.Jul 1 2018, 1:05 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed awight as the assignee of this task.
Vvjjkkii lowered the priority of this task from Unbreak Now! to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: Aklapper, gerritbot.
CommunityTechBot renamed this task from ggbaaaaaaa to ORES deployments blocked by mysterious tin.eqiad.wmnet error.Jul 2 2018, 12:44 PM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to awight.
CommunityTechBot raised the priority of this task from High to Unbreak Now!.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added subscribers: Aklapper, gerritbot.