Page MenuHomePhabricator
Create Task
Maniphest T227132

issues with artifact cache in an-coord1001
Closed, ResolvedPublic
Actions

Description

When deploying from deployment1001 to an-coord1001 we got an error cause target was full

Error looks like:

Unhandled error:
deploy-local failed: <ErrorReturnCode_11> {u'full_cmd': u'/usr/bin/git fat pull', u'stderr': u'\nrsync: write failed on "/srv/deployment/analytics/refinery-cache/revs/4e9894c0db04ee39be0f094d0ef11ebbff198834/.git/fat/objects/ecfba6edeb07a17a0d1b09981aff917e068d146c": No space left on device (28)\nrsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.2]\nrsync error: error in file IO (code 11) at io.c(1642) [sender=3.1.2]\n', u'stdout': u'receiving file list ... \n 100 files...\r182 files to consider\n003d9b4

Looks like some artifacts might not be getting deleted when thy should as our config is set to keep 2:

nuria@deploy1001:/srv/deployment/analytics/refinery/scap$ more scap.cfg
[global]
git_repo: analytics/refinery
git_deploy_dir: /srv/deployment
git_repo_user: analytics-deploy
ssh_user: analytics-deploy
server_groups: canary, default
canary_dsh_targets: target-canary
dsh_targets: targets
git_submodules: False
git_fat: True
cache_revs: 2

Does this scap config take effect in the source as well as the target of the deploy?

Details

SubjectRepoBranchLines +/-
Set deploy targets for notebook environmentanalytics/refinery/scapmaster+2 -0
Deploy to notebook hosts as a separate environment to avoid deploying artifactsanalytics/refinery/scapmaster+14 -2
Customize query in gerrit

Related Objects

Event Timeline

• Nuria created this task.Jul 2 2019, 6:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 2 2019, 6:54 PM
• Nuria added a project: Release-Engineering-Team.Jul 2 2019, 6:55 PM
Ottomata subscribed.Jul 2 2019, 6:55 PM

This also often affects other hosts with relatively small /srv partitions, like notebook* hosts.

• Nuria added a comment.Jul 2 2019, 6:55 PM

Can the release engineering chime as to whether scap config settings should also delete artifacts from the target of the deploy?

• Nuria added a subscriber: greg.Jul 2 2019, 6:56 PM

ping @greg

Milimetric assigned this task to Ottomata.Jul 8 2019, 3:37 PM
Milimetric triaged this task as High priority.
Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.
Milimetric added a project: Analytics-Kanban.
Stashbot added a comment.Jul 11 2019, 8:31 PM

Mentioned in SAL (#wikimedia-analytics) [2019-07-11T20:31:54Z] <ottomata> resized /srv on an-coord1001 from 60G to 115G - T227132

Ottomata added a comment.Jul 11 2019, 8:36 PM

I believe this happens on an-coord1001 and notebook* hosts because their /srv partitions are relatively small. When the disk fills up during scap deploy, the scap deploy aborts, and does not remove old cached deploys. Upon future successful deploys it does.

We had some room on an-coord1001's volumegroup, so I resized it from 60G to 115G. refinery is 15G, so hopefully this will be fine on an-coord1001 from now on.

On notebooks however, /srv is used for both deployment and user /home dirs. Users are often filling up /srv with their homes on notebook hosts. I'm not sure what we can do about this problem there other than implementing some usage quotas...or expanding disk space hardware...or doing a distributed Hadoop based notebook project with the goal of dropping support for individual notebook hosts.

Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.Jul 11 2019, 8:36 PM
• Nuria moved this task from Done to Ready to Deploy on the Analytics-Kanban board.Jul 12 2019, 4:19 PM
• Nuria moved this task from Ready to Deploy to In Progress on the Analytics-Kanban board.

Let's figure out how to deploy only what we need to notebook hosts

gerritbot added a comment.Jul 12 2019, 7:30 PM

Change 522542 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/scap@master] Deploy to notebook hosts as a separate environment to avoid deploying artifacts

https://gerrit.wikimedia.org/r/522542

gerritbot added a project: Patch-For-Review.Jul 12 2019, 7:30 PM
gerritbot added a comment.Jul 15 2019, 1:50 PM

Change 522542 merged by Ottomata:
[analytics/refinery/scap@master] Deploy to notebook hosts as a separate environment to avoid deploying artifacts

https://gerrit.wikimedia.org/r/522542

Ottomata added a comment.Jul 15 2019, 1:53 PM

Docs updated: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Deploy/Refinery#Deploying_to_notebook*_hosts

gerritbot added a comment.Jul 15 2019, 2:03 PM

Change 523171 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/scap@master] Set deploy targets for notebook environment

https://gerrit.wikimedia.org/r/523171

gerritbot added a comment.Jul 15 2019, 2:04 PM

Change 523171 merged by Ottomata:
[analytics/refinery/scap@master] Set deploy targets for notebook environment

https://gerrit.wikimedia.org/r/523171

Maintenance_bot removed a project: Patch-For-Review.Jul 15 2019, 2:10 PM
Ottomata added a comment.Jul 15 2019, 2:23 PM

I've manually removed a bunch of old refinery artifact jar versions from the refinery deploy on notebook hosts to free up space.

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Jul 15 2019, 2:24 PM
• mmodell subscribed.Jul 18 2019, 4:24 PM
In T227132#5301532, @Nuria wrote:

Can the release engineering chime as to whether scap config settings should also delete artifacts from the target of the deploy?

I', not 100% sure what you are asking for here but I'll take my best stab at it:

When cleaning up any old revisions from the deployment cache, scap completely removes the directory, so that should remove artifacts as well.

Does that answer your question or did I misunderstand?

• mmodell added a comment.Jul 18 2019, 4:29 PM

Does this scap config take effect in the source as well as the target of the deploy?

The cached revs are only used on the target, not on the source.

greg edited projects, added Release-Engineering-Team (Deployment services); removed Release-Engineering-Team.Jul 18 2019, 4:34 PM
Ottomata added a comment.Jul 18 2019, 4:57 PM

We're having a problem where the cached revs are not removed in the case of a scap failure. Occasionally we run out of disk space on a target, scap deploy fails, we free up space, deploy again, and then there are more cached revs than there should be.

• Nuria added a comment.EditedJul 18 2019, 5:06 PM

@mmodell: this an example of such a problem: https://phabricator.wikimedia.org/T228347

• Nuria moved this task from Done to In Progress on the Analytics-Kanban board.Jul 18 2019, 8:40 PM
Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Jul 24 2019, 5:39 PM
• Nuria closed this task as Resolved.Feb 27 2020, 7:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits