Page MenuHomePhabricator
Create Task
Maniphest T390736

Create a container image containing the necessary sync utilities for publishing dump files
Closed, ResolvedPublic
Actions

Description

The dumps running on the snapshot servers use rsync to publish the data files from the intermediate storage (dumpsdata) servers to the distribution (clouddumps) servers.

Once we have the dumps running on Kubernetes, we will also need to have a means to publish them in the same way.

This might be rsync, but we might also want to experiment with some other tools, such as lsyncd or parsyncfp2 (which is only available in trixie).

To begin with, we will need an image that includes these sync-utils, so that we can experiment.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Add changelog to trigger build and publish pipelinerepos/data-engineering/sync-utils!2btullisbump_buildmain
Add data-engineering/sync-utils to the trusted runnersrepos/releng/gitlab-trusted-runner!111btullisadd_sync_utilsmain
Add the initial build pipeline and sync utilitiesrepos/data-engineering/sync-utils!1btullisinitial_pipelinemain
Customize query in GitLab

Related Objects
Search...

Event Timeline

BTullis created this task.Apr 1 2025, 2:41 PM
BTullis triaged this task as High priority.
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.
CodeReviewBot added a project: Patch-For-Review.Apr 3 2025, 9:41 AM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/sync-utils/-/merge_requests/1

Add the initial build pipeline and sync utilities

CodeReviewBot added a comment.Apr 3 2025, 10:22 AM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/sync-utils/-/merge_requests/1

Add the initial build pipeline and sync utilities

Maintenance_bot removed a project: Patch-For-Review.Apr 3 2025, 10:30 AM
CodeReviewBot added a project: Patch-For-Review.Apr 3 2025, 10:31 AM

btullis opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/111

Add data-engineering/refinery to the trusted runners

CodeReviewBot added a comment.Apr 3 2025, 10:40 AM

btullis updated https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/111

Add data-engineering/sync-utils to the trusted runners

BTullis moved this task from In Progress to Needs Review on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Apr 3 2025, 10:44 AM

I have created a basic image to start with.
Currently awaiting a review from RelEng so that it can be added to the trusted runners.

I have decided to leave parsyncfp2 and hdfs-rsync out of the image for the moment, since rsync is really what we need to begin testing.

Maintenance_bot removed a project: Patch-For-Review.Apr 3 2025, 11:30 AM
CodeReviewBot added a comment.Apr 3 2025, 2:43 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/111

Add data-engineering/sync-utils to the trusted runners

CodeReviewBot added a project: Patch-For-Review.Apr 7 2025, 1:31 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/sync-utils/-/merge_requests/2

Add changelog to trigger build and publish pipeline

CodeReviewBot added a comment.Apr 7 2025, 1:57 PM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/sync-utils/-/merge_requests/2

Add changelog to trigger build and publish pipeline

BTullis closed this task as Resolved.Apr 7 2025, 1:59 PM
BTullis moved this task from Needs Review to Done on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.

This is now complete.

btullis@barracuda:~$ docker run -it docker-registry.wikimedia.org/repos/data-engineering/sync-utils:2025-04-07-135737-7291841795ffa86db311e48e39684e0faf9e36b1@sha256:097a5bba821bce585815d9837eeeb20a113b1cd54745bc823adda69ae3792b20
Unable to find image 'docker-registry.wikimedia.org/repos/data-engineering/sync-utils:2025-04-07-135737-7291841795ffa86db311e48e39684e0faf9e36b1@sha256:097a5bba821bce585815d9837eeeb20a113b1cd54745bc823adda69ae3792b20' locally
docker-registry.wikimedia.org/repos/data-engineering/sync-utils@sha256:097a5bba821bce585815d9837eeeb20a113b1cd54745bc823adda69ae3792b20: Pulling from repos/data-engineering/sync-utils
cfba3dca3abc: Pull complete 
7173becacac9: Pull complete 
1c5f71d8ae47: Pull complete 
dc98657bbee5: Pull complete 
Digest: sha256:097a5bba821bce585815d9837eeeb20a113b1cd54745bc823adda69ae3792b20
Status: Downloaded newer image for docker-registry.wikimedia.org/repos/data-engineering/sync-utils@sha256:097a5bba821bce585815d9837eeeb20a113b1cd54745bc823adda69ae3792b20
runuser@2d0272461ebe:/home/sync-utils$ rsync --version
rsync  version 3.2.7  protocol version 32
Copyright (C) 1996-2022 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
    hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, ACLs,
    xattrs, optional secluded-args, iconv, prealloc, stop-at, no crtimes
Optimizations:
    SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 sha1 none
Compress list:
    zstd lz4 zlibx zlib none
Daemon auth list:
    sha512 sha256 sha1 md5 md4

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.
runuser@2d0272461ebe:/home/sync-utils$
brouberol moved this task from Done to Reported on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Apr 10 2025, 1:02 PM
brouberol moved this task from Reported to Done on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Apr 10 2025, 1:05 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Apr 11 2025, 9:15 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits