Page MenuHomePhabricator

rsync the diff since mail was held on sodium
Closed, ResolvedPublic

Details

Related Gerrit Patches:
operations/puppet : productionmailman: don't use "c" option with rsync
operations/puppet : productionmailman: also rsync qfiles
operations/puppet : productionmailman: rsync for local import
operations/puppet : productionmailman: rsync command for exim copy

Event Timeline

Dzahn created this task.Aug 24 2015, 11:50 PM
Dzahn claimed this task.
Dzahn raised the priority of this task from to High.
Dzahn updated the task description. (Show Details)
Dzahn added subscribers: ori, MZMcBride, Dzahn and 5 others.

Change 237001 had a related patch set uploaded (by Dzahn):
mailman: rsync command for exim copy

https://gerrit.wikimedia.org/r/237001

Change 237001 merged by Dzahn:
mailman: rsync command for exim copy

https://gerrit.wikimedia.org/r/237001

Change 237008 had a related patch set uploaded (by Dzahn):
mailman: rsync for local import

https://gerrit.wikimedia.org/r/237008

Change 237008 merged by Dzahn:
mailman: rsync for local import

https://gerrit.wikimedia.org/r/237008

Dzahn added a comment.Sep 9 2015, 2:01 AM

in preparation for this i already deleted a whole bunch of spam from sodium, obvious spam from gTLDs such as .racing, .space and .xyz was especially popular among the older held messages.

it is down to things that are just a few days old and not very obvious spam

Dzahn added a comment.Sep 9 2015, 3:11 PM

archives, lists, qfiles and heldmsg's transferred over

Dzahn closed this task as Resolved.Sep 9 2015, 3:11 PM
Dzahn reopened this task as Open.Sep 9 2015, 6:46 PM

Change 237299 had a related patch set uploaded (by Dzahn):
mailman: also rsync qfiles

https://gerrit.wikimedia.org/r/237299

Change 237299 merged by Dzahn:
mailman: also rsync qfiles

https://gerrit.wikimedia.org/r/237299

Dzahn added a comment.Sep 11 2015, 8:36 PM

since rsync scripts have been changed and use other options, i ran new tests:

initial run as dry-run:

sent 2038665 bytes  received 6328 bytes  29424.36 bytes/sec
total size is 2873269305  speedup is 1405.03 (DRY RUN)

real    114m25.086s
user    0m12.470s
sys     1m0.080s

intial run, real:

sent 54669718 bytes  received 40125 bytes  304790.21 bytes/sec
total size is 2848527589  speedup is 52.07

real    136m2.074s
user    12m20.380s
sys     1m20.020s

second run, about 7 hours after the last one:

sent 27587140 bytes  received 8667 bytes  73687.07 bytes/sec
total size is 2879319104  speedup is 104.34

real    105m21.346s
user    3m29.980s
sys     1m3.040s
Dzahn added a comment.Sep 14 2015, 2:52 PM

one more time, about 20 hours later

sent 5435891 bytes received 10238 bytes 37952.12 bytes/sec
total size is 2837146704 speedup is 520.95

real 101m38.155s
user 3m38.360s
sys 1m5.030s

so we're seeing static values below 2 hours but not less than an hour and forty minutes? Still seems high but if we are directly rsyncing, it would fall under a well timed window.

Dzahn closed this task as Resolved.Sep 18 2015, 3:00 PM
Dzahn set Security to None.