We've been getting regular cronspam from the kiwix rsync job on labstore 1006 and 1007
@Error: max connections (6) reached -- try again later rsync error: error starting client-server protocol (code 5) at main.c(1675) [Receiver=3.1.3]
Wen I hopped on the hosts I saw several copies running. I shot them all yesterday, but we are back to getting the same spam again today. We should find out what's up on the kiwix end; additionally the rsyncer should not start up if it is already running.
I've seen some in the past month, indeed.
@ERROR: max connections (6) reached -- try again later rsync error: error starting client-server protocol (code 5) at main.c(1675) [Receiver=3.1.3]
That was labstore1007 on Apr 14. We had one on Apr 8 as well.
Thanks for this. We should also not start a new rsync on our side if one is already running on the same host. And someone needs to doublecheck that there are timeouts so that we won't have processes hanging and never completing, although I don't think that's the case for the recent incidents.
@ArielGlenn What is the status here? We have kept a bit more slots and we have as moved to a stronger server, 2 months ago.
I don't really know, someone from WMCS should look at this and see if everything looks ok.
Has @dcaro's patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/812819 resolved this issue?
I saw two emails today with this:
@ERROR: max connections (10) reached -- try again later rsync error: error starting client-server protocol (code 5) at main.c(1675) [Receiver=3.1.3] Error while running rsync, check the logs...
So I guess not.
I last saw this on September 17th. Other than new hardware, I don't know of any other changes. I'm hopeful this may not recur.
During September we have detected that one of the mirror was full and therefore rsync client was keeping download and redownloading things. As a consequence a significant part of the bandwidth and the rsync daemon slots were always busy. We still lack of monitoring solution for this on our side honestly. We should figure a tool and a process to better monitor the server/service. If you have propositions, they are welcome.
We just had an instance of this in the past day. So it's still a thing.
Note that we did see an error message from several hosts, including labstore1007. This has the spare::system role and should not be trying to pull down anything from kiwix. Is it possible that the systemd unit responsible for that job did not get cleaned up completely before the spare role was assigned? I guess the same check could be made of labstore1006 too, just in case.
Hello Kelson, We would like to know if the space problem has been resolved as we received the same error message today.
Additionally, we'd like to know why only one of your mirror servers was full. Does it have less disk size than others? Also, If any, it would be great to know what monitoring tools you use; For instance to detect mirror downtime. Depending on what you use, new checks could be added.
I'm waiting on the OK from @BTullis and you before I actually decom the old labstores; so yes, there are some latent jobs still running there. I just now disabled the kiwix syncs so perhaps that will make things happier going forwards (assuming this issue was ever on our end).
As far I know all our mirrors work fine, @Rgaudin do you have more information why the rsync client of Wikimedia has still problems?
Additionally, we'd like to know why only one of your mirror servers was full. Does it have less disk size than others? Also, If any, it would be great to know what monitoring tools you use; For instance to detect mirror downtime. Depending on what you use, new checks could be added.
We don't manage our mirrors ourself. The mirrors are provided by charities, universities or internet providers. They mirror other stuff as well and we have no control on them. We use Mirrorbrain to manage our master server and check what our mirrors provide for files. The mirrors all using rsync to sync against our master.
That disk-full issue on a (foreign) mirror was first resolved then happened again. It was eventually fixed on October 18th.
We currently allow 10 concurrent connections to our rsync server.
As per http://download.kiwix.org/mirrors.html, we have 9 official mirrors, including the Wikimedia one.
It is still possible for anonymous user to sync anything as the rsync server is open and users are invited to.
Hannah and I talked about the issue and here's what we've come up with.
The last time we had the too many connections error was on October 18th, so that lines up nicely with your report of problems on your side.
If we start seeing this issue often, we could try a couple of things:
Right now maybe the best thing to do is leave things as they are though.
As to monitoring connections from your downstream mirrors to see if one of them is behaving badly, I don't have any suggestions for that. Maybe someone else watching this task can chime in.
Hello @Kelson,
We’ve been seeing a lot of these recently;
@ERROR: max connections (10) reached -- try again later…
Do we have this because an upstream mirror is running out of space, as it did before, or is there another issue? Could you kindly look into this for us on your end, please? Many thanks!
No, this is not due to a mirror being full. As I mentioned before, we don't have much spare slots and this is an open server. At the moment for instance, 12.138.99.218 is using one slot. @Kelson should we increase our number of slots?
@Hokwelum, we've decided to separate the official-mirrors and the public/anonymous services. This should prevent random slot grabbing by anonymous users. This would also be a safer base to investigate any additional issue in the future. @Kelson and I will be in touch with the contact person for this mirror to update the link (it's a different module).
@Hokwelum @ArielGlenn, we've updated our config and you should update yours. The rsync module has now changed from download.kiwix.org to wmf.download.kiwix.org
So reusing the example from http://download.kiwix.org/README:
rsync -vzrlptD --delete master.download.kiwix.org::download.kiwix.org/zim/ ./zim/
becomes
rsync -vzrlptD --delete master.download.kiwix.org::wmf.download.kiwix.org/zim/ ./zim/
Please note that this module is IP restricted and you should connect from 208.80.154.0/23. Please advise if this is not a correct range. I inferred it from clouddumps1001.wikimedia.org which currently resolves to 208.80.154.142 in AS14907.
We could also use the reverse entry or clouddumps*.wikimedia.org ; whatever feels less prone to change in your opinion.
Please confirm when you've switched module
If you write @ plus ERROR in the text, you are subscribing me to the bug. Please don't. To reduce mail traffic, I am not asking for excuses. Just remember not to do it again. Can you please remove me from the subscriber list? I can only use backspace to remove the last subscriber.
Change 884965 had a related patch set uploaded (by Hokwelum; author: Hokwelum):
[operations/puppet@production] Change kiwix rsync module
Hey WMCS folks, see the above patch for the change of upstream hostname for kiwix downloads, I figure these are your boxes so you ought to give the thumbs up/merge. @Andrew you have been added as a reviewer but feel free to swap anyone else in as you see fit.
Also for WMCS folks, there's a pending question about how they should filter for access to our hosts on their end; I tend to think that the wildcard hostname is not the best approach, since these servers already have a different name than the last ones. While both clouddumps hosts are in the subnet they specify, do we want to be broader than that? Is there a chance of moving them or adding new ones on a different subnet? Thanks in advance!
Change 884965 merged by Andrew Bogott:
[operations/puppet@production] Change kiwix rsync module
I merged the patch, we'll see if the jobs cheer up.
For filtering...
I'm inclined to say that clouddumps*.wikimedia.org is likely to last longer than an IP range, although I agree that nothing is certain in the long run. IPs seem likely to change sooner than hostnames, but we'll probably just have to communicate if/when something unexpected happens with hostnames.
I'm only barely following this task, but kiwix dumps just now failed again:
@ERROR: max connections (2) reached -- try again later rsync error: error starting client-server protocol (code 5) at main.c(1817) [Receiver=3.2.3] Error while running rsync, check the logs...
Ah! Kinda glad to read it because it confirms our hunch: your script misbehaves somehow and end up opening several connections: that limit of 2 only applies to you.
I've increased it to 6; please take a look at it as this is not the intended behavior, I believe.
Today it says:
@ERROR: max connections (6) reached -- try again later rsync error: error starting client-server protocol (code 5) at main.c(1817) [Receiver=3.2.3] Error while running rsync, check the logs...
So something interesting is happening!
Change 885815 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] dumps::web::fetches::kiwix: splay rsyncs so we don't overload the kiwix server
Change 885815 merged by Andrew Bogott:
[operations/puppet@production] dumps::web::fetches::kiwix: splay rsyncs so we don't overload the kiwix server
I have adjusted the runs so now there will be one run every 4 hours (and alternate download hosts). Let's see if that's slow enough.
Hey WMCS folks, we've been getting the errors again after the last patch; we decided to do some testing on the script just after the loop completes and discovered some processes lingering around in TIME-WAIT and one in FIN-WAIT1. On the remote end, these may not be deemed open connections and so might have no effect on the connection problem.
We've also noticed that the errors these days have occurred around 4 a.m. or 4 p.m. on both hosts. All successful runs take 4 to 5 minutes to complete, unless the previous run did not complete, in this case, it takes less than 20 minutes to complete.
Fri-Sat-Sun: 16.15 run, error reported from clouddumps1002
@ ERROR: max connections (6) reached -- try again later
All other runs good.
I don't see anything else happening on that host, at least in the logs, and grepping through puppet shows nothing else doing anything with kiwix, even checking for the ip address of the upstream host rather than the string "kiwix". Maybe someone on the kiwix end can keep a log of the incoming connections to our module for a day and see what there is to be seen? @Kelson what do you think?
Here's the log:
2023/02/06 12:15:03 [332] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 12:15:03 [332] building file list 2023/02/06 12:15:07 [332] sent 29894523 bytes received 74 bytes total size 2632336060003 2023/02/06 12:15:07 [333] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 12:15:07 [333] building file list 2023/02/06 12:15:08 [333] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/06 12:15:08 [334] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 12:15:09 [334] building file list 2023/02/06 12:15:10 [334] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/06 12:15:10 [335] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 12:15:11 [335] building file list 2023/02/06 12:15:12 [335] sent 11433 bytes received 29 bytes total size 8818811045 2023/02/06 12:15:13 [336] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 12:15:13 [336] building file list 2023/02/06 12:15:13 [336] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/06 12:15:14 [337] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 12:15:15 [337] building file list 2023/02/06 12:15:16 [337] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/06 12:15:16 [338] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 12:15:16 [338] building file list 2023/02/06 12:15:17 [338] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/06 12:15:18 [339] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 12:15:18 [339] building file list 2023/02/06 12:15:19 [339] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/06 20:15:02 [11473] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 20:15:02 [11473] building file list 2023/02/06 20:17:49 [11473] sent 5392483796 bytes received 304 bytes total size 2632959017646 2023/02/06 20:17:52 [11474] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 20:17:52 [11474] building file list 2023/02/06 20:17:53 [11474] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/06 20:17:53 [11475] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 20:17:53 [11475] building file list 2023/02/06 20:17:54 [11475] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/06 20:17:54 [11476] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 20:17:55 [11476] building file list 2023/02/06 20:17:55 [11476] sent 11433 bytes received 29 bytes total size 8818811045 2023/02/06 20:17:56 [11477] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 20:17:56 [11477] building file list 2023/02/06 20:17:57 [11477] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/06 20:17:57 [11478] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 20:17:57 [11478] building file list 2023/02/06 20:17:58 [11478] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/06 20:17:58 [11479] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 20:17:59 [11479] building file list 2023/02/06 20:17:59 [11479] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/06 20:18:02 [11480] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/06 20:18:02 [11480] building file list 2023/02/06 20:18:03 [11480] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/07 00:15:03 [12131] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 00:15:04 [12131] building file list 2023/02/07 00:25:05 [12131] sent 15501922979 bytes received 783 bytes total size 2632684099279 2023/02/07 00:25:06 [12132] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 00:25:06 [12132] building file list 2023/02/07 00:25:07 [12132] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/07 00:25:07 [12133] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 00:25:07 [12133] building file list 2023/02/07 00:25:08 [12133] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/07 00:25:08 [12134] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 00:25:09 [12134] building file list 2023/02/07 00:25:09 [12134] sent 11433 bytes received 29 bytes total size 8818811045 2023/02/07 00:25:10 [12135] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 00:25:10 [12135] building file list 2023/02/07 00:25:11 [12135] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/07 00:25:11 [12136] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 00:25:11 [12136] building file list 2023/02/07 00:25:12 [12136] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/07 00:25:12 [12137] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 00:25:13 [12137] building file list 2023/02/07 00:25:14 [12137] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/07 00:25:14 [12138] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 00:25:14 [12138] building file list 2023/02/07 00:25:15 [12138] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/07 04:15:02 [12790] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 04:15:03 [12790] building file list 2023/02/07 04:21:57 [12790] sent 12541133560 bytes received 494 bytes total size 2632586709442 2023/02/07 04:21:58 [12791] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 04:21:58 [12791] building file list 2023/02/07 04:21:58 [12791] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/07 04:21:59 [12792] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 04:21:59 [12792] building file list 2023/02/07 04:22:00 [12792] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/07 04:22:00 [12793] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 04:22:01 [12793] building file list 2023/02/07 04:22:01 [12793] sent 11433 bytes received 29 bytes total size 8818811045 2023/02/07 04:22:02 [12794] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 04:22:02 [12794] building file list 2023/02/07 04:22:03 [12794] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/07 04:22:03 [12795] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 04:22:03 [12795] building file list 2023/02/07 04:22:04 [12795] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/07 04:22:04 [12796] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 04:22:04 [12796] building file list 2023/02/07 04:22:05 [12796] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/07 04:22:06 [12797] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 04:22:06 [12797] building file list 2023/02/07 04:22:06 [12797] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/07 08:15:04 [13450] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 08:15:04 [13450] building file list 2023/02/07 08:16:27 [13450] sent 2461490306 bytes received 57 bytes total size 2632586709442 2023/02/07 08:16:28 [13451] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 08:16:28 [13451] building file list 2023/02/07 08:16:29 [13451] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/07 08:16:29 [13452] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 08:16:30 [13452] building file list 2023/02/07 08:16:30 [13452] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/07 08:16:31 [13453] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 08:16:31 [13453] building file list 2023/02/07 08:16:32 [13453] sent 11433 bytes received 29 bytes total size 8818811045 2023/02/07 08:16:32 [13454] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 08:16:32 [13454] building file list 2023/02/07 08:16:33 [13454] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/07 08:16:33 [13455] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 08:16:34 [13455] building file list 2023/02/07 08:16:34 [13455] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/07 08:16:35 [13456] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 08:16:35 [13456] building file list 2023/02/07 08:16:35 [13456] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/07 08:16:36 [13457] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/07 08:16:36 [13457] building file list 2023/02/07 08:16:37 [13457] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/07 12:15:00 [14113] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 12:15:01 [14113] building file list 2023/02/07 12:19:58 [14113] sent 9406966641 bytes received 78 bytes total size 2632807333566 2023/02/07 12:19:58 [14114] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 12:19:59 [14114] building file list 2023/02/07 12:19:59 [14114] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/07 12:20:00 [14115] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 12:20:00 [14115] building file list 2023/02/07 12:20:01 [14115] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/07 12:20:01 [14116] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 12:20:01 [14116] building file list 2023/02/07 12:20:02 [14116] sent 11433 bytes received 29 bytes total size 8818811045 2023/02/07 12:20:03 [14117] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 12:20:03 [14117] building file list 2023/02/07 12:20:03 [14117] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/07 12:20:04 [14118] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 12:20:05 [14118] building file list 2023/02/07 12:20:05 [14118] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/07 12:20:06 [14119] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 12:20:06 [14119] building file list 2023/02/07 12:20:07 [14119] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/07 12:20:07 [14120] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/07 12:20:08 [14120] building file list 2023/02/07 12:20:08 [14120] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/08 00:15:14 [26056] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 00:15:14 [26056] building file list 2023/02/08 01:00:30 [26056] sent 62894756987 bytes received 248 bytes total size 2635693415668 2023/02/08 01:00:30 [26218] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 01:00:31 [26218] building file list 2023/02/08 01:00:31 [26218] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/08 01:00:32 [26219] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 01:00:32 [26219] building file list 2023/02/08 01:00:33 [26219] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/08 01:00:33 [26220] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 01:00:33 [26220] building file list 2023/02/08 01:00:34 [26220] sent 11433 bytes received 29 bytes total size 8818811045 2023/02/08 01:00:35 [26221] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 01:00:35 [26221] building file list 2023/02/08 01:00:35 [26221] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/08 01:00:36 [26222] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 01:00:36 [26222] building file list 2023/02/08 01:00:37 [26222] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/08 01:00:37 [26223] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 01:00:38 [26223] building file list 2023/02/08 01:00:39 [26223] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/08 01:00:40 [26224] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 01:00:40 [26224] building file list 2023/02/08 01:00:41 [26224] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/08 04:15:02 [26746] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/08 04:15:03 [26746] building file list 2023/02/08 04:44:46 [26746] sent 53487888811 bytes received 204 bytes total size 2635693415668 2023/02/08 04:44:47 [26747] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/08 04:44:47 [26747] building file list 2023/02/08 04:44:48 [26747] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/08 04:44:51 [26748] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/08 04:44:51 [26748] building file list 2023/02/08 04:44:52 [26748] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/08 04:44:52 [26749] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/08 04:44:52 [26749] building file list 2023/02/08 04:44:53 [26749] sent 11433 bytes received 29 bytes total size 8818811045 2023/02/08 04:44:53 [26750] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/08 04:44:54 [26750] building file list 2023/02/08 04:44:54 [26750] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/08 04:44:55 [26751] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/08 04:44:55 [26751] building file list 2023/02/08 04:44:56 [26751] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/08 04:44:56 [26752] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/08 04:44:56 [26752] building file list 2023/02/08 04:44:57 [26752] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/08 04:44:57 [26753] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/08 04:44:58 [26753] building file list 2023/02/08 04:44:58 [26753] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/08 08:15:09 [27405] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 08:15:10 [27405] building file list 2023/02/08 08:31:34 [27405] sent 31334519385 bytes received 122 bytes total size 2638614075013 2023/02/08 08:31:37 [27406] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 08:31:38 [27406] building file list 2023/02/08 08:31:38 [27406] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/08 08:31:39 [27407] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 08:31:39 [27407] building file list 2023/02/08 08:31:40 [27407] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/08 08:31:40 [27408] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 08:31:40 [27408] building file list 2023/02/08 08:31:41 [27408] sent 11433 bytes received 29 bytes total size 8818811045 2023/02/08 08:31:44 [27409] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 08:31:44 [27409] building file list 2023/02/08 08:31:45 [27409] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/08 08:31:48 [27410] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 08:31:48 [27410] building file list 2023/02/08 08:31:49 [27410] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/08 08:31:49 [27411] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 08:31:50 [27411] building file list 2023/02/08 08:31:50 [27411] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/08 08:31:53 [27412] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/08 08:31:54 [27412] building file list 2023/02/08 08:31:54 [27412] sent 3750 bytes received 29 bytes total size 10671786579
Edited to add logs up to now ; turning logs off
Hello @Rgaudin, we have a slightly different log from Feb 6 at 12:15 to Feb 8 at 08:15 and it looks like most of the parts missing are the runs where we get an error. Please can we get the missing logs where there was maybe a connection error or any other errors
Here is what we have from our end;
Feb 6 12:15:01 clouddumps1001 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 6 12:15:19 clouddumps1001 systemd[1]: kiwix-mirror-update.service: Succeeded. Feb 6 20:15:01 clouddumps1001 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 6 20:18:03 clouddumps1001 systemd[1]: kiwix-mirror-update.service: Succeeded. Feb 6 20:18:03 clouddumps1001 systemd[1]: kiwix-mirror-update.service: Consumed 46.972s CPU time. Feb 7 04:15:01 clouddumps1001 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 7 04:22:06 clouddumps1001 systemd[1]: kiwix-mirror-update.service: Succeeded. Feb 7 04:22:06 clouddumps1001 systemd[1]: kiwix-mirror-update.service: Consumed 1min 52.436s CPU time. Feb 7 12:15:00 clouddumps1001 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 7 12:20:08 clouddumps1001 systemd[1]: kiwix-mirror-update.service: Succeeded. Feb 7 12:20:08 clouddumps1001 systemd[1]: kiwix-mirror-update.service: Consumed 1min 21.977s CPU time. Feb 7 20:15:01 clouddumps1001 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 7 20:15:02 clouddumps1001 systemd[1]: kiwix-mirror-update.service: Main process exited, code=exited, status=5/NOTINSTALLED Feb 7 20:15:02 clouddumps1001 systemd[1]: kiwix-mirror-update.service: Failed with result 'exit-code'. Feb 8 04:15:01 clouddumps1001 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 8 04:44:58 clouddumps1001 systemd[1]: kiwix-mirror-update.service: Succeeded. Feb 8 04:44:58 clouddumps1001 systemd[1]: kiwix-mirror-update.service: Consumed 7min 58.804s CPU time. ------------------------------------------- Feb 6 16:15:13 clouddumps1002 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 6 16:15:13 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Main process exited, code=exited, status=5/NOTINSTALLED Feb 6 16:15:13 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Failed with result 'exit-code'. Feb 7 00:15:02 clouddumps1002 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 7 00:25:15 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Succeeded. Feb 7 00:25:15 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Consumed 2min 8.498s CPU time. Feb 7 08:15:03 clouddumps1002 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 7 08:16:37 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Succeeded. Feb 7 08:16:37 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Consumed 20.592s CPU time. Feb 7 16:15:05 clouddumps1002 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 7 16:15:07 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Main process exited, code=exited, status=5/NOTINSTALLED Feb 7 16:15:07 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Failed with result 'exit-code'. Feb 8 00:15:13 clouddumps1002 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 8 01:00:41 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Succeeded. Feb 8 01:00:41 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Consumed 8min 51.312s CPU time. Feb 8 08:15:09 clouddumps1002 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 8 08:31:54 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Succeeded. Feb 8 08:31:54 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Consumed 4min 37.113s CPU time.
Unfortunately, rsync doesn't log connections that it forbids itself…
I've relaunched with max verbosity (9) but I don't think it will be of any help as unauthorized connections are not logged at all ; and yes, those do return as exit-code 5 on the client.
I've made two changes though:
Please check it over the next couple days and let me know if there's any change on your side. I'll post the log as well.
We got an error yesterday 9th Feb from clouddumps1002 at 5 pm UTC
The error we got was:
Error while running rsync, check the logs...
Please can we get the logs around the time we got these errors?
Feb 9 16:15:05 clouddumps1002 systemd[1]: Started Regular jobs to update kiwix mirror. Feb 9 17:00:36 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Main process exited, code=exited, status=24/n/a Feb 9 17:00:36 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Failed with result 'exit-code'. Feb 9 17:00:36 clouddumps1002 systemd[1]: kiwix-mirror-update.service: Consumed 2min 35.279s CPU time.
Around 5pm:
2023/02/09 17:00:35 [245] file has vanished: "/zim/wikipedia/wikipedia_pcm_all_nopic_2023-02.zim" (in wmf.download.kiwix.org) 2023/02/09 17:00:35 [245] file has vanished: "/zim/wikipedia/wikiquote_guw_all_maxi_2023-02.zim" (in wmf.download.kiwix.org) 2023/02/09 17:00:35 [245] file has vanished: "/zim/wikipedia/wikiquote_guw_all_nopic_2023-02.zim" (in wmf.download.kiwix.org) 2023/02/09 17:00:36 [245] sent 9541255002 bytes received 2556593 bytes total size 2629646620338
Based on rsync(1), 24 exit-code is “Partial transfer due to vanished source files” which matches the log.
Indeed I removed those files which have been incorrectly placed in this folder.
Have you seen any new connection refusal?
And here's the full log:
2023/02/09 16:15:07 [245] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/09 16:15:07 [245] building file list 2023/02/09 17:00:35 [245] file has vanished: "/zim/wikipedia/wikipedia_pcm_all_nopic_2023-02.zim" (in wmf.download.kiwix.org) 2023/02/09 17:00:35 [245] file has vanished: "/zim/wikipedia/wikiquote_guw_all_maxi_2023-02.zim" (in wmf.download.kiwix.org) 2023/02/09 17:00:35 [245] file has vanished: "/zim/wikipedia/wikiquote_guw_all_nopic_2023-02.zim" (in wmf.download.kiwix.org) 2023/02/09 17:00:36 [245] sent 9541255002 bytes received 2556593 bytes total size 2629646620338 2023/02/09 20:15:01 [2975] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:15:02 [2975] building file list 2023/02/09 20:28:43 [2975] sent 18977328574 bytes received 2180205 bytes total size 2634775858113 2023/02/09 20:28:43 [2976] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:44 [2976] building file list 2023/02/09 20:28:44 [2976] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/09 20:28:45 [2977] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:45 [2977] building file list 2023/02/09 20:28:46 [2977] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/09 20:28:46 [2978] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:47 [2978] building file list 2023/02/09 20:28:48 [2978] sent 4017935 bytes received 55 bytes total size 8822816481 2023/02/09 20:28:48 [2979] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:49 [2979] building file list 2023/02/09 20:28:49 [2979] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/09 20:28:50 [2980] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:50 [2980] building file list 2023/02/09 20:28:50 [2980] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/09 20:28:51 [2981] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:51 [2981] building file list 2023/02/09 20:28:52 [2981] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/09 20:28:52 [2982] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:52 [2982] building file list 2023/02/09 20:28:53 [2982] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/10 00:15:05 [3635] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:15:05 [3635] building file list 2023/02/10 00:30:45 [3635] sent 23789242393 bytes received 177120 bytes total size 2643096456307 2023/02/10 00:30:46 [3639] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:46 [3639] building file list 2023/02/10 00:30:47 [3639] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/10 00:30:47 [3640] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:47 [3640] building file list 2023/02/10 00:30:48 [3640] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/10 00:30:49 [3641] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:49 [3641] building file list 2023/02/10 00:30:50 [3641] sent 4017935 bytes received 55 bytes total size 8822816481 2023/02/10 00:30:51 [3642] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:51 [3642] building file list 2023/02/10 00:30:51 [3642] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/10 00:30:52 [3643] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:52 [3643] building file list 2023/02/10 00:30:53 [3643] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/10 00:30:53 [3644] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:53 [3644] building file list 2023/02/10 00:30:54 [3644] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/10 00:30:55 [3645] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:55 [3645] building file list 2023/02/10 00:30:55 [3645] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/10 04:15:02 [4296] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:15:02 [4296] building file list 2023/02/10 04:23:55 [4296] sent 16807714136 bytes received 79925 bytes total size 2645140744294 2023/02/10 04:23:56 [4297] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:23:56 [4297] building file list 2023/02/10 04:23:57 [4297] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/10 04:23:57 [4298] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:23:57 [4298] building file list 2023/02/10 04:23:58 [4298] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/10 04:23:58 [4299] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:23:59 [4299] building file list 2023/02/10 04:23:59 [4299] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/10 04:24:00 [4300] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:24:00 [4300] building file list 2023/02/10 04:24:01 [4300] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/10 04:24:01 [4301] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:24:02 [4301] building file list 2023/02/10 04:24:02 [4301] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/10 04:24:02 [4302] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:24:03 [4302] building file list 2023/02/10 04:24:03 [4302] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/10 04:24:04 [4303] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:24:04 [4303] building file list 2023/02/10 04:24:05 [4303] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/10 08:15:01 [4960] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:15:01 [4960] building file list 2023/02/10 08:24:17 [4960] sent 17768480487 bytes received 401 bytes total size 2652516927605 2023/02/10 08:24:17 [4962] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:18 [4962] building file list 2023/02/10 08:24:18 [4962] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/10 08:24:19 [4963] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:19 [4963] building file list 2023/02/10 08:24:19 [4963] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/10 08:24:20 [4964] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:20 [4964] building file list 2023/02/10 08:24:21 [4964] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/10 08:24:21 [4965] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:21 [4965] building file list 2023/02/10 08:24:22 [4965] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/10 08:24:22 [4966] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:23 [4966] building file list 2023/02/10 08:24:23 [4966] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/10 08:24:24 [4967] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:24 [4967] building file list 2023/02/10 08:24:25 [4967] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/10 08:24:25 [4968] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:25 [4968] building file list 2023/02/10 08:24:26 [4968] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/10 12:15:02 [5628] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:15:02 [5628] building file list 2023/02/10 12:21:36 [5628] sent 11950208902 bytes received 287 bytes total size 2641881572176 2023/02/10 12:21:36 [5629] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:37 [5629] building file list 2023/02/10 12:21:37 [5629] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/10 12:21:38 [5630] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:38 [5630] building file list 2023/02/10 12:21:39 [5630] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/10 12:21:40 [5631] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:40 [5631] building file list 2023/02/10 12:21:41 [5631] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/10 12:21:41 [5632] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:42 [5632] building file list 2023/02/10 12:21:42 [5632] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/10 12:21:43 [5633] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:43 [5633] building file list 2023/02/10 12:21:43 [5633] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/10 12:21:44 [5634] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:44 [5634] building file list 2023/02/10 12:21:45 [5634] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/10 12:21:45 [5635] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:45 [5635] building file list 2023/02/10 12:21:46 [5635] sent 3750 bytes received 29 bytes total size 10671786579
Hello @Rgaudin, we haven’t had any errors since the last one, which is a good thing :-) let's see how things are over the weekend, and thanks for helping with this.
Looking at the log, I can see you connect every four hours so I guess this is working fine now.
I've disabled the logs : I guess the ticket can now be closed.
2023/02/09 16:15:07 [245] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/09 16:15:07 [245] building file list 2023/02/09 17:00:35 [245] file has vanished: "/zim/wikipedia/wikipedia_pcm_all_nopic_2023-02.zim" (in wmf.download.kiwix.org) 2023/02/09 17:00:35 [245] file has vanished: "/zim/wikipedia/wikiquote_guw_all_maxi_2023-02.zim" (in wmf.download.kiwix.org) 2023/02/09 17:00:35 [245] file has vanished: "/zim/wikipedia/wikiquote_guw_all_nopic_2023-02.zim" (in wmf.download.kiwix.org) 2023/02/09 17:00:36 [245] sent 9541255002 bytes received 2556593 bytes total size 2629646620338 2023/02/09 20:15:01 [2975] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:15:02 [2975] building file list 2023/02/09 20:28:43 [2975] sent 18977328574 bytes received 2180205 bytes total size 2634775858113 2023/02/09 20:28:43 [2976] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:44 [2976] building file list 2023/02/09 20:28:44 [2976] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/09 20:28:45 [2977] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:45 [2977] building file list 2023/02/09 20:28:46 [2977] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/09 20:28:46 [2978] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:47 [2978] building file list 2023/02/09 20:28:48 [2978] sent 4017935 bytes received 55 bytes total size 8822816481 2023/02/09 20:28:48 [2979] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:49 [2979] building file list 2023/02/09 20:28:49 [2979] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/09 20:28:50 [2980] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:50 [2980] building file list 2023/02/09 20:28:50 [2980] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/09 20:28:51 [2981] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:51 [2981] building file list 2023/02/09 20:28:52 [2981] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/09 20:28:52 [2982] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/09 20:28:52 [2982] building file list 2023/02/09 20:28:53 [2982] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/10 00:15:05 [3635] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:15:05 [3635] building file list 2023/02/10 00:30:45 [3635] sent 23789242393 bytes received 177120 bytes total size 2643096456307 2023/02/10 00:30:46 [3639] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:46 [3639] building file list 2023/02/10 00:30:47 [3639] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/10 00:30:47 [3640] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:47 [3640] building file list 2023/02/10 00:30:48 [3640] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/10 00:30:49 [3641] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:49 [3641] building file list 2023/02/10 00:30:50 [3641] sent 4017935 bytes received 55 bytes total size 8822816481 2023/02/10 00:30:51 [3642] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:51 [3642] building file list 2023/02/10 00:30:51 [3642] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/10 00:30:52 [3643] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:52 [3643] building file list 2023/02/10 00:30:53 [3643] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/10 00:30:53 [3644] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:53 [3644] building file list 2023/02/10 00:30:54 [3644] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/10 00:30:55 [3645] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 00:30:55 [3645] building file list 2023/02/10 00:30:55 [3645] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/10 04:15:02 [4296] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:15:02 [4296] building file list 2023/02/10 04:23:55 [4296] sent 16807714136 bytes received 79925 bytes total size 2645140744294 2023/02/10 04:23:56 [4297] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:23:56 [4297] building file list 2023/02/10 04:23:57 [4297] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/10 04:23:57 [4298] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:23:57 [4298] building file list 2023/02/10 04:23:58 [4298] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/10 04:23:58 [4299] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:23:59 [4299] building file list 2023/02/10 04:23:59 [4299] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/10 04:24:00 [4300] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:24:00 [4300] building file list 2023/02/10 04:24:01 [4300] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/10 04:24:01 [4301] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:24:02 [4301] building file list 2023/02/10 04:24:02 [4301] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/10 04:24:02 [4302] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:24:03 [4302] building file list 2023/02/10 04:24:03 [4302] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/10 04:24:04 [4303] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 04:24:04 [4303] building file list 2023/02/10 04:24:05 [4303] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/10 08:15:01 [4960] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:15:01 [4960] building file list 2023/02/10 08:24:17 [4960] sent 17768480487 bytes received 401 bytes total size 2652516927605 2023/02/10 08:24:17 [4962] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:18 [4962] building file list 2023/02/10 08:24:18 [4962] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/10 08:24:19 [4963] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:19 [4963] building file list 2023/02/10 08:24:19 [4963] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/10 08:24:20 [4964] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:20 [4964] building file list 2023/02/10 08:24:21 [4964] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/10 08:24:21 [4965] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:21 [4965] building file list 2023/02/10 08:24:22 [4965] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/10 08:24:22 [4966] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:23 [4966] building file list 2023/02/10 08:24:23 [4966] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/10 08:24:24 [4967] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:24 [4967] building file list 2023/02/10 08:24:25 [4967] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/10 08:24:25 [4968] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 08:24:25 [4968] building file list 2023/02/10 08:24:26 [4968] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/10 12:15:02 [5628] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:15:02 [5628] building file list 2023/02/10 12:21:36 [5628] sent 11950208902 bytes received 287 bytes total size 2641881572176 2023/02/10 12:21:36 [5629] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:37 [5629] building file list 2023/02/10 12:21:37 [5629] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/10 12:21:38 [5630] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:38 [5630] building file list 2023/02/10 12:21:39 [5630] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/10 12:21:40 [5631] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:40 [5631] building file list 2023/02/10 12:21:41 [5631] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/10 12:21:41 [5632] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:42 [5632] building file list 2023/02/10 12:21:42 [5632] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/10 12:21:43 [5633] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:43 [5633] building file list 2023/02/10 12:21:43 [5633] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/10 12:21:44 [5634] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:44 [5634] building file list 2023/02/10 12:21:45 [5634] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/10 12:21:45 [5635] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 12:21:45 [5635] building file list 2023/02/10 12:21:46 [5635] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/10 16:15:02 [6342] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 16:15:02 [6342] building file list 2023/02/10 16:19:31 [6342] sent 8122215216 bytes received 57 bytes total size 2642078925145 2023/02/10 16:19:32 [6343] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 16:19:32 [6343] building file list 2023/02/10 16:19:33 [6343] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/10 16:19:36 [6344] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 16:19:36 [6344] building file list 2023/02/10 16:19:37 [6344] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/10 16:19:39 [6345] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 16:19:40 [6345] building file list 2023/02/10 16:19:40 [6345] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/10 16:19:43 [6346] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 16:19:44 [6346] building file list 2023/02/10 16:19:44 [6346] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/10 16:19:47 [6347] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 16:19:47 [6347] building file list 2023/02/10 16:19:48 [6347] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/10 16:19:48 [6348] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 16:19:49 [6348] building file list 2023/02/10 16:19:49 [6348] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/10 16:19:50 [6349] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/10 16:19:50 [6349] building file list 2023/02/10 16:19:50 [6349] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/10 20:15:02 [7003] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 20:15:02 [7003] building file list 2023/02/10 20:29:04 [7003] sent 25589818334 bytes received 82 bytes total size 2659542264673 2023/02/10 20:29:05 [7004] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 20:29:05 [7004] building file list 2023/02/10 20:29:06 [7004] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/10 20:29:06 [7005] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 20:29:07 [7005] building file list 2023/02/10 20:29:07 [7005] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/10 20:29:10 [7006] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 20:29:10 [7006] building file list 2023/02/10 20:29:11 [7006] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/10 20:29:12 [7007] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 20:29:12 [7007] building file list 2023/02/10 20:29:13 [7007] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/10 20:29:13 [7008] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 20:29:13 [7008] building file list 2023/02/10 20:29:14 [7008] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/10 20:29:14 [7009] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 20:29:15 [7009] building file list 2023/02/10 20:29:15 [7009] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/10 20:29:16 [7010] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/10 20:29:16 [7010] building file list 2023/02/10 20:29:16 [7010] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/11 00:15:04 [7662] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 00:15:05 [7662] building file list 2023/02/11 00:25:27 [7662] sent 17467701851 bytes received 61 bytes total size 2642333198474 2023/02/11 00:25:27 [7663] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 00:25:28 [7663] building file list 2023/02/11 00:25:28 [7663] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/11 00:25:31 [7664] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 00:25:32 [7664] building file list 2023/02/11 00:25:32 [7664] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/11 00:25:33 [7665] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 00:25:33 [7665] building file list 2023/02/11 00:25:34 [7665] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/11 00:25:34 [7666] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 00:25:35 [7666] building file list 2023/02/11 00:25:35 [7666] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/11 00:25:38 [7667] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 00:25:38 [7667] building file list 2023/02/11 00:25:39 [7667] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/11 00:25:39 [7668] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 00:25:39 [7668] building file list 2023/02/11 00:25:40 [7668] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/11 00:25:40 [7669] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 00:25:41 [7669] building file list 2023/02/11 00:25:41 [7669] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/11 04:15:04 [8321] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 04:15:04 [8321] building file list 2023/02/11 04:15:06 [8321] sent 98772 bytes received 32 bytes total size 2642333198474 2023/02/11 04:15:07 [8322] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 04:15:07 [8322] building file list 2023/02/11 04:15:08 [8322] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/11 04:15:11 [8323] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 04:15:11 [8323] building file list 2023/02/11 04:15:12 [8323] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/11 04:15:12 [8324] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 04:15:12 [8324] building file list 2023/02/11 04:15:13 [8324] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/11 04:15:13 [8325] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 04:15:14 [8325] building file list 2023/02/11 04:15:14 [8325] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/11 04:15:15 [8326] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 04:15:15 [8326] building file list 2023/02/11 04:15:16 [8326] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/11 04:15:16 [8327] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 04:15:16 [8327] building file list 2023/02/11 04:15:17 [8327] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/11 04:15:17 [8328] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 04:15:18 [8328] building file list 2023/02/11 04:15:18 [8328] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/11 08:15:04 [9011] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 08:15:05 [9011] building file list 2023/02/11 08:15:06 [9011] sent 98765 bytes received 29 bytes total size 2642333198474 2023/02/11 08:15:09 [9012] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 08:15:09 [9012] building file list 2023/02/11 08:15:10 [9012] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/11 08:15:10 [9013] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 08:15:11 [9013] building file list 2023/02/11 08:15:11 [9013] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/11 08:15:14 [9014] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 08:15:15 [9014] building file list 2023/02/11 08:15:15 [9014] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/11 08:15:16 [9015] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 08:15:16 [9015] building file list 2023/02/11 08:15:16 [9015] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/11 08:15:17 [9016] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 08:15:17 [9016] building file list 2023/02/11 08:15:18 [9016] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/11 08:15:18 [9017] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 08:15:18 [9017] building file list 2023/02/11 08:15:19 [9017] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/11 08:15:20 [9018] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 08:15:20 [9018] building file list 2023/02/11 08:15:20 [9018] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/11 12:15:04 [9678] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 12:15:04 [9678] building file list 2023/02/11 12:38:56 [9678] sent 42608389828 bytes received 65 bytes total size 2684931089570 2023/02/11 12:38:57 [9679] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 12:38:57 [9679] building file list 2023/02/11 12:38:58 [9679] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/11 12:38:58 [9680] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 12:38:58 [9680] building file list 2023/02/11 12:38:59 [9680] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/11 12:38:59 [9681] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 12:39:00 [9681] building file list 2023/02/11 12:39:00 [9681] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/11 12:39:01 [9682] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 12:39:01 [9682] building file list 2023/02/11 12:39:02 [9682] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/11 12:39:02 [9683] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 12:39:03 [9683] building file list 2023/02/11 12:39:03 [9683] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/11 12:39:03 [9684] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 12:39:04 [9684] building file list 2023/02/11 12:39:04 [9684] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/11 12:39:05 [9685] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 12:39:05 [9685] building file list 2023/02/11 12:39:06 [9685] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/11 16:15:05 [10345] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 16:15:05 [10345] building file list 2023/02/11 16:37:58 [10345] sent 42608389828 bytes received 65 bytes total size 2684931089570 2023/02/11 16:37:58 [10346] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 16:37:58 [10346] building file list 2023/02/11 16:37:59 [10346] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/11 16:38:02 [10347] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 16:38:02 [10347] building file list 2023/02/11 16:38:03 [10347] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/11 16:38:03 [10348] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 16:38:04 [10348] building file list 2023/02/11 16:38:04 [10348] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/11 16:38:05 [10349] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 16:38:05 [10349] building file list 2023/02/11 16:38:06 [10349] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/11 16:38:06 [10350] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 16:38:06 [10350] building file list 2023/02/11 16:38:07 [10350] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/11 16:38:07 [10351] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 16:38:07 [10351] building file list 2023/02/11 16:38:08 [10351] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/11 16:38:09 [10352] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/11 16:38:09 [10352] building file list 2023/02/11 16:38:09 [10352] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/11 20:15:02 [11003] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 20:15:02 [11003] building file list 2023/02/11 20:15:03 [11003] sent 98806 bytes received 29 bytes total size 2684931089570 2023/02/11 20:15:06 [11004] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 20:15:06 [11004] building file list 2023/02/11 20:15:07 [11004] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/11 20:15:07 [11005] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 20:15:07 [11005] building file list 2023/02/11 20:15:08 [11005] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/11 20:15:13 [11006] rsync on wmf.download.kiwix.org/zim/wikiquote/ from UNKNOWN (208.80.154.142) 2023/02/11 20:15:14 [11006] building file list 2023/02/11 20:15:14 [11006] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/11 20:15:15 [11007] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 20:15:15 [11007] building file list 2023/02/11 20:15:16 [11007] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/11 20:15:19 [11008] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 20:15:19 [11008] building file list 2023/02/11 20:15:20 [11008] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/11 20:15:20 [11009] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 20:15:20 [11009] building file list 2023/02/11 20:15:21 [11009] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/11 20:15:22 [11010] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/11 20:15:22 [11010] building file list 2023/02/11 20:15:22 [11010] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/12 00:15:13 [11661] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 00:15:14 [11661] building file list 2023/02/12 00:15:15 [11661] sent 98806 bytes received 29 bytes total size 2684931089570 2023/02/12 00:15:18 [11662] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 00:15:20 [11662] building file list 2023/02/12 00:15:21 [11662] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/12 00:15:22 [11663] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 00:15:22 [11663] building file list 2023/02/12 00:15:22 [11663] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/12 00:15:23 [11664] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 00:15:23 [11664] building file list 2023/02/12 00:15:24 [11664] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/12 00:15:24 [11665] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 00:15:26 [11665] building file list 2023/02/12 00:15:26 [11665] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/12 00:15:27 [11666] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 00:15:27 [11666] building file list 2023/02/12 00:15:27 [11666] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/12 00:15:28 [11667] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 00:15:28 [11667] building file list 2023/02/12 00:15:29 [11667] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/12 00:15:29 [11668] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 00:15:29 [11668] building file list 2023/02/12 00:15:30 [11668] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/12 04:15:03 [12320] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 04:15:03 [12320] building file list 2023/02/12 04:15:04 [12320] sent 98806 bytes received 29 bytes total size 2684931089570 2023/02/12 04:15:04 [12321] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 04:15:05 [12321] building file list 2023/02/12 04:15:05 [12321] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/12 04:15:06 [12322] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 04:15:06 [12322] building file list 2023/02/12 04:15:07 [12322] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/12 04:15:07 [12323] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 04:15:07 [12323] building file list 2023/02/12 04:15:08 [12323] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/12 04:15:08 [12324] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 04:15:09 [12324] building file list 2023/02/12 04:15:09 [12324] sent 11284 bytes received 29 bytes total size 308934156392 2023/02/12 04:15:10 [12325] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 04:15:10 [12325] building file list 2023/02/12 04:15:10 [12325] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/12 04:15:11 [12326] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 04:15:11 [12326] building file list 2023/02/12 04:15:12 [12326] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/12 04:15:12 [12327] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 04:15:13 [12327] building file list 2023/02/12 04:15:13 [12327] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/12 08:15:01 [12983] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 08:15:01 [12983] building file list 2023/02/12 08:15:02 [12983] sent 98806 bytes received 29 bytes total size 2684931089570 2023/02/12 08:15:03 [12984] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 08:15:03 [12984] building file list 2023/02/12 08:15:04 [12984] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/12 08:15:07 [12985] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 08:15:07 [12985] building file list 2023/02/12 08:15:08 [12985] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/12 08:15:08 [12986] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 08:15:08 [12986] building file list 2023/02/12 08:15:09 [12986] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/12 08:15:09 [12987] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 08:15:10 [12987] building file list 2023/02/12 08:15:23 [12987] sent 369606253 bytes received 74 bytes total size 308935386457 2023/02/12 08:15:23 [12988] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 08:15:23 [12988] building file list 2023/02/12 08:15:24 [12988] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/12 08:15:27 [12989] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 08:15:27 [12989] building file list 2023/02/12 08:15:28 [12989] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/12 08:15:28 [12990] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 08:15:29 [12990] building file list 2023/02/12 08:15:29 [12990] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/12 12:15:04 [13644] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 12:15:05 [13644] building file list 2023/02/12 12:16:28 [13644] sent 2573509420 bytes received 325 bytes total size 2687503870865 2023/02/12 12:16:29 [13645] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 12:16:29 [13645] building file list 2023/02/12 12:16:30 [13645] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/12 12:16:30 [13646] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 12:16:30 [13646] building file list 2023/02/12 12:16:31 [13646] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/12 12:16:34 [13647] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 12:16:34 [13647] building file list 2023/02/12 12:16:35 [13647] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/12 12:16:36 [13648] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 12:16:36 [13648] building file list 2023/02/12 12:16:49 [13648] sent 369606253 bytes received 74 bytes total size 308935386457 2023/02/12 12:16:49 [13649] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 12:16:49 [13649] building file list 2023/02/12 12:16:50 [13649] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/12 12:16:50 [13650] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 12:16:51 [13650] building file list 2023/02/12 12:16:51 [13650] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/12 12:16:52 [13651] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 12:16:52 [13651] building file list 2023/02/12 12:16:53 [13651] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/12 16:15:04 [14309] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 16:15:05 [14309] building file list 2023/02/12 16:18:07 [14309] sent 5715977164 bytes received 363 bytes total size 2690645571431 2023/02/12 16:18:08 [14310] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 16:18:08 [14310] building file list 2023/02/12 16:18:08 [14310] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/12 16:18:11 [14311] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 16:18:12 [14311] building file list 2023/02/12 16:18:12 [14311] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/12 16:18:15 [14312] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 16:18:16 [14312] building file list 2023/02/12 16:18:16 [14312] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/12 16:18:17 [14313] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 16:18:17 [14313] building file list 2023/02/12 16:18:17 [14313] sent 11265 bytes received 29 bytes total size 308935386457 2023/02/12 16:18:18 [14314] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 16:18:18 [14314] building file list 2023/02/12 16:18:19 [14314] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/12 16:18:19 [14315] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 16:18:19 [14315] building file list 2023/02/12 16:18:20 [14315] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/12 16:18:21 [14316] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/12 16:18:21 [14316] building file list 2023/02/12 16:18:21 [14316] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/12 20:15:02 [14971] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 20:15:03 [14971] building file list 2023/02/12 20:17:31 [14971] sent 4852578665 bytes received 95 bytes total size 2692355165463 2023/02/12 20:17:31 [14972] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 20:17:31 [14972] building file list 2023/02/12 20:17:32 [14972] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/12 20:17:32 [14973] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 20:17:33 [14973] building file list 2023/02/12 20:17:33 [14973] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/12 20:17:34 [14974] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 20:17:34 [14974] building file list 2023/02/12 20:17:35 [14974] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/12 20:17:35 [14975] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 20:17:36 [14975] building file list 2023/02/12 20:17:36 [14975] sent 11265 bytes received 29 bytes total size 308935386457 2023/02/12 20:17:37 [14976] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 20:17:37 [14976] building file list 2023/02/12 20:17:37 [14976] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/12 20:17:38 [14977] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 20:17:38 [14977] building file list 2023/02/12 20:17:39 [14977] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/12 20:17:39 [14978] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/12 20:17:40 [14978] building file list 2023/02/12 20:17:40 [14978] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/13 00:15:02 [15629] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 00:15:03 [15629] building file list 2023/02/13 00:16:15 [15629] sent 1710111005 bytes received 57 bytes total size 2692355165463 2023/02/13 00:16:16 [15630] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 00:16:16 [15630] building file list 2023/02/13 00:16:16 [15630] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/13 00:16:17 [15631] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 00:16:17 [15631] building file list 2023/02/13 00:16:18 [15631] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/13 00:16:18 [15632] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 00:16:20 [15632] building file list 2023/02/13 00:16:21 [15632] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/13 00:16:22 [15633] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 00:16:22 [15633] building file list 2023/02/13 00:16:22 [15633] sent 11265 bytes received 29 bytes total size 308935386457 2023/02/13 00:16:23 [15634] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 00:16:23 [15634] building file list 2023/02/13 00:16:23 [15634] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/13 00:16:24 [15635] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 00:16:26 [15635] building file list 2023/02/13 00:16:27 [15635] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/13 00:16:27 [15636] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 00:16:27 [15636] building file list 2023/02/13 00:16:28 [15636] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/13 04:15:02 [16291] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/13 04:15:03 [16291] building file list 2023/02/13 04:20:15 [16291] sent 8875221018 bytes received 76 bytes total size 2701228120518 2023/02/13 04:20:16 [16292] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/13 04:20:16 [16292] building file list 2023/02/13 04:20:17 [16292] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/13 04:20:17 [16293] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/13 04:20:18 [16293] building file list 2023/02/13 04:20:18 [16293] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/13 04:20:19 [16294] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/13 04:20:19 [16294] building file list 2023/02/13 04:20:19 [16294] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/13 04:20:20 [16295] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/13 04:20:20 [16295] building file list 2023/02/13 04:20:21 [16295] sent 11265 bytes received 29 bytes total size 308935386457 2023/02/13 04:20:21 [16296] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/13 04:20:21 [16296] building file list 2023/02/13 04:20:22 [16296] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/13 04:20:22 [16297] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/13 04:20:23 [16297] building file list 2023/02/13 04:20:23 [16297] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/13 04:20:24 [16298] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1001.wikimedia.org (208.80.154.142) 2023/02/13 04:20:24 [16298] building file list 2023/02/13 04:20:24 [16298] sent 3750 bytes received 29 bytes total size 10671786579 2023/02/13 08:15:13 [16951] rsync on wmf.download.kiwix.org/zim/wikipedia/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 08:15:14 [16951] building file list 2023/02/13 08:28:57 [16951] sent 26272547435 bytes received 101 bytes total size 2718621200499 2023/02/13 08:28:57 [16952] rsync on wmf.download.kiwix.org/zim/wikibooks/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 08:28:58 [16952] building file list 2023/02/13 08:28:58 [16952] sent 12278 bytes received 29 bytes total size 53052004296 2023/02/13 08:28:59 [16953] rsync on wmf.download.kiwix.org/zim/wikinews/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 08:28:59 [16953] building file list 2023/02/13 08:29:00 [16953] sent 5087 bytes received 29 bytes total size 38555482203 2023/02/13 08:29:00 [16954] rsync on wmf.download.kiwix.org/zim/wikiquote/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 08:29:00 [16954] building file list 2023/02/13 08:29:01 [16954] sent 11473 bytes received 29 bytes total size 8822816481 2023/02/13 08:29:01 [16955] rsync on wmf.download.kiwix.org/zim/wikisource/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 08:29:02 [16955] building file list 2023/02/13 08:29:02 [16955] sent 11265 bytes received 29 bytes total size 308935386457 2023/02/13 08:29:03 [16956] rsync on wmf.download.kiwix.org/zim/wikiversity/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 08:29:03 [16956] building file list 2023/02/13 08:29:03 [16956] sent 2779 bytes received 29 bytes total size 21072308227 2023/02/13 08:29:04 [16957] rsync on wmf.download.kiwix.org/zim/wiktionary/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 08:29:04 [16957] building file list 2023/02/13 08:29:05 [16957] sent 23451 bytes received 29 bytes total size 122147058159 2023/02/13 08:29:05 [16958] rsync on wmf.download.kiwix.org/zim/wikivoyage/ from clouddumps1002.wikimedia.org (208.80.154.71) 2023/02/13 08:29:06 [16958] building file list 2023/02/13 08:29:06 [16958] sent 3750 bytes received 29 bytes total size 10671786579
We'll be closing this now because everything looks okay, and we didn’t get any errors over the weekend, but we are not sure what fixed this. It could be the last update @Rgaudin made by increasing the max connection or specifying a dedicated lock file for our module.
Yes, My understanding is that the lock file is probably what fixed it. man is not very informative. We could look for answers in rsync source code but it's not worth the effort to me at the moment. Thank you all for your time.