Thank you, @Krinkle. I'll look into this and let you know if I have questions!

still in progress?

In T335761#8845049, @Protsack.stephan wrote:

Sorry somehow missed this tag, I'm going to take a look into logs to see if there's something interesting there.

I have a quick question, how the checksum is being fetched? I mean before or after the download was initiated?

In T335761#8856843, @jberkel wrote:

Another question, where are the enterprise dumps stored on toolforge now? They seem to have stopped updating October last year.

$ ls /public/dumps/public/other/enterprise_html/runs/
20220720  20220801  20220820  20220901	20220920  20221001

Currently, there isn’t any way for you to track the progress of the dumps because we don’t produce them locally; we fetch them via an API from WME. At the time of the run, the WME lets us access a list of wikis and namespace, this information is also retrieved on the fly rather than from a static list, and they could change on the next run.

The 2023-04-20 run was not completed because of a token refresh failure we experienced (here https://phabricator.wikimedia.org/T335368) The current run is still in progress and we can’t say for now if the files are missing or not; we’ll have to wait for the dumps to be completed.

We saw this when the job that dumps metadata for all revisions was running for this wiki; This is the only wiki impacted so far

One way this could be done might be to add the checksum for dumps created this month via a script and run the script daily for new dump files in some directory trees. Most dumps in https://dumps.wikimedia.org/other belong to other teams and are not maintained by us (Platform Engineering). They are also produced by different scripts, some on different hosts.

Hello @jbond, the errors result from the manual rsync of data we were running on the new dumpsdata hosts. You might see them on dumpdata1007 which started rsync today, the rest are completed. The errors are just during the rsync, and they aren’t serious; Please ignore them :-)

Closing this because the run took four days to complete, this is better than a week!

Hello @EBernhardson, The dumps were completed within a few days, and the resources used on the host are within limits. This is better and faster than when they took more than a week to complete. Thank you!

The dumps run are done and we didn’t get any errors. Thank you @Ladsgroup

Hello @hashar, are you still seeing these errors, and when did they start?

Hello, just checking in to find out what is going on with the OS installation on dumpsdata1006, and please, when will it be ready for use?

So far, we haven’t had any errors; we'll be certain when the full run completes in a day or two, and we can update the task by then.

We'll be closing this now because everything looks okay, and we didn’t get any errors over the weekend, but we are not sure what fixed this. It could be the last update @Rgaudin made by increasing the max connection or specifying a dedicated lock file for our module.

Hello @Rgaudin, we haven’t had any errors since the last one, which is a good thing :-) let's see how things are over the weekend, and thanks for helping with this.

yeah, we still see them, and we got this on Feb 2, 8:40 am UTC.

We got an error yesterday 9th Feb from clouddumps1002 at 5 pm UTC
The error we got was:

Error while running rsync, check the logs...

Please can we get the logs around the time we got these errors?

Hello @Rgaudin, we have a slightly different log from Feb 6 at 12:15 to Feb 8 at 08:15 and it looks like most of the parts missing are the runs where we get an error. Please can we get the missing logs where there was maybe a connection error or any other errors

@ArielGlenn From the last Memory and CPU usage during the last run (please see the image above), we are good to decompress blocks using multiple threads because currently, the last run used less than 25% CPU and memory usage also isn’t much. I think we are good to go!

The job starts on the 26th of every month, and the last run was completed on the 2nd of February. I have attached the MEMORY and CPU usage before and after the job was completed.

Hey WMCS folks, we've been getting the errors again after the last patch; we decided to do some testing on the script just after the loop completes and discovered some processes lingering around in TIME-WAIT and one in FIN-WAIT1. On the remote end, these may not be deemed open connections and so might have no effect on the connection problem.

We’ve been seeing a lot of these recently;

In T222349#8485437, @bking wrote:

Per above patches and discussion with @akosiaris , we are going to try using the rsync protocol instead of NFS. Do the clouddumps servers also rate-limit rsync?

In response to my previous comment on the runtime, The last run started on Oct 11th and ended on Oct 21st. It took 10 days in total to complete the run.

Today's report:

In T265056#8315327, @EBernhardson wrote:

With these errors dated oct 4-6 it's not clear that they would have been running the new search_after code. Hopefully this fails less on this weeks run. I've checked the current log outputs and all seems happy (except bclwikiquote which was a brand new wiki without an index yet), but it's only up to commonswiki_file. Unfortunately this weeks dumps with the new code seem to be running signficantly slower, might have to investigate that.

We've used a similar dumping strategy to import search indices into yarn, there it's able to dump all wikis in ~3.5 hours, although it's issuing 72 shard queries in parallel whereas the cirrus version does 1 index at a time with however many shards it has (from 1 to 32).

In T260223#8302299, @Kelson wrote:

During September we have detected that one of the mirror was full and therefore rsync client was keeping download and redownloading things. As a consequence a significant part of the bandwidth and the rsync daemon slots were always busy. We still lack of monitoring solution for this on our side honestly. We should figure a tool and a process to better monitor the server/service. If you have propositions, they are welcome.

Yesterday's report:

Thank you @Andrew for helping to fix this! and @jberkel the dumps are now available!

I'm sorry @Milimetric, @BCornwall, and @BTullis, I didn’t check my spam earlier but I just saw the mail now :-)

Yesterday's report:

The first sql/xml dumps run (1st of every month) is a full run, this means it contains all historical data and takes longer to dump. Currently we still have the wikidata dumps ongoing, and expect it to complete tomorrow. This gives us a margin of a few days in case there are errors causing us to delay or rerun some parts.

Yesterday's report:

Yesterday’s report:

we have deployed a patch so that we'll get email notification the next time something like this happens.

Today's report:

Hello Ryan, please can someone on your team make a gerrit patch with the neccessary changes and we can +2 and deploy. Here is the page that has to be modified.

Today's report:

Hello @Dsharpe and @Aklapper .. I have been able to link the right account and thank you for pointing that out. Please confirm and let me know if there is anything I need to help out with.

Today's report:

Today's report:

Today's report:

Today's report:

Today's report: