This is now complete, dumps-0 has been replaced with dumps-10.

Hi @Andrew, yes the NFS server is being used extensively. The cinder volumes are only used for temporary storage (which we discussed in T274029), so I don't foresee that the NFS setup will go away anytime soon.

I'm not in a position to support/oppose this request, but having more hands helping out in the extension would be a good thing to have.

Thanks for noticing @Krinkle. I own the tool and it is intended to be running, but I was not monitoring it recently and did not notice that it has already been down for so long. If a redirect can be created, that would be great!

This was fixed in https://gerrit.wikimedia.org/r/670350 and T259360.

Thanks! Ideally we can work with 5 TB as the size of the dumps are getting bigger (enwiki is now at 1.6 TB, but was at around only 1 TB when T159930 was filed), so I really hope if the Cloud Services team would consider giving us this quota. Again, the space used should hover around 10 GB most of the time, the large space is to handle the situation when large wikis like enwiki or wikidatawiki are being archived.

I got the second volume to attach successfully and everything seems to be working fine so far, except that I/O seems to be slightly slower than using instance storage (but if I'm not wrong they should be writing to the same Ceph storage?).

Thanks! I tried to attach one of the Cinder volumes to dumps-1 and it seems to be working. However, it seems like it is not possible to attach a second Cinder volume to another instance, is it possible to look into this issue?

I think it is a good plan, let's go ahead with it.

Nothing to be done here. Dumps have always been archived since 2012.

Thanks for creating this task. Currently, dumps-1 to dumps-5 do not need to be backed up at all (especially @Andrew mentioned in T159930#5234371 that we should delete and re-create the instances often). Likely in the future, any instances of dumps-[1-9] do not need any backups (or really any instance with the type dumps-temporary-file-storage).

Some wikis that have an underscore in the database name still keep their historical dumps since July 2017, probably the dumps generation process forgot to remove them when starting a new dump.

Will it be possible to lift the bandwidth cap for specific hosts within CloudVPS? Currently, the limit is at 5 MB/s which is painfully slow when it comes to archiving the dumps, Wikibase dumps alone is at 560 GB, it will take almost the whole week to archive just one item, not forgetting that there are also other dumps that will take up the bandwidth as well.

Should be similar to T43417 but now to check the action that the user is performing.

The extension is quite outdated due to lots of regression, so a rewrite is probably needed to bring it back up-to-date.

I originally created this badge as one that can be self-awarded, so that we can recognize the work of volunteers. However, I think the policy on badges changed so it is not possible to award them anymore without the help of an administrator. Ideally, anyone who wishes to be recognized as a volunteer can claim this badge for themselves.

It is currently blocked by some other task not tracked on Phabricator. Should I still close this task?

Just an update that I have got the files archived to the Internet Archive: https://archive.org/search.php?query=subject%3A%22shorturls%22%20AND%20subject%3A%22wikimedia%22

The actual system will take a while but let me try to manually get the old files uploaded first, which should take about 2-3 weeks. Approximately how many old copies of this dump will we be keeping, if we are not keeping all of them here?

Unfortunately my time as a volunteer is scarce too. I am already rushing to clear this backlog of files as fast as I can and would have appreciated this grace period for me to get the project back in order. If there is really nothing that can be done, please feel free to delete the files and we can close this task.

Yes, I am still looking into this. Ideally I would be able to give an update somewhere in the fourth quarter of this year.

Very likely a regression caused by rOMWCbf462e, should be fixable by reverting rEWMI034cea for InfoPage.php.

Currently, the storage space on this new instance flavor is for storing of files temporarily as it is getting uploaded to the Internet Archive. How it should work is that for certain datasets that are not currently available via NFS, we will download them, store them in the instance itself and then upload them to the Internet Archive. If we can do this in parallel (i.e. downloading and uploading), it would be even better, though it might cause quite a significant disk I/O, which we can fine tune at a later stage if necessary.

Something’s not right here, there has always been a need for large disks.

I'm done with moving the stuff that I need over to dumps-0. Now it's left with @Nemo_bis to sign off once he verifies that there is nothing on the server that we need.

This fix is not trivial. The original patch has been reverted in https://gerrit.wikimedia.org/r/#/c/476278/ until further testing can be done.

In T204503#4781342, @Krenair wrote:

In T204503#4781336, @Hydriz wrote:

In T204503#4781298, @Krenair wrote:

In T204503#4781283, @Hydriz wrote:

I tried to create a replacement instance dumps-0 in the new region but I was not able to log in from the bastion host.

Were you getting network errors or access denied? Are the security group rules set to allow SSH in from eqiad1-r IPs? What was showing in the instance's console log in horizon?

Access denied, public key is denied. Security groups should be fine since the SSH connection can be made, up till the authentication part. Console log shows nothing out of the ordinary. It's worth noting that other instances work fine in both regions.

Interesting, so the new instance is coming online broken. Do you apply any classes to the instance by default? Are you sure there are no puppet errors in the console log? Perhaps someone could try as root?

In T204503#4781298, @Krenair wrote:

In T204503#4781283, @Hydriz wrote:

I tried to create a replacement instance dumps-0 in the new region but I was not able to log in from the bastion host.

Were you getting network errors or access denied? Are the security group rules set to allow SSH in from eqiad1-r IPs? What was showing in the instance's console log in horizon?

I tried to create a replacement instance dumps-0 in the new region but I was not able to log in from the bastion host. I hence deleted it, but the project quotas seems to be incorrectly showing 6 instances when it should be 4. Can this be fixed before we attempt to recreate this instance from scratch in the new region?

Apparently done when T188726 was resolved. Files are in /public/dumps/public/other/mediacounts.

Per last comment, re-add the tag when the technical architecture has actually started.

I have looked through all the comments, but I still don't see any actions related to the WikimediaIncubator extension. This task is about Wikimedia Incubator, which fits into incubator.wikimedia.org, but it is not defining what needs to be done about the extension. Can you perhaps put a list of items that needs to be done by the extension in the task description?

@Amire80 Anything that the extension can do in this task? It seems to me that this task is more of a process issue than a software issue related to the WikimediaIncubator extension.