This is not about disk size, or number of bytes, it is about adding complexity to a system that already isn't stable. As @Nuria was saying, if we go back to a use case, we might find a way to provide a solution. I'm pretty sure that WDQS isn't the solution here.
This is addressed by T221631, closing.
Given the previous comment from 2016, it looks like things are working well enough. Feel free to re-open as needed.
This sounds like a hugely complex problem to address. It sounds like the classical cache invalidation / propagation with a bit of a termination problem. It is extremely unlikely that we will ever be able to address something of that scale of complexity.
@Abit is this something that is now supported on Commons via Structured Data?
Adding this amount of data to WDQS does not seem to be a good idea. We might want to redefine the higher level problem that we are trying to address here, and maybe implement it in a different way.
We might reopen this if we observe performance of space issues related to this. At the moment there isn't a clear benefit.
Thu, Feb 20
Sorry again for the delay. We are focusing on stabilizing WDQS at the moment. We don't have a good understanding of the impact of federation on the stability of the service, especially since this 'SPARQL engine they have is "shaky"'. So it is unlikely we will had more complexity until we are in a much more stable situation.
Wed, Feb 19
The scalability of WDQS should not depend on users canceling queries manually. We have other ideas to improve performances.
Fri, Feb 14
Thu, Feb 13
Closing this as it seems that we have the rate limiting that we want at this point. We'll continue relying on external mirrors for the time being.
Mon, Feb 10
Wed, Feb 5
Jan 24 2020
Jan 22 2020
This should probably be implemented as part of a more generic throttling / access strategy. It does not make sense to invest in this if we're keeping it just for WDQS.
To some extent, this is linked to T221938. We don't want to start address this before we have a good vision of where we are going in term of overall scaling strategy.
We've already implemented something similar for Gerrit. Making a general solution seems more work than addressing each use case separately.