For reference and context: T408387: CloudVPS instance for ProVe

We can do the DB synchronisation using a user's private key to access the Cloud VPS machine/database. Is this actually allowed and would there be a user-independent solution to do this that does not depend on a user's private key?

It is not strictly prohibited as far as I know. I would recommend creating a dedicated "service account", i.e. creating a separate Developer Account that is not tied to a specific person but has a name like prove-sync, then generate a dedicated SSH key to that user, and use that key.

Would frequent calls between the Cloud VPS instance and external server(s) violate the use policy? In particular we are concerned that this type of connection has a fixed number of calls before the external server is blacklisted (cf. https://wikitech.wikimedia.org/wiki/Robot_policy). In fact, we have been inadvertently blocked once, temporarily.

What are the external servers? And what kind of calls are you planning? Can you share more details of when/why you were blocked temporarily?

Is there a more elegant way to do all this or someone we can speak to to discuss the detail of potential solutions?

From your diagram the "cache sync" does not need to happen via SSH, it could also happen via an API that runs in your external server or in the Cloud VPS server, and is contacted by the other server over HTTPS. The API could use whatever authentication method you like, and you would not need an additional Developer Account as described above.

You could also consider periodically syncing the cache to an Object Storage bucket, that can be accessed from both servers. In this case, you should access the bucket using a dedicated Developer Account that is not tied to a person, as you would do in the SSH case.

Dear Francesco,

Thank you very much for these answers.

We have been fixing some other related technical problems, due to differences in apache modules and some related packages.

Regarding your answers below.

1. We will create the "service-account" as you suggest. This seems like a good way of compartmentalising things.

2. You asked: A) What are the external servers? B) And what kind of calls are you planning? C) Can you share more details of when/why you were blocked temporarily?

A) Once we separate the functionality, inference (i.e., reference verification proper, which requires GPUs) will be done outside CloudVPS. User communication will remain in Cloud VPS and we hope this will enable the use of the service as an official gadget eventually. So the "external" servers in this case will be specific servers linked to the AI infrastructure needed to do the verifications and insert the results into the DB in Cloud VPS for user service.

B) We will need to inject verification results into the Cloud VPS instance and we will need to queue requests from Cloud VPS into the (external) inference server(s). Additionally, external calls from this server(s) will be made for the WikiData API to get the relevant information about the item being processed. Of course we will attempt to limit the calls, but this will be largely user-dependent. We believe these are all Restfull API calls.

C) Regarding the blocking, it happened when requesting information from ProVe's server to the WikiData API. We currently have 3 ProVe processing endpoints running in parallel (externally). We did not know about the robot policy before we did this, so we were rate limited and eventually blocked from making calls to the Wikidata API. Following this (and various conversations here), we updated the endpoints so that the headers in the calls followed the instructions in the robot policy, and since then we have not had any issues. We hope this is all compliant. Please let us know otherwise.

3. Thank you for your suggestion about syncing... We are looking at options and seeing what's most efficient for the database in questions (mongoDB). I can't immediately see how we could use the Object storage, but we will look into it too.

If you see any specific issues with the proposed architecture and plans, please let us know. Would it be possible to have a quick chat at some stage at all?

Thanks,

Odinaldo

Would it be possible to have a quick chat at some stage at all?

See https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication

In T416590#11740124, @Odinaldo wrote:

3. Thank you for your suggestion about syncing... We are looking at options and seeing what's most efficient for the database in questions (mongoDB). I can't immediately see how we could use the Object storage, but we will look into it too.

It looks like the MongoDB would live outside of Toolforge/Cloud VPS, but just to be sure you are aware MongoDB's Server Side Public License is not considered an Open Source license by the Open Source Initiative (OSI) and thus is not installable in Toolforge or Cloud VPS per our Terms of Use.

Dear @bd808,

Many thanks for picking this up Bryan. As part of the separation of the functionality outlined above, we are revisiting the database provision on both sides.

Are there any particular OSI-compliant database servers recommended for use within Cloud VPS? For example, are there examples of best practice, what integrates better, etc? We are thinking of postgreSQL. Would that work?

Kind regards,

Odinaldo

In T416590#11843859, @Odinaldo wrote:

Are there any particular OSI-compliant database servers recommended for use within Cloud VPS? For example, are there examples of best practice, what integrates better, etc? We are thinking of postgreSQL. Would that work?

MariaDB is the most commonly used RDBMS in both Toolforge and Cloud VPS projects. This is a side effect of it being the database server used for Wikimedia's production wikis and the primary database used when developing MediaWiki. Postgresql is also found in some projects in our environment. I'm sure there are many more that are possible; MongoDB stood out because I know they invented a license that was withdrawn from OSI consideration when it was obvious that it would be rejected as non-free ("No Discrimination Against Fields of Endeavor").

Our Database as a Service offering currently supports both MariaDB (12.2.2) and PostgreSQL (18.3). Toolforge tools with modest database needs (<25GB, ideally much less) can use the ToolsDB manged service. Anything distributed under an OSI-approved license should be possible to deploy in a Cloud VPS project.

This is extremely useful, thanks @bd808. We will migrate to a fully compliant RDBMS and both MariaDB and PostgreSQL seem like good candidates. We are prioritising faster access and scalability. The Database as a Service offering you mentioned looks like a great solution. It has the added benefit of removing the computational DB load from the project server. Our tool cannot run in Toolforge, so we would be looking at Trove, which poses some questions:

1. Are quota increases realistic? I read that instances default to 10GB. This should be OK to start with, but as we will process a large proportion of Wikidata's statements, it will not be sufficient long term.
2. List Item Secondly, this database (say within Trove), will need to be updated from outside Cloud VPS. Will that be a problem? The documentation mentions it should be possible through a SSH tunnel to a machine within the same project.

Many thanks again for your support!

Can you say more about database quotas, what do you anticipate the growth to be over time, to what maximum limit?

Also noting this ticket to let you know that this request has my team's attention now (we manage Cloud VPS) and will be looking at this more closely. Up until now the ticket was tagged in such a way that we didn't see it outright (apologies)

In T416590#12120473, @BLiviero-WMF wrote:

Can you say more about database quotas, what do you anticipate the growth to be over time, to what maximum limit?

Dear @BLiviero-WMF ,

First of all, thank you very much for picking this up.

At the moment, our database is in mongoDB and its size is around 60GB. We are aware that mongoDB is not fully open-source and compliant with WMF guiidelines, so we are converting it to an postgreSQL database.

Our intention is to eventually cover the whole of Wikidata (all statements). There are 1.7 billion statements, but not all have references (and hence will not have a record on our DB). We are surveying and estimating the size at the moment (will have an answer in a few weeks). So the database will grow over time, but will reach an upper limit, proportional to the number of statements in Wikidata.

Is this problematic with Trove? This is all part of a wider discussion involving the points in my main message, which I'm very happy to discuss when you've had some time to consider our proposed architecture.

Thanks!

Thank you @Odinaldo for reaching out and the useful information. I don't think 60GB is problematic, I'm looking forward to the size estimation for the full database. Also in terms of sizing I'd like to know if you have a sense of system load in terms of requests per second coming from users? thank you

Our tool cannot run in Toolforge

Can you tell us more about this? The default compute quotas for toolforge are a bit low but as long as your service scales horizontally across multiple pods we can adjust ram + cpu quotas as necessary.

Overall we (the admins) have a strong preference for toolforge because it makes life a lot easier for the users in the long run :)

Hi @Andrew,

Apologies for the slow response (there has been some staff unavailability slowing processes).

Since the initial choice for Cloud VPS, we have realised that we cannot run the monolithic architecture in it, because it does not have GPUs to run the NLP models. This means we had to propose the hybrid architecture above.

We don't really mind whether to host the user communication part in Toolforge or CloudVPS, as long as the infrastructure can support the service.

At the moment, we have distinct modules (inference, database, user communication) running in separate docker containers.

We need to establish a few things:

1. Can we have a service in WMF (Toolforge or CloudVPS) that is able to host the user communication module? It needs to communicate with an approved DB server (postgreSQL) of a reasonable size running somewhere (e.g., in Trove). This DB needs to be regularly updated from an external server (outside WMF).

2. If the answer is yes, then our proposal is to run the inference server outside WMF (so we can run NLP models and scale inference horizontally) and for this the database server needs to be in a location that both the user communication module and the inference server can access/modify without restrictions on number of requests/bandwidth.

3. Now the next question would be where can this database server run? Is Trove an option for Toolforge? (We know it is an option for CloudVPS) If not, then how can we run a DB server that is accessible by Toolforge?

4. The choice of where the user communication module will run will also depend on the ability of that service to host the module (container). Are there memory/requests/bandwidth restrictions in Toolforge? Is Toolforge appropriate for this?

Apologies this is not a simple response, but there are several complex aspects to consider. Again really appreciate your help and any advice!

Best,

Odinaldo

Thanks for the followup.

1. Probably! The only limitation here is that services hosted in toolforge will communicate via proxies so if you are doing something weirder than standard tcp/ip we might want to talk about that first. There will also be limits on DNS; you'll need to use our standard dns format which will be something like <toolname>.toolforge.org

2. That seems ok. I don't think we're likely to offer public GPU support anytime soon. Bandwidth/connection limits will always exist since hardware is not infinite, but the limits are not extraordinary.

3. It's pretty standard to do this in toolforge. You would wind up with a cloud-vps project restricted to trove-use only where you can create and maintain your own postgres db (this works and is fairly reliable although the web UI for postgres is not great.) Then the tool running on the toolforge k8s cluster could access that database.

4. Toolforge is probably appropriate. Admins can adjust toolforge container quotas as long as they don't exceed the actual size of a single k8s worker node. Do you know how much cpu/ram you would need per pod?

I hope that's everything :)