+1 LGTM, in the future would be good IMHO to explore exactly what went wrong with Toolforge, hosting code there would definitely require less maintenance

Hi @MusikAnimal, @Ottomata, the name pageviews is already taken by the web proxy pageviews.wmcloud.org, we would need either changing the webproxy, or this project name so it does not collide (see T360294: [cloud-vps] creating a new project can override existing DNS entries in the wmcloud.org domain), let me know which one and we can continue with this.

Oh! Can we just use the existent project then? Probably just needs us added as admins.

In T433305#12168403, @Ottomata wrote:

Oh! Can we just use the existent project then? Probably just needs us added as admins.

Maybe, the project is commtech so @MusikAnimal should be able to give you the access (I'll need at least +1 from him to do myself). I would recommend creating it's own project (better per-product, than per-team), but that requires either using a different name or taking down the current https://pageviews.wmcloud.org until it's setup in the new project.

Let me know what you want to do, cheers!

In T433305#12168403, @Ottomata wrote:

Oh! Can we just use the existent project then? Probably just needs us added as admins.

The existent project is the commtech catch-all project that needs to spin down as a legacy team focused project with no team.

name pageviews is already taken by the web proxy pageviews.wmcloud.org, we would need either

Oh, sorry, maybe I misunderstood, but I assumed this meant that there was an existent project named pageviews in which the web proxy was configured.

If so, I was suggesting to use the existent pageviews project.

In T433305#12168577, @Ottomata wrote:

name pageviews is already taken by the web proxy pageviews.wmcloud.org, we would need either

Oh, sorry, maybe I misunderstood, but I assumed this meant that there was an existent project named pageviews in which the web proxy was configured.

If so, I was suggesting to use the existent pageviews project.

Sorry for the confusion, let me rephrase/elaborate. The issue is that the commtech project has registered a web proxy by the name pageviews.wmcloud.org, and currently we can't create projects if they collide with an existing web proxy (because the project creates a DNS zone by the same name). So the ideal action would be to de-register the web proxy in the commtech project (that will break the web service until it brought up later), and create the new pageviews project and create the web proxy there.

Ah, got it.

@MusikAnimal in that case, got another name suggestion? Or, should we use the analytics project after all? Probably not given "better per-product, than per-team" :)

+1 LGTM, in the future would be good IMHO to explore exactly what went wrong with Toolforge, hosting code there would definitely require less maintenance

Every tool I've ever maintained eventually hits a breaking point on Toolforge if receives a lot of traffic. Such is the case with Pageviews. I wish I could explain better, but for what it's worth, it's actually easier for me to maintain a VPS project (at least one as simple as this). Communities are now also used to the pageviews.wmcloud.org web proxy, so I'd rather not go back.

In T433305#12168598, @dcaro wrote:

Sorry for the confusion, let me rephrase/elaborate. The issue is that the commtech project has registered a web proxy by the name pageviews.wmcloud.org, and currently we can't create projects if they collide with an existing web proxy (because the project creates a DNS zone by the same name). So the ideal action would be to de-register the web proxy in the commtech project (that will break the web service until it brought up later), and create the new pageviews project and create the web proxy there.

Eek… okay. Let's just call the project something different then, I guess… how about pageviews-app? Then I should be able to get it all set up, delete the pageviews web proxy, and create a new pageviews proxy in pageviews-app, right?

I don't think downtime is acceptable if it can be avoided. The project name is internal, anyway.

Are there any other relevant tools you might want to have a non commtech VPS project to migrate too? If so, we could think of some more generic name. Otherwise, whatever you like is fine with me.

Not on my end. The other VPS instances on the commtech project are planned to be removed entirely. Popular Pages Bot (which runs on commtech-bot02.commtech.eqiad1.wikimedia.cloud) is the only other one that does anything currently – that one I plan to keep running until it dies naturally.

I guess… how about pageviews-app? Then I should be able to get it all set up, delete the pageviews web proxy, and create a new pageviews proxy in pageviews-app, right?

I just want confirmation this will work. The end goal is to keep the pageviews.wmcloud.org web proxy. If we have to, we can just keep things where they are. That's certainly simpler, I just don't like the commtech part being there.

In T433305#12168693, @Ottomata wrote:

Are there any other relevant tools you might want to have a non commtech VPS project to migrate too? If so, we could think of some more generic name. Otherwise, whatever you like is fine with me.

We very much prefer narrowly scoped Cloud VPS projects. https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_project#Reviews_of_Cloud_VPS_Project_requests has some of the reasons.

In T433305#12168685, @MusikAnimal wrote:

Every tool I've ever maintained eventually hits a breaking point on Toolforge if receives a lot of traffic. Such is the case with Pageviews.

This is really hard to refute or to reason about the causes without live examples. Scaling the compute resources allocated to the webservice is one likely avenue. Anything that has involvement with Toolforge's NFS service (legacy hosting containers, logging to disk) are also going to make Toolforge slower than an equivalent Cloud VPS.

I wish I could explain better, but for what it's worth, it's actually easier for me to maintain a VPS project (at least one as simple as this).

The easier for you part is generally also harder for everyone else. Generally more compute and storage will be used and more knowledge will be needed by whoever comes after you to maintain the system.

Communities are now also used to the pageviews.wmcloud.org web proxy, so I'd rather not go back.

We have the redirects project exactly for the use case of preserving URLs across platform migrations.

I'm not attempting to force this project into Toolforge at this time, but I did feel that some clarification of what is possible and reasonable was in order.

Since DE will be owning this...if we could use toolforge I think we would prefer that over a VPS that we have to occasionally replace for debian upgrades ;)

Given the sysadmin docs on Wikitech, setting up a new VM from scratch should be fairly straightforward. In my experience, I spend a lot more time fixing things I maintain on Toolforge. But I am of course keen to respect Data Engineering's desires, as well as Cloud Services' recommendations. It just hasn't worked for me in the past. There is some file read operations in this tool, but not so much writing to files (apart from normal logging, which can be disabled). I'm also using APCu a lot more heavily in the new version, which should make things smoother.

Before I give this a try, I want to also make it clear that this new version does a lot more server-side. I am banking on the concept that this tool (either in isolation or as part of WMCS) will be able to query the various APIs without the standard rate limiting. Would it be possible that some day, we impose the API limitations on WMCS as well, and we'd need a floating IP to allowlist the Pageviews backend? If so, that's another argument for keeping it on VPS.

I am of course keen to respect Data Engineering's desires, as well as Cloud Services' recommendations.

FWIW, I am not strongly opinionated.

I spend a lot more time fixing things I maintain on Toolforge.

I personally have never used Toolforge ;) So your experience here is valuable!

I just want to do whatever folks say is the best practice :)

In T433305#12168954, @MusikAnimal wrote:

Would it be possible that some day, we impose the API limitations on WMCS as well, and we'd need a floating IP to allowlist the Pageviews backend?

No human knows what may or may not be added as new traffic requirements in the indeterminate future. I can say though that a lot more things will break in addition to pageviews if new restrictive rate limits are imposed on the WMCS network space. It also seems quite unlikely that we would have enough free public IPv4 addresses to accommodate all tools that felt they should have a static limit exception in the eventuality.

Also for Toolforge to work, I think we need to get a build service working. The automatic build didn't work for me, so currently I'm manually deploying, first clearing the APCu / Symfony cache in webservice shell and then building the assets in a Node container, which takes a whopping 1 GB of RAM to reliably run.

That's, uh, not really properly testing that Toolforge can handle the traffic today? Most of the reliability improvements we should see will only be there if the service container is running with --mount=none, which can't be done without a buildservice-based image.

In T433305#12303991, @taavi wrote:

Also for Toolforge to work, I think we need to get a build service working. The automatic build didn't work for me, so currently I'm manually deploying, first clearing the APCu / Symfony cache in webservice shell and then building the assets in a Node container, which takes a whopping 1 GB of RAM to reliably run.

That's, uh, not really properly testing that Toolforge can handle the traffic today? Most of the reliability improvements we should see will only be there if the service container is running with --mount=none, which can't be done without a buildservice-based image.

Your reply sounds like it should have been obvious to know to use --mount=none, but this is the first I've ever heard of such an option. I see now it is documented, but is buried deep in the already-complex instructions for setting up a Toolforge tool :/ After all, my initial tests involved only me using the Toolforge installation, going by the standard documented way to set up a tool, and I was frequently met with the "upstream connect error" error (which started happening again, by the way). So hopefully it's understandable why I was led to believe this wasn't going to work.

Please also keep in mind what might be on the horizon for the Pageviews suite of tools and services, in particular adding a Celery-esque job queue, long-running processes, and potentially a large-ish Trove DB (as opposed to using the file system). I can construct a rootless Docker image so it's a simple push-to-deploy for the next time we need to rebuild a VPS instance.

Anyway, I feel like we're getting off-topic here. The ask of this task was merely to move Pageviews as-is to a VPS project that isn't under the CommTech umbrella. If the answer is no, we should decline this task. I hope it's OK then for the commtech VPS project to remain so I can continue to maintain the service. Otherwise, I may need help getting Toolforge to work. This is a 100% volunteer effort on my end, so it will take me some time to trial-and-error my way through it on my own. If it works reliably, I'm all for it. :) I fully admit I am no seasoned sysadmin, I just know I am able to use VPS with ease given it is pure Debian.

Fair enough. We have been trying to hint at it at the relevant docs (e.g.) but you are right that they're not very discoverable at the moment. (We're probably overdue for a dedicated "crawler mitigation and other performance tuning" help page.)

Anyway, if this is indeed non-trivial to move to Toolforge, then we should do the dedicated project. Let me get a +1 from someone per procedure and then we can figure out what to do with the proxy during project creation. (It is possible, although only by bypassing Horizon and talking to the backend APIs directly, to create a proxy for pageviews.wmcloud.org in the new pageviews project that still points to the old VM in commtech and can be later moved by you to point to a new VM in pageviews. That'll mean deleting the proxy for the couple of minutes it takes to do the actual project creation procedure, however.)

+1 LGTM