Page MenuHomePhabricator

Request creation of WikiLoop VPS project
Closed, ResolvedPublic

Description

Project Name: WikiLoop

Wikitech Usernames of requestors: Xinbenlv, Zzn_google

Purpose: WikiLoop is a series of effort initiated from tech industry and intend to contribute tools and datasets back to open knowledge world.

Brief description: The first a few tools will include https://en.wikipedia.org/wiki/Wikipedia:WikiLoop_Battlefield for example, a counter-vandalism tool.

How soon you are hoping this can be fulfilled: This week or ASAP. We already have the tool built, but due to the following two reasons we can't use what we currently have.

  1. We have been using Heroku, but due to Heroku's IP range(from AWS IP pools) falls into the NOP blocks, the users of our tool can't use our tool to make direct revert or rollback, even if they are logged in.
https://meta.wikimedia.org/wiki/Steward_requests/Global_permissions#Global_IP_block_exempt_for_IP_Range_of_WikiLoop_Battlefield
There is a block configurations to let logged in users edit while on blocked IP: However we usually do not allow it on NOP blocks. We cannot let anyone using your tool be exempt from the block, this is a software restrictions. There's just no way to do that. And since we are not going to unblock AWS — thus Heroku — I think your only bet is to follow GZWDer's advice. — regards, Revi 07:54, 4 November 2019 (UTC)
  1. We have tried Toolforge and it doesn't support our NodeJS version (v10 and above). The Toolforge Kubernetes only supports v6, and the Toolfoge GridEngine only supports v0.10 of nodejs.

Therefore, we have no choice but to request a VPS

Event Timeline

Toolforge has as node v10 container:

$ webservice --backend=kubernetes node10 shell
Defaulting container name to interactive.
Use 'kubectl describe pod/interactive -n bd808-test' to see all of the containers in this pod.
If you don't see a command prompt, try pressing enter.
$ node --version
v10.15.2

@bd808 thank you! I can run it now, but now do I need to change anything to wire the new HTTP proxy to node10 instead of nodejs?
I can visit https://tools.wmflabs.org/wikiloop when using nodejs but it shows 502 bad gateway when using node10

@bd808 thank you! I can run it now, but now do I need to change anything to wire the new HTTP proxy to node10 instead of nodejs?
I can visit https://tools.wmflabs.org/wikiloop when using nodejs but it shows 502 bad gateway when using node10

It usually only takes a few minutes for the service that monitors the Kubernetes cluster to register a new webservice with the front proxy. If you are having problems with that and are not able to see the cause in your webservice's error logs or the Kubernetes pod state then please do open a new ticket with the details you are able to gather so we can help you troubleshoot.

It looks like this may already be resolved. I can see that https://tools.wmflabs.org/wikiloop is responding with a "hello world" page and that it is running the docker-registry.tools.wmflabs.org/toollabs-node10-web:latest image.

Great! Thank you @bd808

However, because the Toolforge currently still don't support custom image yet and mounting to root path, it's still not possible for us to use Toolforge, can we apply for a Cloud VPS machine until at least feature of mounting to root path url becomes available?

Thank you!

@bd808 thank you! I can run it now, but now do I need to change anything to wire the new HTTP proxy to node10 instead of nodejs?
I can visit https://tools.wmflabs.org/wikiloop when using nodejs but it shows 502 bad gateway when using node10

It usually only takes a few minutes for the service that monitors the Kubernetes cluster to register a new webservice with the front proxy. If you are having problems with that and are not able to see the cause in your webservice's error logs or the Kubernetes pod state then please do open a new ticket with the details you are able to gather so we can help you troubleshoot.

It looks like this may already be resolved. I can see that https://tools.wmflabs.org/wikiloop is responding with a "hello world" page and that it is running the docker-registry.tools.wmflabs.org/toollabs-node10-web:latest image.

Thank you @bd808 for reopen it. Please feel free to ask any clarifying questions.

The goal of this app is to make counter-vandalism easier so we hope to build a web-based tool so it's easier for people to use, while also still easier to verify the trust-worthy-ness of users and their selections.

@Xinbenlv it would be helpful if you explained here your reasons beyond needing to update the express routers in your app to work when mounted as https://tools.wmflabs.org/wikiloop/ for using a Cloud VPS project instead of Toolforge.

@bd808 Yes, thank you for asking.

Here are a few reasons:

  1. we rely on the NuxtJS server-side rendering. Therefore on the server-side initial rendering will call itself which is mount at the "/<app_root>", while the Toolforge will mount "/wikiloop/<app_root>".
  2. it makes the development environment largely inconsistent with the app-development environment which makes community developers harder to pick up the development
  3. having a standard development environment also ensure our Continuous Integration and auto-rollout easier, ensuring faster iteration of development.

There are also other technical changes such as cookie domain, caching, mounting API, socket.io etc, needs to be applied only for this individual reason the is soon to be removed by new feature of WMF Toolforge.

I understand that it still takes at least 2 - 4 months for the WMF Toolforge itself to start supporting mounting on root path, which showing that changing the mounting location is non-trivial development task.

Respectfully.

@bd808 is there any other aspects of the request you expect from us? I can supply other information if that helps~ thank you!

It could also work if we do a IP forwarding from WFM Cloud IP range to a GoogleCloud static IP. as per Global IP exempt application the admins has agreed that static IP is policy-wise ok, it's just technically they can't do it on the mediawiki software. Is that something feasible? That way you don't need to allocate additional VPS machines

We will wait for @bd808 to clarify/resolve the concerns before moving on with this project creation. But he is currently traveling and this may be delayed for several weeks.

I'm sorry I cannot be of more help right now.

@aborrero thank you. While we are waiting for @bd808 to follow up with the VPS creation question, can we discuss the IP forwarding option? Shall I open a new ticket or we can discuss it here?

@Xinbenlv please provide extensive description of the different hosts and IP involved for the "IP forwarding" that you are proposing. I don't want to make a statement without having a clear picture.

We will wait for @bd808 to clarify/resolve the concerns before moving on with this project creation. But he is currently traveling and this may be delayed for several weeks.

I am not going to personally block this request moving forward. I am part of the group that generally reviews project requests, but I am not the sole decision maker.

I do find it interesting that there is simultaneously an argument that this software needs to be hosted by Cloud Services to be successful because of access constraints on the Wikimedia wikis, but that the project maintainer is unwilling to alter the software to be compatible with the constraints of the most readily available hosting platform (Toolforge) on the basis that doing so is "non-standard".

@aborrero thank you. While we are waiting for @bd808 to follow up with the VPS creation question, can we discuss the IP forwarding option? Shall I open a new ticket or we can discuss it here?

This is not something that Cloud Services can allow. Abusing the trusted status of the Cloud VPS IP range on the Wikimedia projects to extend this trust to arbitrary external hosting platforms is socially bad. Beyond that, it is a direct violation of the "Using Wikimedia Cloud Services as a network proxy" prohibition in the Cloud Services terms of use.

I understand that it still takes at least 2 - 4 months for the WMF Toolforge itself to start supporting mounting on root path, which showing that changing the mounting location is non-trivial development task.

This is not a fair comparison. Re-achitecting and extending the ingress proxy layer of a shared hosting platform is not the same level of effort or complexity of making a nodejs project support a configurable mount prefix:

var router = express.Router();
router.get('/', ...);
var mountPoint = runOnToolforge ? '/my_tool_name' : '';
app.use(mountPoint, router);

Thank you @bd808 .

This is not a fair comparison.
Re-achitecting and extending the ingress proxy layer of a shared hosting platform is not the same level of effort or complexity of making a nodejs project support a configurable mount prefix.

I certainly agree with you that there are great difficulty of changing the host point of Toolforge as a shared platform itself, I am sure you can give me a long list.

But I think you have pretty much ignored a large part of my answer in engineering challenges: serside-rendering, cache, cookie domain, socket.io event subscription, etc, to just name a few.
And you largely underestimated the challenges of this side of the story.

We have also already experienced a significant performance issue merely running our app on Toolforge (without being able to access it from web), and we have not even tested the real performance under real usage - we could not do it before spending weeks to allowing the inconsistent mount path. And we can't test the authentication through this too.

If you insist it's easier to apply a single line of code change to our app to adopt the forced path mount point, you can try it by creating a pull request

!Demo GIF one-click-to-deploy a dev instance on heroku

But I am sure it's way more complex than your estimation, certainly more than the time it takes for the team to approve creating a Cloud VPS, if time of approval is concern

If the cost of Cloud VPS is the concern, can you also let us know, because I think we will be happy to support the cost of running this app, just like running it on other cloud services we understand there are limit of resources.

To make our discussion more constructive, I wonder, because Toolforge is going to support root-mounting soon, why don't you allow us to keep shipping features rather than spending weeks developing a hacky solution that will be no longer needed in 2-4 months?

We can be the first apps to adopt your new architecture, wouldn't that be a great way to showcase new Toolforge? Wouldn't that help showing how much the new Toolforge has made developers' live easier?

Mentioned in SAL (#wikimedia-cloud) [2019-11-19T19:25:16Z] <jeh> create project with project admins xinbenlv and zzn_google T237297

JHedden claimed this task.
JHedden added subscribers: zzn_google, JHedden.

Hi @Xinbenlv and @zzn_google, The WikiLoop CloudVPS project has been approved and created.

Thank you all for the approval.

Xinbenlv reopened this task as Open.EditedDec 6 2019, 7:40 AM

Thank you @JHedden , do you know if I need to apply for a floating IP addresses?

In fact it would be even better if I can have static IP address

Problem: can't ssh into the machine.
Following https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances

  1. created and launched an instance.
  2. uploaded public ssh keys to both wikitech
  3. try to do
ssh -J zzn_google@primary.bastion.wmflabs.org zzn_google@wikiloop3.wikiloop.eqiad.wmflabs

got

zzn_google@wikiloop1.wikiloop.eqiad.wmflabs: Permission denied (publickey).

  1. check the console log, it *never* shows

Search in the console output for “Finished puppet run”, BEGIN SSH HOST KEY FINGERPRINTS, and BEGIN SSH HOST KEY KEYS.

but instead it has multiple remote catalog failures such as

2019-12-06T07:37:31.113492+00:00 host-172-16-0-132 puppet-agent[13867]: Loading facts
[ 109.212321] rc.local[433]: [1;31mError: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: illegal comma separated argument list at /etc/puppet/private/modules/passwords/manifests/init.pp:207:23 on node wikiloop1.wikiloop.eqiad.wmflabs[0m
[ 109.213519] rc.local[433]: [1;31mError: Could not retrieve catalog; skipping run[0m
2019-12-06T07:37:36.184841+00:00 host-172-16-0-132 puppet-agent[13867]: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: illegal comma separated argument list at /etc/puppet/private/modules/passwords/manifests/init.pp:207:23 on node wikiloop1.wikiloop.eqiad.wmflabs
2019-12-06T07:37:36.185610+00:00 host-172-16-0-132 puppet-agent[13867]: Could not retrieve catalog; skipping run

Am I doing it right?

2019-12-06T07:37:36.184841+00:00 host-172-16-0-132 puppet-agent[13867]: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: illegal comma separated argument list at /etc/puppet/private/modules/passwords/manifests/init.pp:207:23 on node wikiloop1.wikiloop.eqiad.wmflabs

There was a problem with one of the puppet manifests earlier today. Could you please delete and recreate your virtual machine?

Thank you @JHedden , do you know if I need to apply for a floating IP addresses?

In fact it would be even better if I can have static IP address

You can find more information on using floating IPs with CloudVPS at https://wikitech.wikimedia.org/wiki/Help:Addresses

Please also note that cloudVPS provides a HTTP/HTTPS proxy https://wikitech.wikimedia.org/wiki/Help:Proxy. This would redirect requests from http://<name>.wmflabs.org/ to your virtual machine

Hi @JHedden

That was very helpful, we are now able to create a web service at http://wikiloop-battlefield-dev.wmflabs.org, pointing to http://172.16.0.129:8000 (instace name = wikiloop3)

Our next question is how do we allow using the domain names of "wikiloop.org", such as http://dev.battlefield.wikiloop.org.

The reason we would like to use wikiloop.org domain instead of wmflabs.org because we plan to brand and manage all our tools with our wikiloop.org domain. We currently have

In the longer vision we hope to include other tech industry companies to contribute to WikiLoop efforts,, so there will be more efforts and tools published under the same domain.
I tried to use a DNS record CNAME to point http://dev-wmf.battlefield.wikiloop.org to http://wikiloop-battlefield-dev.wmflabs.org, but it seems not working because it's a web proxy

Could you advice us how to do it?

The project was created in T237297#5676306 and hence this task is resolved. As it was reopened for a separate question in T237297#5717521 for a floating IP address and as that is now handled in T240414: Request for Floating/Public IP address for WikiLoop, I am resolving this task again, as separate topics should be in separate tasks. Thanks for your understanding!