Unless there's a public interface for e.g. Hadoop you won't be able to route there from a Labs VM. Having a public IP doesn't help with that.

Are you sure you want a scratch start for this and not use existing Analytics infrastructure? Have you discussed this plan with anyone over there?

(No real objection to granting this resource request, if you're sure it's actually going to be useful to you)

In the near future the instance will also have to be able to connect to Spark from RStudio using {sparklyr} on our Hadoop cluster (production).

Connecting from labs to production hadoop will not really be possible.

@Andrew @Addshore

(1) As of Hadoop, I do not know whether there is a public interface, but I can ask. Let me check and I will get back to you.

(2) As of the following: "Are you sure you want a scratch start for this and not use existing Analytics infrastructure? Have you discussed this plan with anyone over there?" - No, but I am eager to learn who would be the right person to approach and ask.

(2) As of the general question on whether am I sure that I need the Lab instance running for this project, please advise in respect to the following constraints:

• The idea is to have an RStudio Shiny dashboard running from a Labs instance, accessible for other people (WMDE, WMF, interested parties); the dashboard will have to be connected to a db back-end, postgreSQL ideally;

• That postgreSQL back-end will be initially fed from mariaDB replicas (fetch mariaDB -> pre-process -> postgreSQL -> Dashboard); however, in the future, and I mean: near future, I would like to have the following workflow implemented: Hadoop/Spark (Cluster) -> postgreSQL (Instance, close to the dashboard) -> Dashboard (Instance).

From your questions and suggestions, I can see that the bottleneck here seems to be the Hadoop/Spark (Cluster) -> postgreSQL (Instance, supporting the dashboard) step. @Addshore says no connection of Labs to Hadoop on production. I need help to understand how to bridge in that case: maybe running a Lab instance is not a solution; is there an alternative? For example, can I have an RStudio and Shiny Server + postgreSQL installed somewhere close to Hadoop/Spark on production (I guess not)?

Thanks,
Goran

In T163672#3213056, @GoranSMilovanovic wrote:

(2) As of the following: "Are you sure you want a scratch start for this and not use existing Analytics infrastructure? Have you discussed this plan with anyone over there?" - No, but I am eager to learn who would be the right person to approach and ask.

The Analytics team own the #analytics-cluster and would be the best people to ask. @Ottomata perhaps.

@Addshore says no connection of Labs to Hadoop on production. I need help to understand how to bridge in that case: maybe running a Lab instance is not a solution; is there an alternative?

It should be possible to do the processing of the data from hadoop within the analytics cluster and then transfer data with no PI out for use on the Shiny Server

@Addshore Ok. Thanks. Let me talk to Analytics then and we will decide what is the best option when I learn about exporting data from the Analytics Cluster.

@Andrew Thanks for support. Let's not rush with a Lab instance until I learn more from the Analytics; please leave the ticket opened.

@GoranSMilovanovic It might also be worth talking to the Analysts on Discovery (Mikhail and Chelsy) - they use R extensively. One thing we do have on our Hadoop Cluster is Jupyter Notebooks to talk to Hadoop and the internal mysql data store -
and these are the helpful instructions Discovery folks wrote to use it with R https://meta.wikimedia.org/wiki/Discovery/Analytics#PAWS_Internal

Approved, I will set this up shortly.

That will do just fine: 'wikidataconcepts'. Thank you.

Regards,
Goran

Thank you very much.