@ItamarWMDE what would be your guess about this?

seems like the containers are still up, I suspect this has to do with ShinyProxy actually, rather than the container fully crashing

@ItamarWMDE @Manuel I will see to take a look at it ASAP.

Thank you @GoranSMilovanovic that would be super helpful!

@GoranSMilovanovic Is there a prognosis when this bug will be solved?

I could understand that it’s not the priority, but could we hope for a soon resolution ?

@Otourly I am currently focused on my professional obligations. I will have some more time following December 26, but then I will also need to rest and spend time with my family. So, as soon as the time permits, I will take a look at this. Also, if the issue here is related to ShinyProxy indeed - as @ItamarWMDE hypothesized - then my help is not even necessary. I do not know if anyone tried to test that hypothesis or not in the previous months, but it would definitely be my first choice in addressing why the dashboard stopped.

I just discover that there is an issue with Cognate and the newly created Wiktionary projects. Two examples:

• • On the Igbo Wiktionary: the entry agba links to all other Wiktionary version, but the same page on the other Wiktionary versions does not link to the Igbo page (for example, here is the French page that does not contain any link to the Igbo Wiktionary)
• On the Gorontalo Wiktionary: the page taluhu links to all other Wiktionary version, but the same page on the other Wiktionary versions does not link to the Gorontalo page (for example, here is the French page that does not contain any link to the Gorontalo Wiktionary)

I did not check if there is already a Phabricator ticket for this problem (if not I will create one) but it may be related to the problem described in this ticket.

@GoranSMilovanovic on October 31st you stated: I will see to take a look at it ASAP. On December 16th you suggested that a solution might have something to do with "ShinyProxy". Unfortunately there seems to be no one else around who is able to do something about it. Now other problems are added, which may or may not be related. Could you please find time to look at these problems soon, or find someone else who is able to do so?

Dans T321666#8504825, @Pamputt a écrit :

I just discover that there is an issue with Cognate and the newly created Wiktionary projects. Two examples:

• • On the Igbo Wiktionary: the entry agba links to all other Wiktionary version, but the same page on the other Wiktionary versions does not link to the Igbo page (for example, here is the French page that does not contain any link to the Igbo Wiktionary)
• On the Gorontalo Wiktionary: the page taluhu links to all other Wiktionary version, but the same page on the other Wiktionary versions does not link to the Gorontalo page (for example, here is the French page that does not contain any link to the Gorontalo Wiktionary)

I did not check if there is already a Phabricator ticket for this problem (if not I will create one) but it may be related to the problem described in this ticket.

I have created T326432 to track this specific issue.

I have found some time to inspect the problem - and the problem is very strange.

All Wikidata Analytics + Wiktionary Analytics Docker images are being rebuild right now in CloudVPS. The services will be unavailable for some time.

For a moment, I was able to put the Wiktionary Cognate Dashboard back online on https://wiktionary-cognate.wmcloud.org/Wiktionary_CognateDashboard/, but it was crushing for some other reasons (related, or not related, to T326432)

I will be reporting back ASAP, but please take into you consideration that my time to work on these issues is quite limited. Thank you.

Ok, we have a problem.

@Manuel @ItamarWMDE @WMDE-leszek @Lydia_Pintscher

After having rebuilt only two of the Wikidata Analytics images on wikidata-analytics in the wmde-dashboards CloudVPS project - in an attempt to actually cure something else, namely the Wiktionary Cognate Dashboard - the whole system became irresponsive following the same pattern that @ItamarWMDE and I have already faced (and failed to understand) in 2022 during our work on the wmdeanalytics CloudVPS projects. It is simple:

• the containers time-out under ShinyProxy.

Currently, not only the Wiktionary Cognate Dashboard, but the whole Wikidata Analytics suite is unavailable.

The problem in CloudVPS has nothing to do with any known ShinyProxy related problematics:

• I can run the Docker containers manually and they respond without any errors

• I can also replicate the whole Wikidata Analytics suite, with exactly the same architecture (for R specialists: ShinyProxy + containers from images built under the Golem framework) as we have on our CloudVPS virtual machines, on a separate, test server, and it delivers there without any hiccups;

• ergo: this has something do with the setup of the CloudVPS machines, be it some Puppet run that changed something, or something else that a DevOps engineer might be able to help around - but that I can certainly not help around.

It would be a pity to lose the functionality of a five year long development of Wikidata Analytics and Wiktionary Cognate (2017 - 2022) for reasons related to system engineering. And please do not tell me that it is something peculiar, R related thing - R is the 13th most popular programming language on the Tiobe index in 2022, it has native support in standard MLOps solutions like AWS SageMaker, etc. So it's not R, and it is not ShinyProxy, nor it is anything else R related; it is something related to the underlying setup of the CloudVPS machines.

Let me know when you find someone who can make a deep dive into this.

What I will probably be able to do in the following weeks is to prepare a production environment on one my (DataKolektiv) server and run Wikidata Analytics and Wiktionary Cognate in a production environment from there, so that the community can have the dashboards available.

As of the CloudVPS related problems in running these two systems under ShinyProxy, there is nothing else that I can do; you need to find someone with expertise in web-servers (ShinyProxy is nothing more than a web-server) and virtualization to identify and solve the problem. From all my previous attempts, Wikidata Analytics and Wiktionary Cognate run safely in any environment that I have tested except CloudVPS.

Hey @GoranSMilovanovic, sorry it took so long to respond. We will definitely look into this topic more in depth in the future, especially when our new data analyst begins.

Regardless, however, since we now see that dashboards are not accessible on both old, and new instances of wikidata-analytics(-1) and wiktionary-cognate(-1), would it be safe to remove the old instances from wmde-dashboards, or do we need to back up some information before this is done?

@ItamarWMDE Is it possible to be a little more specific about the intended timeframe? It would be useful to know when the new data analyst will begin and when s/he is expected to give an assessment of the time needed to restore the Cognate dashboard.

Hello @MarcoSwart thank you for the question. Unfortunately, I cannot tell what the specific timeframe is for this particular task. I am responding to this ticket in my role as a tech lead in order to find out if we can shut down instances on CloudVPS and conserve some resources that are undoubtably needed in our server infrastructure.

For the record, ping @AndrewTavis_WMDE ;)

Thank you, @VIGNERON! Appreciate getting pinged here so I can get an overview 😊 We (@Manuel, @ItamarWMDE and I) will be discussing the state of the R dashboards in early June. We'll be able to give a better estimate on this and other related tasks at that time :)

Is this issue still being worked on?

Hi @Bicolino34 👋 Thanks for reaching out :) We are still working on tasks related to this dashboard - at least bringing back some of the data processes.