Ok, I have just realized that the right page is https://research.wikimedia.org/index.html, but the README in the repo links to GitHub.

We are done with the upgrade, so the resources can be scaled back.

Hi all,

In T314214#8117322, @rook wrote:

ram was already listed as 16384

In T306103#8113155, @Andrew wrote:

If you need increased quotas in order to create new Bullseye VMs before deleting the old Stretch VMs (or need more storage quota for a bigger Cinder volume) please open a quota request (https://phabricator.wikimedia.org/project/view/2880/) and link here.

Hi all,

Additional context on why do we need this.

is there anything we can do to help with this? Any info we can provide?
Let us know and thanks in advance.

In T288963#7287895, @Peachey88 wrote:

@CristianCantoro Could you please provide a description that you want to be on the project?

In T288069#7260113, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2021-08-04T16:11:41Z] <dcaro> rebooted the VM and it's back up, with prompt on virsh, and reachable through ssh, CristianCantoro can you try and confirm?(T288069)

In T288069#7259813, @CristianCantoro wrote:

Should I rebuild it with the "Rebuild instance" command from horizon?

In T288069#7259675, @dcaro wrote:

@CristianCantoro having spent some time trying to debug the issue, it seems it's going to be tricky to continue debugging, would you mind trying rebuilding the instance? (if you don't mind losing what's there now)

In T288069#7258910, @dcaro wrote:

The machine seems to have failed to setup the local disks when running cloud-init:

Thanks for the pointers, however, let me clarify a little bit the background of our request:

1. we are already working with mwxml and we are already working in a streaming fashion on the dumps. (Also, let me point out that all the code we are writing is public in this GitHub organization;
2. the main problem in terms of storage is the WikiConv dataset, which is quite large, as it is a collection of all discussions from a given Wikipedia. We can try to find another solution for at least preprocessing this dataset so that we are able to do everything with less data;
3. at this moment, it is quite urgent for us to be able to deploy the servers and test our code, so we would like to find a reasonable compromise to be able to deploy our system.

thanks for your question. At the moment we are processing the following datasets:

• MediaWiki history dumps
• Wikipedia XML dumps
• WikiConv dataset

the analysis that we do are applied to all languages, with the exclusion of WikiConv for which we only focus on ca, en, es, it.

In T267162#6860744, @CristianCantoro wrote:

Any idea of why this is necessary?

I have created an instance of a VPS within this project, for which I have an internal IP address (172.16.3.146).

My 2cents: I created a new query by mistake, it is a draft and the fact that I cannot delete it is super annoying. I am ok with the idea of not deleting published queries.

Thanks, @srodlund, I think this bug can be closed now.

In T266710#6588791, @srodlund wrote:

I have spec it out a little more, but I don't think it would take too long to put together a similar chart. The main question is where you would be most likely to discover it when setting up an instance.

Do you feel that this would be a good place? https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_Instances

In T251777#6140199, @CristianCantoro wrote:

My fear is that selecting a page by id would not be exactly equivalent to select it by title and since these data come from web requests, using the page title would be the "right" way to do it.

My 2 cents:

• Redirects are very useful and something to be taken into account when working on Wikipedia - the paper linked by Isaac is the prime reference in that regard - however, handling them is quite tricky. I have some experience in that, having built snapshots of the graph of Wikilinks for several Wikipedia over several years ([shameless plug] see the paper, the code is on GitHub). I have also worked with redirects and the pageviews data (code on GitHub). In short, it would be useful to have the redirects solved, but it is a project on its own, IMHO.
• Page ids are very useful, but beware that all kind of quicky things can happen to them over time with page moves, deletions and re-creations, etc. My fear is that selecting a page by id would not be exactly equivalent to select it by title and since these data come from web requests, using the page title would be the "right" way to do it. Note that this is kind of a subproblem to the redirect problem in some sense. I would like to put together some concrete examples of what I mean, but I really do not have the time to do that at the moment.

It seems that I have had a problem with Caddy's cache plugin. Now, the server is back up.

I've fixed this, it seems that Caddy implemented a directive to handle this case, here's the pull request caddy#2144.

In T232213#5472235, @Reedy wrote:

Have you considered asking Caddy for support with their software? :)

In T232213#5471251, @Reedy wrote:

https://github.com/caddyserver/caddy/issues/1011 looks a lot like the issue you're experiencing

In T232213#5471202, @Reedy wrote:

I could reproduce this bug on multiple browsers and I have no reason to believe this is a client-related issue, nor I think this is a configuration issue of my proxy.

What does your proxy do with the Location: header? As it kinda seems like it's not capturing it and potentially just passing it through

In T232188#5470736, @Aklapper wrote:

There are many other small quirks, such as the fact that [[Special:Random]] redirects automatically to the main domain (wikipedia.org), maybe I should file a separate bug for this.

Yes, please do, with good steps to reproduce. :)

In T222151#5146583, @Aklapper wrote:

Which website is this about? I assume LDAP on wikitech.wikimedia.org and not SUL on mediawiki.org?

I am taking on, together with @Geofrizz and @Alessandro.palmas, the administration of these machines.

I have also published the pagecounts-ez files, for the period from 2007-12-09 to 2011-11-15, these are the same files as are available through the Google Drive link above, but hosted by my University.
http://cricca.disi.unitn.it/datasets/pagecounts-ez/.

it may be of interest that I have published the sorted pagecounts-raw dataset. You can find it at: http://cricca.disi.unitn.it/datasets/pagecounts-raw-sorted/.
There are more info at this page: http://disi.unitn.it/~consonni/datasets/wikipedia-pagecounts-raw-sorted/.

In T188041#4445976, @Milimetric wrote:

That works, let me know if you need to take them down before I get to copy them, and I'll try to squeeze it in.

In T188041#4438495, @Milimetric wrote:

@CristianCantoro where do you have access? The files have to end up on terbium at some point, but I can move them to the right place if you put them anywhere.

In T199461#4420546, @Reedy wrote:

This isn't a regression, it's a requested "feature" right? ie there wasn't checksums before that are now gone

I am done with the computation, I have processed all pages untill 2011-11-15. I have 1432 files averaging ~400MB in size, for a total of 581GB total. I can transfer them to WMF server if you tell me where.

Sibling iOS bug T198693

In T188041#4296933, @Milimetric wrote:

This is great, thanks very much Cristian. I've been out with an injury for a few weeks. Now I have to catch up with other stuff but I'll get back to this not too far in the future.

Ok, I have written a script that uses GNU Parallel to process multiple days at the same time. Using 6 cores I was able to process 23 days worth of data in a little more than 4 hours, as expeced.

Ok, I am done writing the new "streaming" script. It takes ~70 minutes on a single core to process one day. About the RAM, it takes 20GB at peak (when reading the input data and sorting the rows), but then it uses ~4GB, and it is using just one core.

@Samwilson, thanks for the heads up. I have added you as a maintaner of the tool wscontest.

So, I am basically writing another script that does not use Spark but simply process the data in a streaming fashion (the basic idea of the algorithm is: take one day worth of data, sort them by page and then process the data stream one line at a time).

I have run other tests and they took between 8 to 9.5 h using between 34GB to 36.5GB on a single machine with 8 cores. Also, I limited the problem with the data from 2007-12-10 to a few files. (I suspect the root of the problem may be some corrupt file).

I have run the script over 1 day worth of data (2007-12-11) , it took a little more than 8 hours (484 minuts) and around 34GB of RAM on a single machine with 8 cores. I am testing on another day (2007-12-12).

I worked on this during the Wikimedia hackathon and now I have a final version of the code that computes the daily total and the compact string representation for hourly views from the pagecounts-raw data.

In T188041#4210717, @Milimetric wrote:

The code is in this repo if anybody cares to take a look:
https://github.com/CristianCantoro/merge-pageviews

I only took a very quick look so apologies if I missed some nuance, but I think you might be trying to crunch all the data in memory before outputting, which would definitely use up all your memory (pandas is not very light and the files are big). So here's a complicated writeup https://indico.io/blog/fast-method-stream-data-from-big-data-sources/ that essentially boils down to: "use the yield concept" to process only what you need in memory and dump it back out to your output before going on to the next thing.

But of course, this is mostly solved for you in Hadoop where you can just write some logic and the parallelism is HDFS's job, with no work from you. You can even put a Hive table on top of all the source files and then do the transformation with a simple SQL statement.

In T188041#4208340, @Milimetric wrote:

Thanks so much Cristian, I'll wait for your update. Now, I thought of another thing. If you transform the data on your machine, you'd have to then upload it which may take a really long time.

Instead of that, do you want to run the script(s) on our machines? It would duplicate the processing but eliminate the transfer, which is more of a bottleneck. Either way is ok with me, whatever you prefer. If you do want to run it on our machines, I can help debug the logic and launch it myself, or we can get you access to do it yourself.

@Milimetric, no problem.

In T193759#4190449, @Milimetric wrote:

@CristianCantoro: I'm sorry I didn't think of this, but isn't this what pagecounts-ez already did? https://dumps.wikimedia.org/other/pagecounts-ez/merged/

Oh but you have them going back to 2007. Hm, maybe using your data we can resolve this task: https://phabricator.wikimedia.org/T188041 and then it would be available in a fairly compressed way for everyone.

I'm coming from T193759, I can help with this. Is the script doing the merge available? I can run it on one of my machines and let it run even for several days.

Anyway, I am totally ok with uploading these data, I think I just need a server where to save them.

In T193759#4179085, @Milimetric wrote:

In the meantime, maybe we can host the data on dumps as files?

Thank you! I was able to login!

In T193647#4175604, @Framawiki wrote:

[...] Can you create a text file in your $HOME on Toolforge that references this bug and then ping us here?

I was able to:

• create a Wikitech account named "CristianCantoro SUL": https://wikitech.wikimedia.org/w/index.php?title=User:CristianCantoro_SUL&redirect=no
• create a Mediawiki account named "CristianCantoro SUL": https://www.mediawiki.org/w/index.php?title=User:CristianCantoro_SUL&redirect=no

In T193616#4174462, @Aklapper wrote:

That's what I tried to explain in T193616#4174283. The workaround would be creating yet another LDAP account, link to that LDAP account from your Phab account, then unlink the SUL account from your Phab account, then link that (now free) SUL account to your other Phab account that you want to keep.

[...]

I already explained above that this is not possible.

In T193616#4174283, @Aklapper wrote:

However, that requires that the SUL or LDAP account is not yet connected to another existing Phab account, which is the case here. :-/

Ok, if I go (logged in as CristianCantoro_SUL) to https://phabricator.wikimedia.org/settings/user/CristianCantoro_SUL/page/external/ and I try to disconnect it I get the following message:

I see that most of my activity is done with CristianCantoro, I would like to keep this. I think I can disconnect CristianCantoro_SUL and re-connect my Mediawiki accounts to CristianCantoro. I will try it now.

Commenting here with the other account to confirm the request.

I second what Legoktm is saying: adding the option (or default) of using Orbot to route the traffic of the Wikipedia is completely independent of creating a Tor hidden service.