This project ended up going a different direction.

I'm not available to work on this, @JAllemandou's data served my purpose but it seems like there was some interest in maintaining a table like this.

Great! Can I get some help transferring this data?

Thank you @akosiaris. For your info, my revisions deadline was just extended from September 9 to October 15th. I'll be able to wrap up this project by the end of September. But I might still be working after the 9th.

I got reviews back today. They are pretty positive, but one reviewer asked for additional summary statistics that weren't originally reported. I'm also thinking of adding a small analysis to the appendix that will help address another reviewer's concern. The revisions are due September 9th. Can we renew an extension through then? Thank you!

This all sounds fine with me. No CSCW reviews yet. I'll update this thread when the time comes.

Okay,
For the project with @Halfak any risks would arise from the internal histories of historical ores scores of revisions. The rest of the data used in the project is the publicly available wikimedia histories and derived from the open source ores projects.. I expect that this will be low-risk since I'm aware of plans to release the ores scores publicly. I'm not sure, but this might relate to other fields in the ores scores table I used. In the data that I saved in my home directory, the only fields I kept from the scores table are the revision scores.

Hi @leila,
I don't have reviews back yet. I expect them to come next week. If it's a hassle we can wait and see if the reviewers want anything. There were a few things I wanted to double-check in any case but that isn't essential. Even if I don't need access to the data to address reviews, I'd like to have a copy of my home directory on stat1006.

@Nuria this is work with the scoring team so I think the WMF collaborator is likely to be @calbon or someone on their team.

Thanks @elukey and @Halfak for catching this. I'm not actively working on this project, but I expect that reviews will come back next week and I may have some work to do for the revisions. I also want to package up code and publicly available data for public release.

Hi! here's my request for Kerberos credentials for Hadoop access on stat100X and notebook100X. My username is nathante. I'm a volunteer researcher and I've contracted with the WMF.

Thanks!

Hey @Tgr. Here's my best guess at what the historical thresholds were and when they changed. Missing values indicate that no threshold was set for a given class of edit. These numbers are the result of a fairly complicated process based on parsing old configuration files and loading old versions of the models and I'm still troubleshooting some aspects of it. So I will really appreciate it if you can let me know whether this looks right to you. Thanks!

@Tgr, It sounds like you have the old scores right?

Hi Tgr. I'm working on this! I should be able to send the threshholds over in the next day or so. This is very much a research project so buyer beware! You can checkout my code at https://github.com/groceryheist/ores_bias_project/blob/master/ores_archaeologist.py

Hey @Ottomata,
it turns out that I think stat1006 is a better fit for my purposes since it has ORES dependencies (mainly hunspell) that were missing on the notebook machine. So once I finish moving that part of the project over to stat1006 I can reduce my usage on notebook1004.

Yeah I would say so. There's always room to improve it i.e. to support more tools and wikis. I also haven't done quality checks for wikis that aren't in our study.

I deleted a couple Gb that I don't need. Unfortunately most of the space I'm using is from ORES assets so I can't really store it in Hadoop. Maybe I should move this work to a different machine with more space? It's a bit inconvenient to have to move between different machines for different tasks though. Is my current usage OK for the month or so? After that I'll clean up.

Sorry I lost track of this bug until today. I think it is really regrettable to turn off the instrumentation. The utility of the data is greatly lessened by gaps in the collection window.My understanding is that the instrument should only send two events for each page view. The sampling rate has been quite high at 10%, explaining the high number of events.

Thanks Nuria!

Hi Nuria. I'm proposing to start with a one-off release that I can handle easily. I can also do some work to set up automated scheduled releases, but I don't want to commit to owning it in the long run.

I do not think we should be in a rush to remove this instrumentation.

Thank you!

I'm making a list of people who helped with labeling campaigns for the different ores projects.

The changeslisthighlights and changeslistfilters schemas were deleted along with the data. So we don't have the data that we would want to have for this.

I created a task T224918 for that analysis.

Ok I see. A hostile dependency could be a big problem. I'm not looking to argue, just sincerely curious. I get involved managing a sort of ad-hoc spark setup on the UW cluster, so maybe I can learn something useful :)

Having said this, Andrew is planning to work on the Spark 2.4.2 upgrade and he will take a look if pandas could be upgraded as well :)

I see, for Python packages I usually use pip instead of Debian since python tends to move much faster than Debian. Of course, I'm just managing this for myself and not supporting a whole organization :), But I'm also curious about why you use Debian for this.

@elukey, thanks. It seems like I'm experiencing a regression then. I can work around it for now. See you tomorrow!

@Nuria yes. My understanding is that they are when pp_propname == "wikibase_item"

My ultimate goal is to identify, from a random sample of ~500,000 to ~50,000,000 edits from different language Wikipedias.

1. Which edits are to biographical articles.
2. The gender or sex of the subject of the biographical articles.

Thank you Nuria. Are you saying that we'll be able to sqoop the prop_tables in May at the earliest? Would it be okay to lookup a sizable number of pages in the prop_tables in the meantime? I'm thinking on the order of 20,000 pages per language.

Also
https://sr.wikipedia.org/wiki/Special:ORESModels has a strange threshold (0,1) for goodfaith.

Wikipedia-to-Wikidata linkage patterns (T209891#4798717, using the page_props table)

@Ladsgroup Oh sweet thanks I'll do that.

@Harej Indeed. I was already planning to do something very similar to this in the course of my project. I may be actively working on some of these subtasks starting next week.

A handful of thoughts:

Good question. I don't think so, unless there are additional schemas that we might need to join with that have keys other than page_id or revision_id. We already record namespace.

A related problem is that pages can move. Right now we record page_title, but different pages can have the same_page title at different times. It would also make downstream analysis much more convenient to have page_id in the schema.

Created task https://phabricator.wikimedia.org/T205454 for LDAP access

I still don't have access to SWAP. I understand that I need to be added to the nda LDAP group.

@RobH: Great. Thanks!

Here's the contract that I signed and sent to @ovasileva : REMOVED
It includes a "Contractor Confidentiality Agreement. Is this the NDA we are looking for?
Per the contract, the end date is November 16th 2018.