Which tables are you thinking of?

I think the Link Events table, perhaps joined with Collections/Organisations so that the link is clearer?

Hi! I'm the involved researcher. I don't know the schema of these tables, but I can share what I anticipate and hope the data will include.

• Timestamps at which collections / organizations are added to the library.
• Link events, which have revision ids and citation metadata.
• Association between link events and collections / organizations.
• Link events by editors who have recently accessed the Wikipedia library (i.e., within a last week).

I think I can figure out how to merge or join different tables, as long as the keys are fairly clear. Whatever makes the most sense to you should work out.

Note that the first step will be determining if our json dumps are adequate: suggest we provide an example snippet here and check in with our requesting researcher.

note that we do have wikipedia library users associated with linkevent contributions, but it's not retroactive. We can provide data about users affiliated with the library at the time of the data export.

@jsn.sherman

Json should be fine. I can be flexible about formats.

I don't really understand your second comment. It sounds like there's a limitation related to the data you can provide about users?

Sorry to nag, but I'd like to use this data for a project that would have a deadline in May. Any ETA?

In T380735#10530357, @Groceryheist wrote:

Sorry to nag, but I'd like to use this data for a project that would have a deadline in May. Any ETA?

Apologies for the delay! We'll prioritise this next week.

In T380735#10482234, @Groceryheist wrote:

@jsn.sherman

Json should be fine. I can be flexible about formats.

I don't really understand your second comment. It sounds like there's a limitation related to the data you can provide about users?

We can query current library members. If you want data about past links, the library membership checks would be against the current library user set. As members come in and out over time, this will lead to false positive and negative matches when looking at links created in the past.

In T380735#10539266, @jsn.sherman wrote:

In T380735#10482234, @Groceryheist wrote:

@jsn.sherman

Json should be fine. I can be flexible about formats.

I don't really understand your second comment. It sounds like there's a limitation related to the data you can provide about users?

We can query current library members. If you want data about past links, the library membership checks would be against the current library user set. As members come in and out over time, this will lead to false positive and negative matches when looking at links created in the past.

I'm not sure this is accurate, unless I'm misunderstanding - doesn't each LinkEvent have an on_user_list data point, which is based on the time when the event was originally stored?

In T380735#10539302, @Samwalton9-WMF wrote:

In T380735#10539266, @jsn.sherman wrote:

In T380735#10482234, @Groceryheist wrote:

@jsn.sherman

Json should be fine. I can be flexible about formats.

I don't really understand your second comment. It sounds like there's a limitation related to the data you can provide about users?

We can query current library members. If you want data about past links, the library membership checks would be against the current library user set. As members come in and out over time, this will lead to false positive and negative matches when looking at links created in the past.

I'm not sure this is accurate, unless I'm misunderstanding - doesn't each LinkEvent have an on_user_list data point, which is based on the time when the event was originally stored?

It's accurate for live data related to events that have been archived. However, I checked and those are available in the archived events. We're actually working up a task to archive to a public object store, so this could be a good use case. However, we'll have to reconstruct the events to match the current schema and then re-export them to get this historical data.

I'll rework this task to fit with that. We'll need to do two things that are related:

• implement archiving to the object store so that we're no longer using the local filesystem
• do that schema migration and put previous archives into the object store

@Groceryheist I've filed the tasks that relate to fulfilling this request; you'll see movement on these over the coming weeks.

In T380735#10480409, @jsn.sherman wrote:

Note that the first step will be determining if our json dumps are adequate: suggest we provide an example snippet here and check in with our requesting researcher.

For this first step, I just want to confirm if it's okay for me to add a zip file here in this ticket. My understanding of the necessary tables:

• links_linkevent (200 rows as sample)
• Corresponding links_linkevent_url to get the matching url pattern
• Corresponding links_urlpattern to get the collection
• Corresponding organisations_collection
• Corresponding organisations_organisation

In T380735#10572401, @Mimurawil wrote:

In T380735#10480409, @jsn.sherman wrote:

Note that the first step will be determining if our json dumps are adequate: suggest we provide an example snippet here and check in with our requesting researcher.

For this first step, I just want to confirm if it's okay for me to add a zip file here in this ticket. My understanding of the necessary tables:

• links_linkevent (200 rows as sample)
• Corresponding links_linkevent_url to get the matching url pattern
• Corresponding links_urlpattern to get the collection
• Corresponding organisations_collection
• Corresponding organisations_organisation

The data you provided in T386397: Sample data for Wikilink research should be adequate

In T380735#10575842, @jsn.sherman wrote:

In T380735#10572401, @Mimurawil wrote:

In T380735#10480409, @jsn.sherman wrote:

Note that the first step will be determining if our json dumps are adequate: suggest we provide an example snippet here and check in with our requesting researcher.

For this first step, I just want to confirm if it's okay for me to add a zip file here in this ticket. My understanding of the necessary tables:

• links_linkevent (200 rows as sample)
• Corresponding links_linkevent_url to get the matching url pattern
• Corresponding links_urlpattern to get the collection
• Corresponding organisations_collection
• Corresponding organisations_organisation

The data you provided in T386397: Sample data for Wikilink research should be adequate

Thanks @jsn.sherman. I'll start working on the subtasks to provide the archives in the object storage.

In T380735#10552750, @jsn.sherman wrote:

@Groceryheist I've filed the tasks that relate to fulfilling this request; you'll see movement on these over the coming weeks.

🙏 Appreciate it very much!

Note that this has become a tracking task; unassigning and sticking in our team epic column

Checking in to ask if progress so far makes it possible to share this data soon? I want very much to analyze this data in May.

Looks like the data has been uploaded? Is it ready to access? I would love to have it as soon as possible. I need to do some benchmarks to request a compute allocation from my institution by June 1.

In T380735#10839433, @Groceryheist wrote:

Looks like the data has been uploaded? Is it ready to access? I would love to have it as soon as possible. I need to do some benchmarks to request a compute allocation from my institution by June 1.

Yes, the data is ready to be accessed! Let us know if you need anything else.-

That's great! Only I don't know how to access it. Can you help?

Vipin sent me this link to the data https://object.eqiad1.wikimediacloud.org/swift/v1/AUTH_wikilink/archive-linkevents/.

It looks good. I just want to ask if the schema documented somewhere? Specifically I wonder what on_user_list refers to?

The field on_user_list is a flag indicating whether the user was a Wikipedia Library member or had access to the resource via the Wikipedia Library when they made the citation (https://github.com/WikipediaLibrary/externallinks/blob/master/extlinks/links/models.py#L152).

Thank you!

It sounds like this is resolved - is that right? Anything else required here?