- Guidelines for separating code analyzing survey data from the data itself, in order for the code to be public
- Write a proposal for where to store survey data which tends to contain a lot of PII data.
- Consult with data platform to discuss options for data storage.
Description
Description
Details
Details
- Due Date
- Sep 22 2023, 4:00 AM
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fkaelin | T341818 Migrate and consolidate Research teams' code to Gitlab | |||
| Resolved | fkaelin | T342914 Proposal for handling PII survey data |
Event Timeline
Comment Actions
@fkaelin is this on track to be resolved by Thursday? If not, we should probably change the due date?
Comment Actions
Moved the due date back. We have had some initial discussion that touched on this:
- we will use Organizer Lab code T344625 (which involves disentangling PII data and code) to
- describe the nature of the PII data collected
- evaluate the requirements from research for storing such data
- reach out to data-engineering
- suggestions for storing survey data / discuss best practices
- evaluate storing data in the data infra (e.g. ceph / kerberos)
Comment Actions
After learning more about the nature of the PII data from @YLiou_WMF, and some informal discussions with Joseph in data engineering, this proposal for PII survey data is simple: we will continue with the current way of doing this.
- the survey data is stored on locked down google drives. the amount of data is very small, and backed by a privacy policy that is part of the survey process
- data engineering has a number of ongoing efforts that will impact how PII is handled (moving away from kerberos and hdfs to e.g. ceph), this special use case is not a good fit/use case for this work
Instead, the focus for this quarter is to minimize the entangling of the existing code analyzing the survey results and the PII data itself (e.g. T344625), in order for the researchers to make the code available publicly. If as part of this work we identify the need for more guidelines for future work related to PII data, we will capture that in a new task.
Closing this as done.
Comment Actions
@fkaelin thanks for looking into this. Should this kind of data be tracked as part of our internal tracking then? (private link) https://office.wikimedia.org/wiki/Research/Documentation#Internal_documentation . If yes, you can work with @Miriam to make sure it's tracked in this case (and possible future cases).