Page MenuHomePhabricator

New Service Request 'open_nsfw'
Closed, ResolvedPublic

Description

Description: T225664
Timeline: As soon as possible.
Diagram: F29712934
Technologies: Python, Coffee, Docker
Point person: @Mholloway @MusikAnimal @Chtnnh

Event Timeline

fgiunchedi triaged this task as Medium priority.Apr 14 2020, 9:31 AM

@Aklapper anyone I can reach out to for this task?

@Chtnnh: See https://phabricator.wikimedia.org/project/profile/1305/ ; apart from that see the point persons that you listed, I'd say?

I meant someone from the Service deployment team, so that it can be on their radar. T214201 is waiting on this task.

Thank you @Aklapper

@akosiaris Hey! Chaitanya here, a volunteer from the scoring platform team. Need your help deploying this service to production. Please let me know of any follow up questions you may have. Thank you for your time!

Hi,

Is this still something that might happen? I don't see any activity on this task for the last 5 months. Note that it will require a person with deployer rights and considerable investment to deploy it in production. Alternatively WMCS might be an option as well.

Hello!

Yes, we would love to have this service deployed. Although, over the course of the last 5 months, we have developed a newer iteration of the service that utilizes an internally built model and is currently in the process of being deployed on toolforge. Once we have deployed on toolforge and allowed the community to take the model out for a test run, we would love to get the service deployed to production.

@MusikAnimal @Abbasidaniyal @Harshineesriram feel free to add anything I may have missed.

@akosiaris What would the process and expected timeline for deployment look like?

Sounds great! Can you post here when the model is up on toolforge? I'd love to take a look.

@calbon we have hit some snags in deploying the model on toolforge and are in the process of resolving those. But in the meantime you can have a look at the model here. The README contains all the setup and testing instructions.

You taking a look would definitely be helpful!

Hello,

Hello!

Yes, we would love to have this service deployed. Although, over the course of the last 5 months, we have developed a newer iteration of the service that utilizes an internally built model and is currently in the process of being deployed on toolforge. Once we have deployed on toolforge and allowed the community to take the model out for a test run, we would love to get the service deployed to production.

@MusikAnimal @Abbasidaniyal @Harshineesriram feel free to add anything I may have missed.

@akosiaris What would the process and expected timeline for deployment look like?

First and foremost, you would need some team with deployer rights to adopt, own, shepherd to production and most importantly maintain this service. That is your number 1 blocker. SRE would coordinate with that team to facilitate deployment and day-to-day.

Aside from that, the service should conform to https://www.mediawiki.org/wiki/Wikimedia_services_policy. Many of the bulletpoints are about the ownership (e.g. mechanisms for support and response to incidents), some are more technical like having structured logging, structured metrics (we prefer the prometheus format although also support statsd) or be in a language and toolset that is approved WMF wide (e.g. service-runner).

Assuming all of the above is ready, https://wikitech.wikimedia.org/wiki/Deployment_Pipeline is the guide for the steps after having something deployable ready and wanting it deployed.

I can only give you a timeline for that last part, it's usually a quarter(usually a bit less) of a year assuming proper communication beforehand.

If all of this sounds too much, it's because deploying in production is way more involved than setting something up in WMCS where almost none of the above is required.

I understand @akosiaris !

Is it possible to deploy to production as volunteers? As in, is it possible for long time volunteers to have deploy rights? If that is not possible, we would still have some options to consider, maybe @MusikAnimal or @calbon could volunteer based on their interaction with the model and see if it is something they would like to support. But if the volunteer option is possible, it would be preferred.

I am going through the points you have linked to here (https://www.mediawiki.org/wiki/Wikimedia_services_policy) and will get these sorted as soon as possible. I should be able to give you a rough timeline on getting the service deploy ready by next week.

If all goes well, do you think we would be able to deploy before the end of this financial year?

Thank you for the follow-up! :D

I understand @akosiaris !

Is it possible to deploy to production as volunteers? As in, is it possible for long time volunteers to have deploy rights? If that is not possible, we would still have some options to consider, maybe @MusikAnimal or @calbon could volunteer based on their interaction with the model and see if it is something they would like to support. But if the volunteer option is possible, it would be preferred.

It is possible (we do already have volunteers deploying things), you will need to find people in WMF to vouch for you though and go through the legalities of the thing (sign NDAs, L3 agreements, etc).

I am going through the points you have linked to here (https://www.mediawiki.org/wiki/Wikimedia_services_policy) and will get these sorted as soon as possible. I should be able to give you a rough timeline on getting the service deploy ready by next week.

If all goes well, do you think we would be able to deploy before the end of this financial year?

If all goes excellent it might be plausible, but I wouldn't bet on that, the end of this financial year for WMF is June 30.

I see. I think the team (@Harshineesriram, @Abbasidaniyal and I) will have to put some thought into that.

As far as the timeline is concerned, we just want to see this tool deployed ASAP so that it can start having impact.

What is the relation of this task to T214201? Does this one block the other one (=subtask)?

calbon moved this task from Unsorted to In Progress on the Machine-Learning-Team board.
calbon added a subscriber: Mholloway.

I am tentatively removing Service-deployment-requests as I don't see how serviceops (the onwer of that tag) has anything to do with this for now. Feel free to readd. I am also adding serviceops-radar so that as a team we don't lose all visibility into this.

elukey subscribed.

This task needs a bit more clarification, we already have an experimental model server for nsfw content. Putting back in "Unsorted" status so the ML team can re-asses the work to be done.

This work is superceded by the NSFW model made by @Htriedman and deployed by @AikoChou to experiment model server. Closing ticket.