Thank you for providing details about the logo detection project, @mfossati! The ML team is excited to explore hosting it on LiftWing.

We have reviewed the demo you provided and would like to ask a few questions to help us effectively build, host, and provide an API to query the logo detection model server:

1.API input and preprocessing:
In the demo, the model accepts an image directory as input and preprocesses the images within specific subdirectories like 'logo' and 'out_of_domain' (see directory structure in screenshot below).

Could you please clarify how the images will be sent to the API? Will they be sent as one image or multiple images? Will you provide image links or serialized image objects? Additionally, what is the expected size of the images that will be sent?

2.API output:
The demo currently visualizes prediction results in a plotted grid as shown in the screenshot below.

LiftWing typically returns API responses as JSON objects (see example). Could you please specify the expected response format from the API?

In T358676#9629389, @kevinbazira wrote:

Thank you for providing details about the logo detection project, @mfossati! The ML team is excited to explore hosting it on LiftWing.

Cool cool @kevinbazira , thanks for picking this up!
To give you more context, the interaction with the model will happen within Commons Upload Wizard, most likely at its Upload step.

Could you please clarify how the images will be sent to the API? Will they be sent as one image or multiple images?

Multiple images. Users can upload multiple images, but typically it's only one.

Will you provide image links or serialized image objects?

Serialized image objects. More specifically, the Upload Wizard currently uses chunked uploading.

Additionally, what is the expected size of the images that will be sent?

The image size is arbitrary, it will depend on what users are uploading to Upload Wizard. The current upper bound is 5.37 GB, but this is usually for video files.
In any case, the model requires inputs of size 224 x 224 pixels, and the pre-processing step will take care of rescaling to that size.

LiftWing typically returns API responses as JSON objects (see example). Could you please specify the expected response format from the API?

The essentials are predictions with their probability scores. I think that a JSON array of per-file objects is a good fit:

[
    {
        "filename": "my_uploaded_file.jpg",
        "target: "logo",
        "prediction": 0.999,
        "out_of_domain": 0.001 
    },

    ...

]

In the example you provided, I also see the latency key, which looks like a nice to have.

Thank you for providing more context @mfossati. I shared this information with the team, and they have a few more questions to clarify the implementation details for the logo detection API:

In T358676#9637065, @mfossati wrote:

In T358676#9629389, @kevinbazira wrote:

Could you please clarify how the images will be sent to the API? Will they be sent as one image or multiple images?

Multiple images. Users can upload multiple images, but typically it's only one.

To prevent potential DOS vulnerabilities, we need to establish a limit on the number of images that can be sent to the API in a single request. Currently, the upload wizard restricts uploads to 50 files. Would you like us to maintain this limit for the API as well?

To give you more context, the interaction with the model will happen within Commons Upload Wizard, most likely at its Upload step.

Will you provide image links or serialized image objects?

Serialized image objects. More specifically, the Upload Wizard currently uses chunked uploading.

The Upload Wizard documentation warns against processing images from the Upload Stash during the upload step due to potential security risks. Will you implement security checks for serialized image objects before sending them to the API?

LiftWing typically returns API responses as JSON objects (see example). Could you please specify the expected response format from the API?

The essentials are predictions with their probability scores. I think that a JSON array of per-file objects is a good fit.
...
In the example you provided, I also see the latency key, which looks like a nice to have.

Great. Could you please provide a sample API input that specifies the expected parameters and the encoding format for serialized images?

In T358676#9645058, @kevinbazira wrote:

To prevent potential DOS vulnerabilities, we need to establish a limit on the number of images that can be sent to the API in a single request. Currently, the upload wizard restricts uploads to 50 files. Would you like us to maintain this limit for the API as well?

Yes, please.

Will you provide image links or serialized image objects?

Serialized image objects. More specifically, the Upload Wizard currently uses chunked uploading.

The Upload Wizard documentation warns against processing images from the Upload Stash during the upload step due to potential security risks. Will you implement security checks for serialized image objects before sending them to the API?

I spoke with my team and we think that consuming the stash URL doesn't pose security risks. As a result, instead of sending image objects to the LiftWing API, we'll send URLs.

Great. Could you please provide a sample API input that specifies the expected parameters and the encoding format for serialized images?

I think we can send a POST request with the following JSON body:

[
    {
        "filename": "my_uploaded_file.jpg",
        "url": "https://commons.wikimedia.org/wiki/Special:UploadStash/file/my_stash_filekey.png",
        "target" : "logo"
    },

    ...

]

Please let me know if you prefer form-encoded data instead.

Thank you for sharing this information, @mfossati. Based on the requirements you've shared so far, we have worked on a first pass of the prototype that takes the input JSON you specified, preprocesses it similar to the way you did in the demo, and returns the output JSON you specified (see P58917#237712). Please test it and let us know whether we've captured the key requirements correctly before we proceed working on input validation and sanitization, image limits, error handling, etc.

mfossati updated https://gitlab.wikimedia.org/mfossati/scriptz/-/merge_requests/6

Improve functionality

Hey @kevinbazira , I went through P58917, took the liberty of versioning it, and added some changes. Please have a look at https://gitlab.wikimedia.org/mfossati/scriptz/-/merge_requests/6.
Key changes involve:

• JSON output
• input dataset, which shouldn't have labels
• label mode, which shouldn't be binary, since the model is actually multiclass

@kevinbazira: I'm hitting this ignored exception when running the code:

Exception ignored in: <function AtomicFunction.__del__ at 0x157402660>
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/cnn/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py", line 291, in __del__
TypeError: 'NoneType' object is not subscriptable

Not harmful, but worth a check: are you hitting that, too?

Thank you for versioning the liftwing_prototype and making changes @mfossati! I tested the changes locally and got results that I shared in P58917#237822. Please have a look whenever you get a minute.

In T358676#9662292, @mfossati wrote:

@kevinbazira: I'm hitting this ignored exception when running the code:

Exception ignored in: <function AtomicFunction.__del__ at 0x157402660>
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/cnn/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py", line 291, in __del__
TypeError: 'NoneType' object is not subscriptable

Not harmful, but worth a check: are you hitting that, too?

I am not able to reproduce this issue. I've tried running the prototype in two enviroments:

1. GColab with python 3.10.12, keras==3.0.4, keras-cv==0.8.1, and tensorflow==2.15.0.
2. local venv with python 3.9.2, keras==3.0.4, keras-cv==0.8.1, and tensorflow==2.16.1.

Could this be caused by an incompatility in the python version shown in the error message: python3.11?

In T358676#9664492, @kevinbazira wrote:

Thank you for versioning the liftwing_prototype and making changes @mfossati! I tested the changes locally and got results that I shared in P58917#237822. Please have a look whenever you get a minute.

See answer in P58917#237851.

I am not able to reproduce this issue. I've tried running the prototype in two enviroments:

1. GColab with python 3.10.12, keras==3.0.4, keras-cv==0.8.1, and tensorflow==2.15.0.
2. local venv with python 3.9.2, keras==3.0.4, keras-cv==0.8.1, and tensorflow==2.16.1.

Could this be caused by an incompatility in the python version shown in the error message: python3.11?

Interesting, looks like I'm hitting it with both 3.10.12 and 3.12.2 Python versions in a mamba environment, both on a Linux and a Mac machine. Also hitting it with 3.11.5 in a venv environment on a Mac machine.
Not a blocker, but I suggest to try to resolve this before going to production.

mfossati merged https://gitlab.wikimedia.org/mfossati/scriptz/-/merge_requests/6

Improve functionality

kevinbazira opened https://gitlab.wikimedia.org/mfossati/scriptz/-/merge_requests/7

lw_prototype: validate input data

mfossati merged https://gitlab.wikimedia.org/mfossati/scriptz/-/merge_requests/7

lw_prototype: validate input data

kevinbazira opened https://gitlab.wikimedia.org/mfossati/scriptz/-/merge_requests/8

lw_prototype: image download error handling

mfossati merged https://gitlab.wikimedia.org/mfossati/scriptz/-/merge_requests/8

lw_prototype: image download error handling

kevinbazira opened https://gitlab.wikimedia.org/mfossati/scriptz/-/merge_requests/9

lw_prototype: add LogoDetectionModel class

Hi @mfossati ! Thanks a lot for all this great work!
I was wondering if you had tried to train the same model using pytorch as a keras backend instead of tensorflow. The reason I'm asking is totally unrelated to the model itself but has to do with technical challenges of maintaining multiple images and backends. There is ongoing work on our side to provide better support for pytorch (related task).
This is more of a question so we can provide better support and not a request from our side as we'd be supporting keras/tensorflow models as well.

mfossati merged https://gitlab.wikimedia.org/mfossati/scriptz/-/merge_requests/9

lw_prototype: add LogoDetectionModel class

In T358676#9683661, @isarantopoulos wrote:

I was wondering if you had tried to train the same model using pytorch as a keras backend instead of tensorflow.

Hey @isarantopoulos: no, I haven't.

The prototype looks good to me, I'm excited to see this effort move to the next level!
@kevinbazira, I've especially appreciated the tightness of our development iterations 😄 .

Thanks @mfossati! <3
It's great to hear you're excited about moving to the next milestone.
Rest assured, in T361803, we'll maintain the tight development iterations and ensure you're kept in the loop at every key milestone as we work towards hosting the logo-detection model-server on LiftWing.