Page MenuHomePhabricator

Implement sqlite db
Open, Needs TriagePublic5 Estimated Story Points

Description

The current POC Image Suggestion API has been facing memory issues when trying to parse the large .tsv files that holds all the Image Matching Algorithm data. Specifically, the largest file (enwiki.tsv) was causing memory issues in the node api.
To remedy this, we will instead implement a sqlite database for data storage.

Event Timeline

Change 670624 had a related patch set uploaded (by Nikki Nikkhoui; owner: Nikki Nikkhoui):
[mediawiki/services/image-suggestion-api@master] Sqlite

https://gerrit.wikimedia.org/r/670624

nnikkhoui set the point value for this task to 5.

Change 670624 merged by jenkins-bot:
[mediawiki/services/image-suggestion-api@master] Sqlite

https://gerrit.wikimedia.org/r/670624

Change 676145 had a related patch set uploaded (by BPirkle; author: BPirkle):

[mediawiki/services/image-suggestion-api@master] Parsing fix, related error message improvements

https://gerrit.wikimedia.org/r/676145

Change 676145 merged by jenkins-bot:

[mediawiki/services/image-suggestion-api@master] Parsing fix, related error message improvements

https://gerrit.wikimedia.org/r/676145

Change 676155 had a related patch set uploaded (by Nikki Nikkhoui; author: Nikki Nikkhoui):

[mediawiki/services/image-suggestion-api@master] Sqlite version change

https://gerrit.wikimedia.org/r/676155

Change 677316 had a related patch set uploaded (by Nikki Nikkhoui; author: Nikki Nikkhoui):

[mediawiki/services/image-suggestion-api@master] Add join table to image and pages

https://gerrit.wikimedia.org/r/677316

Change 677316 merged by jenkins-bot:

[mediawiki/services/image-suggestion-api@master] Add join table to image and pages

https://gerrit.wikimedia.org/r/677316

Change 676155 merged by jenkins-bot:

[mediawiki/services/image-suggestion-api@master] Sqlite version change

https://gerrit.wikimedia.org/r/676155

Change 677735 had a related patch set uploaded (by Nikki Nikkhoui; author: Nikki Nikkhoui):

[mediawiki/services/image-suggestion-api@master] DB Schema change. Promise resolve bugfix.

https://gerrit.wikimedia.org/r/677735

Change 677988 had a related patch set uploaded (by Nikki Nikkhoui; author: Nikki Nikkhoui):

[mediawiki/services/image-suggestion-api@master] Handle tab chars as falsey/missing value

https://gerrit.wikimedia.org/r/677988

Change 677988 merged by jenkins-bot:

[mediawiki/services/image-suggestion-api@master] Handle tab chars as falsey/missing value

https://gerrit.wikimedia.org/r/677988