Page MenuHomePhabricator
Create Task
Maniphest T276863

Implement sqlite db
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

The current POC Image Suggestion API has been facing memory issues when trying to parse the large .tsv files that holds all the Image Matching Algorithm data. Specifically, the largest file (enwiki.tsv) was causing memory issues in the node api.
To remedy this, we will instead implement a sqlite database for data storage.

Details

SubjectRepoBranchLines +/-
DB Schema change. Promise resolve bugfix.mediawiki/services/image-suggestion-apimaster+93 -102
Handle tab chars as falsey/missing valuemediawiki/services/image-suggestion-apimaster+4 -4
Sqlite version changemediawiki/services/image-suggestion-apimaster+236 -216
Add join table to image and pagesmediawiki/services/image-suggestion-apimaster+43 -19
Parsing fix, related error message improvementsmediawiki/services/image-suggestion-apimaster+13 -7
Sqlitemediawiki/services/image-suggestion-apimaster+426 -397
Customize query in gerrit

Related Objects

Event Timeline

• nnikkhoui created this task.Mar 8 2021, 10:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 8 2021, 10:43 PM
• nnikkhoui moved this task from Backlog to Ready on the Platform Team Workboards (Image Suggestion API) board.Mar 8 2021, 10:43 PM
• sdkim added a project: Image-Suggestions.Mar 10 2021, 9:19 PM
gerritbot added a comment.Mar 11 2021, 12:27 AM

Change 670624 had a related patch set uploaded (by Nikki Nikkhoui; owner: Nikki Nikkhoui):
[mediawiki/services/image-suggestion-api@master] Sqlite

https://gerrit.wikimedia.org/r/670624

gerritbot added a project: Patch-For-Review.Mar 11 2021, 12:27 AM
• nnikkhoui claimed this task.Mar 12 2021, 4:27 PM
• nnikkhoui moved this task from Ready to In progress on the Platform Team Workboards (Image Suggestion API) board.
• nnikkhoui set the point value for this task to 5.
• sdkim added a project: Image-Suggestion-API.Mar 12 2021, 5:14 PM
• sdkim moved this task from Backlog to v1 on the Image-Suggestion-API board.
• sdkim moved this task from v1 to Proof of Concept on the Image-Suggestion-API board.Mar 12 2021, 5:24 PM
• nnikkhoui moved this task from In progress to In review on the Platform Team Workboards (Image Suggestion API) board.Mar 30 2021, 3:41 PM
gerritbot added a comment.Mar 30 2021, 3:41 PM

Change 670624 merged by jenkins-bot:
[mediawiki/services/image-suggestion-api@master] Sqlite

https://gerrit.wikimedia.org/r/670624

• nnikkhoui mentioned this in rMSISa9a8c0a01696: Sqlite.Mar 30 2021, 3:41 PM
• nnikkhoui moved this task from In review to Done on the Platform Team Workboards (Image Suggestion API) board.Mar 30 2021, 4:00 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 30 2021, 4:10 PM
• nnikkhoui moved this task from Done to In progress on the Platform Team Workboards (Image Suggestion API) board.Mar 31 2021, 5:52 PM
gerritbot added a comment.Mar 31 2021, 8:12 PM

Change 676145 had a related patch set uploaded (by BPirkle; author: BPirkle):

[mediawiki/services/image-suggestion-api@master] Parsing fix, related error message improvements

https://gerrit.wikimedia.org/r/676145

gerritbot added a project: Patch-For-Review.Mar 31 2021, 8:12 PM
gerritbot added a comment.Mar 31 2021, 9:13 PM

Change 676145 merged by jenkins-bot:

[mediawiki/services/image-suggestion-api@master] Parsing fix, related error message improvements

https://gerrit.wikimedia.org/r/676145

BPirkle mentioned this in rMSIS20eeb3846dc5: Parsing fix, related error message improvements.Mar 31 2021, 9:13 PM
gerritbot added a comment.Mar 31 2021, 9:23 PM

Change 676155 had a related patch set uploaded (by Nikki Nikkhoui; author: Nikki Nikkhoui):

[mediawiki/services/image-suggestion-api@master] Sqlite version change

https://gerrit.wikimedia.org/r/676155

gerritbot added a comment.Apr 6 2021, 7:27 PM

Change 677316 had a related patch set uploaded (by Nikki Nikkhoui; author: Nikki Nikkhoui):

[mediawiki/services/image-suggestion-api@master] Add join table to image and pages

https://gerrit.wikimedia.org/r/677316

gerritbot added a comment.Apr 6 2021, 10:49 PM

Change 677316 merged by jenkins-bot:

[mediawiki/services/image-suggestion-api@master] Add join table to image and pages

https://gerrit.wikimedia.org/r/677316

• nnikkhoui mentioned this in rMSIS3fcc14d527f6: Add join table to image and pages.Apr 6 2021, 10:50 PM
gerritbot added a comment.Apr 7 2021, 5:56 PM

Change 676155 merged by jenkins-bot:

[mediawiki/services/image-suggestion-api@master] Sqlite version change

https://gerrit.wikimedia.org/r/676155

• nnikkhoui mentioned this in rMSIS37d9d3dfb8d9: Sqlite version change.Apr 7 2021, 5:56 PM
gerritbot added a comment.Apr 8 2021, 1:32 PM

Change 677735 had a related patch set uploaded (by Nikki Nikkhoui; author: Nikki Nikkhoui):

[mediawiki/services/image-suggestion-api@master] DB Schema change. Promise resolve bugfix.

https://gerrit.wikimedia.org/r/677735

gerritbot added a comment.Apr 8 2021, 5:49 PM

Change 677988 had a related patch set uploaded (by Nikki Nikkhoui; author: Nikki Nikkhoui):

[mediawiki/services/image-suggestion-api@master] Handle tab chars as falsey/missing value

https://gerrit.wikimedia.org/r/677988

gerritbot added a comment.Apr 8 2021, 9:00 PM

Change 677988 merged by jenkins-bot:

[mediawiki/services/image-suggestion-api@master] Handle tab chars as falsey/missing value

https://gerrit.wikimedia.org/r/677988

• nnikkhoui mentioned this in rMSIS9767d2e8b093: Handle tab chars as falsey/missing value.Apr 8 2021, 9:01 PM
• nnikkhoui mentioned this in rMSISedb991e7d5d5: Synchronous event stream to populate DB.Apr 19 2021, 9:24 PM
• nnikkhoui moved this task from In progress to Done on the Platform Team Workboards (Image Suggestion API) board.Apr 20 2021, 11:15 PM
gerritbot added a comment.May 28 2021, 1:53 PM

Change 677735 abandoned by Nikki Nikkhoui:

[mediawiki/services/image-suggestion-api@master] DB Schema change. Promise resolve bugfix.

Reason:

https://gerrit.wikimedia.org/r/677735

• nnikkhoui closed this task as Resolved.May 28 2021, 1:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL