Page MenuHomePhabricator

Usmanmuhd (Muhammad Usman)
User

Projects

User does not belong to any projects.

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Feb 25 2019, 1:41 PM (28 w, 6 d)
Availability
Available
LDAP User
Usmanmuhd
MediaWiki User
Muhdusman [ Global Accounts ]

Recent Activity

Wed, Sep 4

Usmanmuhd closed T218971: GSoC 2019 Proposal: Improve article recommendation pipeline as Resolved.
Wed, Sep 4, 1:15 AM · Google-Summer-of-Code (2019)
Usmanmuhd closed T217655: Improve article recommendation pipeline as Resolved.
Wed, Sep 4, 1:15 AM · Article-Recommendation, Google-Summer-of-Code (2019)

Wed, Aug 28

bmansurov awarded T218971: GSoC 2019 Proposal: Improve article recommendation pipeline a Like token.
Wed, Aug 28, 5:09 PM · Google-Summer-of-Code (2019)

Mon, Aug 26

srishakatux awarded T218971: GSoC 2019 Proposal: Improve article recommendation pipeline a Love token.
Mon, Aug 26, 10:03 PM · Google-Summer-of-Code (2019)
Usmanmuhd committed rMSRA379bce01fa0e: Replace Wikidata Query Service with MediaWiki API (authored by Usmanmuhd).
Replace Wikidata Query Service with MediaWiki API
Mon, Aug 26, 10:08 AM
Usmanmuhd closed T226264: Recommendation-API CI testing is flaky due to frequent 429s from Wikidata Query Service as Resolved.
Mon, Aug 26, 9:22 AM · Wikidata-Query-Service, Wikidata, Recommendation-API
Usmanmuhd added a comment to T226264: Recommendation-API CI testing is flaky due to frequent 429s from Wikidata Query Service.

T216750: Article recommendation API: replace WDQS with MW API fix has been merged. This will no longer be an issue.

Mon, Aug 26, 9:21 AM · Wikidata-Query-Service, Wikidata, Recommendation-API
Usmanmuhd closed T216750: Article recommendation API: replace WDQS with MW API as Resolved.
Mon, Aug 26, 9:14 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd closed T216750: Article recommendation API: replace WDQS with MW API, a subtask of T217655: Improve article recommendation pipeline, as Resolved.
Mon, Aug 26, 9:14 AM · Article-Recommendation, Google-Summer-of-Code (2019)

Sat, Aug 24

Usmanmuhd committed rRARD9830d3e496e1: Bulk import data to MySQL in chunks (authored by Usmanmuhd).
Bulk import data to MySQL in chunks
Sat, Aug 24, 1:26 PM
Usmanmuhd closed T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks, a subtask of T213566: Transferring data from Hadoop to production MySQL database, as Resolved.
Sat, Aug 24, 11:15 AM · Research-Backlog, serviceops-radar, Core Platform Team Legacy (Watching / External), Services (watching), User-Marostegui, Operations, Article-Recommendation, Analytics
Usmanmuhd closed T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks, a subtask of T217655: Improve article recommendation pipeline, as Resolved.
Sat, Aug 24, 11:15 AM · Article-Recommendation, Google-Summer-of-Code (2019)
Usmanmuhd closed T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks as Resolved.
Sat, Aug 24, 11:15 AM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation
Usmanmuhd added a comment to T218971: GSoC 2019 Proposal: Improve article recommendation pipeline.

Final Evaluation Summary:

Sat, Aug 24, 1:11 AM · Google-Summer-of-Code (2019)

Fri, Aug 23

Usmanmuhd added a comment to T230859: Merge the patch for GSoC.

@Aklapper Wanted to get the patches merged before the GSoC deadline, nothing else.

Fri, Aug 23, 3:08 AM · Release-Engineering-Team, Google-Summer-of-Code (2019), Research, Recommendation-API

Thu, Aug 22

Usmanmuhd added a comment to T230859: Merge the patch for GSoC.

I also have to another patch (https://gerrit.wikimedia.org/r/#/c/research/article-recommender/deploy/+/527571/) merged. The deadline is in ~ 3 days from today on. It would be great if it could be merged before that.

Thu, Aug 22, 4:25 PM · Release-Engineering-Team, Google-Summer-of-Code (2019), Research, Recommendation-API

Mon, Aug 19

Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@leila As GSoC is ending could you please help me get this patch merged?

Mon, Aug 19, 12:57 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API

Sun, Aug 18

Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

@bmansurov Updated the patch. Instead of committing the change after each chunk, it is committing all the chunks at the end. So if it fails at any point running the same previous command will import the tsv file without any problems.

Sun, Aug 18, 1:24 AM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation

Sat, Aug 17

Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

@bmansurov How would I move from the temp table to the actual table?

Sat, Aug 17, 7:51 AM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation

Aug 13 2019

Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

@bmansurov Updated the patch. Please take a look.

Aug 13 2019, 12:27 PM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation

Aug 11 2019

Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

@bmansurov Made the changes as required. Please take a look.

Aug 11 2019, 7:18 AM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation

Aug 9 2019

Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

@bmansurov I'm deleting the chunks once it's imported to keep a track of which all chunks have already been imported and which are yet to be imported. I did not really understand the advantages of placing it in /tmp if I'm going to delete the files anyway.

Aug 9 2019, 3:31 PM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation

Aug 4 2019

Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

@bmansurov Added the ability to continue when it stops at a given chunk.
Example:

Aug 4 2019, 8:19 AM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation

Aug 2 2019

Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

@bmansurov The working currently is:

  1. Chunks the file and stores in temp/<dir>/chunk-<i>.tsv
  2. Imports each chunk, executes the sql command and commits each transaction at the end of each chunk.
  3. Deletes the directory.
Aug 2 2019, 2:54 PM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation
Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

@bmansurov Thanks, figured out the error. I was using python3 along with the python-mysql.connector which is for python2.

Aug 2 2019, 2:41 AM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation

Aug 1 2019

Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

@bmansurov I shifted to a different system and now when I run python3 deploy.py import_languages 20181130 localhost 3306 recommendationapi usman db_password.txt --language_file 20181130/language.tsv I get the error as below. Tried various settings after searching online. None seem to work. Basically the error is on LOAD DATA LOCAL INFILE commands.

Aug 1 2019, 5:35 PM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation

Jul 29 2019

Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

Yeah, I think the UNIQUE constraint would be (wikidata_id, normalized_rank, source_id, target_id).

Jul 29 2019, 4:18 PM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation
Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

Yeah, that's a great idea. In case of a failure while importing a chunk (which can arise due to invalid data present in the chunk), how do we make sure that the values already entered inside that chunk are not entered again?
One approach would be to validate the chunk each time before importing it.

Jul 29 2019, 7:08 AM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation

Jul 27 2019

Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

@bmansurov As we are using LOAD DATA there is no way apart from splitting the file into chunks and then inserting each file separately.

Jul 27 2019, 2:55 PM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation
Usmanmuhd added a comment to T211980: 'morelike' recommendation API: Bulk import data to MySQL in chunks.

@bmansurov How do I get the chunks that need to be imported? If I need to generate the chunks by myself, what's the procedure to generate it?

Jul 27 2019, 11:09 AM · Research-Backlog, Google-Summer-of-Code (2019), Article-Recommendation
Usmanmuhd created T229157: ruwiki API returns no meaningful response.
Jul 27 2019, 7:38 AM · MediaWiki-API, Core Platform Team

Jul 26 2019

Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov pushed the fix. Please take a look now.

Jul 26 2019, 12:52 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov We had reduced the gpvimlimit in this case to 50 because we were having problems with ruwiki. I tried increasing the gpvimlimit and looks like it's been resolved.

Jul 26 2019, 6:45 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov There is a problem with handling it this way. We don't know the value that has to be passed until the previous request is finished. Trying to figure out the right implementation for this.

Jul 26 2019, 4:23 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API

Jul 25 2019

Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov Should I handle the llcontinue in getArticles itself? i.e. if llcontinue then call mwApiGet again from the same place.

Jul 25 2019, 5:08 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov made the changes and also enabled the tests.

Jul 25 2019, 6:59 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API

Jul 24 2019

Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov I think that is required when we have more than 10 languages being returned. In our case it's just one that is being returned. Should we really have that still?

Jul 24 2019, 4:35 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API

Jul 22 2019

Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov The Q406 item is coming from the following query: https://ru.wikipedia.org/w/api.php?action=query&format=json&prop=pageprops|langlinks|langlinkscount&ppprop=wikibase_item|disambiguation&generator=mostviewed&gpvimlimit=50&gpvimoffset=450&lllang=uz .

Jul 22 2019, 2:27 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API

Jul 16 2019

Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov Pushed the patch for getArticlesBySeed.

Jul 16 2019, 5:22 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

Edit1: Figured it out.

Jul 16 2019, 1:57 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API

Jul 15 2019

Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov Is this query right for getting items by pageviews? https://uz.wikipedia.org/w/api.php?format=json&action=query&prop=pageprops|langlinks&ppprop=wikibase_item|disambiguation&lllang=en&generator=mostviewed

Jul 15 2019, 8:42 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API

Jul 12 2019

Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov Oh I confused sitelink_count with index. In our case we don't have sitelink count. Should we fetch it from the API or should we just phase it out?

Jul 12 2019, 12:37 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov Given a seed, I am retrieving the wikibase_item, title and index. I should sort using index in ascending or descending?

Jul 12 2019, 6:29 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

How do I pass multiple props and ppprops? When I do this it's always returning 404. Same when I swapped out | for a ,.

const parameters = {
        format: 'json',
        action: 'query',
        prop: 'pageprops|langlinks',
        ppprop: 'wikibase_item|disambiguation',
        lllang: target,
        generator: 'search',
        gsrlimit: 500,
        gsrsearch: `morelike:${seed}`
};
Jul 12 2019, 5:12 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

Yeah, it works now. Thanks.

Jul 12 2019, 4:48 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

Even after deleting the node_modules and npm install, the error still persists.

Jul 12 2019, 4:36 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov When I run npm run test | bunyan I am getting the following error:

Jul 12 2019, 3:59 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API

Jun 29 2019

Usmanmuhd added a comment to T218971: GSoC 2019 Proposal: Improve article recommendation pipeline.

Thanks for the great feedback!

Jun 29 2019, 12:32 AM · Google-Summer-of-Code (2019)

Jun 28 2019

Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

I can't use ppprop or lllang along with action='wbgetentities'. It gives "*": "Unrecognized parameters: ppprop, lllang."

Jun 28 2019, 12:51 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

Oh okay.
So the flow would be:

  1. Get all the entities and corresponding data.
  2. Remove all the items containing enwiki in sitelinks.
  3. Remove all the items having 'disambiguation' in the labels.
Jun 28 2019, 7:31 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API

Jun 26 2019

Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

I meant to say that SparQL is excluding the entity.

Jun 26 2019, 4:22 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

I removed all entities which have 'enwiki', I got ['Q4077077', 'Q4427926', 'Q24287657', 'Q52686724']. The extra entity I am getting is 'Q4077077'. Seems like it's getting excluded in SparQL due to ?article schema:about ?item ..
Which attribute is being checked in this part?

Jun 26 2019, 1:57 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API

Jun 24 2019

Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

I don't think there is a way to get only the counts.
For this: https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q3986754|Q4224|Q4429859|Q306403|Q2498038|Q271534|Q274306|Q205707|Q229651|Q222|Q4077077|Q4427926|Q2983218|Q166502|Q3023357|Q1924847|Q34436|Q19865538|Q24287657|Q42296351|Q52686724|Q47300912|Q64768584&props=sitelinks&format=json
we get the counts correctly but also there are extra items other than the query(https://query.wikidata.org/#SELECT%20%3Fitem%20%28COUNT%28%3Fsitelink%29%20as%20%3Fcount%29%20WHERE%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20VALUES%20%3Fitem%20%7B%20wd%3AQ3986754%20wd%3AQ4224%20wd%3AQ4429859%20wd%3AQ306403%20wd%3AQ2498038%20wd%3AQ271534%20wd%3AQ274306%20wd%3AQ205707%20wd%3AQ229651%20wd%3AQ222%20wd%3AQ4077077%20wd%3AQ4427926%20wd%3AQ2983218%20wd%3AQ166502%20wd%3AQ3023357%20wd%3AQ1924847%20wd%3AQ34436%20wd%3AQ19865538%20wd%3AQ24287657%20wd%3AQ42296351%20wd%3AQ52686724%20wd%3AQ47300912%20wd%3AQ64768584%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20FILTER%20NOT%20EXISTS%20%7B%20%3Fitem%20wdt%3AP31%20wd%3AQ4167410%20.%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20OPTIONAL%20%7B%20%3Fsitelink%20schema%3Aabout%20%3Fitem%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20FILTER%20NOT%20EXISTS%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Farticle%20schema%3Aabout%20%3Fitem%20.%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Farticle%20schema%3AisPartOf%20%3Chttps%3A%2F%2Fen.wikipedia.org%2F%3E%20.%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7D%20GROUP%20BY%20%3Fitem). Searching for 'ruwiki' or 'enwiki' in ['sitelinks'] also does not help either.

Jun 24 2019, 4:35 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@bmansurov Is this API query a right replacement for the SparQL?

Jun 24 2019, 1:59 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API

Jun 22 2019

Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

Hopefully will get it done by Monday or Tuesday.

Jun 22 2019, 2:25 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

@bmansurov I think T216750: Article recommendation API: replace WDQS with MW API should solve the issue. If the issue still persists, I'll come back to it.

Jun 22 2019, 1:37 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research

Jun 20 2019

Usmanmuhd added a comment to T218971: GSoC 2019 Proposal: Improve article recommendation pipeline.

Evaluation 1 summary:

Jun 20 2019, 2:01 PM · Google-Summer-of-Code (2019)

Jun 18 2019

Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

Thanks!
Minor point:

  1. Why is the count:24 even after passing count=5? It works as expected on my local env.
Jun 18 2019, 4:09 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

Thanks, shall I move on to the next task or do I have something else to do before moving on to the next one?

Jun 18 2019, 5:27 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research

Jun 17 2019

Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

A few observations:

  1. https://en.wikipedia.org/api/rest_v1/data/recommendation/article/creation/translation/ru?count=5 returns 24 items itself. Works as expected on local machine.
  2. Should we explore the tests for this API?
  3. Using 50 for sparql currently suffices, but we risk running into 429 error due to this. Should we increase the limit?
Jun 17 2019, 1:46 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd committed rMSRA8920fec6d619: Splits the request to WMAPI in batches (authored by Usmanmuhd).
Splits the request to WMAPI in batches
Jun 17 2019, 1:14 PM
Usmanmuhd committed rMSRA4af41a0a240e: Splits the request to WMAPI in batches (authored by Usmanmuhd).
Splits the request to WMAPI in batches
Jun 17 2019, 1:14 PM
Usmanmuhd committed rMSRA9603d4282492: Splits the request to MediaWiki API and Wikidata query service in batches. (authored by Usmanmuhd).
Splits the request to MediaWiki API and Wikidata query service in batches.
Jun 17 2019, 1:14 PM

Jun 15 2019

Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

Pushed the changes after making the filter() use batches as well.

Jun 15 2019, 2:09 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd committed rMSRAea3acdf0ca96: Splits the request to WMAPI in batches (authored by Usmanmuhd).
Splits the request to WMAPI in batches
Jun 15 2019, 2:04 PM
Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

@bmansurov http://localhost:6927/en.wikipedia.org/v1/article/creation/translation/ru works perfectly.
I checked http://localhost:6927/uz.wikipedia.org/v1/article/creation/translation/ru with both fix-T215222 branch and the master branch. Both give an error.
Further investigation reveals:
This error is being caused in https://query.wikidata.org/sparql.

Jun 15 2019, 5:58 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd updated the task description for T225853: BadMethodCallException for API call on ru.wikipedia: "Call to a member function getPrefixedText() on a non-object (null)".
Jun 15 2019, 3:46 AM · MW-1.34-notes (1.34.0-wmf.16; 2019-07-30), Core Platform Team (Needs Cleaning - Code Health (TEC13)), good first bug, Core Platform Team Workboards (Clinic Duty Team), PageViewInfo, Wikimedia-production-error, MediaWiki-API
Usmanmuhd renamed T225853: BadMethodCallException for API call on ru.wikipedia: "Call to a member function getPrefixedText() on a non-object (null)" from Error with RuWikiAPi to Error with RuWikiAPI.
Jun 15 2019, 2:27 AM · MW-1.34-notes (1.34.0-wmf.16; 2019-07-30), Core Platform Team (Needs Cleaning - Code Health (TEC13)), good first bug, Core Platform Team Workboards (Clinic Duty Team), PageViewInfo, Wikimedia-production-error, MediaWiki-API
Usmanmuhd created T225853: BadMethodCallException for API call on ru.wikipedia: "Call to a member function getPrefixedText() on a non-object (null)".
Jun 15 2019, 2:26 AM · MW-1.34-notes (1.34.0-wmf.16; 2019-07-30), Core Platform Team (Needs Cleaning - Code Health (TEC13)), good first bug, Core Platform Team Workboards (Clinic Duty Team), PageViewInfo, Wikimedia-production-error, MediaWiki-API

Jun 14 2019

Usmanmuhd committed rMSRA02121ecaeec3: Splits the request to WMAPI in batches (authored by Usmanmuhd).
Splits the request to WMAPI in batches
Jun 14 2019, 6:33 PM
Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

Updated the code as requested. Please take a look.

Jun 14 2019, 6:33 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd committed rMSRAd7c7353b5ed6: Splits the request to WMAPI in batches (authored by Usmanmuhd).
Splits the request to WMAPI in batches
Jun 14 2019, 3:46 PM
Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

npm run test | bunyan giving this error:

Jun 14 2019, 2:58 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

Pushed the quick fix for the error. Will report the bug in a while.

Jun 14 2019, 2:54 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd committed rMSRA6af2642d4199: Splits the request to WMAPI in batches (authored by Usmanmuhd).
Splits the request to WMAPI in batches
Jun 14 2019, 2:53 PM
Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

ru.wikipedia.org API behaves much differently from the others.
Example:

Jun 14 2019, 12:42 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

There is a "continue" attribute in the body of the request when we have more items than the limit. Example:

{"batchcomplete":"","continue":{"gpvimoffset":250,"continue":"gpvimoffset||"},"query":{"pages":{"-4":{"ns":2,"title":"User:Geilamir","missing":"","known":""},"-5":{"ns":2,"title":"User:Logan","missing":"","known":""},"-6":{"ns":2,"title":"User:Courcelles","missing":"","known":""},"-7":{"ns":6,"title":"File:Tanzania in its region.svg","missing":"","known":""},"-1":{"ns":-1,"title":"Special:Contributions/84.198.31.211","special":""},"-2":{"ns":-1,"title":"Special:EmailUser/Troubled asset","special":""},"-3":{"ns":-1,"title":"Special:NewPages","special":""},..........................................."4474":{"pageid":4474,"ns":828,"title":"Module:Citation/CS1/Utilities","pageprops":{"wikibase_item":"Q21993353"}},"4445":{"pageid":4445,"ns":828,"title":"Module:No globals","pageprops":{"wikibase_item":"Q16748603"}}}}}

Should I make use of this attribute to fetch all the items or should we limit it to 500 or the number of items which ever is lesser?

Jun 14 2019, 9:47 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd committed rMSRA05738c5c322d: Throw appropriate error when wmAPI returns internal server error. (authored by Usmanmuhd).
Throw appropriate error when wmAPI returns internal server error.
Jun 14 2019, 9:32 AM

Jun 13 2019

Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

Yeah, higher sitelink_count is important. It is being sorted here https://github.com/wikimedia/mediawiki-services-recommendation-api/blob/master/lib/article.creation.translation.js#L182.

Jun 13 2019, 2:09 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

I tested for different number of items. It gives a different output. Basically the elements returned by the API are retrieved from the db along with other data. Example:

Jun 13 2019, 1:18 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

Yeah, it works. How do we handle this case?

Jun 13 2019, 12:28 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

Sent the same request as the one being sent through sandbox:

Jun 13 2019, 8:24 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research
Usmanmuhd committed rMSRA55b0a49ad3b7: Throw appropriate error when wmAPI returns internal server error. (authored by Usmanmuhd).
Throw appropriate error when wmAPI returns internal server error.
Jun 13 2019, 4:25 AM

Jun 12 2019

Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

Yeah, just tested with the valid API call and the invalid API call. response.body.error will not be null in case of error and will be null in case of no error. Will send in a patch in sometime.

Jun 12 2019, 1:00 PM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research

Jun 10 2019

Usmanmuhd added a comment to T215222: Recommendation API translation endpoint stopped working.

How do I reproduce this error?

Both above work fine.

Jun 10 2019, 7:35 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Legacy (Watching / External), Services (watching), Recommendation-API, Research

May 30 2019

Usmanmuhd committed rMSRA6b8ca3e4a2f1: Remove duplicate Wikidata items from article recommendations (authored by Usmanmuhd).
Remove duplicate Wikidata items from article recommendations
May 30 2019, 1:46 PM

May 28 2019

Usmanmuhd added a comment to T216721: Remove duplicate Wikidata items from article recommendations.

Output for http://localhost:6927/uz.wikipedia.org/v1/article/creation/morelike/Kitob

May 28 2019, 6:09 PM · Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd committed rMSRAb292034c51ca: Remove duplicate Wikidata items from article recommendations (authored by Usmanmuhd).
Remove duplicate Wikidata items from article recommendations
May 28 2019, 6:08 PM
Usmanmuhd committed rMSRA140b3af44cac: Remove duplicate Wikidata items from article recommendations (authored by Usmanmuhd).
Remove duplicate Wikidata items from article recommendations
May 28 2019, 4:19 PM

May 6 2019

Usmanmuhd added a comment to T218971: GSoC 2019 Proposal: Improve article recommendation pipeline.

Thanks a lot for selecting me! Looking forward to working on it.

May 6 2019, 6:09 PM · Google-Summer-of-Code (2019)

Apr 9 2019

Usmanmuhd moved T220329: GSoC 2019 Proposal: Build statistics toolset to support WM-HU editor retention grant from Proposals in Progress to Proposals Submitted on the Google-Summer-of-Code (2019) board.
Apr 9 2019, 2:07 PM · Google-Summer-of-Code (2019)
Usmanmuhd updated the task description for T220329: GSoC 2019 Proposal: Build statistics toolset to support WM-HU editor retention grant.
Apr 9 2019, 1:59 PM · Google-Summer-of-Code (2019)
Usmanmuhd updated the task description for T220329: GSoC 2019 Proposal: Build statistics toolset to support WM-HU editor retention grant.
Apr 9 2019, 1:40 PM · Google-Summer-of-Code (2019)
Usmanmuhd added a comment to T220329: GSoC 2019 Proposal: Build statistics toolset to support WM-HU editor retention grant.

@Tgr Added 2 coding tasks. Will go ahead and submit the proposal.

Apr 9 2019, 1:38 PM · Google-Summer-of-Code (2019)
Usmanmuhd updated the task description for T220329: GSoC 2019 Proposal: Build statistics toolset to support WM-HU editor retention grant.
Apr 9 2019, 1:36 PM · Google-Summer-of-Code (2019)
Usmanmuhd updated the task description for T220329: GSoC 2019 Proposal: Build statistics toolset to support WM-HU editor retention grant.
Apr 9 2019, 1:05 PM · Google-Summer-of-Code (2019)
Usmanmuhd updated the task description for T220329: GSoC 2019 Proposal: Build statistics toolset to support WM-HU editor retention grant.
Apr 9 2019, 1:05 PM · Google-Summer-of-Code (2019)
Usmanmuhd moved T216721: Remove duplicate Wikidata items from article recommendations from Proposals Submitted to Microtasks on the Google-Summer-of-Code (2019) board.
Apr 9 2019, 12:37 PM · Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Usmanmuhd updated the task description for T220329: GSoC 2019 Proposal: Build statistics toolset to support WM-HU editor retention grant.
Apr 9 2019, 11:09 AM · Google-Summer-of-Code (2019)
Usmanmuhd added a comment to T220329: GSoC 2019 Proposal: Build statistics toolset to support WM-HU editor retention grant.

Thanks for the review. I will change those parts accordingly.

Apr 9 2019, 10:30 AM · Google-Summer-of-Code (2019)