Page MenuHomePhabricator

Article recommendation API: replace WDQS with MW API
Open, LowPublic

Description

The translation end point of the Recommendation API depends on the Wikidata Query Service (WDQS). Replacing a call to WDQS with a call to the MediaWiki API (MW API) should make the service faster and reduce issues with testing on the beta cluster.

Background info: T213222 and T211397#4806848 and on.

Mentors

Skills Required

  • Javascript

Acceptance Criteria

Event Timeline

bmansurov triaged this task as Low priority.Feb 21 2019, 6:21 PM
bmansurov updated the task description. (Show Details)
bmansurov renamed this task from Can recommendation-api-service work without WDQS? to Article recommendation API: replace WDQS with MW API.Mar 4 2019, 2:41 PM
bmansurov updated the task description. (Show Details)
bmansurov updated the task description. (Show Details)
bmansurov updated the task description. (Show Details)
bmansurov added a subscriber: Stabgan.

@Stabgan take a look at the task and let me know if you're interested in working on it.

bmansurov removed a subscriber: Stabgan.Mar 5 2019, 5:51 PM

@bmansurov this task seems interesting to me and I surely would love to work on it.
Thank you :)

@Shivanshbindal9 awesome. Feel free to assign it to yourself.

hi, Sorry for very late
can I solve this microtask for GSoC participant

@srishakatux can you help me the understanding problem

then how can I participate in GSoC. please guide

Usmanmuhd added a subscriber: Usmanmuhd.EditedJun 24 2019, 1:59 PM

@bmansurov Is this API query a right replacement for the SparQL?

API: https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q4224|Q4427926&props=sitelinks

WDQS: https://query.wikidata.org/#SELECT%20%3Fitem%20%28COUNT%28%3Fsitelink%29%20as%20%3Fcount%29%20WHERE%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20VALUES%20%3Fitem%20%7Bwd%3AQ4224%20wd%3AQ4427926%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20FILTER%20NOT%20EXISTS%20%7B%20%3Fitem%20wdt%3AP31%20wd%3AQ4167410%20.%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20OPTIONAL%20%7B%20%3Fsitelink%20schema%3Aabout%20%3Fitem%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20FILTER%20NOT%20EXISTS%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Farticle%20schema%3Aabout%20%3Fitem%20.%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Farticle%20schema%3AisPartOf%20%3Chttps%3A%2F%2Fuz.wikipedia.org%2F%3E%20.%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7D%20GROUP%20BY%20%3Fitem

The length of the keys in the sitelinks is same both the above cases.

Questions: If we want to get the same from uz.wikipedia.org as actually desired: https://uz.wikipedia.org/w/api.php?action=wbgetentities&ids=Q4224|Q4427926&props=sitelinks I get

{
    "error": {
        "code": "unknown_action",
        "info": "Unrecognized value for parameter \"action\": wbgetentities.",
        "*": "See https://uz.wikipedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes."
    },
    "servedby": "mw1315"
}

Looks like the results are similar. I wonder if the MediaWiki API supports counting results rather than outputting them? Can you see if it does?

Uzwiki doesn't work because it doesn't have info about Wikidata items I think. That's why all wikis should use the Wikidata MediaWiki API.

I don't think there is a way to get only the counts.
For this: https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q3986754|Q4224|Q4429859|Q306403|Q2498038|Q271534|Q274306|Q205707|Q229651|Q222|Q4077077|Q4427926|Q2983218|Q166502|Q3023357|Q1924847|Q34436|Q19865538|Q24287657|Q42296351|Q52686724|Q47300912|Q64768584&props=sitelinks&format=json
we get the counts correctly but also there are extra items other than the query(https://query.wikidata.org/#SELECT%20%3Fitem%20%28COUNT%28%3Fsitelink%29%20as%20%3Fcount%29%20WHERE%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20VALUES%20%3Fitem%20%7B%20wd%3AQ3986754%20wd%3AQ4224%20wd%3AQ4429859%20wd%3AQ306403%20wd%3AQ2498038%20wd%3AQ271534%20wd%3AQ274306%20wd%3AQ205707%20wd%3AQ229651%20wd%3AQ222%20wd%3AQ4077077%20wd%3AQ4427926%20wd%3AQ2983218%20wd%3AQ166502%20wd%3AQ3023357%20wd%3AQ1924847%20wd%3AQ34436%20wd%3AQ19865538%20wd%3AQ24287657%20wd%3AQ42296351%20wd%3AQ52686724%20wd%3AQ47300912%20wd%3AQ64768584%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20FILTER%20NOT%20EXISTS%20%7B%20%3Fitem%20wdt%3AP31%20wd%3AQ4167410%20.%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20OPTIONAL%20%7B%20%3Fsitelink%20schema%3Aabout%20%3Fitem%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20FILTER%20NOT%20EXISTS%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Farticle%20schema%3Aabout%20%3Fitem%20.%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Farticle%20schema%3AisPartOf%20%3Chttps%3A%2F%2Fen.wikipedia.org%2F%3E%20.%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7D%20GROUP%20BY%20%3Fitem). Searching for 'ruwiki' or 'enwiki' in ['sitelinks'] also does not help either.

OK, let me take a look at it later today/tomorrow.

As for your question on IRC, the query is just excluding English Wikipedia from the results.

I removed all entities which have 'enwiki', I got ['Q4077077', 'Q4427926', 'Q24287657', 'Q52686724']. The extra entity I am getting is 'Q4077077'. Seems like it's getting excluded in SparQL due to ?article schema:about ?item ..
Which attribute is being checked in this part?

Wikidata has items and properties (not sure about other things). Items start with Q, and properties with P. So ?article schema:about ?item couldn't exclude Q4077077 because it's an item.

I meant to say that SparQL is excluding the entity.

Output from SparQL: 'Q4427926', 'Q24287657', 'Q52686724'
Output from API Endpoint: 'Q4077077', 'Q4427926', 'Q24287657', 'Q52686724'

'Q4077077' entity:

{
    "entities": {
        "Q4077077": {
            "pageid": 3885557,
            "ns": 0,
            "title": "Q4077077",
            "lastrevid": 922522939,
            "modified": "2019-04-23T07:10:46Z",
            "type": "item",
            "id": "Q4077077",
            "labels": {
                "ru": {
                    "language": "ru",
                    "value": "\u0411\u0430\u043b\u044c\u0437\u0430\u043a\u043e\u0432\u0441\u043a\u0438\u0439 \u0432\u043e\u0437\u0440\u0430\u0441\u0442"
                },
                "en": {
                    "language": "en",
                    "value": "kanskje"
                },
                .......................
                "ast": {
                    "language": "ast",
                    "value": "p\u00e1xina de dixebra"
                }
            },
            "aliases": {},
            "claims": {
                "P31": [
                    {
                        "mainsnak": {
                            "snaktype": "value",
                            "property": "P31",
                            "hash": "0ebb78ffecd3415a2dbc8db900881ff784fead2a",
                            "datavalue": {
                                "value": {
                                    "entity-type": "item",
                                    "numeric-id": 4167410,
                                    "id": "Q4167410"
                                },
                                "type": "wikibase-entityid"
                            },
                            "datatype": "wikibase-item"
                        },
                        "type": "statement",
                        "id": "Q4077077$1F7CD1A0-66D6-4568-B87D-11DE6943F2E8",
                        "rank": "normal"
                    }
                ]
            },
            "sitelinks": {
                "ruwiki": {
                    "site": "ruwiki",
                    "title": "\u0411\u0430\u043b\u044c\u0437\u0430\u043a\u043e\u0432\u0441\u043a\u0438\u0439 \u0432\u043e\u0437\u0440\u0430\u0441\u0442",
                    "badges": []
                }
            }
        }
    },
    "success": 1
}

Your MediaWiki API URL doesn't seem to filter out the disambiguation pages (unlike WDQS) and that item (Q4077077) points to a disambiguation page.

Oh okay.
So the flow would be:

  1. Get all the entities and corresponding data.
  2. Remove all the items containing enwiki in sitelinks.
  3. Remove all the items having 'disambiguation' in the labels.

Is this right or should there be any change?

I think you can get all the info you need in one query. Here's an example when you have a seed article:

https://uz.wikipedia.org/w/api.php?format=json&action=query&prop=pageprops|langlinks&ppprop=wikibase_item|disambiguation&lllang=en&generator=search&gsrlimit=500&gsrsearch=morelike:Yorug%27lik

You'd then only select items that don't have an 'en' langlink, and 'disambiguation' page prop.

I imagine you can construct a similar query when getting recommendations by pageviews.

I can't use ppprop or lllang along with action='wbgetentities'. It gives "*": "Unrecognized parameters: ppprop, lllang."

You don't need action=wbgetentities if you use the URL I shared. That URL gives you all the information you need to display the results.

As for the mismatch, don't use wikidata.org (since we don't need action=wbgetentities), use specific wikis only.

Usmanmuhd added a comment.EditedJul 12 2019, 3:59 AM

@bmansurov When I run npm run test | bunyan I am getting the following error:

> recommendation-api@0.6.0 test /home/usman/gsoc/recommendation-api
> PREQ_CONNECT_TIMEOUT=15 mocha && npm run lint

/home/usman/gsoc/recommendation-api/node_modules/mocha/node_modules/yargs/yargs.js:1163
      else throw err
           ^

Error: Cannot find module 'colors/safe'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:582:15)
    at Function.Module._load (internal/modules/cjs/loader.js:508:25)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/home/usman/gsoc/recommendation-api/node_modules/limitation/node_modules/kad/lib/logger.js:3:14)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/home/usman/gsoc/recommendation-api/node_modules/limitation/node_modules/kad/index.js:16:25)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/home/usman/gsoc/recommendation-api/node_modules/limitation/lib/kad_backend.js:4:26)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/home/usman/gsoc/recommendation-api/node_modules/limitation/index.js:18:18)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/home/usman/gsoc/recommendation-api/node_modules/service-runner/lib/ratelimiter.js:5:21)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/home/usman/gsoc/recommendation-api/node_modules/service-runner/lib/worker.js:10:27)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/home/usman/gsoc/recommendation-api/node_modules/service-runner/lib/master.js:8:16)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/home/usman/gsoc/recommendation-api/node_modules/service-runner/service-runner.js:13:16)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/home/usman/gsoc/recommendation-api/test/utils/server.js:4:23)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/home/usman/gsoc/recommendation-api/test/features/app/app.js:7:16)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at /home/usman/gsoc/recommendation-api/node_modules/mocha/lib/mocha.js:330:36
    at Array.forEach (<anonymous>)
    at Mocha.loadFiles (/home/usman/gsoc/recommendation-api/node_modules/mocha/lib/mocha.js:327:14)
    at Mocha.run (/home/usman/gsoc/recommendation-api/node_modules/mocha/lib/mocha.js:804:10)
    at Object.exports.singleRun (/home/usman/gsoc/recommendation-api/node_modules/mocha/lib/cli/run-helpers.js:207:16)
    at exports.runMocha (/home/usman/gsoc/recommendation-api/node_modules/mocha/lib/cli/run-helpers.js:300:13)
    at Object.exports.handler.argv [as handler] (/home/usman/gsoc/recommendation-api/node_modules/mocha/lib/cli/run.js:296:3)
    at Object.runCommand (/home/usman/gsoc/recommendation-api/node_modules/mocha/node_modules/yargs/lib/command.js:242:26)
    at Object.parseArgs [as _parseArgs] (/home/usman/gsoc/recommendation-api/node_modules/mocha/node_modules/yargs/yargs.js:1104:24)
    at Object.parse (/home/usman/gsoc/recommendation-api/node_modules/mocha/node_modules/yargs/yargs.js:566:25)
    at Object.exports.main (/home/usman/gsoc/recommendation-api/node_modules/mocha/lib/cli/cli.js:63:6)
    at Object.<anonymous> (/home/usman/gsoc/recommendation-api/node_modules/mocha/bin/_mocha:10:23)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:754:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! recommendation-api@0.6.0 test: `PREQ_CONNECT_TIMEOUT=15 mocha && npm run lint`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the recommendation-api@0.6.0 test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/usman/.npm/_logs/2019-07-12T03_58_15_145Z-debug.log

I deleted node_modules and ran npm install still the error persists.

Looks like some dependencies may be missing. @Mholloway I see some recent patches by you. Anything ideas how to fix the above error?

Actually, @Usmanmuhd I also deleted node_modules and did npm install, then the command worked. Can you try again?

Actually, @Usmanmuhd I also deleted node_modules and did npm install, then the command worked. Can you try again?

This was going to be my first suggestion.

Usmanmuhd added a comment.EditedJul 12 2019, 4:36 AM

Even after deleting the node_modules and npm install, the error still persists.

Figured out the error: package-lock.json was causing all the problem.

@Usmanmuhd Is there a package-lock.json file present? If so, try deleting that as well as node_modules/ before doing a fresh npm install.

Figured out the error: package-lock.json was causing all the problem.

Ah, glad you got it worked out!

Yeah, it works now. Thanks.

Usmanmuhd added a comment.EditedJul 12 2019, 5:12 AM

How do I pass multiple props and ppprops? When I do this it's always returning 404. Same when I swapped out | for a ,.

const parameters = {
        format: 'json',
        action: 'query',
        prop: 'pageprops|langlinks',
        ppprop: 'wikibase_item|disambiguation',
        lllang: target,
        generator: 'search',
        gsrlimit: 500,
        gsrsearch: `morelike:${seed}`
};

Edit 1: Figured this out, was a error in a different place. I was manipulating the seed string unnecessarily.

@bmansurov Given a seed, I am retrieving the wikibase_item, title and index. I should sort using index in ascending or descending?

Thanks, @Mholloway!

@Usmanmuhd is index meaningful? I don't think it matters whether you sort it or not. It's up to you.

Usmanmuhd added a comment.EditedJul 12 2019, 12:36 PM

@bmansurov Oh I confused sitelink_count with index. In our case we don't have sitelink count. Should we fetch it from the API or should we just phase it out?

Is langlinkscount the field we are looking for?

@Usmanmuhd Good call. I forgot about that. So we have to make two kinds of requests to the API: by seed and by pageviews. In each case we need to make a request that fetches Wikidata IDs, then we need to make a separate request to sort these IDs by sitelink_count. We cannot phase it out because the purpose of this API is to recommend the most appropriate articles for creation, and filtering by sitelink_count was deemed appropriate when the API was created.

@Usmanmuhd that URL looks correct. As for sitelinks_count, you'd have to checkout the MediaWiki API documentation. I can't remember off the top of my head.

Usmanmuhd added a comment.EditedJul 16 2019, 1:57 PM

@bmansurov

Edit1: Figured it out.

Edit 2:

  1. Is this link [ https://uz.wikipedia.org/w/api.php?format=json&action=query&prop=pageprops|langlinks&ppprop=wikibase_item|disambiguation&lllang=en&generator=mostviewed ] correct?

The output it returns is

{"batchcomplete":"","continue":{"gpvimoffset":10,"continue":"gpvimoffset||"},"query":{"pages":{"-1":{"ns":-1,"title":"Maxsus:CreateAccount","special":""},"-2":{"ns":-1,"title":"Maxsus:ElectronPdf","special":""},"-3":{"ns":-1,"title":"Maxsus:Search","special":""},"-4":{"ns":-1,"title":"Maxsus:CiteThisPage","special":""},"673959":{"pageid":673959,"ns":0,"title":"Abdusalom Azizov","pageprops":{"wikibase_item":"Q39083778"},"langlinks":[{"lang":"en","*":"Abdusalom Azizov"}]},"1":{"pageid":1,"ns":0,"title":"Bosh Sahifa","pageprops":{"wikibase_item":"Q5296"},"langlinks":[{"lang":"en","*":""}]},"1819":{"pageid":1819,"ns":0,"title":"Muhammad","pageprops":{"wikibase_item":"Q9458"},"langlinks":[{"lang":"en","*":"Muhammad"}]},"793":{"pageid":793,"ns":0,"title":"O\u02bbzbekiston","pageprops":{"wikibase_item":"Q265"},"langlinks":[{"lang":"en","*":"Uzbekistan"}]},"4345":{"pageid":4345,"ns":0,"title":"Shavkat Mirziyoyev","pageprops":{"wikibase_item":"Q57808"},"langlinks":[{"lang":"en","*":"Shavkat Mirziyoyev"}]},"656178":{"pageid":656178,"ns":0,"title":"UTC","pageprops":{"wikibase_item":"Q1536"},"langlinks":[{"lang":"en","*":"Coordinated Universal Time"}]}}}}

The above output has no overlap with the actual solution:

{"count":24,"items":[{"wikidata_id":"Q11846678","title":"Saqoqushlar","sitelink_count":13},{"wikidata_id":"Q4056778","title":"Ogahiy","sitelink_count":7},{"wikidata_id":"Q38905192","title":"Poʻlat Bobojonov","sitelink_count":3},{"wikidata_id":"Q12828800","title":"Nukus pedagogika instituti","sitelink_count":3},{"wikidata_id":"Q8249804","title":"Bojalar (guruh)","sitelink_count":2},{"wikidata_id":"Q16709430","title":"Oʻzbek urugʻlari","sitelink_count":2},{"wikidata_id":"Q21645204","title":"Said Ahmad","sitelink_count":2},{"wikidata_id":"Q16656759","title":"Oʻzbekiston Respublikasi Vazirlar Mahkamasi","sitelink_count":2},{"wikidata_id":"Q19848817","title":"Jangi loy","sitelink_count":2},{"wikidata_id":"Q25524958","title":"Chimildiq","sitelink_count":2},{"wikidata_id":"Q28464328","title":"Zarina Nizomiddinova","sitelink_count":2},{"wikidata_id":"Q4207413","title":"Pirimqul Qodirov","sitelink_count":2},{"wikidata_id":"Q4453002","title":"Oʻzbekiston Respublikasi Milliy gvardiyasi harbiy texnika instituti","sitelink_count":2},{"wikidata_id":"Q12829575","title":"Oʻrta Osiyo (eneolit va bronza davri)","sitelink_count":1},{"wikidata_id":"Q12824019","title":"Jinsiy gigiyena","sitelink_count":1},{"wikidata_id":"Q25513686","title":"Pirouzum kislota","sitelink_count":1},{"wikidata_id":"Q25524241","title":"Xeops","sitelink_count":1},{"wikidata_id":"Q25528466","title":"Qogʻonoq suvi","sitelink_count":1},{"wikidata_id":"Q25529284","title":"Shayx Muhammad Sodiq Muhammad Yusuf kitoblari roʻyxati","sitelink_count":1},{"wikidata_id":"Q25533492","title":"Klinik qon tahlili","sitelink_count":1},{"wikidata_id":"Q31283836","title":"Shohruhxon","sitelink_count":1},{"wikidata_id":"Q12826180","title":"Logarifm jadvallari","sitelink_count":1},{"wikidata_id":"Q12829614","title":"Oʻzbekiston telefon kodlari","sitelink_count":1},{"wikidata_id":"Q21663931","title":"Ismlar imlosi","sitelink_count":1}]}
  1. langlinkscount + 1 same as sitelink_count. Can you check please?

Change 523779 had a related patch set uploaded (by Usmanmuhd; owner: Usmanmuhd):
[mediawiki/services/recommendation-api@master] Replace Wikidata Query Service with MediaWiki API

https://gerrit.wikimedia.org/r/523779

Usmanmuhd added a comment.EditedJul 16 2019, 5:22 PM

@bmansurov Pushed the patch for getArticlesBySeed.

Example:
Input: http://localhost:6927/en.wikipedia.org/v1/article/creation/translation/uz/Yorug'lik

Output:

{"count":24,"items":[{"wikidata_id":"Q9430","title":"Okean","sitelink_count":193},{"wikidata_id":"Q523","title":"Yulduz","sitelink_count":191},{"wikidata_id":"Q7944","title":"Zilzila","sitelink_count":170},{"wikidata_id":"Q318","title":"Galaktika","sitelink_count":153},{"wikidata_id":"Q12725","title":"Elektr","sitelink_count":147},{"wikidata_id":"Q11379","title":"Energiya","sitelink_count":146},{"wikidata_id":"Q323","title":"Katta portlash","sitelink_count":138},{"wikidata_id":"Q11982","title":"Fotosintez","sitelink_count":131},{"wikidata_id":"Q11650","title":"Elektronika","sitelink_count":130},{"wikidata_id":"Q11435","title":"Suyuqlik","sitelink_count":121},{"wikidata_id":"Q944","title":"Kvant mexanika","sitelink_count":117},{"wikidata_id":"Q2111","title":"Yorugʻlik tezligi","sitelink_count":113},{"wikidata_id":"Q11438","title":"Qattiq jism","sitelink_count":112},{"wikidata_id":"Q34777","title":"Rentgen nurlari","sitelink_count":112},{"wikidata_id":"Q35197","title":"Koʻzgu","sitelink_count":111},{"wikidata_id":"Q11391","title":"Ultrabinafsha nurlanish","sitelink_count":105},{"wikidata_id":"Q38867","title":"Lazer","sitelink_count":103},{"wikidata_id":"Q14620","title":"Optika","sitelink_count":102},{"wikidata_id":"Q3198","title":"Foton","sitelink_count":99},{"wikidata_id":"Q37547","title":"Astrofizika","sitelink_count":92},{"wikidata_id":"Q768575","title":"Linza","sitelink_count":92},{"wikidata_id":"Q41257","title":"Gers","sitelink_count":88},{"wikidata_id":"Q127234","title":"Albert A. Michelson","sitelink_count":85},{"wikidata_id":"Q72277","title":"Refraksiya","sitelink_count":79}]}

Edit 2: For getArticlesByPageviews.
Input: http://localhost:6927/en.wikipedia.org/v1/article/creation/translation/uz
Output:

{"count":24,"items":[{"wikidata_id":"Q17","title":"Yaponiya","sitelink_count":289},{"wikidata_id":"Q30","title":"Amerika Qoʻshma Shtatlari","sitelink_count":285},{"wikidata_id":"Q38","title":"Italiya","sitelink_count":281},{"wikidata_id":"Q142","title":"Fransiya","sitelink_count":277},{"wikidata_id":"Q15","title":"Afrika","sitelink_count":274},{"wikidata_id":"Q148","title":"Xitoy","sitelink_count":273},{"wikidata_id":"Q145","title":"Birlashgan Qirollik","sitelink_count":264},{"wikidata_id":"Q16","title":"Kanada","sitelink_count":264},{"wikidata_id":"Q408","title":"Avstraliya (davlat)","sitelink_count":258},{"wikidata_id":"Q224","title":"Xorvatiya","sitelink_count":257},{"wikidata_id":"Q79","title":"Misr","sitelink_count":254},{"wikidata_id":"Q220","title":"Rim","sitelink_count":244},{"wikidata_id":"Q262","title":"Jazoir","sitelink_count":242},{"wikidata_id":"Q188","title":"Nemis tili","sitelink_count":240},{"wikidata_id":"Q884","title":"Koreya Respublikasi","sitelink_count":235},{"wikidata_id":"Q283","title":"Suv","sitelink_count":232},{"wikidata_id":"Q2807","title":"Madrid","sitelink_count":231},{"wikidata_id":"Q146","title":"Mushuk","sitelink_count":230},{"wikidata_id":"Q851","title":"Saudiya Arabistoni","sitelink_count":230},{"wikidata_id":"Q432","title":"Islom","sitelink_count":229},{"wikidata_id":"Q822","title":"Livan","sitelink_count":225},{"wikidata_id":"Q49","title":"Shimoliy Amerika","sitelink_count":224},{"wikidata_id":"Q810","title":"Iordaniya","sitelink_count":219},{"wikidata_id":"Q729","title":"Hayvonlar","sitelink_count":219}]}

@bmansurov The Q406 item is coming from the following query: https://ru.wikipedia.org/w/api.php?action=query&format=json&prop=pageprops|langlinks|langlinkscount&ppprop=wikibase_item|disambiguation&generator=mostviewed&gpvimlimit=50&gpvimoffset=450&lllang=uz .

The item being returned is:

"40352": {
                "pageid": 40352,
                "ns": 0,
                "title": "\u0421\u0442\u0430\u043c\u0431\u0443\u043b",
                "pageprops": {
                    "wikibase_item": "Q406"
                },
                "langlinkscount": 206
            },

For other items(example) it is returning:

"9874": {
                "pageid": 9874,
                "ns": 0,
                "title": "\u0424\u0440\u0430\u043d\u0446\u0443\u0437\u0441\u043a\u0438\u0439 \u044f\u0437\u044b\u043a",
                "pageprops": {
                    "wikibase_item": "Q150"
                },
                "langlinks": [
                    {
                        "lang": "uz",
                        "*": "Fransuz tili"
                    }
                ],
                "langlinkscount": 240
            },

Seems like some problem with the API in this case.

@Usmanmuhd Looks like we're not passing all the required parameters to the API. Take a look at https://www.mediawiki.org/wiki/API:Langlinks, specifically llcontinue and lllimit. Once you pass those, you should get langlinks for results in subsequent requests after the first one.

@bmansurov I think that is required when we have more than 10 languages being returned. In our case it's just one that is being returned. Should we really have that still?

Looks like so. Compare the following two URLs:

https://uz.wikipedia.org/w/api.php?action=query&format=json&prop=pageprops|langlinks|langlinkscount&ppprop=wikibase_item|disambiguation&generator=mostviewed&gpvimlimit=50&gpvimoffset=450&lllang=ru

Here is what I see when I visit the above URL:

{
    "query": {
        "pages": {
            "20617": {
                "langlinkscount": 182,
                "pageprops": {
                    "wikibase_item": "Q1982926"
                },
                "title": "Turkum:Koʻp maʼnoli atamalar",
                "ns": 14,
                "pageid": 20617
            },
            "15222": {
                "langlinkscount": 72,
                "pageprops": {
                    "wikibase_item": "Q4097370"
                },
                "title": "Vikipediya:Interviki",
                "ns": 4,
                "pageid": 15222
            },
            "494851": {
                "pageprops": {
                    "wikibase_item": "Q12832530"
                },
                "title": "Zaryadning saqlanish qonuni",
                "ns": 0,
                "pageid": 494851
            },
            "616083": {
                "langlinkscount": 106,
                "pageprops": {
                    "wikibase_item": "Q35694"
                },
                "title": "Yaguar",
                "ns": 0,
                "pageid": 616083
            },
            "4098": {
                "langlinkscount": 230,
                "langlinks": [
                    {
                        "*": "Христианство",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q5043"
                },
                "title": "Xristian dini",
                "ns": 0,
                "pageid": 4098
            },
            "3382": {
                "langlinkscount": 263,
                "langlinks": [
                    {
                        "*": "Вьетнам",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q881"
                },
                "title": "Vyetnam",
                "ns": 0,
                "pageid": 3382
            },
            "3211": {
                "langlinkscount": 264,
                "langlinks": [
                    {
                        "*": "Украина",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q212"
                },
                "title": "Ukraina",
                "ns": 0,
                "pageid": 3211
            },
            "581990": {
                "langlinkscount": 114,
                "pageprops": {
                    "wikibase_item": "Q553"
                },
                "title": "Tish",
                "ns": 0,
                "pageid": 581990
            },
            "682411": {
                "langlinkscount": 1,
                "pageprops": {
                    "wikibase_item": "Q58967593"
                },
                "title": "Tashkent City",
                "ns": 0,
                "pageid": 682411
            },
            "646532": {
                "pageprops": {
                    "wikibase_item": "Q61123221"
                },
                "title": "Tamila",
                "ns": 0,
                "pageid": 646532
            },
            "575011": {
                "pageprops": {
                    "wikibase_item": "Q25519436"
                },
                "title": "Suvchechak",
                "ns": 0,
                "pageid": 575011
            },
            "672693": {
                "pageprops": {
                    "wikibase_item": "Q28669258"
                },
                "title": "Shaytanat",
                "ns": 0,
                "pageid": 672693
            },
            "557548": {
                "langlinkscount": 142,
                "pageprops": {
                    "wikibase_item": "Q12078"
                },
                "title": "Saraton (kasallik)",
                "ns": 0,
                "pageid": 557548
            },
            "638692": {
                "langlinkscount": 178,
                "pageprops": {
                    "wikibase_item": "Q735"
                },
                "title": "Sanʼat",
                "ns": 0,
                "pageid": 638692
            },
            "168604": {
                "langlinkscount": 175,
                "pageprops": {
                    "wikibase_item": "Q6583"
                },
                "title": "Sahroi Kabir",
                "ns": 0,
                "pageid": 168604
            },
            "566400": {
                "pageprops": {
                    "wikibase_item": "Q12823276"
                },
                "title": "Sachratqi",
                "ns": 0,
                "pageid": 566400
            },
            "13569": {
                "langlinkscount": 203,
                "pageprops": {
                    "wikibase_item": "Q544"
                },
                "title": "Quyosh tizimi",
                "ns": 0,
                "pageid": 13569
            },
            "625029": {
                "pageprops": {
                    "wikibase_item": "Q25528098"
                },
                "title": "Qalqonsimon bez",
                "ns": 0,
                "pageid": 625029
            },
            "556327": {
                "langlinkscount": 175,
                "pageprops": {
                    "wikibase_item": "Q9418"
                },
                "title": "Psixologiya",
                "ns": 0,
                "pageid": 556327
            },
            "14737": {
                "langlinkscount": 157,
                "pageprops": {
                    "wikibase_item": "Q10943"
                },
                "title": "Pishloq",
                "ns": 0,
                "pageid": 14737
            },
            "20857": {
                "langlinkscount": 14,
                "pageprops": {
                    "wikibase_item": "Q651133"
                },
                "title": "Oʻzbekiston milliy universiteti",
                "ns": 0,
                "pageid": 20857
            },
            "15796": {
                "langlinkscount": 122,
                "pageprops": {
                    "wikibase_item": "Q1344"
                },
                "title": "Opera",
                "ns": 0,
                "pageid": 15796
            },
            "5756": {
                "title": "O`zbek Adabiyot",
                "ns": 0,
                "pageid": 5756
            },
            "3202": {
                "langlinkscount": 282,
                "langlinks": [
                    {
                        "*": "Норвегия",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q20"
                },
                "title": "Norvegiya",
                "ns": 0,
                "pageid": 3202
            },
            "537733": {
                "pageprops": {
                    "wikibase_item": "Q12828593"
                },
                "title": "Ninachilar",
                "ns": 0,
                "pageid": 537733
            },
            "536567": {
                "langlinkscount": 57,
                "pageprops": {
                    "wikibase_item": "Q188008"
                },
                "title": "Nafas qisishi",
                "ns": 0,
                "pageid": 536567
            },
            "5151": {
                "langlinkscount": 202,
                "pageprops": {
                    "wikibase_item": "Q638"
                },
                "title": "Musiqa",
                "ns": 0,
                "pageid": 5151
            },
            "216751": {
                "langlinkscount": 116,
                "pageprops": {
                    "wikibase_item": "Q712226"
                },
                "title": "Kvadrat kilometr",
                "ns": 0,
                "pageid": 216751
            },
            "492555": {
                "pageprops": {
                    "wikibase_item": "Q25530962"
                },
                "title": "Jiraf (hayvon)",
                "ns": 0,
                "pageid": 492555
            },
            "490431": {
                "langlinkscount": 127,
                "pageprops": {
                    "wikibase_item": "Q8341"
                },
                "title": "Jaz",
                "ns": 0,
                "pageid": 490431
            },
            "3394": {
                "langlinkscount": 218,
                "langlinks": [
                    {
                        "*": "Иордания",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q810"
                },
                "title": "Iordaniya",
                "ns": 0,
                "pageid": 3394
            },
            "498980": {
                "title": "Insulin",
                "ns": 0,
                "pageid": 498980
            },
            "456381": {
                "langlinkscount": 84,
                "pageprops": {
                    "wikibase_item": "Q81788"
                },
                "title": "Giza",
                "ns": 0,
                "pageid": 456381
            },
            "139565": {
                "langlinkscount": 160,
                "pageprops": {
                    "wikibase_item": "Q1410"
                },
                "title": "Gibraltar",
                "ns": 0,
                "pageid": 139565
            },
            "20739": {
                "langlinkscount": 158,
                "pageprops": {
                    "wikibase_item": "Q8087"
                },
                "title": "Geometriya",
                "ns": 0,
                "pageid": 20739
            },
            "455179": {
                "pageprops": {
                    "wikibase_item": "Q12821918"
                },
                "title": "Galley kometasi",
                "ns": 0,
                "pageid": 455179
            },
            "1047": {
                "langlinkscount": 48,
                "langlinks": [
                    {
                        "*": "Ферганская область",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q487089"
                },
                "title": "Fargʻona viloyati",
                "ns": 0,
                "pageid": 1047
            },
            "3392": {
                "langlinkscount": 253,
                "langlinks": [
                    {
                        "*": "Иран",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q794"
                },
                "title": "Eron",
                "ns": 0,
                "pageid": 3392
            },
            "613401": {
                "pageprops": {
                    "wikibase_item": "Q25527311"
                },
                "title": "Elektromagnit toʻlqinlar",
                "ns": 0,
                "pageid": 613401
            },
            "18871": {
                "pageprops": {
                    "wikibase_item": "Q65290668"
                },
                "title": "Delfin",
                "ns": 0,
                "pageid": 18871
            },
            "461310": {
                "pageprops": {
                    "wikibase_item": "Q12821190"
                },
                "title": "Deformatsiya",
                "ns": 0,
                "pageid": 461310
            },
            "21762": {
                "langlinkscount": 67,
                "pageprops": {
                    "wikibase_item": "Q483269"
                },
                "title": "Choʻyan",
                "ns": 0,
                "pageid": 21762
            },
            "22415": {
                "langlinkscount": 150,
                "pageprops": {
                    "wikibase_item": "Q720"
                },
                "title": "Chingizxon",
                "ns": 0,
                "pageid": 22415
            },
            "3180": {
                "langlinkscount": 247,
                "langlinks": [
                    {
                        "*": "Чехия",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q213"
                },
                "title": "Chexiya",
                "ns": 0,
                "pageid": 3180
            },
            "3174": {
                "langlinkscount": 247,
                "langlinks": [
                    {
                        "*": "Белоруссия",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q184"
                },
                "title": "Belarus",
                "ns": 0,
                "pageid": 3174
            },
            "441443": {
                "langlinkscount": 107,
                "pageprops": {
                    "wikibase_item": "Q37853"
                },
                "title": "Barokko",
                "ns": 0,
                "pageid": 441443
            },
            "441171": {
                "pageprops": {
                    "wikibase_item": "Q12819304"
                },
                "title": "Balogʻatga yetish",
                "ns": 0,
                "pageid": 441171
            },
            "2193": {
                "langlinkscount": 247,
                "langlinks": [
                    {
                        "*": "Армения",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q399"
                },
                "title": "Armaniston",
                "ns": 0,
                "pageid": 2193
            },
            "446814": {
                "langlinkscount": 44,
                "pageprops": {
                    "wikibase_item": "Q209387"
                },
                "title": "Anqara jangi",
                "ns": 0,
                "pageid": 446814
            },
            "-1": {
                "special": "",
                "title": "Maxsus:RecentChanges",
                "ns": -1
            }
        }
    },
    "continue": {
        "continue": "||pageprops|langlinkscount",
        "llcontinue": "5151|ru"
    }
}

And the second URL is https://uz.wikipedia.org/w/api.php?action=query&format=json&prop=pageprops|langlinks|langlinkscount&ppprop=wikibase_item|disambiguation&generator=mostviewed&gpvimlimit=50&gpvimoffset=450&lllang=ru&llcontinue=5151|ru

{
    "query": {
        "pages": {
            "20617": {
                "langlinkscount": 182,
                "langlinks": [
                    {
                        "*": "Категория:Страницы значений",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q1982926"
                },
                "title": "Turkum:Koʻp maʼnoli atamalar",
                "ns": 14,
                "pageid": 20617
            },
            "15222": {
                "langlinkscount": 72,
                "langlinks": [
                    {
                        "*": "Википедия:Интервики",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q4097370"
                },
                "title": "Vikipediya:Interviki",
                "ns": 4,
                "pageid": 15222
            },
            "494851": {
                "pageprops": {
                    "wikibase_item": "Q12832530"
                },
                "title": "Zaryadning saqlanish qonuni",
                "ns": 0,
                "pageid": 494851
            },
            "616083": {
                "langlinkscount": 106,
                "pageprops": {
                    "wikibase_item": "Q35694"
                },
                "title": "Yaguar",
                "ns": 0,
                "pageid": 616083
            },
            "4098": {
                "langlinkscount": 230,
                "pageprops": {
                    "wikibase_item": "Q5043"
                },
                "title": "Xristian dini",
                "ns": 0,
                "pageid": 4098
            },
            "3382": {
                "langlinkscount": 263,
                "pageprops": {
                    "wikibase_item": "Q881"
                },
                "title": "Vyetnam",
                "ns": 0,
                "pageid": 3382
            },
            "3211": {
                "langlinkscount": 264,
                "pageprops": {
                    "wikibase_item": "Q212"
                },
                "title": "Ukraina",
                "ns": 0,
                "pageid": 3211
            },
            "581990": {
                "langlinkscount": 114,
                "pageprops": {
                    "wikibase_item": "Q553"
                },
                "title": "Tish",
                "ns": 0,
                "pageid": 581990
            },
            "682411": {
                "langlinkscount": 1,
                "pageprops": {
                    "wikibase_item": "Q58967593"
                },
                "title": "Tashkent City",
                "ns": 0,
                "pageid": 682411
            },
            "646532": {
                "pageprops": {
                    "wikibase_item": "Q61123221"
                },
                "title": "Tamila",
                "ns": 0,
                "pageid": 646532
            },
            "575011": {
                "pageprops": {
                    "wikibase_item": "Q25519436"
                },
                "title": "Suvchechak",
                "ns": 0,
                "pageid": 575011
            },
            "672693": {
                "pageprops": {
                    "wikibase_item": "Q28669258"
                },
                "title": "Shaytanat",
                "ns": 0,
                "pageid": 672693
            },
            "557548": {
                "langlinkscount": 142,
                "pageprops": {
                    "wikibase_item": "Q12078"
                },
                "title": "Saraton (kasallik)",
                "ns": 0,
                "pageid": 557548
            },
            "638692": {
                "langlinkscount": 178,
                "pageprops": {
                    "wikibase_item": "Q735"
                },
                "title": "Sanʼat",
                "ns": 0,
                "pageid": 638692
            },
            "168604": {
                "langlinkscount": 175,
                "pageprops": {
                    "wikibase_item": "Q6583"
                },
                "title": "Sahroi Kabir",
                "ns": 0,
                "pageid": 168604
            },
            "566400": {
                "pageprops": {
                    "wikibase_item": "Q12823276"
                },
                "title": "Sachratqi",
                "ns": 0,
                "pageid": 566400
            },
            "13569": {
                "langlinkscount": 203,
                "langlinks": [
                    {
                        "*": "Солнечная система",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q544"
                },
                "title": "Quyosh tizimi",
                "ns": 0,
                "pageid": 13569
            },
            "625029": {
                "pageprops": {
                    "wikibase_item": "Q25528098"
                },
                "title": "Qalqonsimon bez",
                "ns": 0,
                "pageid": 625029
            },
            "556327": {
                "langlinkscount": 175,
                "pageprops": {
                    "wikibase_item": "Q9418"
                },
                "title": "Psixologiya",
                "ns": 0,
                "pageid": 556327
            },
            "14737": {
                "langlinkscount": 157,
                "langlinks": [
                    {
                        "*": "Сыр",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q10943"
                },
                "title": "Pishloq",
                "ns": 0,
                "pageid": 14737
            },
            "20857": {
                "langlinkscount": 14,
                "langlinks": [
                    {
                        "*": "Национальный университет Узбекистана",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q651133"
                },
                "title": "Oʻzbekiston milliy universiteti",
                "ns": 0,
                "pageid": 20857
            },
            "15796": {
                "langlinkscount": 122,
                "langlinks": [
                    {
                        "*": "Опера",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q1344"
                },
                "title": "Opera",
                "ns": 0,
                "pageid": 15796
            },
            "5756": {
                "title": "O`zbek Adabiyot",
                "ns": 0,
                "pageid": 5756
            },
            "3202": {
                "langlinkscount": 282,
                "pageprops": {
                    "wikibase_item": "Q20"
                },
                "title": "Norvegiya",
                "ns": 0,
                "pageid": 3202
            },
            "537733": {
                "pageprops": {
                    "wikibase_item": "Q12828593"
                },
                "title": "Ninachilar",
                "ns": 0,
                "pageid": 537733
            },
            "536567": {
                "langlinkscount": 57,
                "pageprops": {
                    "wikibase_item": "Q188008"
                },
                "title": "Nafas qisishi",
                "ns": 0,
                "pageid": 536567
            },
            "5151": {
                "langlinkscount": 202,
                "langlinks": [
                    {
                        "*": "Музыка",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q638"
                },
                "title": "Musiqa",
                "ns": 0,
                "pageid": 5151
            },
            "216751": {
                "langlinkscount": 116,
                "pageprops": {
                    "wikibase_item": "Q712226"
                },
                "title": "Kvadrat kilometr",
                "ns": 0,
                "pageid": 216751
            },
            "492555": {
                "pageprops": {
                    "wikibase_item": "Q25530962"
                },
                "title": "Jiraf (hayvon)",
                "ns": 0,
                "pageid": 492555
            },
            "490431": {
                "langlinkscount": 127,
                "pageprops": {
                    "wikibase_item": "Q8341"
                },
                "title": "Jaz",
                "ns": 0,
                "pageid": 490431
            },
            "3394": {
                "langlinkscount": 218,
                "pageprops": {
                    "wikibase_item": "Q810"
                },
                "title": "Iordaniya",
                "ns": 0,
                "pageid": 3394
            },
            "498980": {
                "title": "Insulin",
                "ns": 0,
                "pageid": 498980
            },
            "456381": {
                "langlinkscount": 84,
                "pageprops": {
                    "wikibase_item": "Q81788"
                },
                "title": "Giza",
                "ns": 0,
                "pageid": 456381
            },
            "139565": {
                "langlinkscount": 160,
                "pageprops": {
                    "wikibase_item": "Q1410"
                },
                "title": "Gibraltar",
                "ns": 0,
                "pageid": 139565
            },
            "20739": {
                "langlinkscount": 158,
                "langlinks": [
                    {
                        "*": "Геометрия",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q8087"
                },
                "title": "Geometriya",
                "ns": 0,
                "pageid": 20739
            },
            "455179": {
                "pageprops": {
                    "wikibase_item": "Q12821918"
                },
                "title": "Galley kometasi",
                "ns": 0,
                "pageid": 455179
            },
            "1047": {
                "langlinkscount": 48,
                "pageprops": {
                    "wikibase_item": "Q487089"
                },
                "title": "Fargʻona viloyati",
                "ns": 0,
                "pageid": 1047
            },
            "3392": {
                "langlinkscount": 253,
                "pageprops": {
                    "wikibase_item": "Q794"
                },
                "title": "Eron",
                "ns": 0,
                "pageid": 3392
            },
            "613401": {
                "pageprops": {
                    "wikibase_item": "Q25527311"
                },
                "title": "Elektromagnit toʻlqinlar",
                "ns": 0,
                "pageid": 613401
            },
            "18871": {
                "pageprops": {
                    "wikibase_item": "Q65290668"
                },
                "title": "Delfin",
                "ns": 0,
                "pageid": 18871
            },
            "461310": {
                "pageprops": {
                    "wikibase_item": "Q12821190"
                },
                "title": "Deformatsiya",
                "ns": 0,
                "pageid": 461310
            },
            "21762": {
                "langlinkscount": 67,
                "langlinks": [
                    {
                        "*": "Чугун",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q483269"
                },
                "title": "Choʻyan",
                "ns": 0,
                "pageid": 21762
            },
            "22415": {
                "langlinkscount": 150,
                "langlinks": [
                    {
                        "*": "Чингисхан",
                        "lang": "ru"
                    }
                ],
                "pageprops": {
                    "wikibase_item": "Q720"
                },
                "title": "Chingizxon",
                "ns": 0,
                "pageid": 22415
            },
            "3180": {
                "langlinkscount": 247,
                "pageprops": {
                    "wikibase_item": "Q213"
                },
                "title": "Chexiya",
                "ns": 0,
                "pageid": 3180
            },
            "3174": {
                "langlinkscount": 247,
                "pageprops": {
                    "wikibase_item": "Q184"
                },
                "title": "Belarus",
                "ns": 0,
                "pageid": 3174
            },
            "441443": {
                "langlinkscount": 107,
                "pageprops": {
                    "wikibase_item": "Q37853"
                },
                "title": "Barokko",
                "ns": 0,
                "pageid": 441443
            },
            "441171": {
                "pageprops": {
                    "wikibase_item": "Q12819304"
                },
                "title": "Balogʻatga yetish",
                "ns": 0,
                "pageid": 441171
            },
            "2193": {
                "langlinkscount": 247,
                "pageprops": {
                    "wikibase_item": "Q399"
                },
                "title": "Armaniston",
                "ns": 0,
                "pageid": 2193
            },
            "446814": {
                "langlinkscount": 44,
                "pageprops": {
                    "wikibase_item": "Q209387"
                },
                "title": "Anqara jangi",
                "ns": 0,
                "pageid": 446814
            },
            "-1": {
                "special": "",
                "title": "Maxsus:RecentChanges",
                "ns": -1
            }
        }
    },
    "continue": {
        "continue": "||pageprops|langlinkscount",
        "llcontinue": "139565|ru"
    }
}

If look at the page titled "Opera" in both results, you'll see that the lang links are present in the second result. The difference between the URLs is the llcontinue parameter.

@bmansurov made the changes and also enabled the tests.

@bmansurov Should I handle the llcontinue in getArticles itself? i.e. if llcontinue then call mwApiGet again from the same place.

I think you should handle it similar to this code, i.e. llcontinue should go into the getArticlesBySeed and getArticlesByPageviews functions directly.

@bmansurov There is a problem with handling it this way. We don't know the value that has to be passed until the previous request is finished. Trying to figure out the right implementation for this.

@Usmanmuhd since we're only requesting 500 items, can you set lllimit to 500 instead? This should return all langlinks in the first request.

Usmanmuhd added a comment.EditedFri, Jul 26, 6:45 AM

@bmansurov We had reduced the gpvimlimit in this case to 50 because we were having problems with ruwiki. I tried increasing the gpvimlimit and looks like it's been resolved.

Edit 1: It works correctly by setting gpvimlimit and lllimit both to 500. Should I make the gpvimlimit change or not?

@bmansurov pushed the fix. Please take a look now.

@leila my Gerrit +2 right have been revoked so I couldn't merge the student's patch. Could you help us find someone who can merge the patch? Thanks. Here's the patch: https://gerrit.wikimedia.org/r/#/c/mediawiki/services/recommendation-api/+/523779/

@Usmanmuhd probably could have +2 in this repo, I think he is a GSOC volunteer so not sure of protocol cc-ing @greg and @Aklapper

@leila As GSoC is ending could you please help me get this patch merged?

leila added a comment.Tue, Aug 20, 9:29 PM

@Usmanmuhd thanks for the reminder. I'll ask around again.