Nice question. There is a french ground-truth available. Subsequently, the model will accommodate several ground truths which would be used alongside the language picker.

In T263874#6578380, @Precillieo wrote:

Hello Everyone,
I'd like to raise a question concerning the predictive model we are to build in the later part of the notebook.
Are we predicting the protection types or we are to predict if a page is protected or not??

In T263874#6577188, @SafiaKhaleel wrote:

Thank You so much @Isaac , @Amamgbu and @Tambe You guys have been really helpful.
However, when I use the value wbgetentities for the action parameter, I'm getting the following error:

APIError: badvalue: Unrecognized value for parameter "action": wbgetentities. -- See https://en.wikipedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes.

In T263874#6576412, @SafiaKhaleel wrote:

In T263874#6576411, @Amamgbu wrote:

In T263874#6576410, @SafiaKhaleel wrote:

Hi @Isaac and everyone,
Can anyone give me an idea to find if a page is about a human or not?

You could reference the tutorial given to us by @Isaac. There’s a segment on that in it.

Yes but its for checking if an individual "item" is human or not. I was thinking if we could implement that in pages too

In T263874#6576410, @SafiaKhaleel wrote:

Hi @Isaac and everyone,
Can anyone give me an idea to find if a page is about a human or not?

Hi @Isaac and everyone

In T263874#6569853, @Isaac wrote:

Welcome to all the new applicants since I last posted a welcome! One request for everyone working on this task:

• To get a good sense of how many people are intending to apply to each project (T263646 and/or T263860), I'd ask that you make an initial contribution on the Outreachy site with a link to your current progress in the next two days (so by end-of-day October 23rd).

Thanks! Keep the questions / collaboration coming!

Thanks @Amamgbu But a lot of pages seems to have 'null' value for the pageviews variable.
And another problem I encountered is with the limit upto which we can query page information using title/pageid. Getting only 50 instances at a time is not enough to work on right? Does anyone know how to increase the limit upto 500? (Its said "500 for clients allowed higher limits")
@Isaac

In T263874#6559875, @Thulieblack wrote:

@Amamgbu thanks I have sorted it out 🤗🤗

In T263874#6558281, @SafiaKhaleel wrote:

@Isaac and everyone ,
Can we access the variable 'page_counter' from the page table as it had been removed completely in MediaWiki 1.25. Is there any other method to get views of each page?

@Amamgbu inprop gave me an error I passed the prop*inf that's how I got all the queries

In T263874#6557977, @Thulieblack wrote:

@Amamgbu take a look{F32397114}

In T263874#6557925, @Thulieblack wrote:

Perfect day I need to ask,my API response doesn't show protection type unlike the Dump data,is it something I need to worry about?

@Amamgbu that is correct and @Vanevela pointed to the appropriate prior discussion about this. More details: the restrictiontypes field is just what restrictions could be applied to the page, not which ones are applied -- a fuller description of what you could find in that field can be found here. For most pages, you'll see edit and move and can verify this by choosing a random page without restrictions and querying the API. I'd suggest ignoring the field as it won't tell you much.

In T263874#6556936, @Vanevela wrote:

Hi @Amamgbu
I think that this issue was addressed here, I hope that this helps you.

@Isaac and everyone.

In T263874#6549305, @SIBGHAsheikh wrote:

Hi,
I am extremely sorry for this stupid question, but can anyone please guide me a bit about what is happening in "example of working with Wikidata data" in cell 5. Basically, I am having difficulty in understanding the format of data extraction from json dump. What checks are being confirmed by if and for statements?

Thanks

In T263874#6546833, @SIBGHAsheikh wrote:

Hi,
I am unable to understand "head -46 " in below line code.

!zcat "{DUMP_DIR}{DUMP_FN}" | head -46 | cut -c1-1000

Secondly, I am having error

IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
--NotebookApp.iopub_data_rate_limit.

Current values:
NotebookApp.iopub_data_rate_limit=1000000.0 (bytes/sec)
NotebookApp.rate_limit_window=3.0 (secs)

How to solve this error?

In T263874#6541244, @0xkaywong wrote:

Does anyone know if we can obtain the timestamp for when a protection was added for a page? It seems to be an option for protected titles, but I haven't had any luck in finding out how to do this for protected pages.

Hi everyone, I've been able to decompress the file but please how do I clean the data

In T263874#6534904, @SafiaKhaleel wrote:

In T263874#6534884, @Amamgbu wrote:

In T263874#6534882, @SafiaKhaleel wrote:

In T263874#6534745, @Amamgbu wrote:

In T263874#6534735, @SafiaKhaleel wrote:

@Isaac
The protection data that I got from MediaWiki dump and API results seems to be in different format.
For a particular page, we get data in the following formats..
In MediaWiki dump : (3664672,'edit','autoconfirmed',0,NULL,'infinity',717409)

In API result : {'pageid': 3664672, 'ns': 10, 'title': 'Template:Cyclopaedia 1728', 'contentmodel': 'wikitext', 'pagelanguage': 'en', 'pagelanguagehtmlcode': 'en', 'pagelanguagedir': 'ltr', 'touched': '2020-10-10T18:45:26Z', 'lastrevid': 952065611, 'length': 3572, 'protection': [{'type': 'edit', 'level': 'autoconfirmed', 'expiry': 'infinity'}, {'type': 'move', 'level': 'autoconfirmed', 'expiry': 'infinity'}], 'restrictiontypes': ['edit', 'move']}

Isn't that a problem? like how are we supposed to check the discrepancies between the two if they are in different formats? Moreover the protection data in API results doesnt show user specific restrictions or sysop permissions. Is that ok?

Hi @SafiaKhaleel, it is possible to compare those tuples with the JSON objects through indexing. You can check the tutorial notebook @Isaac shared to us for more details.

@Isaac mentioned that the user specific restrictions was an obsolete field and we should disregard it. For the sysop permissions, it is stored in the JSON data as “level”

Thanks @Amamgbu . I understood that. But the protection type in both the cases also seems to be different. In the API result, all the pages have both edit and move protection as you can see here:
'protection': [{'type': 'edit', 'level': 'autoconfirmed', 'expiry': 'infinity'}, {'type': 'move', 'level': 'autoconfirmed', 'expiry': 'infinity'}]
whereas in MediaWiki dump, there is only one type out of the two: (39620487,'edit','autoconfirmed',0,NULL,'infinity',692808)

There are actually two if you inspect the data well. I had that same issue till i inspected the ids

Oh so do you mean to say there are two page protection entries for a single page in MediaWiki dumps? One for edit and another for move

In T263874#6534882, @SafiaKhaleel wrote:

In T263874#6534745, @Amamgbu wrote:

In T263874#6534735, @SafiaKhaleel wrote:

@Isaac
The protection data that I got from MediaWiki dump and API results seems to be in different format.
For a particular page, we get data in the following formats..
In MediaWiki dump : (3664672,'edit','autoconfirmed',0,NULL,'infinity',717409)

In API result : {'pageid': 3664672, 'ns': 10, 'title': 'Template:Cyclopaedia 1728', 'contentmodel': 'wikitext', 'pagelanguage': 'en', 'pagelanguagehtmlcode': 'en', 'pagelanguagedir': 'ltr', 'touched': '2020-10-10T18:45:26Z', 'lastrevid': 952065611, 'length': 3572, 'protection': [{'type': 'edit', 'level': 'autoconfirmed', 'expiry': 'infinity'}, {'type': 'move', 'level': 'autoconfirmed', 'expiry': 'infinity'}], 'restrictiontypes': ['edit', 'move']}

Isn't that a problem? like how are we supposed to check the discrepancies between the two if they are in different formats? Moreover the protection data in API results doesnt show user specific restrictions or sysop permissions. Is that ok?

Hi @SafiaKhaleel, it is possible to compare those tuples with the JSON objects through indexing. You can check the tutorial notebook @Isaac shared to us for more details.

@Isaac mentioned that the user specific restrictions was an obsolete field and we should disregard it. For the sysop permissions, it is stored in the JSON data as “level”

Thanks @Amamgbu . I understood that. But the protection type in both the cases also seems to be different. In the API result, all the pages have both edit and move protection as you can see here:
'protection': [{'type': 'edit', 'level': 'autoconfirmed', 'expiry': 'infinity'}, {'type': 'move', 'level': 'autoconfirmed', 'expiry': 'infinity'}]
whereas in MediaWiki dump, there is only one type out of the two: (39620487,'edit','autoconfirmed',0,NULL,'infinity',692808)

In T263874#6534735, @SafiaKhaleel wrote:

@Isaac
The protection data that I got from MediaWiki dump and API results seems to be in different format.
For a particular page, we get data in the following formats..
In MediaWiki dump : (3664672,'edit','autoconfirmed',0,NULL,'infinity',717409)

In API result : {'pageid': 3664672, 'ns': 10, 'title': 'Template:Cyclopaedia 1728', 'contentmodel': 'wikitext', 'pagelanguage': 'en', 'pagelanguagehtmlcode': 'en', 'pagelanguagedir': 'ltr', 'touched': '2020-10-10T18:45:26Z', 'lastrevid': 952065611, 'length': 3572, 'protection': [{'type': 'edit', 'level': 'autoconfirmed', 'expiry': 'infinity'}, {'type': 'move', 'level': 'autoconfirmed', 'expiry': 'infinity'}], 'restrictiontypes': ['edit', 'move']}

Isn't that a problem? like how are we supposed to check the discrepancies between the two if they are in different formats? Moreover the protection data in API results doesnt show user specific restrictions or sysop permissions. Is that ok?

In T263874#6534403, @Toluwani7 wrote:

# Wikidata JSON dump we'll start processing (56 GB in size, compressed) so far too large to process the whole thing right now
!ls -shH "{WIKIDATA_DIR}{WIKIDATA_DUMP_FN}"

I'm still going through the Wikidata example, do you know what the shH option might mean? I can't find it online

I am not too sure but I think !ls -shH means list short format with readable file size.

In T263874#6533330, @Thulieblack wrote:

Perfect evening
I'm really confused and need someone to clarify for me in cell 5
To do an example that loops through all pages and extract data how do I do that as the python docs don't really give example a or docs

In T263874#6532670, @Arfat2396 wrote:

In T263874#6532662, @Amamgbu wrote:

@Arfat2396 !zcat is a jupyter inbuilt command. It can also be found on Linux operating system. Mainly used for viewing compressed files without decompressing it.

So, I can use it in Jupyter with the same rules as in Linux! https://www.howtoforge.com/linux-zcat-command/ Thanks!

@Arfat2396 !zcat is a jupyter inbuilt command. It can also be found on Linux operating system. Mainly used for viewing compressed files without decompressing it.

After changing LANGUAGE variable, you should leave it as it is. Just as @Arfat2396 rightly said, you can access a list of supported db names for your language from the link shared.

Hi Shamima, I'm Jesse. Welcome. To your question, yes we are to start with T263874.