Page MenuHomePhabricator

Synchronising Wikidata and Wikipedias using pywikibot - Task 2
Closed, ResolvedPublic

Assigned To
Authored By
Mike_Peel
Mar 30 2021, 6:18 PM
Referenced Files
F34400200: pywikibot error.PNG
Apr 15 2021, 2:16 PM
F34316118: image.png
Apr 11 2021, 8:24 PM
F34275551: Capture.PNG
Apr 7 2021, 3:01 PM
F34249670: image.png
Apr 4 2021, 6:15 PM
F34248761: image.png
Apr 4 2021, 4:00 PM

Description

This is the second task for T276329, Synchronising Wikidata and Wikipedias using pywikibot, aimed at getting you familiar with Pywikibot.

  1. You should register a Wikimedia account if you don't already have one. You can do so at https://www.wikidata.org/w/index.php?title=Special:CreateAccount
  1. You should download and install Python 3 and pywikibot - see instructions at https://www.mediawiki.org/wiki/Manual:Pywikibot/Installation . You will need to set up a user configuration file, and make sure you can log in. Set 'wikipedia' as the base family, and then manually add usernames['wikidata']['wikidata'] = u'Your username' into the generated user-config.py file. If you want, you can set a bot password (see https://www.mediawiki.org/wiki/Manual:Pywikibot/BotPasswords ) and use that - or use your normal Wikimedia login information.
  1. Set up a script that will connect to Wikidata, load the 'Outreachy_1' page that you created in the previous task, and print it out.
  1. Try adding 'Hello' to the end of the page you just loaded, and save it back.
  1. Load a Wikidata item (use 'Q4115189' to start with - it is the sandbox), and print out information from it. You could also try loading some of the other items you looked at in the first task.

Bonus: Try adding a new property to the sandbox item (e.g., 'P31' = 'Q5') using your code, and saving it.

Save your code to a repository, or create a page like https://www.wikidata.org/wiki/User:Mike_Peel/Outreachy_2 (under your username)

Once you are happy, send me a link to your page (by email, on my talk page, or replying to this ticket as you prefer). Make sure to also register it as a contribution on the Outreachy website (https://www.outreachy.org/outreachy-may-2021-internship-round/communities/wikimedia/synchronising-wikidata-and-wikipedias-using-pywiki/contributions/)!

Hints:

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

When submitting my code, should I also send my user_config.py file privately? Since I picked Wikidata as my default family of sites. When I write my script, I don't have to write:

site = pywikibot.Site('en', 'wikidata')

I can instead just write:

site = pywikibot.Site()

and by default it chooses wikidata. This might cause some inconsistencies when sending my script to someone else.

It's generally better to write code that is stand-alone and others can run, so in this case it's worth including '('en', 'wikidata')' in the main script, even if it's not strictly needed for you to run it. You shouldn't normally share your user config file.

It's generally better to write code that is stand-alone and others can run, so in this case it's worth including '('en', 'wikidata')' in the main script, even if it's not strictly needed for you to run it. You shouldn't normally share your user config file.

Ok, yeah, was just making sure. Thanks.

Load a Wikidata item (use 'Q4115189' to start with - it is the sandbox), and print out information from it. You could also try loading some of the other items you looked at in the first task.

I printed out all of the information relating to the sandbox by looping through the item's data using

item.get()

and printing out every key value pair one-by-one, is this good enough or should I format my output more or maybe print out specific information?

is this good enough or should I format my output more or maybe print out specific information?

That's fine for this task - the aim is to get the idea of how the data is stored and how you can retrieve it. But if you want to format it more, please do so. :)

Ok, thanks for the clarification!

@Mike_Peel I am getting an error while configuring Pywikibot. It says:
python: can't open file 'pwb.py': [Errno 2] No such file or directory

Please suggest how should I fix this.

@Mike_Peel I am getting an error while configuring Pywikibot. It says:
python: can't open file 'pwb.py': [Errno 2] No such file or directory

Please suggest how should I fix this.

Are you in the same directory as pywikibot or it is otherwise in the python path? You might need to change into the 'core' subdirectory.

@Mike_Peel I am getting an error while configuring Pywikibot. It says:
python: can't open file 'pwb.py': [Errno 2] No such file or directory

Please suggest how should I fix this.

Are you in the same directory as pywikibot or it is otherwise in the python path? You might need to change into the 'core' subdirectory.

I started with the core-stable directory and later switched to the Pywikibot directory where I tried to configure Pywikibot.

I think you need to be in 'core' to run that command, since that is where 'pwb.py' is located. I think you can then move the user script to a different directory afterwards and still be able to 'import pywikibot' though.

I think you need to be in 'core' to run that command, since that is where 'pwb.py' is located. I think you can then move the user script to a different directory afterwards and still be able to 'import pywikibot' though.

I did that. A new error popped up right now. It asks me to pip install "requests>=2.20.1" and once I did that, it says: The system cannot find the file specified.

Hmm, I'm not sure then. Possibly it's related to T120222 . Could you start a new task for this via https://phabricator.wikimedia.org/maniphest/task/edit/form/1/ - and include the full set of steps and the OS versions you're using, please?

Hmm, I'm not sure then. Possibly it's related to T120222 . Could you start a new task for this via https://phabricator.wikimedia.org/maniphest/task/edit/form/1/ - and include the full set of steps and the OS versions you're using, please?

Sure. I will do that.

I have linked the task below, @Mike_Peel. Please let me know if I need to add more details to it.

https://phabricator.wikimedia.org/T279162

Thanks, I've pinged a few people to see if they can help (it's been a long time since I've used Windows!).

I think I'm up to date with replies now, if I've missed replying to you then please tell me!

Hi @Mike_Peel, this is my Task 2 page, https://www.wikidata.org/wiki/User:Poornima7/Outreachy_2

Kindly suggest any modifications/corrections required.

PS: I'm not able to add any claim to any wikidata page. It crashes with :

Traceback (most recent call last):
File "pwb.py", line 363, in <module>
  if not main():
File "pwb.py", line 355, in main
  run_python_file(filename,
File "pwb.py", line 74, in run_python_file
  exec(compile(source, filename, 'exec', dont_inherit=True),
File ".\scripts\userscripts\test.py", line 15, in <module>
  text.addClaim(claim, summary=u'Adding claim')
AttributeError: 'str' object has no attribute 'addClaim'
CRITICAL: Exiting due to uncaught exception <class 'AttributeError'>

Thank you

Can anyone tell me how to create a new page under my username in wikidata? like user:naznin haque/outreachy_2 ?

Can anyone tell me how to create a new page under my username in wikidata? like user:naznin haque/outreachy_2 ?

Enter that URL into the address bar, go there, and you can then create the page.

@Mike_Peel I tried to configure Pywikibot but I am facing some issues while doing so. Here is the screenshot of the commands I am running.

image.png (997×955 px, 132 KB)

Please guide me on what must I do to fix this.

@Mike_Peel I tried to configure Pywikibot but I am facing some issues while doing so. Here is the screenshot of the commands I am running.

image.png (997×955 px, 132 KB)

Please guide me on what must I do to fix this.

@AnkitaxPriya Would you once try to login through your username only i.e AnkitaxPriya. Meaning not keeping botname bot1 and setting password for it and see if your problem is resolving.

After you run this command: $ python pwb.py generate_user_files , you have to do below step manually by going inside the User Config File which was generated in this step.

this step --> usernames['wikidata']['wikidata'] = u'Your username' into the generated user-config.py file.

After this try to login with $ python pwb.py login command.

Hope it works.

@Mike_Peel I tried to configure Pywikibot but I am facing some issues while doing so. Here is the screenshot of the commands I am running.

image.png (997×955 px, 132 KB)

Please guide me on what must I do to fix this.

@AnkitaxPriya Would you once try to login through your username only i.e AnkitaxPriya. Meaning not keeping botname bot1 and setting password for it and see if your problem is resolving.

After you run this command: $ python pwb.py generate_user_files , you have to do below step manually by going inside the User Config File which was generated in this step.

this step --> usernames['wikidata']['wikidata'] = u'Your username' into the generated user-config.py file.

After this try to login with $ python pwb.py login command.

Hope it works.

@Srishti0gupta so we have to rename the user-config.py.sample file to user-config.py and then manually enter our username there?

@Mike_Peel I tried to configure Pywikibot but I am facing some issues while doing so. Here is the screenshot of the commands I am running.

image.png (997×955 px, 132 KB)

Please guide me on what must I do to fix this.

@AnkitaxPriya Would you once try to login through your username only i.e AnkitaxPriya. Meaning not keeping botname bot1 and setting password for it and see if your problem is resolving.

After you run this command: $ python pwb.py generate_user_files , you have to do below step manually by going inside the User Config File which was generated in this step.

this step --> usernames['wikidata']['wikidata'] = u'Your username' into the generated user-config.py file.

After this try to login with $ python pwb.py login command.

Hope it works.

@Srishti0gupta so we have to rename the user-config.py.sample file to user-config.py and then manually enter our username there?

Hello @AnkitaxPriya ~ When you run the script of "python pwb.py generate_user_files", it automatically creates a file with the name of user-config.py in the same directory. And as suggested in the description above, add the following line to the code in that file:

usernames['wikidata']['wikidata'] = u'Your username'

which in your case, should be:

usernames['wikidata']['wikidata'] = u'AnkitaxPriya'

Hope this helps :)

Thank you so much for the help @Srishti0gupta and @Nizz009. It worked!

Thank you so much for the help @Srishti0gupta and @Nizz009. It worked!

Great @AnkitaxPriya

Hello, I submitted my task 2 code by email.
Thanks,

@Mike_Peel kindly review my Task 2 at https://www.wikidata.org/wiki/User:AnkitaxPriya/Outreachy_2 and do suggest some improvements (if needed).

Regards,
Ankita

@Mike_Peel, @MSGJ, I have completed task 2. I have mailed you.
Though here is the link:
https://www.wikidata.org/wiki/User:Tru2198/Outreachy_2
Await your feedback,
Thank you!

hi @Mike_Peel , @MSGJ
I am getting the following error when i try to run my code.
pywikibot.exceptions.NoPage: Page [[wikidata:Outreachy 1]] doesn't exist.
CRITICAL: Exiting due to uncaught exception <class 'pywikibot.exceptions.NoPage'>

Capture.PNG (465×1 px, 32 KB)

Can you please help me in removing this error.

thank you
pushpanjali kumari

Hello @Pushp24, this may be because of one of the following reasons:

  1. You didn't login to your Wikidata account properly.
  2. You might have entered a different username by mistake.
  3. You are just mentioning the page "Outreachy_1" instead of the full link to your page.

BTW, it'd be more helpful if you could share the Python script you wrote. Not much is clear as of now.

thanks @MSGJ for your help, by the way, due to some technical issue i changed my page. link is following:
https://www.wikidata.org/wiki/User:Pushp24/Outreachy_1

I think I've given feedback to everyone that's tried this task so far - mostly on talk pages or by email. Please let me know if I missed yours!

hello @Mike_Peel , @MSGJ

following is the link for my task2 -
https://www.wikidata.org/wiki/User:Pushp24/Outreachy_2

I am looking forward for your suggestion and improvement.

Regards
Pushpanjali Kumari

hello @Mike_Peel , @MSGJ

following is the link for my task2 -
https://www.wikidata.org/wiki/User:Pushp24/Outreachy_2

I am looking forward for your suggestion and improvement.

Regards
Pushpanjali Kumari

I've replied at https://www.wikidata.org/wiki/User_talk:Pushp24/Outreachy_2

image.png (152×785 px, 19 KB)

Can anyone point me to where I can find documentation for the reload function in line 15 of example.py? It seems to be causing the above error when I execute my code. My terminal defaults to UTF-8, so I could for the sake of the project just comment out the code I'm sure just to take a look at things. However, I've been through documentation for the imported modules and can't find anything, nor could I find the code defined in the script itself. Thanks

image.png (152×785 px, 19 KB)

Can anyone point me to where I can find documentation for the reload function in line 15 of example.py? It seems to be causing the above error when I execute my code. My terminal defaults to UTF-8, so I could for the sake of the project just comment out the code I'm sure just to take a look at things. However, I've been through documentation for the imported modules and can't find anything, nor could I find the code defined in the script itself. Thanks

You actually don't need to do any of this if you're using Python 3. The example code is written in Python 2, where UTF-8 wasn't the standard, it is the standard in Python 3. Reload is not even a function in Python 3 either because it's unnecessary for most cases.

Sources:
https://stackoverflow.com/questions/961162/reloading-module-giving-nameerror-name-reload-is-not-defined
https://stackoverflow.com/questions/14083111/should-i-use-encoding-declaration-in-python-3

image.png (152×785 px, 19 KB)

Can anyone point me to where I can find documentation for the reload function in line 15 of example.py? It seems to be causing the above error when I execute my code. My terminal defaults to UTF-8, so I could for the sake of the project just comment out the code I'm sure just to take a look at things. However, I've been through documentation for the imported modules and can't find anything, nor could I find the code defined in the script itself. Thanks

You actually don't need to do any of this if you're using Python 3. The example code is written in Python 2, where UTF-8 wasn't the standard, it is the standard in Python 3. Reload is not even a function in Python 3 either because it's unnecessary for most cases.

Sources:
https://stackoverflow.com/questions/961162/reloading-module-giving-nameerror-name-reload-is-not-defined
https://stackoverflow.com/questions/14083111/should-i-use-encoding-declaration-in-python-3

That makes so much sense. Thanks for helping me out, Amine! I'm still developing my syntax so I can better ask questions online for issues that I encounter, so it's incredibly helpful that you included the source pages so I can learn from this!

Hello @Mike_Peel and other applicants. I have had issues with installing pywikibot had similar problems @AnkitaxPriya on my windows changed to linux and i am having issues
please how are we to create a script that will link to wikidata?

Hello @Mike_Peel and other applicants. I have had issues with installing pywikibot had similar problems @AnkitaxPriya on my windows changed to linux and i am having issues
please how are we to create a script that will link to wikidata?

Hey @Fonyuy237 what problems are you facing? Can you elaborate please.

Hello @Mike_Peel and other applicants. I have had issues with installing pywikibot had similar problems @AnkitaxPriya on my windows changed to linux and i am having issues
please how are we to create a script that will link to wikidata?

Hello @Fonyuy237 I was getting such an error because of the Python version I had on my local machine. The script didn't work on Python 3.8. Try installing the latest version of Python (preferably Python 3.9) and then run the scripts.

Hello @Mike_Peel and other applicants. I have had issues with installing pywikibot had similar problems @AnkitaxPriya on my windows changed to linux and i am having issues
please how are we to create a script that will link to wikidata?

Please show the error messages/problems you are getting and we can help - or use PAWS if needed ( https://www.mediawiki.org/wiki/Manual:Pywikibot/PAWS )

Hello @Fonyuy237! What's your OS configuration? I faced a problem with the login step on my linux system (pop-os) too, and after some research, the issue: https://phabricator.wikimedia.org/T212723 helped me solve the problem.

Greetings again everyone.
@Mike_Peel I am happy with my code for this second task: https://www.wikidata.org/wiki/User:Tambe_Tabitha/Outreachy_2.
Please can you review it?.

Hello @Mike_Peel and other applicants. I have had issues with installing pywikibot had similar problems @AnkitaxPriya on my windows changed to linux and i am having issues
please how are we to create a script that will link to wikidata?

Hey, if you copy the code from here https://www.wikidata.org/wiki/User:Mike_Peel/Outreachy_2 and paste it to a file that you have created in core/scripts/userscripts , you can then run the file (script) from core using

python pwb.py name_of_your_script.py

When you do this, then you would have successfully connected to wikidata and printed the text from the wiki page that you created earlier. if you are not very clear on which particular line of code connects to wikidata (since the enwiki variable was instead initialized to wikipedia), know that it is the line where enwiki_repo is formed. That is,

enwiki_repo = enwiki.data_repository()

If you need guidance on any of the other sub-tasks, check out the mentor's example code here : https://bitbucket.org/mikepeel/wikicode/src/master/example.py , I found the functions in it very useful.

For any other specific issue, you can simply tag me here and I'll have a look at it.

Best

Hello @Mike_Peel and @MSGJ Here is the link to my page.
https://www.wikidata.org/wiki/User:DaneshwariK/Outreachy_2
Please review and provide feedback regarding this. Thanks!

Hello @Fonyuy237 , maybe you can work on PAWS since python, bash, and jupyter-notebook support is provided there itself. Only configuration required is for pywikibot and user-config.py file. If you need any help for setup on PAWS, maybe I can help you.

Hello @Mike_Peel and @MSGJ,

I would like to submit my contribution for Task 2 -> Outreachy_2.
Any feedback would be highly appreciated.

Thank you,
Nina

@Mike_Peel, @MSGJ,

I am getting the error :
pywikibot.exceptions.NoUsername: Username "Tru2198" does not have read permissions on wikipedia:en
.The supplied credentials could not be authenticated.

CRITICAL: Exiting due to uncaught exception <class 'pywikibot.exceptions.NoUsername'>

Here is the image

pywikibot error.PNG (712×1 px, 84 KB)

I have

usernames['wikipedia']['en'] = 'Tru2198'

usernames['wikidata']['wikidata']= u'Tru2198', set in user_config.py

Also, I get:
Failed OAuth authentication for wikipedia:test: The authorization headers in your request are for a user that does not exist here

when I try to add a date property in the wikidata

Kindly help!

usernames['wikipedia']['en'] = 'Tru2198'

Try changing that to u'Tru2198' - but in general it looks OK otherwise... Make sure you're entering your password correctly?

@Mike_Peel, @MSGJ,

I am getting the error :
pywikibot.exceptions.NoUsername: Username "Tru2198" does not have read permissions on wikipedia:en
.The supplied credentials could not be authenticated.

CRITICAL: Exiting due to uncaught exception <class 'pywikibot.exceptions.NoUsername'>

Here is the image

pywikibot error.PNG (712×1 px, 84 KB)

I have

usernames['wikipedia']['en'] = 'Tru2198'

usernames['wikidata']['wikidata']= u'Tru2198', set in user_config.py

Also, I get:
Failed OAuth authentication for wikipedia:test: The authorization headers in your request are for a user that does not exist here

when I try to add a date property in the wikidata

Kindly help!

If your browser had been inactive for a while, try logging out of wikipedia, refreshing the page and logging in again. You could also, restart your terminal.

Does it work?
Does the error change?

@Mike_Peel, @MSGJ,

I am getting the error :
pywikibot.exceptions.NoUsername: Username "Tru2198" does not have read permissions on wikipedia:en
.The supplied credentials could not be authenticated.

CRITICAL: Exiting due to uncaught exception <class 'pywikibot.exceptions.NoUsername'>

Here is the image

pywikibot error.PNG (712×1 px, 84 KB)

I have

usernames['wikipedia']['en'] = 'Tru2198'

usernames['wikidata']['wikidata']= u'Tru2198', set in user_config.py

Also, I get:
Failed OAuth authentication for wikipedia:test: The authorization headers in your request are for a user that does not exist here

when I try to add a date property in the wikidata

Kindly help!

If your browser had been inactive for a while, try logging out of wikipedia, refreshing the page and logging in again. You could also, restart your terminal.

Does it work?
Does the error change?

No. It does not. I restarted my entire terminal. Now I am planning to reinstall user_config.py altogether.

Thank you

usernames['wikipedia']['en'] = 'Tru2198'

Try changing that to u'Tru2198' - but in general it looks OK otherwise... Make sure you're entering your password correctly?

Yes, my password is correct.
I am running my code on paws as jupyter notebook and have also checked on VS code by implementing the script as .py file where the error is:

No user is logged in on wikidata:wikidata

Yes, my password is correct.
I am running my code on paws as jupyter notebook and have also checked on VS code by implementing the script as .py file where the error is:

No user is logged in on wikidata:wikidata

run

python pwb.py login

follow the steps, try again and us what happens

No. It does not. I restarted my entire terminal. Now I am planning to reinstall user_config.py altogether.

Thank you

Oh this would probably work. I reinstalled everything when I had a similar error, at the start. I was just considering that there should be a less crude way to do it.
In any case, keep us updated.

No. It does not. I restarted my entire terminal. Now I am planning to reinstall user_config.py altogether.

Thank you

Oh this would probably work. I reinstalled everything when I had a similar error, at the start. I was just considering that there should be a less crude way to do it.
In any case, keep us updated.

Thank you very much for the time and patience. Yes, I will definitely keep you in the loop

No. It does not. I restarted my entire terminal. Now I am planning to reinstall user_config.py altogether.

Thank you

Oh this would probably work. I reinstalled everything when I had a similar error, at the start. I was just considering that there should be a less crude way to do it.
In any case, keep us updated.

Thank you very much for the time and patience. Yes, I will definitely keep you in the loop

At this point, I reinstalled the pywikibot and I can successfully log in to en:Wikipedia.

Thank you for the time and patience.

That's weird - you shouldn't need to access test.wikipedia.org... Perhaps open that as a separate ticket, and I can tag the relevant people to see if they can find the problem?

That's weird - you shouldn't need to access test.wikipedia.org... Perhaps open that as a separate ticket, and I can tag the relevant people to see if they can find the problem?

I caught my error. It was the repository. I made a successful update on the publication date. I apologize for the haste.

Hello @Mike_Peel and other applicants. I have had issues with installing pywikibot had similar problems @AnkitaxPriya on my windows changed to linux and i am having issues
please how are we to create a script that will link to wikidata?

Hey, if you copy the code from here https://www.wikidata.org/wiki/User:Mike_Peel/Outreachy_2 and paste it to a file that you have created in core/scripts/userscripts , you can then run the file (script) from core using

python pwb.py name_of_your_script.py

When you do this, then you would have successfully connected to wikidata and printed the text from the wiki page that you created earlier. if you are not very clear on which particular line of code connects to wikidata (since the enwiki variable was instead initialized to wikipedia), know that it is the line where enwiki_repo is formed. That is,

enwiki_repo = enwiki.data_repository()

If you need guidance on any of the other sub-tasks, check out the mentor's example code here : https://bitbucket.org/mikepeel/wikicode/src/master/example.py , I found the functions in it very useful.

For any other specific issue, you can simply tag me here and I'll have a look at it.

Best

Thanks alot

Hello @Mike_Peel and other applicants. I have had issues with installing pywikibot had similar problems @AnkitaxPriya on my windows changed to linux and i am having issues
please how are we to create a script that will link to wikidata?

Hey, if you copy the code from here https://www.wikidata.org/wiki/User:Mike_Peel/Outreachy_2 and paste it to a file that you have created in core/scripts/userscripts , you can then run the file (script) from core using

python pwb.py name_of_your_script.py

When you do this, then you would have successfully connected to wikidata and printed the text from the wiki page that you created earlier. if you are not very clear on which particular line of code connects to wikidata (since the enwiki variable was instead initialized to wikipedia), know that it is the line where enwiki_repo is formed. That is,

enwiki_repo = enwiki.data_repository()

If you need guidance on any of the other sub-tasks, check out the mentor's example code here : https://bitbucket.org/mikepeel/wikicode/src/master/example.py , I found the functions in it very useful.

For any other specific issue, you can simply tag me here and I'll have a look at it.

Best

Thanks alot

Please when i try to save property i receive an error which says
"API error failed-save: The save has failed" any help on that

Hello, this is the link to my outreach page 2 please check it out and comment
https://www.wikidata.org/wiki/User:Fonyuy237/Outreachy_2