Refactor the AQS 2.0 Geo Analytics code as per the code review comments and standard practices.
Description
Description
Event Timeline
Comment Actions
@SGupta-WMF you mentioned in stand up you would park this one for further work later. I'll move it to the pause column.
Comment Actions
- @BPirkle which editors endpoint are you referring to ? A specific one or all of them?
- Also I will need to check which ever one you highlight locally correct ?
Comment Actions
Just editors/by-country.
Per our spreadsheet, the "editors" service only implements the one Cassandra-backed "editors" endpoint. The other "editors" endpoints will be provided by one or more other services that we still need to implement.
Comment Actions
Test status : QA FAIL
Scenario 1:
Testing the endpoint with an invalid project
import requests
prod_url = 'https://wikimedia.org/api/rest_v1/metrics/editors/by-country/not.valid/100..-edits/2020/12'
header = {"accept": "application/json",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"}
response = requests.get(prod_url, headers=header)
print(response.status_code)
print(response.json())in prod this returns : 404 status code
and the json response:
{'type': 'https://mediawiki.org/wiki/HyperSwitch/errors/not_found', 'title': 'Not found.', 'method': 'get', 'detail': 'The date(s) you used are valid, but we either do not have data for those date(s), or the project you asked for is not loaded yet. Please check https://wikimedia.org/api/rest_v1/?doc for more information.', 'uri': '/analytics.wikimedia.org/v1/editors/by-country/not.valid/100..-edits/2020/12'}Local:
this returns status 400
json response:
Traceback (most recent call last):
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/venv/lib/python3.9/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/editors/test_local_editors_by_country.py", line 14, in <module>
print(response.json())
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/venv/lib/python3.9/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)Reported in T327139
Scenario 2:
Testing the endpoint with an invalid activity level
Prod:
import requests
prod_url = 'https://wikimedia.org/api/rest_v1/metrics/editors/by-country/en.wikipedia/50..99-edits/2020/12'
header = {"accept": "application/json",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"}
response = requests.get(prod_url, headers=header)
print(response.status_code)
print(response.json())This returns 400 status code and the json:
{'type': 'https://mediawiki.org/wiki/HyperSwitch/errors/bad_request', 'title': 'Invalid parameters', 'method': 'get', 'detail': "data.params['activity-level'] should be equal to one of the allowed values: [5..99-edits, 100..-edits]", 'uri': '/wikimedia.org/v1/metrics/editors/by-country/en.wikipedia/50..99-edits/2020/12'}Local:
This returns 400 status code and the json error:
Traceback (most recent call last):
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/venv/lib/python3.9/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/editors/test_local_editors_by_country.py", line 14, in <module>
print(response.json())
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/venv/lib/python3.9/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)Reported in T327140
Scenario 3:
Testing the endpoint with years with no editors by country data
Hint: From 2017 and earlier years there is not data for editors by country. Data began in the year 2018
Prod:
import requests
prod_url = 'https://wikimedia.org/api/rest_v1/metrics/editors/by-country/en.wikipedia/en.wikipedia/5..99-edits/2000/06'
header = {"accept": "application/json",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"}
response = requests.get(prod_url, headers=header)
print(response.status_code)
print(response.json())This returns status code 404 and json response:
{'type': 'https://mediawiki.org/wiki/HyperSwitch/errors/not_found#route', 'title': 'Not found.', 'method': 'get', 'uri': '/wikimedia.org/v1/metrics/editors/by-country/en.wikipedia/en.wikipedia/5..99-edits/2000/06'}Local:
This returns wrong status code 400 and wrong jsonn response:
Traceback (most recent call last):
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/venv/lib/python3.9/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/editors/test_local_editors_by_country.py", line 14, in <module>
print(response.json())
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/venv/lib/python3.9/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)Reported in T327141
Scenario 4:
Testing the endpoint with years with invalid month
Prod:
import requests
prod_url = 'https://wikimedia.org/api/rest_v1/metrics/editors/by-country/en.wikipedia/100..-edits/2019/13'
header = {"accept": "application/json",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"}
response = requests.get(prod_url, headers=header)
print(response.status_code)
print(response.json())This returns 400 status code and json reponse:
{'type': 'https://mediawiki.org/wiki/HyperSwitch/errors/invalid_request', 'method': 'get', 'detail': ['Given year/month is invalid date'], 'uri': '/analytics.wikimedia.org/v1/editors/by-country/en.wikipedia/100..-edits/2019/13'}Local:
This returns 400 and the wrong json:
Traceback (most recent call last):
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/venv/lib/python3.9/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/editors/test_local_editors_by_country.py", line 14, in <module>
print(response.json())
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/venv/lib/python3.9/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)Reported in T327142
Scenario 5:
Testing the endpoint with years with invalid year
Prod:
import requests
prod_url = 'https://wikimedia.org/api/rest_v1/metrics/editors/by-country/en.wikipedia/100..-edits/wrong/12'
header = {"accept": "application/json",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"}
response = requests.get(prod_url, headers=header)
print(response.status_code)
print(response.json())This gives a status of 400 and the json response below:
{'type': 'https://mediawiki.org/wiki/HyperSwitch/errors/invalid_request', 'method': 'get', 'detail': ['Given year/month is invalid date'], 'uri': '/analytics.wikimedia.org/v1/editors/by-country/en.wikipedia/100..-edits/wrong/12'}Local :
This gives 400 status and the json exception:
Traceback (most recent call last):
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/venv/lib/python3.9/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/editors/test_local_editors_by_country.py", line 14, in <module>
print(response.json())
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/venv/lib/python3.9/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)Reported in T327144
Scenario 6:
Testing a positive scenario with valid, project, activity level, year and month
prod:
import requests
prod_url = 'https://wikimedia.org/api/rest_v1/metrics/editors/by-country/en.wikipedia/100..-edits/2019/11'
header = {"accept": "application/json",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"}
response = requests.get(prod_url, headers=header)
print(response.status_code)
print(response.json())This returns 200 status and valid json:
Local:
This returns a 400 status with json error:
Traceback (most recent call last):
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/venv/lib/python3.9/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/editors/test_local_editors_by_country.py", line 14, in <module>
print(response.json())
File "/Users/emekachukwukere/PycharmProjects/pythonProject1/venv/lib/python3.9/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)Reported in T327143
Scenario 7:
Testing the content-type header
prod:
The content-type in the header returns :
'content-type': 'application/json; charset=utf-8'
local:
The content-type in the header returns :
'Content-Type': 'application/json'
Reported in T327145
For the other possitive scenarios, the tests are passing with the right status codes, data, data types, schemas and the values been returned. Defects found have reported in the aformentioned phab tickets. I will put this ticket back in progress
Comment Actions
Based on discussion with @BPirkle a health check on the editors_by_endpoint should suffice to make a pass on this. I will be moving this task to the done column.
Let me know if anyone has any objections