Change Details

As discussed in last week's team meeting, my task for this week is to write the program that will take data from Toolhub's database and insert it into ours. The initial data dump should be relatively simple: GET request to `api/tools` and pass every result to an `add_new_tool()` function. The workflow for handling updates is a little more complicated. After a lengthy session with the test server API I've concluded that we'd want to make periodic requests to the `/api/auditlogs` and run our operations on the results. I've outlined the functions that I'm planning to write and the path the data will take here: [[ https://public.paws.wmcloud.org/User:NicoleLBee/Toolhunt_Making_Updates.ipynb#A-pathway-for-checking-updates: | A pathway for making updates to the Toolhunt db ]]. You will probably notice that that is just a section at the bottom of a much longer notebook: [[ https://public.paws.wmcloud.org/User:NicoleLBee/Toolhunt_Making_Updates.ipynb |Data Fetching & Insertion: UPDATES Toolhub -> Toolhunt ]]; the full notebook includes a rather lengthy investigation of the differences between the `/auditlogs` and `/recent` endpoints and the structure of various responses. It isn't really necessary to read all of this, but the additional information would clarify some of my decisions. Comments/questions welcomed.

As discussed in last week's team meeting, my task for this week is to write the program that will take data from Toolhub's database and insert it into ours. We'd previously discussed a more complicated approach involving diffs, but @bd808's comment below has gotten me to reconsider a simpler alternative. Let's call them the "hard way" and the "easy way." **The Hard Way**: Initial data dump via GET request to `/api/tools` and subsequent updates made with requests to `/api/auditlogs` followed by more requests and a lot of additional checks. If we went this route, I've outlined the functions that I'd plan to write and the path the data will take here: [[ https://public.paws.wmcloud.org/User:NicoleLBee/Toolhunt_Making_Updates.ipynb#A-pathway-for-checking-updates: | A pathway for making updates to the Toolhunt db ]]. You will probably notice that that is just a section at the bottom of a much longer notebook: [[ https://public.paws.wmcloud.org/User:NicoleLBee/Toolhunt_Making_Updates.ipynb |Data Fetching & Insertion: UPDATES Toolhub -> Toolhunt ]]; the full notebook includes a rather lengthy investigation of the differences between the `/auditlogs` and `/recent` endpoints and the structure of various responses. It isn't really necessary to read all of this, but the additional information would clarify some of my decisions. **The Easy Way**: Pull all tool data on every update. Details and functions here: [[ https://public.paws.wmcloud.org/User:NicoleLBee/Toolhub_to_Toolhunt_Simplified.ipynb | Toolhub to Toolhunt: Simplified ]] Comments/questions welcome.

As discussed in last week's team meeting, my task for this week is to write the program that will take data from Toolhub's database and insert it into ours. The initial data dump should be relatively simple: GET request to `api/tools` and pass every result to an `add_new_tool()` function. We'd previously discussed a more complicated approach involving diffs, but @bd808's comment below has gotten me to reconsider a simpler alternative. Let's call them the "hard way" and the "easy way." The workflow for handling updates is a little more complicated. After a lengthy session with the test server API I've concluded that we'd want to make periodic requests to the `/api/auditlogs` and run our operations on the results. I've outlined the functions that I'm planning to write and the path the data will take here: [[ https://public.paws.wmcloud.org/User:NicoleLBee/Toolhunt_Making_Updates.ipynb#A-pathway-for-checking-updates: | A pathway for making updates to the Toolhunt db ]]**The Hard Way**: Initial data dump via GET request to `/api/tools` and subsequent updates made with requests to `/api/auditlogs` followed by more requests and a lot of additional checks. You will probably notice that that is just a section at the bottom of a much longer notebook: [[ https://public.paws.wmcloud.org/User:NicoleLBee/Toolhunt_Making_Updates.ipynb |Data Fetching & Insertion: UPDATES Toolhub -> Toolhunt ]];If we went this route, the full notebook includes a rather lengthy investigation of the differences between the `/auditlogs` and `/recent` endpoints and the structure of various responses. It isn't really necessary to read all of this, but the additional information would clarify some of my decisionsI've outlined the functions that I'd plan to write and the path the data will take here: [[ https://public.paws.wmcloud.org/User:NicoleLBee/Toolhunt_Making_Updates.ipynb#A-pathway-for-checking-updates: | A pathway for making updates to the Toolhunt db ]]. Comments/questions welcomedYou will probably notice that that is just a section at the bottom of a much longer notebook: [[ https://public.paws.wmcloud.org/User:NicoleLBee/Toolhunt_Making_Updates.ipynb |Data Fetching & Insertion: UPDATES Toolhub -> Toolhunt ]]; the full notebook includes a rather lengthy investigation of the differences between the `/auditlogs` and `/recent` endpoints and the structure of various responses. It isn't really necessary to read all of this, but the additional information would clarify some of my decisions. **The Easy Way**: Pull all tool data on every update. Details and functions here: [[ https://public.paws.wmcloud.org/User:NicoleLBee/Toolhub_to_Toolhunt_Simplified.ipynb | Toolhub to Toolhunt: Simplified ]] Comments/questions welcome.