Page MenuHomePhabricator
Create Task
Maniphest T294125

Duplicate URLs do not provide user visible reason when rejected
Closed, ResolvedPublicBUG REPORT
Actions

Description

The Url registration parts of the Toolhub frontend were the earliest API calls we added and do not follow the patterns that we have built up over the subsequent year of work.

Things to fix in vue/src/store/user.js:

  • registerUrl should call this._vm.$notify.success(...) to signal the user when a URL registration succeeded.
  • registerUrl should use getFailurePayload and looping over the returned error collection to notify the user of errors in registration.
  • unregisterUrl should call this._vm.$notify.success(...) to signal the user when a URL removal succeeded.
  • unregisterUrl should getFailurePayload and looping over the returned error collection to notify the user of errors in registration.
  • user.spec.js tests should be updated to validate the new behaviors.

(optional) move crawler URL related functions and state out of vue/src/store/user.js into vue/src/store/crawler.js to match the backend API organization of the endpoints

Original bug report:

  1. Tried to add https://magnustools.toolforge.org/toolinfo_buggregator.json on https://toolhub.wikimedia.org/add-or-remove-tools?tab=urls but failed with "bad request". Tried http instead of https and it worked.
  1. I now see the URL in a list of one. Is that added now? Do I need to do something? Will it auto-update? I have reloaded the page but eg the "drawshield" tool that is in the toolinfo.json is still not found.

Details

SubjectRepoBranchLines +/-
toolhub: Bump container version to 2021-12-23-121200-productionoperations/deployment-chartsmaster+1 -1
Bug: Fix Inadequate notification bug in url registration actionswikimedia/toolhubmain+315 -215
Customize query in gerrit

Event Timeline

Magnus created this task.Oct 22 2021, 2:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 22 2021, 2:11 PM
Magnus added a comment.Oct 22 2021, 2:12 PM

Ah it looks like the crawler runs once every hour: https://toolhub.wikimedia.org/crawler-history

That could be stated on the "add toolinfo.json" page somewhere.

bd808 renamed this task from "Submit a toolinfo.json URL" issues to Duplicate URLs do not provide user visible notification when submitted.Oct 22 2021, 9:22 PM
bd808 renamed this task from Duplicate URLs do not provide user visible notification when submitted to Duplicate URLs do not provide user visible reason when rejected.
bd808 subscribed.Oct 22 2021, 9:27 PM

I had already added https://magnustools.toolforge.org/toolinfo_buggregator.json via the "Toolhub" user's list of URLs which explains why it was rejected at least. I noticed your http version and have since removed the "Toolhub" owned URL and updated the one you own to use https.

The URL submission screen in general needs some work. It seems to not give a "URL added" notice on success, and poor error notification on failure.

bd808 added a comment.Oct 22 2021, 9:30 PM

It would be nice to also revisit the UI choice of only showing each user the URLs that they themselves have submitted with no way through the UI to see all of the registered URLs. All URLs are shown in the crawler reports, but that is pretty obscure.

Magnus added a comment.Oct 27 2021, 12:05 PM

So it does not look like the crawler used the JSON I provided. For example, "drawshield" is clearly in https://magnustools.toolforge.org/toolinfo_buggregator.json but it does not show up in Toolhub search.

bd808 added a comment.Oct 27 2021, 6:19 PM
In T294125#7461200, @Magnus wrote:

So it does not look like the crawler used the JSON I provided. For example, "drawshield" is clearly in https://magnustools.toolforge.org/toolinfo_buggregator.json but it does not show up in Toolhub search.

If you check the crawler run history for https://magnustools.toolforge.org/toolinfo_buggregator.json you can see that drawshield and many other records are being rejected for not conforming to the toolinfo/1.0 schema (Hay's Directory schema) required fields. From the 2021-10-27T18:05 run:

ERROR: Toolinfo record mm_cradle missing description.
ERROR: Toolinfo record mm_erwin85 missing description.
ERROR: Toolinfo record mm_file_candidates missing description.
ERROR: Toolinfo record mm_wd4wp missing description.
ERROR: Toolinfo record mm_glamorgan missing description.
ERROR: Toolinfo record mm_botstatus missing description.
ERROR: Toolinfo record mm_dynamic missing description.
ERROR: Toolinfo mm_bong already seen at https://magnustools.toolforge.org/toolinfo.json
ERROR: Toolinfo mm_treeviews already seen at https://magnustools.toolforge.org/toolinfo.json
ERROR: Toolinfo record mm_authority_control missing description.
ERROR: Toolinfo record mm_autolinker_demo missing description.
ERROR: Toolinfo record mm_autotools missing description.
ERROR: Toolinfo record mm_can_i_haz_files missing description.
ERROR: Toolinfo record mm_commonsapi_magnustools missing description.
ERROR: Toolinfo record mm_csstest missing description.
ERROR: Toolinfo record mm_flommons_include missing description.
ERROR: Toolinfo record mm_flommons missing description.
ERROR: Toolinfo record mm_js_toolinfo missing description.
ERROR: Toolinfo record mm_ke missing description.
ERROR: Toolinfo record mm_last_tools missing description.
ERROR: Toolinfo record mm_legacy missing description.
ERROR: Toolinfo record mm_logger missing description.
ERROR: Toolinfo record mm_main missing description.
ERROR: Toolinfo record mm_mashwiki missing description.
ERROR: Toolinfo record mm_metronom missing description.
ERROR: Toolinfo record mm_missing_webtools missing description.
ERROR: Toolinfo record mm_multistatus missing description.
ERROR: Toolinfo record mm_oauth_uploader missing description.
ERROR: Toolinfo record mm_pipeline missing description.
ERROR: Toolinfo record mm_prep_art missing description.
ERROR: Toolinfo record mm_random_image_commons_subcat missing description.
ERROR: Toolinfo record mm_spacer missing description.
ERROR: Toolinfo record mm_toolstatus missing description.
ERROR: Toolinfo record mm_uploader_distribution missing description.
ERROR: Toolinfo record mm_wdsearch missing description.
ERROR: Toolinfo record mm_wd_useful missing description.
ERROR: Toolinfo record mm_whatelse missing description.
ERROR: Toolinfo record mm_mobilegame missing description.
ERROR: Toolinfo record mm_visualtool missing description.
ERROR: Toolinfo record mm_importer missing description.
ERROR: Toolinfo record mm_geneawiki missing description.
ERROR: Toolinfo record mm_geneawiki2 missing description.
ERROR: Toolinfo record mm_geneawiki3 missing description.
ERROR: Toolinfo record mm_geneawiki4 missing description.
ERROR: Toolinfo record mm_geneawiki5 missing description.
ERROR: Toolinfo record mm_referee missing description.
ERROR: Toolinfo record mm_sourcemd missing description.
ERROR: Toolinfo record mm_topicmatcher missing description.
ERROR: Toolinfo record mm_add_name_labels missing description.
ERROR: Toolinfo record mm_barcode2wikidata missing description.
ERROR: Toolinfo record mm_best_cats missing description.
ERROR: Toolinfo record mm_bioacoustica missing description.
ERROR: Toolinfo record mm_canned_queries missing description.
ERROR: Toolinfo record mm_canned_queries_wikidata-todo missing description.
ERROR: Toolinfo record mm_commons2json missing description.
ERROR: Toolinfo record mm_compare_lists missing description.
ERROR: Toolinfo record mm_coords2json missing description.
ERROR: Toolinfo record mm_country_stats missing description.
ERROR: Toolinfo record mm_creator2 missing description.
ERROR: Toolinfo record mm_last_changed_items missing description.
ERROR: Toolinfo record mm_list2wd missing description.
ERROR: Toolinfo record mm_merge_candidates_by_shared_identifier missing description.
ERROR: Toolinfo record mm_missing_coordinates_all missing description.
ERROR: Toolinfo record mm_missing_images_test missing description.
ERROR: Toolinfo record mm_mobile_upload_demo missing description.
ERROR: Toolinfo record mm_needleman missing description.
ERROR: Toolinfo record mm_odnb_redirect missing description.
ERROR: Toolinfo record mm_oneoff missing description.
ERROR: Toolinfo record mm_papertrail missing description.
ERROR: Toolinfo record mm_pasleim_projectmerge_game_api missing description.
ERROR: Toolinfo record mm_pasleim_projectmerge_game missing description.
ERROR: Toolinfo record mm_pasleim_projectmerge missing description.
ERROR: Toolinfo record mm_personendaten_wikidata-todo missing description.
ERROR: Toolinfo mm_quick_statements already seen at http://wikidata-todo.toolforge.org/toolinfo.json
ERROR: Toolinfo record mm_random_item_without_instance missing description.
ERROR: Toolinfo record mm_relabel missing description.
ERROR: Toolinfo record mm_relations missing description.
ERROR: Toolinfo record mm_picturethis missing description.
ERROR: Toolinfo record mm_drawshield missing description.
ERROR: Toolinfo record mm_distributedgame missing description.
Magnus added a comment.Oct 28 2021, 12:25 PM

Thanks, I added descriptions and toolhub now picked it up.

Do I understand correctly that the entire JSON file is rejected if even one tool does not have a description? That would be a bit excessive.

bd808 added a comment.Oct 28 2021, 11:44 PM
In T294125#7465060, @Magnus wrote:

Thanks, I added descriptions and toolhub now picked it up.

Do I understand correctly that the entire JSON file is rejected if even one tool does not have a description? That would be a bit excessive.

Only the invalid records are omitted from storage, but the entire URL is marked as having errors during the run in the database so that it is easier to find these problems. The crawler actually tries pretty hard to make things valid so that we capture as many records as we can at this point.

• sdkim triaged this task as High priority.Nov 1 2021, 5:23 PM
bd808 moved this task from Backlog to Groomed/Ready on the Toolhub board.Nov 5 2021, 3:14 PM
bd808 updated the task description. (Show Details)Nov 11 2021, 12:16 AM
bd808 updated the task description. (Show Details)Nov 15 2021, 6:24 PM
Raymond_Ndibe claimed this task.Nov 16 2021, 1:25 PM

assigned this to myself

bd808 changed the task status from Open to In Progress.Nov 20 2021, 12:07 AM
bd808 moved this task from Groomed/Ready to In Progress on the Toolhub board.
gerritbot added a comment.Nov 20 2021, 12:33 AM

Change 740286 had a related patch set uploaded (by Raymond Ndibe; author: Raymond Ndibe):

[wikimedia/toolhub@main] Bug: Fix Inadequate notification bug in url registration actions

https://gerrit.wikimedia.org/r/740286

gerritbot added a project: Patch-For-Review.Nov 20 2021, 12:33 AM
gerritbot added a comment.Dec 4 2021, 12:58 AM

Change 740286 merged by jenkins-bot:

[wikimedia/toolhub@main] Bug: Fix Inadequate notification bug in url registration actions

https://gerrit.wikimedia.org/r/740286

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2021, 1:10 AM
bd808 closed this task as Resolved.Dec 6 2021, 9:51 PM
bd808 moved this task from In Progress to Review on the Toolhub board.
gerritbot added a comment.Dec 21 2021, 5:19 PM

Change 749220 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/deployment-charts@master] toolhub: Bump container version to 2021-12-20-122341-production

https://gerrit.wikimedia.org/r/749220

gerritbot added a project: Patch-For-Review.Dec 21 2021, 5:19 PM
gerritbot added a comment.Jan 5 2022, 5:23 PM

Change 749220 merged by jenkins-bot:

[operations/deployment-charts@master] toolhub: Bump container version to 2021-12-23-121200-production

https://gerrit.wikimedia.org/r/749220

Maintenance_bot removed a project: Patch-For-Review.Jan 5 2022, 6:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits