Greetings to you, @Arcstur and @Ederporto.

I have been working through the project and paying attention to Wishlist #3.

To gain a clearer insight into the issue, I created a small interactive prototype that recreates duplicate reference detection with normalised identifiers (URL, DOI, ISBN) and a basic reuse flow.

🔗 Live Demo: https://wish3-duplicate-reference-prototype.vercel.app/
🔗 Repository: https://github.com/Jikugodwill/wish3-duplicate-reference-prototype

The prototype is concentrated on:

• Normalizing identifiers
• Finding duplicates on the fly.
• Recommending the use of old references.

I took the approach as a workflow point of view to see how this would fit into the Visual Editor experience.

Regarding the microtasks:

• JavaScript task: T418285 completed.
• I am still working on T418286 (Python task), which I will submit soon.

It will be my pleasure to know how well this direction fits the planned implementation.

Thanks!

Hi @Arcstur,

Thank you for the detailed discussion so far — it has been very helpful.

I’ve been going through the ideas around Wishlist #3, especially about identifier normalization and duplicate detection. Based on this, I am trying to understand how a simple approach could work as a starting point.

For example, I am thinking of:

• Normalizing identifiers (like cleaning URLs, DOIs, ISBNs)
• Comparing within the same type first
• Then exploring how cross-type matching might be handled later

Would this be a good direction to begin with for a basic implementation?

I’d really appreciate any guidance on how to approach this as a beginner.

Thank you!

In T418284#11788072, @Adwivedii wrote:

One thing worth thinking about is that each type of identifier needs its own cleaning before comparing. For example, https://doi.org/10.1100/abc and 10.1100/abc are the same DOI, and 978-3-16-148410-0 and 9783161484100 are the same ISBN, just written differently. So the checker needs to clean and normalize each type separately before deciding whether two references are duplicates or not.

A trickier situation is when different identifier types refer to the same source, such as a URL and a DOI for the same article. Simple cleaning and normalization will not detect this due to different formats. Handling this would need some external lookup to see what each identifier actually points to. I'm curious whether this (cross-type matching) is in scope for first version or whether same-type matching is the right starting point. Would be better to confirm with the mentors @Arcstur @Ederporto .

Another edge case I am thinking about is when some references on a Wikipedia page are not added through the Visual Editor. They may be written directly in the wikitext or added through scripts. If the checker only analyzes references processed by Visual Editor, it might miss duplicates. Should the checker rely only on what the Visual Editor detects, or should it analyze the full page wikitext? Would love to know how you are thinking about this @Arcstur @Ederporto.

Also, once a duplicate is detected, what should happen next? Should it work like Visual Editor's existing named reference feature, where it shows the user the existing reference and asks if they want to reuse it?

Hi @Arcstur, @Ederporto,

I have submitted Task 1 & 2, (T418285 & T418286) and look forward to your feedback.

In the meantime, I am exploring more of Wishlist #3 and Wishlist #8. It would be great to have your guidance on my previous comment when you get a chance.

Thank you for your time!

In T418284#11788783, @Jikugodwill wrote:

Greetings to you, @Arcstur and @Ederporto.

I have been working through the project and paying attention to Wishlist #3.

To gain a clearer insight into the issue, I created a small interactive prototype that recreates duplicate reference detection with normalised identifiers (URL, DOI, ISBN) and a basic reuse flow.

🔗 Live Demo: https://wish3-duplicate-reference-prototype.vercel.app/
🔗 Repository: https://github.com/Jikugodwill/wish3-duplicate-reference-prototype

The prototype is concentrated on:

• Normalizing identifiers
• Finding duplicates on the fly.
• Recommending the use of old references.

I took the approach as a workflow point of view to see how this would fit into the Visual Editor experience.

Regarding the microtasks:

• JavaScript task: T418285 completed.
• I am still working on T418286 (Python task), which I will submit soon.

It will be my pleasure to know how well this direction fits the planned implementation.

Thanks!

Quick update:

I’ve now completed both microtasks:

• T418285 (JavaScript)
• T418286 (Python)

🔗 Microtasks repository: https://github.com/Jikugodwill/microtask-python

Thanks!

Hi @mentor,

I implemented a simple prototype for duplicate reference detection based on Wishlist #8.

Approach

• Normalized identifiers (DOI/URL) by removing prefixes like https://doi.org/
• Compared references within the same type (starting simple)
• Stored references using browser localStorage
• Checked for duplicates before adding

Prototype

• GitHub:https://github.com/KalyaniRayala/Duplicate-Checker
• Live Demo: https://vercel.com/kalyanirayalas-projects/duplicate-checker

What I learned

• Importance of normalization before comparison
• Handling edge cases like DOI vs URL formats
• Basic duplicate detection logic

Next Steps

• Add cross-type matching (DOI ↔ URL)
• Improve UI and feedback
• Explore API-based validation (CrossRef)

I would appreciate any feedback on whether this approach is aligned with the project direction.

Thank you!

@Shehrbano_Ali: Please ask general questions in Zulip instead.

Hello @Arcstur @Ederporto,

I would like to inform you that I have updated the README and improved code readability. Kindly consider these changes.

Thank you!!

The discussion here about identifier normalization is really interesting. With my background in JavaScript, I’ve been looking into how we can implement these checks in real-time within the Visual Editor without impacting the user's typing performance.

I've been researching the ve.ui.MWReferenceDialog to understand how we might trigger a 'Potential Duplicate' alert when a user adds a URL. I'm curious to hear the mentors' thoughts on whether we should aim for a client-side check for immediate feedback, or if leveraging a MediaWiki API for normalization would be more reliable for the first version? @Arcstur @Ederporto

Hi @Arcstur I have created my final application Hello,

https://phabricator.wikimedia.org/T423000

Thank you

Hi @Arcstur ,

I finished Task 2 and then updated it to use multithreading (ThreadPoolExecutor), so it checks multiple URLs at once instead of one by one.

It runs much faster now, especially with larger datasets. Feedback is welcome!

Hi @Arcstur and @Ederporto,

I have completed both microtasks and submitted my final application. Thank you for the opportunity to work on this project.

While exploring Wishlist #3, I’ve been thinking about how duplicate references could be handled more effectively in the Visual Editor. One idea I found interesting is normalizing identifiers such as URLs, DOIs, and ISBNs before comparison, so that the same reference written in different formats can still be detected as a duplicate.

For example, small differences like prefixes (https://, doi.org/) or formatting variations could lead to duplicates if not handled properly, so normalizing them first might improve accuracy.

I’m still learning and would really appreciate any feedback on whether this approach aligns with the expected direction for the project, or how I can improve my contributions.

Thank you for your time and guidance.

Thank you @Arcstur for the feedback. Here are the changes made:

Task 1: Replaced the hardcoded month names array with Intl.DateTimeFormat as suggested. This also makes it easy to switch to Portuguese month names which aligns well with the Lusophone context of this project.

Task 2: Simplified the code by removing the color-coded output, retry internet logic, summary, and timestamped CSV as these features were not required by the task. Retained features include specific-error classification, User-Agent header, and duplicate detection . Refined the duplicate detection logic after analyzing that previous logic was missing on the real duplicates. The updated approach correctly differentiates between URL differences that are meaningful and those that are not, which also connects to the core challenge described in Wishlist #3.

Please let me know if you have any further feedback.

Thank you @Arcstur for the valuable feedback
Here are the changes made.

Task 1: Replaced hardcoded month names with Intl.DateTimeFormat.
Also fixed timezone issue - Dates are now parsed as UTC to prevent the "off by one" bug in UTC-3 and other timezones

Task 2: Removed complex CLI tool features - No more argparse, command-line options, or concurrent processing
Removed unnecessary classes - Eliminated the URLStatusChecker class
Removed advanced features - No retry logic, no statistics, no file saving
Simplified requirements - Only need requests library now

Thanks for your valuable inputs you have given
I appreciate.

I read through my task feedback, and it was such a great reality check. I used to think writing tons of comments made me a better coder, but the mentor reminded me that good code should just speak for itself.
I also learned a lot about tunnel vision and over-engineering. On Task 1, I was so focused on fixing a timezone bug that I hardcoded the month names, completely missing that using Intl.DateTimeFormat would solve both the bug and localization at once. And on Task 2, I built a GET fallback for 403 HEAD errors, not realizing that servers blocking HEAD almost always block GET too. It was a great reminder to truly understand how protocols work in the real world instead of just writing complex workarounds.
I have submitted the final application:)

Thank you @Arcstur for your feedback!

I have made the suggested updates:

Task 1: JSON Data Formatting

Fixed the date issue by handling the creation date in local time to avoid timezone-related shifts.

Task 2: URL Status Checker

Updated the implementation to use a HEAD request instead of GET, since only the status code is required.

Thanks again for the feedback.

Hi @Arcstur and @Ederporto,

Thanks for the detailed feedback on the tasks. I have made the necessary improvements to my microtasks and updated my Outreachy proposal here: https://phabricator.wikimedia.org/T423451. I would appreciate any further feedback or suggestions.

I have already submitted the final application.

Best regards,

Hi everyone 👋
I’ve been working further on my Wikimedia Outreachy Task 1 and made a few improvements to my initial submission.
So far I’ve added sorting, improved data validation, refactored the code into cleaner modules, and enhanced the UI with a card layout, summary count, and loading state. I’ve also updated the README to better explain the project.
I’m open to any feedback or suggestions 👍

Hi everyone 👋

I’ve continued refining my Task 1 and Task 2 work.

Added sorting, validation, and small UI improvements for Task 1.
For Task 2, I improved performance with concurrency, added retries, and a simple summary report.

Would really appreciate any feedback , happy to improve further 🙏

Note there is an existing phabricator task for wishlist item #3 :

https://phabricator.wikimedia.org/T126488

The successful applicant should claim the phabricator task I linked above, so no one else works on it in the meantime, thanks!

Congratulations @AnaMRRuiz on being selected for Outreachy Round 32! Please see the next steps here: Outreachy Participants Page

Looking forward to hearing more updates from you in the coming days. Welcome to the Wikimedia Foundation, we’re excited to have you with us and hope to see you around!

@AnaMRRuiz and team, reminder to please update the project page: https://www.mediawiki.org/wiki/Outreachy/Past_projects#Round_32

Thank you!

@LGoto sorry for the delay on updating that page, now that is complete