Change Details

This task aims to evaluate the feasibility of running automated checks on the verifiability of existing claim+citation pairs on Wikipedia. To this end it will provide a dataset of citation+claim pairs for testing different models/approaches along with insights about claim extraction and accessibility of sources. == Basic workflow The focus of this task is existing claim+citations on Wikipedia. It will not assess the verifiability of claims that lack citations, which would require an additional step of locating potential sources. Steps: * For a given citation in a Wikipedia article, extract: ** The claim associated with the citation. This has various options for how to process the Wikipedia content to get "clean" text. And then there are multiple potential approaches for identifying the specific claim: *** Simple heuristics for collecting a fixed-size chunk of content preceding the citation *** Slightly more complex grabbing the sentence where the citation appears *** More complex using a language model to isolate the atomic claim most likely associated with the citation ** The URL of the external source (if it exists). This requires some basic logic to link an inline citation to its reference (simple) and then extract the URL(s) to assess. This will range from no external URL to a single link to multiple (e.g., original + archived). * For a given external source URL, extract: ** The content of the page -- i.e. raw HTML or PDF if it points to a file. In today's internet, this is far from assured. Many sites block non-human traffic (quite understandably) or put the actual content behind paywalls. ** The "cleaned" text from the page content. HTML can be very verbose and contain a lot of noise (styling etc.) so you probably don't want to feed this directly into any language model. Instead you want just the actual text. While this is straightforward for some sites, in others the text might be buried in tabs or flashy visual elements. And you likely want to remove generic boilerplate text (menu options, legalese, etc.) from the text to also reduce the amount of noise. * For a given claim + external source text: ** Assess whether the source supports the claim. Again, multiple potential approaches; *** Binary yes supports or no does not. *** Three-way classification: supports, explicitly rejects the claim, not enough information to support or reject *** Even more complicated assessments that allow for partial support -- e.g., if a claim has multiple parts to it and some are supported but not all. == Potential production connections * Edit Check: {T399642} (specifically see T399642#11375222 for some early thinking / related work) * [[https://www.mediawiki.org/wiki/Article_guidance|Article Creation Guidance]] * Suggested Edits -- e.g., companion task to directing newcomers to articles with citation-needed tags.

This task aims to evaluate the feasibility of running automated checks on the verifiability of existing claim+citation pairs on Wikipedia. To this end it will provide a dataset of citation+claim pairs for testing different models/approaches along with insights about claim extraction and accessibility of sources. == Basic workflow The focus of this task is existing claim+citations on Wikipedia. It will not assess the verifiability of claims that lack citations, which would require an additional step of locating potential sources. Steps: * For a given citation in a Wikipedia article, extract: ** The claim associated with the citation. This has various options for how to process the Wikipedia content to get "clean" text. And then there are multiple potential approaches for identifying the specific claim: *** Simple heuristics for collecting a fixed-size chunk of content preceding the citation *** Slightly more complex grabbing the sentence where the citation appears *** More complex using a language model to isolate the atomic claim most likely associated with the citation ** The URL of the external source (if it exists). This requires some basic logic to link an inline citation to its reference (simple) and then extract the URL(s) to assess. This will range from no external URL to a single link to multiple (e.g., original + archived). * For a given external source URL, extract: ** The content of the page -- i.e. raw HTML or PDF if it points to a file. In today's internet, this is far from assured. Many sites block non-human traffic (quite understandably) or put the actual content behind paywalls. ** The "cleaned" text from the page content. HTML can be very verbose and contain a lot of noise (styling etc.) so you probably don't want to feed this directly into any language model. Instead you want just the actual text. While this is straightforward for some sites, in others the text might be buried in tabs or flashy visual elements. And you likely want to remove generic boilerplate text (menu options, legalese, etc.) from the text to also reduce the amount of noise. * For a given claim + external source text: ** Assess whether the source supports the claim. Again, multiple potential approaches; *** Binary yes supports or no does not. *** Three-way classification: supports, explicitly rejects the claim, not enough information to support or reject *** Even more complicated assessments that allow for partial support -- e.g., if a claim has multiple parts to it and some are supported but not all. == Potential production connections * Edit Check: {T399642} (specifically see T399642#11375222 for some early thinking / related work) * [[https://www.mediawiki.org/wiki/Article_guidance|Article Creation Guidance]] * Suggested Edits -- e.g., companion task to directing newcomers to articles with citation-needed tags. == See Also * [[https://app.asana.com/1/3758245663860/project/1213607804837821/overview/1213607914027936|Asana ticket (internal)]]

This task aims to evaluate the feasibility of running automated checks on the verifiability of existing claim+citation pairs on Wikipedia. To this end it will provide a dataset of citation+claim pairs for testing different models/approaches along with insights about claim extraction and accessibility of sources. == Basic workflow The focus of this task is existing claim+citations on Wikipedia. It will not assess the verifiability of claims that lack citations, which would require an additional step of locating potential sources. Steps: * For a given citation in a Wikipedia article, extract: ** The claim associated with the citation. This has various options for how to process the Wikipedia content to get "clean" text. And then there are multiple potential approaches for identifying the specific claim: *** Simple heuristics for collecting a fixed-size chunk of content preceding the citation *** Slightly more complex grabbing the sentence where the citation appears *** More complex using a language model to isolate the atomic claim most likely associated with the citation ** The URL of the external source (if it exists). This requires some basic logic to link an inline citation to its reference (simple) and then extract the URL(s) to assess. This will range from no external URL to a single link to multiple (e.g., original + archived). * For a given external source URL, extract: ** The content of the page -- i.e. raw HTML or PDF if it points to a file. In today's internet, this is far from assured. Many sites block non-human traffic (quite understandably) or put the actual content behind paywalls. ** The "cleaned" text from the page content. HTML can be very verbose and contain a lot of noise (styling etc.) so you probably don't want to feed this directly into any language model. Instead you want just the actual text. While this is straightforward for some sites, in others the text might be buried in tabs or flashy visual elements. And you likely want to remove generic boilerplate text (menu options, legalese, etc.) from the text to also reduce the amount of noise. * For a given claim + external source text: ** Assess whether the source supports the claim. Again, multiple potential approaches; *** Binary yes supports or no does not. *** Three-way classification: supports, explicitly rejects the claim, not enough information to support or reject *** Even more complicated assessments that allow for partial support -- e.g., if a claim has multiple parts to it and some are supported but not all. == Potential production connections * Edit Check: {T399642} (specifically see T399642#11375222 for some early thinking / related work) * [[https://www.mediawiki.org/wiki/Article_guidance|Article Creation Guidance]] * Suggested Edits -- e.g., companion task to directing newcomers to articles with citation-needed tags. == See Also * [[https://app.asana.com/1/3758245663860/project/1213607804837821/overview/1213607914027936|Asana ticket (internal)]]