Change Details

The training of the quality model was replicated in T360815. This task focuses on taking the step of swapping out the wikitext-based features for HTML-based features in the training of the model. It will build on the final notebook (with Chinese data included) from that previous task: https://public-paws.wmcloud.org/User:DJames-WMF/Quality_Model_Training.ipynb == 1. Getting started [x] Get familiar with [[https://pypi.org/project/mwparserfromhtml/|mwparserfromhtml]] library. It does a lot of things (many of which are irrelevant to this task) but you can see a direct example of how we can replace wikitext-based feature extraction with HTML-based feature extraction for references here: https://public-paws.wmcloud.org/User:Isaac%20(WMF)/HTML-dumps/references-wikitext-vs-html.ipynb. You will notice that the `wikitext_to_refs` function is very similar to existing code for extracting references in `get_article_features` in your notebook. The `html_to_refs` replacement is even simpler then and that's what you'll be switching it to. [x] Duplicate [[https://public-paws.wmcloud.org/User:DJames-WMF/Quality_Model_Training.ipynb|your notebook]] so you can update it while retaining a copy of your previous results/code. == 2. Switch wikitext to HTML features For a given article, the notebook currently fetches its wikitext and extracts the features from it. We want to instead fetch its HTML and extract the same features from that. [] Replace `get_article_wikitext` with a function called `get_article_html` that takes the same parameters (`lang` and `revid`). I've actually already started this elsewhere so you can re-use the code called `get_article_parsoid` in [[https://public-paws.wmcloud.org/User:Isaac%20(WMF)/HTML-dumps/wikipedia-plaintext-extraction.ipynb#Getting-individual-articles-from-the-Parsoid-API|this notebook]] but that gets the current version of an article (not a specific revision ID). To get the specific `revid`, you'll have to instead switch to the [[https://www.mediawiki.org/wiki/RESTBase/service_migration#Parsoid_endpoints|revision-oriented API endpoint]] in the code. The function should return a string that is the HTML for an article revision. [] Rewrite `get_article_features` to take the article HTML instead of wikitext. Each feature count that we return at the end (`[page_length, refs, wikilinks, categories, media, headings]`) will now need to be calculated using functions in `mwparserfromhtml` from the HTML. [] Clean-up: a bunch of global variables that were used for wikitext processing can now also be removed (variables related to category/media/reference wikitext extraction). And presumably the `re` import can also be removed. == 3. Replicate and compare! [] Run the new code from start-to-finish! [] In markdown at the top of the notebook, write a summary that includes your new model coefficients and how similar they are to the feature weights based on wikitext features. == 4. Optional explorations [] Complete [[https://gitlab.wikimedia.org/repos/research/html-dumps/-/issues/126|this issue]] on updating the mwparserfromhtml documentation. This will be very helpful for future users and is a good way to get accustomed to our Gitlab infrastructure / code review process.

The training of the quality model was replicated in T360815. This task focuses on taking the step of swapping out the wikitext-based features for HTML-based features in the training of the model. It will build on the final notebook (with Chinese data included) from that previous task: https://public-paws.wmcloud.org/User:DJames-WMF/Quality_Model_Training.ipynb == 1. Getting started [x] Get familiar with [[https://pypi.org/project/mwparserfromhtml/|mwparserfromhtml]] library. It does a lot of things (many of which are irrelevant to this task) but you can see a direct example of how we can replace wikitext-based feature extraction with HTML-based feature extraction for references here: https://public-paws.wmcloud.org/User:Isaac%20(WMF)/HTML-dumps/references-wikitext-vs-html.ipynb. You will notice that the `wikitext_to_refs` function is very similar to existing code for extracting references in `get_article_features` in your notebook. The `html_to_refs` replacement is even simpler then and that's what you'll be switching it to. [x] Duplicate [[https://public-paws.wmcloud.org/User:DJames-WMF/Quality_Model_Training.ipynb|your notebook]] so you can update it while retaining a copy of your previous results/code. == 2. Switch wikitext to HTML features For a given article, the notebook currently fetches its wikitext and extracts the features from it. We want to instead fetch its HTML and extract the same features from that. [x] Replace `get_article_wikitext` with a function called `get_article_html` that takes the same parameters (`lang` and `revid`). I've actually already started this elsewhere so you can re-use the code called `get_article_parsoid` in [[https://public-paws.wmcloud.org/User:Isaac%20(WMF)/HTML-dumps/wikipedia-plaintext-extraction.ipynb#Getting-individual-articles-from-the-Parsoid-API|this notebook]] but that gets the current version of an article (not a specific revision ID). To get the specific `revid`, you'll have to instead switch to the [[https://www.mediawiki.org/wiki/RESTBase/service_migration#Parsoid_endpoints|revision-oriented API endpoint]] in the code. The function should return a string that is the HTML for an article revision. [x] Rewrite `get_article_features` to take the article HTML instead of wikitext. Each feature count that we return at the end (`[page_length, refs, wikilinks, categories, media, headings]`) will now need to be calculated using functions in `mwparserfromhtml` from the HTML. [x] Clean-up: a bunch of global variables that were used for wikitext processing can now also be removed (variables related to category/media/reference wikitext extraction). And presumably the `re` import can also be removed. == 3. Replicate and compare! [x] Run the new code from start-to-finish! [x] In markdown at the top of the notebook, write a summary that includes your new model coefficients and how similar they are to the feature weights based on wikitext features. == 4. Optional explorations [] Complete [[https://gitlab.wikimedia.org/repos/research/html-dumps/-/issues/126|this issue]] on updating the mwparserfromhtml documentation. This will be very helpful for future users and is a good way to get accustomed to our Gitlab infrastructure / code review process.

The training of the quality model was replicated in T360815. This task focuses on taking the step of swapping out the wikitext-based features for HTML-based features in the training of the model. It will build on the final notebook (with Chinese data included) from that previous task: https://public-paws.wmcloud.org/User:DJames-WMF/Quality_Model_Training.ipynb == 1. Getting started [x] Get familiar with [[https://pypi.org/project/mwparserfromhtml/|mwparserfromhtml]] library. It does a lot of things (many of which are irrelevant to this task) but you can see a direct example of how we can replace wikitext-based feature extraction with HTML-based feature extraction for references here: https://public-paws.wmcloud.org/User:Isaac%20(WMF)/HTML-dumps/references-wikitext-vs-html.ipynb. You will notice that the `wikitext_to_refs` function is very similar to existing code for extracting references in `get_article_features` in your notebook. The `html_to_refs` replacement is even simpler then and that's what you'll be switching it to. [x] Duplicate [[https://public-paws.wmcloud.org/User:DJames-WMF/Quality_Model_Training.ipynb|your notebook]] so you can update it while retaining a copy of your previous results/code. == 2. Switch wikitext to HTML features For a given article, the notebook currently fetches its wikitext and extracts the features from it. We want to instead fetch its HTML and extract the same features from that. [x] Replace `get_article_wikitext` with a function called `get_article_html` that takes the same parameters (`lang` and `revid`). I've actually already started this elsewhere so you can re-use the code called `get_article_parsoid` in [[https://public-paws.wmcloud.org/User:Isaac%20(WMF)/HTML-dumps/wikipedia-plaintext-extraction.ipynb#Getting-individual-articles-from-the-Parsoid-API|this notebook]] but that gets the current version of an article (not a specific revision ID). To get the specific `revid`, you'll have to instead switch to the [[https://www.mediawiki.org/wiki/RESTBase/service_migration#Parsoid_endpoints|revision-oriented API endpoint]] in the code. The function should return a string that is the HTML for an article revision. [x] Rewrite `get_article_features` to take the article HTML instead of wikitext. Each feature count that we return at the end (`[page_length, refs, wikilinks, categories, media, headings]`) will now need to be calculated using functions in `mwparserfromhtml` from the HTML. [x] Clean-up: a bunch of global variables that were used for wikitext processing can now also be removed (variables related to category/media/reference wikitext extraction). And presumably the `re` import can also be removed. == 3. Replicate and compare! [x] Run the new code from start-to-finish! [x] In markdown at the top of the notebook, write a summary that includes your new model coefficients and how similar they are to the feature weights based on wikitext features. == 4. Optional explorations [] Complete [[https://gitlab.wikimedia.org/repos/research/html-dumps/-/issues/126|this issue]] on updating the mwparserfromhtml documentation. This will be very helpful for future users and is a good way to get accustomed to our Gitlab infrastructure / code review process.