weekly update:

• no update (this was a shortened week and I was busy with catching up after the break and to preparing for the team offsite next week)

weekly update:

• Starting looking into one of the most recent benchmark dataset (D-Wikipedia)
• explored access, dataset size, potential quality issues

weekly update:

• Started to explore the T5 model class on stat1008 and managed to run inference with GPUs on benchmark data
• Next steps are:
  • figure out different options and their impact on performance, memory, latency
  • figure out how to fine-tune the model on training data for simplification with benchmark dataset; going through some related code examples, e.g. from descartes’ article description generation https://github.com/wikimedia/descartes

weekly update:

• Re-implementing a previous model to figure out how to fine-tune a model for simplification. I am starting with this example script from a recent paper using an English T5 model.

weekly update:

• Succesfully built a pipeline to fine-tune a language model on DWikipedia benchmark dataset. As a test case I started with the T5-small. Qualitatively evaluated with standard simplification metrics (SARI, BLEU). Results are comparable with state-of-the-art results described in the corresponding paper (Document-Level Text Simplification: Dataset, Criteria and Baseline)
• Qualitative evaluation of model output seems promising. Example output
  • Input (article from enwiki):

in economics , a free market is a system in which the prices for goods and services are self-regulated by the open market and by consumers . in a free market , the laws and forces of supply and demand are free from any intervention by a government or other authority , and from all forms of economic privilege , monopolies and artificial scarcities . proponents of the concept of free market contrast it with a regulated market in which a government intervenes in supply and demand through various methods such as tariffs used to restrict trade and to protect the local economy . in an idealized free-market economy , prices for goods and services are set freely by the forces of supply and demand and are allowed to reach their point of equilibrium without intervention by government policy.scholars contrast the concept of a free market with the concept of a coordinated market in fields of study such as political economy , new institutional economics , economic sociology and political science . all of these fields emphasize the importance in currently existing market systems of rule-making institutions external to the simple forces of supply and demand which create space for those forces to operate to control productive output and distribution . although free markets are commonly associated with capitalism within a market economy in contemporary usage and popular culture , free markets have also been advocated by anarchists , socialists and some proponents of cooperatives and advocates of profit sharing.criticism of the theoretical concept may regard systems with significant market power , inequality of bargaining power , or information asymmetry as less than free , with regulation being necessary to control those imbalances in order to allow markets to function more efficiently as well as produce more desirable social outcomes .

• Output (model):

a free market is a system in which prices for goods and services are self-regulated by the open market and by consumers. in an idealized free-market economy, prices for goods and services are set freely by the forces of supply and demand. the laws and forces of supply and demand are free from any intervention by a government or other authority, and from all forms of economic privilege, monopolies and artificial scarcities.

• Next steps: Scaling the model: i) larger model variants (use -base/large instead of -small), ii) longer input sequences to simplify full document (use long-T5 instead of t5), iii) multilingual training (use mT5 instead of t5 ). Then we can apply to our custom multilingual dataset. This will require some tweaking to run the model in our infrastructure.

weekly:

• added small improvements to the model (adding prefix "simplify: " to the model input, adding other evaluation metrics such as flesch-kincaid grade level)
• running experiments with multilingual models (mT5) on D-Wikipedia benchmark.
• if benchmark evaluation in English yields similarly good results, next step is to evaluate on other languages

weekly update:

• trained and evaluated multilingual model (mT5-small) on the D-Wikipedia benchmark dataset (only English). The performance of simplification via SARI is similar to that of the monolingual model (if not slightly larger).
• next step is training/evaluating for the non-English datasets

weekly update:

• evaluated fine-tuned model on multilingual dataset
• performance on English datasets is good: evaluation metrics comparable to previously reported SOTA results and qualitative inspection of output looks promising
• performance on non-English datasets is poor: evaluation metrics (SARI, BLEU) very low. manual inspection shows that the model's output is often in English despite the input being in another language. the resulting simplification is often not bad, just in the wrong language. this explains why the metrics are bad.
• this means that multilingual modeling requires additional efforts to get it right as current state is unsatisfactory: i) fine-tuning other models, e.g., those where target-language can be fixed, or ii) additional fine-tuning with multilingual data to ensure better language alignment.

weekly update:

• Refactored code for more systematic comparison of different models
• Currently running different experiments with different model sizes and pre-traing strategies