Change Details

In T369288, we developed a model for generating simple summaries for Wikipedia articles. While successfully tested in our infrastructure on a small sample of articles (~8K), one of the main limitations of the model is that inference time is substantial (>10s per sample). As a result, in its current state, the model cannot be used to generate simple summaries at scale (for millions of articles). Therefore, in this task we aim to test different options for optimizing the model inference. We do not only aim to improve inference time but also make sure to preserve a sufficient level of quality of the model’s output. The majority of work will be done on the ml-lab instances, but for some experiments we need to generate a docker image to run batch model inference pipelines using AMD GPU as workloads on the DSE cluster. List of potential optimization approaches: * Quantization https://phabricator.wikimedia.org/T377848 * Flash attention https://phabricator.wikimedia.org/T371344 * Distributed compute * (stretch) other (such as pruning) Metrics for monitoring: * Inference: Time, memory, etc * Quality: Guardrail metrics (simplicity, fluency, meaning preservations), etc Implementation: [ ] Baseline [ ] Optimization 1 [ ] Optimization 2

In T369288, we developed a model for generating simple summaries for Wikipedia articles. While successfully tested in our infrastructure on a small sample of articles (~8K), one of the main limitations of the model is that inference time is substantial (>10s per sample). As a result, in its current state, the model cannot be used to generate simple summaries at scale (for millions of articles). Therefore, in this task we aim to test different options for optimizing the model inference. We do not only aim to improve inference time but also make sure to preserve a sufficient level of quality of the model’s output. The majority of work will be done on the ml-lab instances, but for some experiments we need to generate a docker image to run batch model inference pipelines using AMD GPU as workloads on the DSE cluster. List of potential optimization approaches: * Quantization https://phabricator.wikimedia.org/T377848 * Flash attention https://phabricator.wikimedia.org/T371344 * Distributed compute * (stretch) other (such as pruning) Metrics for monitoring: * Inference: Time, memory, etc ( [[ https://gitlab.wikimedia.org/repos/research/llm_evaluation/-/tree/main/llmperf?ref_type=heads | llmperf ]] ) * Quality: simplicity, fluency, meaning preservations, etc ([[ https://gitlab.wikimedia.org/repos/research/simple-summaries/-/blob/main/simple-summary_eval-guardrail_example.ipynb | guardrail metrics from simple summaries ]]) Implementation: [ ] Baseline [ ] Optimization 1...N [ ] (stretch) Replicate methodology with classification model for peacock detection from SDS 1.2.1 B Roles: * optimization and inference metrics: @fkaelin (ResEng) * quality/guardrail metrics (@MGerlach )

In T369288, we developed a model for generating simple summaries for Wikipedia articles. While successfully tested in our infrastructure on a small sample of articles (~8K), one of the main limitations of the model is that inference time is substantial (>10s per sample). As a result, in its current state, the model cannot be used to generate simple summaries at scale (for millions of articles). Therefore, in this task we aim to test different options for optimizing the model inference. We do not only aim to improve inference time but also make sure to preserve a sufficient level of quality of the model’s output. The majority of work will be done on the ml-lab instances, but for some experiments we need to generate a docker image to run batch model inference pipelines using AMD GPU as workloads on the DSE cluster. List of potential optimization approaches: * Quantization https://phabricator.wikimedia.org/T377848 * Flash attention https://phabricator.wikimedia.org/T371344 * Distributed compute * (stretch) other (such as pruning) Metrics for monitoring: * Inference: Time, memory, etc ( [[ https://gitlab.wikimedia.org/repos/research/llm_evaluation/-/tree/main/llmperf?ref_type=heads | llmperf ]] ) * Quality: Guardrail metrics (simplicity, fluency, meaning preservations), etc ([[ https://gitlab.wikimedia.org/repos/research/simple-summaries/-/blob/main/simple-summary_eval-guardrail_example.ipynb | guardrail metrics from simple summaries ]]) Implementation: [ ] Baseline [ ] Optimization 1...N [ ] Optimization 2(stretch) Replicate methodology with classification model for peacock detection from SDS 1.2.1 B Roles: * optimization and inference metrics: @fkaelin (ResEng) * quality/guardrail metrics (@MGerlach )