| achou | |
| Dec 17 2024, 2:41 PM |
| F58412248: visualization (2).png | |
| Feb 17 2025, 4:07 PM |
| F58138817: 14. mean_latency_comparison.png | |
| Jan 7 2025, 1:55 PM |
| F58138821: 13. mean_latency_comparison.png | |
| Jan 7 2025, 1:55 PM |
| F58138824: 12. mean_latency_comparison.png | |
| Jan 7 2025, 1:55 PM |
| F58138830: 11. mean_latency_comparison.png | |
| Jan 7 2025, 1:55 PM |
| F58138834: 10. mean_latency_comparison.png | |
| Jan 7 2025, 1:55 PM |
| F58138838: 9. mean_latency_comparison.png | |
| Jan 7 2025, 1:55 PM |
| F58035640: aya-expanse-32B.png | |
| Dec 21 2024, 3:52 AM |
We want a simple way to produce plots that show how input and output token size affect latency. For example, the heatmap below demonstrates this for Llama 3.1 8B models, showing latency comparisons on varying token sizes between the base model and its AWQ and GPTQ quantized versions.
Tools we're investigating are:
optimum-benchmark
In the meeting, we discussed finding a way to easily run the HF optimum benchmark on ml-lab. I built a tool to enable us to do this:
https://gitlab.wikimedia.org/repos/machine-learning/huggingface-optimum-benchmark-automation
At the moment, one can use this tool to set up an environment, run benchmarks for non-quantized models, produce reports, and plot bar charts with mean latency comparisons (similar to F58025719). We shall add more features to it once we've established that we can get the analysis we want from HF optimum benchmark reports (like how input and output token size affect latency).
optimum-benchmark
I refactored the benchmark automation tool to use the HF optimum benchmark CLI instead of the Python SDK since the CLI relies on hydra, which simplifies the process of running multiple benchmarks.
The tool now produces two types of charts:
1.Bar Chart: shows mean latency comparisons for each inference stage in a benchmark report.
2.Heatmap: shows the relationship between mean latency of the decode stage and varying input/output sizes (using sequence_length and max_new_tokens)
Since we are now able to see how input and output token sizes affect latency from these benchmark charts, I'll prepare to add support for both quantized and accelerated models.
llmperf
Below are the results from llmperf benchmarking for aya-expanse-8b and 32b original and quantized models (AWQ, Bitsandbytes, and GPTQ):
Note that the white parts in the GPTQ plots indicate missing results due to the default temperature setting being too low (0.001). Since the temperature parameter wasn't configurable in llmperf, token generation encountered a "RuntimeError: probability tensor contains either inf, nan or element < 0" (P71712). Following discussion, Muniza has made the temperature configurable, allowing us to increase its value to resolve this issue.
The process of llmperf benchmarking
For this experiment, I used these prebuilt AWQ and GPTQ quantized models:
-Orion-zhen/aya-expanse-8b-AWQ
-Orion-zhen/aya-expanse-32b-AWQ
-2z299/aya-expanse-8b-GPTQ-4bit
-2z299/aya-expanse-32b-GPTQ-4bit
python3 -m llmperf --config-dir experiments --log-dir tmp/llmperf --result-dir results --experiment 'aya_expanse_*'The results are saved as JSON files. For example:
$ cat results/aya_8b_gptq_i128_b1_g64.json | jq '.' { "performance": { "latency": 1.712948, "prefill": 0.141301, "throughput": 40.08533722903425 }, "ram_usage": { "min_mbs": 834.002944, "peak_mbs": 6415.106048 }, "vram_usage": { "0": { "min_mbs": 11.231232, "peak_mbs": 7316.946944 }, "1": { "min_mbs": 11.231232, "peak_mbs": 12.140544 } } }
The logs are saved in tmp/llmperf/ as specified in the command, allowing us to check for any issues during the experiments.
Note that if we want to save the charts to images, vl-convert-python is required:
pip install vl-convert-python
Then we can save an Altair chart object as a png:
chart.save("aya-expanse-32B.png")
I was able to reproduce this outside of llmperf using the same model that @achou used (2z299/aya-expanse-8b-GPTQ-4bit) with this snippet:
I was curious to see how this error was related to sampling and the way sampling works in transformers is by passing the outputs from the final layer or logits to multiple LogitsProcessor in succession (exactly which logit processors are used depends on the arguments passed to generate but some like the TemperatureLogitsWarper are always there when do_sample=True). So I wrote a custom processor that just prints out the logits before running any other processors:
from transformers import LogitsProcessor, LogitsProcessorList class LogitsPrinter(LogitsProcessor): def __call__(self, input_ids: torch.Tensor, scores: torch.Tensor) -> torch.Tensor: print(scores) return scores generated_tokens = model.generate( ..., logits_processor=LogitsProcessorList([LogitsPrinter()]), )
And looking at the logs, we can see that the logits being passed to the processor contain NaNs (so something does not go wrong during sampling) and the only reason why changing the temperature works is because it changes what tokens get selected earlier in the generation process:
--------------------------------- # temperature: 0.5 ########## token idx: 256 logits: tensor([[-1.1982, -2.6641, -1.2686, ..., -1.3418, -1.3418, -1.3418]], device='cuda:0') next token: 21 ########## token idx: 257 logits: tensor([[nan, nan, nan, ..., nan, nan, nan]], device='cuda:0') RuntimeError: probability tensor contains either inf, nan or element < 0 -------------------------------- # temperature: 0.8 ########## token idx: 256 logits: tensor([[-1.1982, -2.6641, -1.2686, ..., -1.3418, -1.3418, -1.3418]], device='cuda:0') next token: 1695 ########## token idx: 257 logits: tensor([[1.3027, 2.2012, 1.6729, ..., 1.2598, 1.2598, 1.2598]], device='cuda:0') next token: 2075 ...
So it looks like the actual issue is that the model is producing logits with NaNs for certain inputs. After inserting some breakpoints and stepping through inference, I noticed that the hidden states start containing inf after passing through down_proj in the CohereMLP block in one of the layers and then all downstream operations start producing tensors with NaNs. See L178 in these logs:
.optimum-benchmark
The benchmark automation tool now supports running benchmarks with quantization and/or flash attention. Bitsandbytes quantized models are not yet supported (see P71793#287869)
Below are bar charts showing the performance of the aya-expanse-8b model under different conditions: unquantized, quantized (gptq and awq), and accelerated with Flash Attention 2 (fa2).
1.unquantized:
# run benchmark CONFIG_NAME=aya_expanse_8b MULTIRUN_CONFIG="scenario.input_shapes.sequence_length=64,128 +scenario.generate_kwargs.max_new_tokens=64,128" make run_multiple_benchmarks # generate chart REPORT_PATHS="sweeps/aya_expanse_8b/+scenario.generate_kwargs.max_new_tokens=64,scenario.input_shapes.sequence_length=64/benchmark_report.json sweeps/aya_expanse_8b/+scenario.generate_kwargs.max_new_tokens=64,scenario.input_shapes.sequence_length=128/benchmark_report.json sweeps/aya_expanse_8b/+scenario.generate_kwargs.max_new_tokens=128,scenario.input_shapes.sequence_length=64/benchmark_report.json sweeps/aya_expanse_8b/+scenario.generate_kwargs.max_new_tokens=128,scenario.input_shapes.sequence_length=128/benchmark_report.json" PLOT_TYPE="bar" make analyse_benchmark_reports
2.unquantized + fa2:
# run benchmark USE_FLASH_ATTENTION=true CONFIG_NAME=aya_expanse_8b_fa2 MULTIRUN_CONFIG="scenario.input_shapes.sequence_length=64,128 +scenario.generate_kwargs.max_new_tokens=64,128" make run_multiple_benchmarks # generate chart REPORT_PATHS="sweeps/aya_expanse_8b_fa2/+scenario.generate_kwargs.max_new_tokens=64,scenario.input_shapes.sequence_length=64/benchmark_report.json sweeps/aya_expanse_8b_fa2/+scenario.generate_kwargs.max_new_tokens=64,scenario.input_shapes.sequence_length=128/benchmark_report.json sweeps/aya_expanse_8b_fa2/+scenario.generate_kwargs.max_new_tokens=128,scenario.input_shapes.sequence_length=64/benchmark_report.json sweeps/aya_expanse_8b_fa2/+scenario.generate_kwargs.max_new_tokens=128,scenario.input_shapes.sequence_length=128/benchmark_report.json" PLOT_TYPE="bar" make analyse_benchmark_reports
3.gptq quantized:
# run benchmark QUANTIZATION_SCHEME=gptq CONFIG_NAME=aya_expanse_8b_gptq_4bit MULTIRUN_CONFIG="scenario.input_shapes.sequence_length=64,128 +scenario.generate_kwargs.max_new_tokens=64,128" make run_multiple_benchmarks # generate chart REPORT_PATHS="sweeps/aya_expanse_8b_gptq_4bit/+scenario.generate_kwargs.max_new_tokens=64,scenario.input_shapes.sequence_length=64/benchmark_report.json sweeps/aya_expanse_8b_gptq_4bit/+scenario.generate_kwargs.max_new_tokens=64,scenario.input_shapes.sequence_length=128/benchmark_report.json sweeps/aya_expanse_8b_gptq_4bit/+scenario.generate_kwargs.max_new_tokens=128,scenario.input_shapes.sequence_length=64/benchmark_report.json sweeps/aya_expanse_8b_gptq_4bit/+scenario.generate_kwargs.max_new_tokens=128,scenario.input_shapes.sequence_length=128/benchmark_report.json" PLOT_TYPE="bar" make analyse_benchmark_reports
4.gptq quantized + fa2:
# run benchmark USE_FLASH_ATTENTION=true QUANTIZATION_SCHEME=gptq CONFIG_NAME=aya_expanse_8b_gptq_4bit_fa2 MULTIRUN_CONFIG="scenario.input_shapes.sequence_length=64,128 +scenario.generate_kwargs.max_new_tokens=64,128" make run_multiple_benchmarks # generate chart REPORT_PATHS="sweeps/aya_expanse_8b_gptq_4bit_fa2/+scenario.generate_kwargs.max_new_tokens=64,scenario.input_shapes.sequence_length=64/benchmark_report.json sweeps/aya_expanse_8b_gptq_4bit_fa2/+scenario.generate_kwargs.max_new_tokens=64,scenario.input_shapes.sequence_length=128/benchmark_report.json sweeps/aya_expanse_8b_gptq_4bit_fa2/+scenario.generate_kwargs.max_new_tokens=128,scenario.input_shapes.sequence_length=64/benchmark_report.json sweeps/aya_expanse_8b_gptq_4bit_fa2/+scenario.generate_kwargs.max_new_tokens=128,scenario.input_shapes.sequence_length=128/benchmark_report.json" PLOT_TYPE="bar" make analyse_benchmark_reports
5.awq quantized:
# run benchmark QUANTIZATION_SCHEME=awq CONFIG_NAME=aya_expanse_8b_awq_4bit.yaml MULTIRUN_CONFIG="scenario.input_shapes.sequence_length=64,128 +scenario.generate_kwargs.max_new_tokens=64,128" make run_multiple_benchmarks # generate chart REPORT_PATHS="sweeps/aya_expanse_8b_awq_4bit/+scenario.generate_kwargs.max_new_tokens=64,scenario.input_shapes.sequence_length=64/benchmark_report.json sweeps/aya_expanse_8b_awq_4bit/+scenario.generate_kwargs.max_new_tokens=64,scenario.input_shapes.sequence_length=128/benchmark_report.json sweeps/aya_expanse_8b_awq_4bit/+scenario.generate_kwargs.max_new_tokens=128,scenario.input_shapes.sequence_length=64/benchmark_report.json sweeps/aya_expanse_8b_awq_4bit/+scenario.generate_kwargs.max_new_tokens=128,scenario.input_shapes.sequence_length=128/benchmark_report.json" PLOT_TYPE="bar" make analyse_benchmark_reports
6.awq quantized + fa2:
# run benchmark USE_FLASH_ATTENTION=true QUANTIZATION_SCHEME=awq CONFIG_NAME=aya_expanse_8b_awq_4bit_fa2 MULTIRUN_CONFIG="scenario.input_shapes.sequence_length=64,128 +scenario.generate_kwargs.max_new_tokens=64,128" make run_multiple_benchmarks # generate chart REPORT_PATHS="sweeps/aya_expanse_8b_awq_4bit_fa2/+scenario.generate_kwargs.max_new_tokens=64,scenario.input_shapes.sequence_length=64/benchmark_report.json sweeps/aya_expanse_8b_awq_4bit_fa2/+scenario.generate_kwargs.max_new_tokens=64,scenario.input_shapes.sequence_length=128/benchmark_report.json sweeps/aya_expanse_8b_awq_4bit_fa2/+scenario.generate_kwargs.max_new_tokens=128,scenario.input_shapes.sequence_length=64/benchmark_report.json sweeps/aya_expanse_8b_awq_4bit_fa2/+scenario.generate_kwargs.max_new_tokens=128,scenario.input_shapes.sequence_length=128/benchmark_report.json" PLOT_TYPE="bar" make analyse_benchmark_reports
In conclusion, the unquantized model generally has higher latencies compared to its quantized counterparts. Both quantization methods (gptq and awq) lead to reduced inference times, but their performance varies slightly. While gptq models load faster but decode slightly slower, awq models take longer to load yet decode a little faster. fa2 significantly reduces decode latency for the unquantized model, but in quantized models its decode performance seems to increase in gptq models but not awq models.
I updated optimum-benchmark repo to use GPTQModel but I am receiving error:
I will investigate the following issues:
Unfortunately optimum-benchmark does not support GPTQModel yet. Even if I installed the gptqmodel wheel it looks for auto_gptq and fails check the above error. So, that means that we cannot use it for benchmarking at the moment.
It is also clear in the setup.py file in optimum-benchmark upstream:
"vllm": ["vllm"], # optional dependencies "torchao": ["torchao"], "autoawq": ["autoawq"], "auto-gptq": ["optimum", "auto-gptq"], "sentence-transformers": ["sentence-transformers"], "bitsandbytes": ["bitsandbytes"],
They are still using auto_gptq.
I will try to experiment with llmperf.
llmperf
Below are the results from llmperf benchmarking for aya-expanse-8b and 32b original and quantized models (AWQ, Bitsandbytes, and GPTQ):
- aya-expanse-8b
- aya-expanse-32b
Note that the white parts in the GPTQ plots indicate missing results due to the default temperature setting being too low (0.001). Since the temperature parameter wasn't configurable in llmperf, token generation encountered a "RuntimeError: probability tensor contains either inf, nan or element < 0" (P71712). Following discussion, Muniza has made the temperature configurable, allowing us to increase its value to resolve this issue.
The process of llmperf benchmarking
- Add experiment configs (yaml files)
For this experiment, I used these prebuilt AWQ and GPTQ quantized models:
-Orion-zhen/aya-expanse-8b-AWQ
-Orion-zhen/aya-expanse-32b-AWQ
-2z299/aya-expanse-8b-GPTQ-4bit
-2z299/aya-expanse-32b-GPTQ-4bit
- Run experiments using the command:
python3 -m llmperf --config-dir experiments --log-dir tmp/llmperf --result-dir results --experiment 'aya_expanse_*'The results are saved as JSON files. For example:
$ cat results/aya_8b_gptq_i128_b1_g64.json | jq '.' { "performance": { "latency": 1.712948, "prefill": 0.141301, "throughput": 40.08533722903425 }, "ram_usage": { "min_mbs": 834.002944, "peak_mbs": 6415.106048 }, "vram_usage": { "0": { "min_mbs": 11.231232, "peak_mbs": 7316.946944 }, "1": { "min_mbs": 11.231232, "peak_mbs": 12.140544 } } }The logs are saved in tmp/llmperf/ as specified in the command, allowing us to check for any issues during the experiments.
- Load the JSON results and produce plots using this notebook.
Note that if we want to save the charts to images, vl-convert-python is required:
pip install vl-convert-pythonThen we can save an Altair chart object as a png:
chart.save("aya-expanse-32B.png")
Is it possible to add some more info on the prerequisites such as:
I am following the instructions in the documentation (and the above one) but I cannot reproduce the results.
When I ran the python3 -m llmperf \ ... command I received ModuleNotFoundError: No module named 'torch'
I installed the torch==2.5.1 --extra-index-url https://download.pytorch.org/whl/rocm6.2/ version but then it does not export any results.
It assigns the job to GPU for 2 seconds and then it stops without any errors or anything like that.
Do you have any idea what is going wrong ?
- What virtual environment manager you used to built the env (I see uv.lock)
- Which torch version you used?
I created a conda env with python 3.11, and used export PYTHONPATH="$(conda info --base)/envs/{your-condo-env-name}/lib/python3.11/site-packages:/srv/pytorch-rocm/venv/lib/python3.11/site-packages" to use our ROCm-enabled PyTorch libraries in /srv
Thank you for your help @AikoChou.
I thinks there is an issue with typing_extensions.cpython-311.pyc.
The llmperf tries to install typing-extensions>=4.12.2 but we are using typing_extensions-4.9.0 in the PYTHONPATH for rocm.
I followed these steps:
The llmperf tries to install typing-extensions>=4.12.2 but we are using typing_extensions-4.9.0 in the PYTHONPATH for rocm.
Ah right, I ran into the same issue before. I solved it by using the --user option to install llmperf and then adding the path to PYTHONPATH: export "PYTHONPATH=/home/aikochou/.local/lib/python3.11/site-packages/:$PYTHONPATH"
Hi @gkyziridis, apologies if the documentation doesn't make it clear but the path that you're passing to --config-dir needs to exist and contain experiment config files. If you've already cloned llm-evaluation, pass --config-dir llmperf/experiments/quantization and if you're already aware and have your own config files under experiments, please ignore this!
Thanks for your time @achou and @MunizaA but I am still having the same issue, which probably has to do with typing_extensions.
I used conda env
I used the --user flag to pip install llmperf
export "PYTHONPATH=/home/gkyziridis/.local/lib/python3.11/site-packages/:$PYTHONPATH" python3 -m llmperf --config-dir llmperf/experiments/quantization --log-dir /tmp/llmperf --result-dir llmperf/results --experiment 'llama31_8b*' [-] ModuleNotFoundError: No module named 'torch'`
export PYTHONPATH="$(conda info --base)/envs/llenv/lib/python3.11/site-packages:/srv/pytorch-rocm/venv/lib/python3.11/site-packages" python3 -m llmperf --config-dir llmperf/experiments/quantization --log-dir /tmp/llmperf --result-dir llmperf/results --experiment 'llama31_8b*' [-] ImportError: cannot import name 'TypeIs' from 'typing_extensions' (/srv/pytorch-rocm/venv/lib/python3.11/site-packages/typing_extensions.py)
@gkyziridis One thing you could try is with your PYTHONPATH set to "$(conda info --base)/envs/llenv/lib/python3.11/site-packages:/srv/pytorch-rocm/venv/lib/python3.11/site-packages, install typing_extensions via:
pip install --upgrade --ignore-installed typing-extensions
This will ignore the version installed under pytorch-rocm and install the latest version in your conda env which would be the first thing on your sys.path.
Thank you @MunizaA for your time, that fixes the issue.
I think llmperf does not recognise the gptqmodel, it tries to use auto-gptq same as in the optimum-benchmark repo discussed in: T382343#10548614.
Steps I followed:
$ cat llmperf/experiments/quantization/aya_expanse_32b_gptq_4bit.yaml # Create a new experiment for local quantized `aya model` via `gptqmodel` name: aya_expanse_32b_gptq_4bit batch_size: [1] sequence_length: [64, 128] tasks: - kind: generation max_new_tokens: 64 - kind: generation max_new_tokens: 128 model_config: model_name: /srv/home/gkyziridis/aya-expanse-32b-4bit-4batch-1024textsize-512nsamples low_cpu_mem_usage: true device: cuda:0 device_map: cuda:0 torch_dtype: float16
$ export PYTHONPATH="$(conda info --base)/envs/llenv/lib/python3.11/site-packages:/srv/pytorch-rocm/venv/lib/python3.11/site-packages" $ pip install --upgrade --ignore-installed typing-extensions $ pip install 'accelerate>=0.26.0' $ pip install optimum[amd] $ pip install gptqmodel-1.7.4.dev0-cp311-cp311-linux_x86_64.whl --user $ pip install lm_eval $ python3 -m llmperf --config-dir llmperf/experiments/quantization --log-dir llmperf/tmp/llmperf --result-dir results --experiment 'aya_expanse_32b_gptq_4bit.yaml' INFO:root:Reading config file: aya_expanse_32b_gptq_4bit.yaml INFO:root:Launching experiment aya_expanse_32b_gptq_4bit as an isolated process INFO:root:Running experiment for seq len: 64, batch size: 1 amdgpu.ids: No such file or directory amdgpu.ids: No such file or directory INFO:root:Running experiment for seq len: 128, batch size: 1 amdgpu.ids: No such file or directory amdgpu.ids: No such file or directory INFO:root:Running experiment for seq len: 64, batch size: 1 amdgpu.ids: No such file or directory amdgpu.ids: No such file or directory INFO:root:Running experiment for seq len: 128, batch size: 1 amdgpu.ids: No such file or directory amdgpu.ids: No such file or directory
$ cat llmperf/tmp/llmperf/aya_expanse_32b_gptq_4bit_i64_b1_g64/experiment.err Process SpawnProcess-1: Traceback (most recent call last): File "/home/gkyziridis/miniconda3/envs/llenv/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/gkyziridis/miniconda3/envs/llenv/lib/python3.11/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/gkyziridis/.local/lib/python3.11/site-packages/llmperf/experiment.py", line 83, in _run llm = HFLM(config=self.model_config, task=task.kind) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gkyziridis/.local/lib/python3.11/site-packages/llmperf/llms/huggingface.py", line 91, in __init__ self.model = TASK_TO_MODEL[self.task].from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gkyziridis/.local/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained return model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gkyziridis/.local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3620, in from_pretrained hf_quantizer.validate_environment( File "/home/gkyziridis/.local/lib/python3.11/site-packages/transformers/quantizers/quantizer_gptq.py", line 59, in validate_environment raise ImportError( ImportError: Loading a GPTQ quantized model requires the auto-gptq library (`pip install auto-gptq`)
llmperf does not directly use auto-gptq (or checks for its availability) but instead loads the model using transformers.AutoModelForCausalLM.from_pretrained and the latest version of transformers still relies on auto-gptq for loading gptq quantized models. Support for gptqmodel has landed in the main branch though and installing from that (for now) allows you to run benchmarks using gptqmodel:
$ git clone https://github.com/huggingface/transformers.git && cd transformers $ pip install .
Then running your benchmarks:
$ python3 -m llmperf --config-dir llmperf/experiments/quantization --log-dir llmperf/tmp/llmperf --result-dir results --experiment 'aya_expanse_32b_gptq_4bit.yaml'would yield:
$ cat results/aya_expanse_32b_gptq_4bit_i64_b1_g64.json | jq { "performance": { "latency": 3.457892, "prefill": 0.303436, "throughput": 19.971747902015434 }, "ram_usage": { "min_mbs": 988.28288, "peak_mbs": 21549.416448 }, "vram_usage": { "0": { "min_mbs": 11.227136, "peak_mbs": 12.853248 }, "1": { "min_mbs": 11.227136, "peak_mbs": 24758.054912 } } }
Thank you so much @MunizaA for your help. I finally made it to work.
Much appreciated!!!
Performance Benchmark
Aya-expanse-32B quantization via GPTQModel.
It can be easily observable that the latency between the pre-trained model and the quantized one is almost half.
