A Coding Implementation to Compress and Benchmark Instruction-Tuned LLMs with FP8, GPTQ, and SmoothQuant Quantization using llmcompressor
AI News

A Coding Implementation to Compress and Benchmark Instruction-Tuned LLMs with FP8, GPTQ, and SmoothQuant Quantization using llmcompressor

import subprocess, sys def pip(*pkgs): subprocess.check_call([sys.executable, “-m”, “pip”, “install”, “-q”, *pkgs]) pip(“llmcompressor”, “compressed-tensors”, “transformers>=4.45”, “accelerate”, “datasets”) import os, gc, time, json, math from pathlib import Path import torch from transformers import AutoModelForCausalLM, AutoTokenizer from datasets […]