TurboQuant vs GPTQ vs AWQ: Why Google's Method Needs No Retraining

TurboQuant vs GPTQ vs AWQ is a comparison of different layers of an AI inference stack. TurboQuant is described by Google Research as a method for key-value cache compression and vector search. GPTQ and AWQ are post-training methods for reducing the storage cost of model weights. A lower bit width does not make these methods interchangeable.

The distinction matters when an engineer plans a local LLM deployment. Weight quantization changes the model file and the memory required to load the model. KV cache compression changes the memory used as a request runs and the context grows. A system may use weight quantization and KV cache compression together, but the two operations answer different capacity questions.

Google Research's TurboQuant announcement, published March 24, 2026, describes PolarQuant and Quantized Johnson-Lindenstrauss, or QJL, for compact vector representations. The GPTQ paper describes one-shot weight quantization based on approximate second-order information. The AWQ paper describes activation-aware weight-only quantization that uses offline activation statistics. The methods should be compared on their actual target and data requirements.

What You'll Learn

• What TurboQuant, GPTQ, and AWQ actually quantize.
• How calibration data and activation statistics differ.
• Why weight memory and KV cache memory are separate.
• How to choose tests for quality, latency, and capacity.

First Question: What Gets Quantized?

Quantization reduces the number of bits used to represent numerical data. The data can be model weights, activations, keys, values, or vectors in a search index. The method's name tells only part of the story. The target tensor, timing, metadata, and downstream kernel determine the practical result.

GPTQ and AWQ are described in their papers as weight quantization methods. They act on parameters that already exist in a trained model. TurboQuant is described for the KV cache and vector search. It acts on data created or stored during inference and retrieval.

TurboQuant: A KV Cache and Vector Search Method

Google Research describes TurboQuant as a compression approach for high-dimensional vectors. One application is the KV cache used by an LLM during generation. Another is vector search, where a system stores embeddings and retrieves items close to a query.

The method combines PolarQuant and QJL. PolarQuant uses a random rotation and a polar transformation to prepare a vector for low-bit representation. QJL uses a one-bit residual idea and an estimator to reduce bias in the score calculation. The design focuses on preserving task-relevant relationships rather than reconstructing every source number exactly.

Google reports experiments across LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, and L-Eval with open-source Gemma and Mistral models. The announcement says the discussed needle-in-a-haystack results reduced KV memory by at least 6x. It also reports up to 8x performance for attention-logit calculation with a 4-bit configuration over 32-bit unquantized keys on H100 accelerators.

GPTQ: One-Shot Weight Quantization

The GPTQ paper describes GPTQ as a one-shot weight quantization method based on approximate second-order information. Its goal is to reduce the storage and computation burden of large generative models while retaining useful model behavior.

GPTQ operates on weights rather than the KV cache. The cited paper reports experiments at 3 or 4 bits per weight for large GPT models and says the method can preserve accuracy relative to an uncompressed baseline in those settings. The paper also reports a 175 billion parameter experiment and a specific approximate four GPU hour quantization time.

Those numbers describe the paper's experiments. They do not mean every model takes the same time or every deployment receives the same quality result. The hardware, model architecture, weight layout, group size, kernel, and evaluation set all affect the outcome.

GPTQ is useful when the main constraint is loading model weights into available memory. It does not by itself solve the growth of the KV cache during long-context generation. A deployment may need a separate cache strategy if active requests, context length, or batch size become the main limit.

AWQ: Activation-Aware Weight Quantization

The AWQ paper describes Activation-aware Weight Quantization as a hardware-friendly, low-bit, weight-only method. Its central idea is that not all weights matter equally to the final output. The method uses activation distributions to identify salient weight channels and protect them through an equivalent scaling transformation.

The AWQ abstract says activation statistics are collected offline. It also says AWQ does not rely on backpropagation or reconstruction. That makes it different from a claim that no data is used. The method does not use a retraining loop, but it does use statistics from sample inputs to guide the weight transformation.

The paper reports a TinyChat implementation for 4-bit on-device LLM and vision-language-model inference. It reports more than 3x speedup over a Hugging Face FP16 implementation on desktop and mobile GPUs in the cited experiments. This is a paper-specific result for its implementation and hardware comparisons.

Calibration, Statistics, and No Retraining

Calibration is a broad term. In weight quantization, it can mean running representative inputs to collect activation ranges, estimate scales, or choose parameters that reduce reconstruction error. A method may avoid gradient updates while still using sample data.

The GPTQ paper describes a one-shot process. AWQ describes offline activation statistics and a scaling transformation. TurboQuant is described by Google as a KV cache method that can quantize the cache without training or fine-tuning. These statements are compatible because the methods operate on different objects and stages.

The legacy claim that GPTQ and AWQ both always require the same calibration dataset should not be repeated without qualification. The papers describe data and statistics in different ways. A production tool may also add its own calibration or conversion step. Engineers should inspect the actual implementation and configuration rather than infer requirements from a method label.

Weight Memory Versus KV Cache Memory

Weight memory is mostly present before the first request. If the model weights are too large for the device, a weight quantizer can reduce the loading requirement. GPTQ and AWQ are relevant to this problem because their cited papers focus on weight quantization.

KV cache memory grows as tokens are processed. It can become the larger constraint for long prompts, long conversations, retrieval-heavy applications, and high concurrency. TurboQuant is relevant to this problem because its Google Research description focuses on KV cache compression.

A server may benefit from both. It can load a model with a weight quantizer and compress the cache while requests run. The combined system needs separate quality tests because errors in weights and errors in cached attention data can interact.

For a broader AI infrastructure view, read our PolarQuant and QJL explainer. Our earlier TurboQuant analysis covers the geometric and residual stages in more detail.

Accuracy Should Be Measured by Task

Weight quantization can be evaluated with perplexity, language-model benchmarks, coding tests, mathematics tests, and domain prompts. AWQ's paper discusses language modeling and domain-specific benchmarks. GPTQ's paper compares quantized models with uncompressed baselines.

KV cache compression needs different checks. Long-context retrieval can reveal whether information remains available after many tokens. The system can also compare answer quality, attention scores, latency, and memory. A result that looks strong on a short prompt may not apply to a long request.

Vector search uses another measure. Google describes the 1@k recall ratio for a GloVe experiment. Search teams should report the index size, recall, query latency, and update behavior together. A smaller index that loses the nearest result may not fit the product's requirement.

Speed Claims Need a Fixed Baseline

GPTQ's paper reports end-to-end inference speedups in specific GPU settings. AWQ's paper reports a TinyChat speed result for its implementation. Google Research reports an operation-level attention-logit performance result for a 4-bit TurboQuant configuration on H100 hardware.

These are not the same measurement. End-to-end latency includes scheduling, prompt processing, model layers, sampling, memory movement, and output transfer. An attention-logit kernel is one part of that path. A fair comparison must state the hardware, precision baseline, batch, sequence length, kernel version, and whether the figure is per-operation or end-to-end.

When to Use Each Approach

Use a weight quantizer when model weights are the immediate memory problem. GPTQ is a candidate when a one-shot weight conversion fits the model and evaluation process. AWQ is a candidate when activation-aware channel protection and a hardware-friendly weight-only path fit the target runtime.

Use KV cache compression when context memory or active-request capacity is the main problem. TurboQuant is a candidate for testing when the serving stack can support its compressed cache and score calculation. Its reported results should be treated as a starting point for a workload-specific experiment.

Use more than one method only after measuring the combined behavior. Weight quantization may change the baseline quality. Cache compression may change long-context behavior. The two effects should be tested separately before they are combined.

Our AI agent ROI guide can help frame infrastructure tests around service outcomes. The AI builder guide covers application-level choices that may influence traffic and context patterns.

How to Run a Fair Comparison

Fix the model and workload first. Use the same prompts, context lengths, batch shapes, output limits, and evaluation set. Record the original precision format and every metadata choice that changes memory.

Measure four results. Measure quality against the original model. Measure peak weight and cache memory separately. Measure per-token and end-to-end latency. Measure failure and fallback behavior for unsupported sequence shapes or quality thresholds.

For calibration-based weight methods, document the sample data and its relationship to production traffic. For TurboQuant, document the cache format, bit width, transform, estimator, and kernel. This makes the comparison reproducible and prevents a method-specific result from being presented as a general property.

Calibration Happens at a Different Layer

Calibration and conversion should be documented as part of the deployment record. Weight methods may use sample inputs to select scales or protect salient channels. KV cache compression operates after the model begins processing a request. The two stages can share a test set, but they do not share the same data object or failure mode.

For model selection context, read our AI model comparison guide. It covers model-level tradeoffs that remain separate from quantization method choice.

Conclusion: Choose by the Bottleneck

TurboQuant, GPTQ, and AWQ are not direct substitutes. GPTQ and AWQ address model-weight compression. TurboQuant addresses KV cache and vector data according to Google's description. A comparison that ignores the target object can produce a misleading answer even when every individual fact sounds reasonable.

GPTQ offers a one-shot weight quantization approach based on approximate second-order information. AWQ uses activation-aware protection for salient weight channels and an offline statistics step. TurboQuant uses PolarQuant and QJL to target compact cache and vector representations without model training or fine-tuning in the cited description.

The right choice depends on the bottleneck. If the model does not fit, test weight quantization. If the model fits but long-context concurrency does not, test cache compression. If the product is a search index, measure recall and query cost. In every case, compare the real workload rather than relying on a headline bit width.