Training usually stores each weight as a 16-bit or 32-bit floating point number. Quantization rewrites those weights at lower precision, commonly 8-bit or 4-bit integers, and keeps a scaling factor so the original range can be approximated.
The gain is mechanical. Halving the bits per weight roughly halves the memory the model occupies, and smaller weights move through memory bandwidth faster, which is the real bottleneck in most serving setups. A 70-billion-parameter model that needs two high-end GPUs at 16-bit often fits on one at 4-bit.
Quality does drop. How much depends on the method. Post-training quantization converts an already trained model in minutes and is usually good enough for chat workloads. Quantization-aware training simulates the lower precision during fine-tuning and holds accuracy better, at the cost of a training run.
The honest way to choose is to measure. Run your own evaluation set at each precision level and look at where the curve bends, because published benchmarks rarely match the task you actually ship.