Make the model run on lower GPU at the expense of time/speed

The feature, motivation and pitch

Add command-line flags (e.g., --max-model-len) that allow users to: Manually cap the context window (e.g., to 4096 or 8192) to fit within smaller VRAM buffers.

Trade off processing speed or context size to allow the model to run on 12GB, 10GB, or 8GB GPUs.

This would make the tool accessible to a much wider range of researchers and hobbyists who are running local workstations rather than server-grade hardware.

Alternatives

No response

Additional context

No response