[Feature Request] LLaVA 1.6 LoRA fine-tuning example

Author: tctrautmanCreated Mar 21, 2024Updated May 14, 2026

Building on the amazing work by @mzbac and @nkasmanoff in https://github.com/ml-explore/mlx-examples/pull/461, I'd really love an example of how LLaVA 1.6 (aka llava next) can be fine-tuned with a LoRA.

I might be able to make progress on this myself, but it'll take me some time. Any help or thoughts on how to best approach this would be appreciated. (Especially from @mzbac and/or @nkasmanoff.)