gemma-tuner-multimodal

Gemma Multimodal Fine-Tuner

Fine-tune Gemma on text, images, and audio — on your Mac, on data that doesn't fit on your Mac.

• ️ Image + text LoRA — captioning and VQA on local CSV.
• ️ Audio + text LoRA — Apple-Silicon-native, no CUDA required.
• Text-only LoRA — instruction or completion on CSV.
• ☁️ Stream from GCS / BigQuery — train on terabytes without filling your SSD.
• Runs on Apple Silicon — MPS-native, no NVIDIA box required.

Source: github.com/mattmireles/gemma-tuner-multimodal (public).

Watch your model learn

Loss curve. Attention heatmap. Gradient signal strength. Memory pressure. Token-by-token predictions — all updating in real time, in your browser, while the model trains on your Mac. No TensorBoard. No notebook. One flag in your config, one URL in your terminal.

→ Setup takes 30 seconds

LoRA for Gemma 4 & 3n — why not just use…?

Fine-tune Gemma on text, images, or audio without renting an H100 or copying a terabyte of data to your laptop. All three modalities run on Apple Silicon.

Text-only fine-tuning (instruction or completion on CSV) is supported: set modality = text in your profile and use local CSV splits under data/datasets/<name>/. See Text-only fine-tuning below.

Image + text fine-tuning (captioning or VQA on local CSV) uses modality = image, image_sub_mode, and image_token_budget; see Image fine-tuning below. v1 is local CSV only (same constraint as text-only).

How it works: Hugging Face Gemma checkpoints + PEFT LoRA, supervised fine-tuning in gemma_tuner/models/gemma/finetune.py, exported as a merged HF / SafeTensors tree by gemma_tuner/scripts/export.py. For Core ML conversion and GGUF inference tooling, see README/guides/README.md — this repo's training path is Gemma-only by design.

Deeper reading: README/guides/README.md · README/specifications/Gemma3n.md

What you can build with this

• Domain-specific ASR — fine-tune on medical dictation, legal depositions, call-center recordings, or any field where off-the-shelf Whisper / Gemma mishears the jargon.
• Domain-specific vision — captioning or VQA on receipts, charts, screenshots, manufacturing defects, medical imagery — any visual domain where generic models hallucinate.
• Document & screen understanding — train on screenshot → structured-output pairs for UI agents, OCR-adjacent pipelines, or chart QA.
• Accent, dialect, and low-resource language adaptation — adapt a base Gemma model to underrepresented voices and languages with your own labeled audio.
• Multimodal assistants — extend Gemma's text reasoning with image or audio grounding for transcription, captioning, and Q&A pipelines.
• Private, on-device pipelines — train and run entirely on your Mac. Data never leaves the machine; weights never touch a third-party API.

If your data lives in GCS or BigQuery, you can do all of this on a laptop without copying terabytes locally — the dataloader streams shards on demand.

Supported models

Training targets Gemma multimodal (text + image + audio) checkpoints loaded via base_model in config/config.ini and routed to gemma_tuner/models/gemma/finetune.py. The default file ships these [model:…] entries (LoRA on top of the Hub weights):

Add your own [model:your-name] section with group = gemma and a compatible base_model if you need another any-to-any Gemma 3n / Gemma 4 E2B–E4B checkpoint. Larger Gemma 4 weights on Hugging Face (for example 26B or 31B class) use a different Transformers architecture than this trainer’s AutoModelForCausalLM audio path—they are not supported here yet.

Wizard time and memory hints come from gemma_tuner/wizard/base.py (ModelSpecs).

Architecture

Run layout (typical):

output/
├── {id}-{profile}/
│   ├── metadata.json
│   ├── metrics.json
│   ├── checkpoint-*/
│   └── adapter_model/          # LoRA artifacts when applicable

Configuration: hierarchical INI—defaults, groups, models, datasets, then profiles—read by gemma_tuner/core/config.py. Set GEMMA_TUNER_CONFIG if you invoke the CLI outside the repo root.

Requirements

Installation

1. Create a Python 3.10+ virtual environment

macOS's built-in Python is 3.9, which is too old. Install a newer one with Homebrew:

brew install [email protected]

Then create and activate a virtual environment:

python3.12 -m venv .venv
source .venv/bin/activate

Every command below assumes the venv is active. To reactivate in a new terminal: source .venv/bin/activate.

2. Confirm you are on arm64 (Apple Silicon)

python -c "import platform; print(platform.machine())"
# arm64  -> good
# x86_64 -> Python is running under Rosetta; install a native arm64 Python and recreate the venv

A native arm64 Python is available from python.org or Homebrew (brew install [email protected]).

3. Install PyTorch

pip install torch torchaudio

4. Install this package

pip install -e .

5. Authenticate with Hugging Face

Gemma weights are gated. Accept the license on the model card, then either log in or export a token:

huggingface-cli login
# or:  export HF_TOKEN=hf_...

6. Gemma 4 (optional)

The base install (pip install -e .) pins Transformers ≥5.5 — both Gemma 3n and Gemma 4 families work out of the box. Gemma 4 checkpoints need a slightly newer PEFT:

pip install -r requirements/requirements-gemma4.txt

finetune and export are family-aware. A few non-training commands (gemma_generate, multimodal probing, ASR eval) still reject Gemma 4 ids until those code paths are upgraded.

7. Run the wizard

gemma-macos-tuner wizard

The wizard is the primary UI: it picks the model, walks you through dataset and hyperparameter selection, and starts training. On first run it creates config/config.ini for you from the committed config/config.ini.example template (the live config is gitignored because the wizard writes local paths and GCP project IDs into it).

If a command fails, run gemma-macos-tuner system-check first to surface environment issues.

Zero to training in 90 seconds

The repo ships a 16-row instruction-tuning dataset at data/datasets/sample-text/ — translations, summaries, trivia, haiku, JSON conversion. Small enough to finish in under a minute. Large enough to prove the full pipeline works: data loading, tokenization, LoRA, checkpointing, export.

gemma-macos-tuner wizard

Pick Instruction tuning → gemma-3n-e2b-it → sample-text, accept the defaults, and watch it train. First run downloads ~5 GB of base weights from Hugging Face (step 5 above must be done). Every run after that starts in seconds.

Or skip the wizard entirely:

gemma-macos-tuner finetune sample-text

Once the sample run finishes, drop your own CSV under data/datasets/<your-name>/ and run the wizard again — it picks up new datasets automatically.

What kind of data do I need?

All training data is CSV under data/datasets/<name>/, with one row per example and a header row. The required columns depend on the modality. Each dataset directory holds at least:

data/datasets/<name>/
├── train.csv
└── validation.csv

There is no JSONL / Parquet / Hugging Face dataset format requirement — just CSV. The column names are configurable via prompt_column, text_column, and image_path_column in your profile; the names below are the defaults used by config/config.ini.example.

Text instruction (modality = text, text_sub_mode = instruction)

id,prompt,response
1,Translate to French: Good morning.,Bonjour.
2,What is the capital of Japan?,Tokyo.

The prompt is masked from the loss; the model only learns to generate response. This is what the bundled sample-text dataset uses.

Text completion (modality = text, text_sub_mode = completion)

id,text
1,"Once upon a time, in a small village by the sea, ..."

A single text column; the full sequence is trained (no prompt mask). Useful for domain pretraining-style adaptation.

Image + text (modality = image)

id,image_path,caption
1,images/receipt_001.jpg,"Total: $42.18, paid in cash"
2,images/receipt_002.jpg,"Subtotal $19.99, tax $1.60, total $21.59"

image_path is resolved relative to the dataset directory (or an absolute path). For VQA, set image_sub_mode = vqa and use image_path,question,answer columns. See [Image fi