Qwen-Image text to image lora trainer
An open-source implementation for training LoRA (Low-Rank Adaptation) layers for Qwen/Qwen-Image, Qwen/Qwen-Image-Edit, and FLUX.1-dev models by FlyMy.AI.
Agentic Infra for GenAI. FlyMy.AI is a B2B infrastructure for building and running GenAI Media agents.
** Useful Links:**
diffusers16.10.2025
02.09.2025
20.08.2025
09.08.2025
08.08.2025
utils/validate_dataset.py)** Under Development:** We are actively working on improving the code and adding test coverage. The project is in the refinement stage but ready for use.
** Development Plans:**
Requirements:
Clone the repository and navigate into it:
git clone https://github.com/FlyMyAI/flymyai-lora-trainer
cd flymyai-lora-trainerInstall required packages:
pip install -r requirements.txtInstall the latest diffusers from GitHub:
pip install git+https://github.com/huggingface/diffusersDownload pre-trained LoRA weights (optional):
# Qwen LoRA weights
git clone https://huggingface.co/flymy-ai/qwen-image-realism-lora
# FLUX LoRA weights
git clone https://huggingface.co/flymy-ai/flux-dev-anne-hathaway-lora
# Or download specific files
wget https://huggingface.co/flymy-ai/qwen-image-realism-lora/resolve/main/flymy_realism.safetensors
wget https://huggingface.co/flymy-ai/flux-dev-anne-hathaway-lora/resolve/main/pytorch_lora_weights.safetensorsThe training data should follow the same format for both Qwen and FLUX models, where each image has a corresponding text file with the same name:
dataset/
├── img1.png
├── img1.txt
├── img2.jpg
├── img2.txt
├── img3.png
├── img3.txt
└── ...For control-based image editing, the dataset should be organized with separate directories for target images/captions and control images:
dataset/
├── images/ # Target images and their captions
│ ├── image_001.jpg
│ ├── image_001.txt
│ ├── image_002.jpg
│ ├── image_002.txt
│ └── ...
└── control/ # Control images
├── image_001.jpg
├── image_002.jpg
└── ...my_training_data/
├── portrait_001.png
├── portrait_001.txt
├── landscape_042.jpg
├── landscape_042.txt
├── abstract_design.png
├── abstract_design.txt
└── style_reference.jpg
└── style_reference.txtFor FLUX character training (portrait_001.txt):
ohwx woman, professional headshot, studio lighting, elegant pose, looking at cameraFor Qwen landscape training (landscape_042.txt):
Mountain landscape at sunset, dramatic clouds, golden hour lighting, wide angle viewFor FLUX portrait training (abstract_design.txt):
ohwx woman, modern portrait style, soft lighting, artistic compositionYou can verify your data structure using the included validation utility:
python utils/validate_dataset.py --path path/to/your/datasetThis will check that:
To begin training with your configuration file (e.g., train_lora_4090.yaml), run:
accelerate launch train_4090.py --config ./train_configs/train_lora_4090.yamlTo begin training with your configuration file (e.g., train_lora.yaml), run:
accelerate launch train.py --config ./train_configs/train_lora.yamlMake sure train_lora.yaml is correctly set up with paths to your dataset, model, output directory, and other parameters.
To begin training with your configuration file (e.g., train_full_qwen_image.yaml), run:
accelerate launch train_full_qwen_image.py --config ./train_configs/train_full_qwen_image.yamlMake sure train_full_qwen_image.yaml is correctly set up with paths to your dataset, model, output directory, and other parameters.
The proposed method was tested on an NVIDIA H200 GPU environment.
After training, you can load your trained model from the checkpoint directory for inference.
Simple Example:
…Complete Example Script:
python inference_trained_model_gpu_optimized.pyCheckpoint Structure:
The trained model is saved in the following structure:
checkpoint/
├── config.yaml # Training configuration
└── transformer/ # Trained transformer weights
├── config.json
├── diffusion_pytorch_model.safetensors.index.json
└── diffusion_pytorch_model-00001-of-00005.safetensors
└── ... (multiple shard files)For control-based image editing training, use the specialized training script:
accelerate launch train_qwen_edit_lora.py --config ./train_configs/train_lora_qwen_edit.yamlThe configuration file train_lora_qwen_edit.yaml should include:
img_dir: Path to target images and captions directory (e.g., ./extracted_dataset/train/images)control_dir: Path to control images directory (e.g., ./extracted_dataset/train/control)To begin training with your configuration file (e.g., train_full_qwen_edit.yaml), run:
accelerate launch train_full_qwen_edit.py --config ./train_configs/train_full_qwen_edit.yamlfrom diffusers import DiffusionPipeline
import torch
model_name = "Qwen/Qwen-Image"
# Load the pipeline
if torch.cuda.is_available():
torch_dtype = torch.bfloat16
device = "cuda"
else:
torch_dtype = torch.float32
device = "cpu"
pipe = DiffusionPipeline.from_pretrained(model_name, torch_dtype=torch_dtype)
pipe = pipe.to(device)from diffusers import QwenImageEditPipeline
import torch
from PIL import Image
# Load the pipeline
pipeline = QwenImageEditPipeline.from_pretrained("Qwen/Qwen-Image-Edit")
pipeline.to(torch.bfloat16)
pipeline.to("cuda")For Qwen-Image:
# Load LoRA weights
pipe.load_lora_weights('flymy-ai/qwen-image-realism-lora', adapter_name="lora")For Qwen-Image-Edit:
# Load trained LoRA weights
pipeline.load_lora_weights("/path/to/your/trained/lora/pytorch_lora_weights.safetensors")You can find LoRA weights here
No trigger word required
……Input Image:
Prompt: "Make a shot in the same scene of the left hand securing the edge of the cutting board while the right hand tilts it, causing the chopped tomatoes to slide off into the pan, camera angle shifts slightly to the left to center more on the pan."
Output without LoRA:
Output with LoRA:
FLUX.1-dev is a powerful text-to-image model that excels at generating high-quality portraits and character images. Our LoRA training implementation allows you to fine-tune FLUX for specific characters or styles.
To begin FLUX LoRA training with your configuration file, run:
accelerate launch train_flux_lora.py --config ./train_configs/train_flux_config.yamlMake sure train_flux_config.yaml is correctly set up with paths to your dataset, model, output directory, and other parameters.
from diffusers import DiffusionPipeline
import torch
model_name = "black-forest-labs/FLUX.1-dev"
# Load the pipeline
if torch.cuda.is_available():
torch_dtype = torch.bfloat16
device = "cuda"
else:
torch_dtype = torch.float32
device = "cpu"
pipe = DiffusionPipeline.from_pretrained(model_name, torch_dtype=torch_dtype)
pipe = pipe.to(device)# Load LoRA weights
pipe.load_lora_weights('flymy-ai/flux-dev-anne-hathaway-lora', adapter_name="lora")You can find our pre-trained FLUX LoRA weights here
Trigger word required: "ohwx woman"
…Below are examples of images generated with our FLUX Anne Hathaway LoRA model:
Prompt: "ohwx woman portrait selfie"
Prompt: "ohwx woman perfectly symmetrical young female face close-up, presented with double exposure overlay blending nature textures like leaves and water"
Prompt: "ohwx woman Macro photography style close-up of female face with light makeup, focused on eyes and lips, illuminated by golden hour sunlight for warm tones"
No open issues yet, or sync has not completed.