Training ControlNet, exclude text caption

Anyone know how to to finetune ControlNet on image-to-image tasks, without considering the text prompt at all during the training process? I.e. for image restoration tasks like super resolution/denoising

I know at inference time, we can reduce the unconditional_guidance_scale, and at training time we could just pass in an empty string as a workaround, but I figured the optimal solution would involve completely excluding CLIP from the finetuning process