#365·heretic

Trying to use Heretic to abliterate Gemma 4 31B QAT

Author: Roman215Created Jun 9, 2026Updated Jun 30, 2026

So this is my first attempt at using heretic, and given that I didn't particularly find any good abliterated alternatives of the new QAT models yet, I wanted to take it upon myself. Initially I tried running the normal command like so:

heretic google/gemma-4-31B-it-qat-q4_0-unquantized

I'm using bitsandbytes because I only have an RTX 3090 so without Q4 quantization I would have no hope at abliterating a 31B model. This runs fine in concept, but I set a max of 2000 trials and over 2-3 days it finished 1000 of them. The issue is that the best outcome I have so far still has around 63 refusals with a divergence of around 0.02, which is "fine" but I'm aiming for less than 10 refusals like the professional abliterated models I see.

I read that the ARA method is better for abliterating newer models like Gemma 4. So I checked out the ara branch. It seems like if I run that branch with bitsandbytes set, I get the following exception:

RuntimeError: linalg.vector_norm: Expected a floating point or complex tensor as input. Got Byte

I did some digging, and it seems like the bitsandbytes support was added, but then I assume never added to the ara branch's ara_abliterate method? As mentioned above, without bitsandbytes, I would have no hope of abliterating the 31B model. Any plans to update the ARA branch with bitsandbytes support? Or am I using the plain abliteration wrong if even after 1000 trials the best outcome still has 63 refusals?