I tried many times but couldn't get this tool to work.
Author: luyangliu616Created Jun 3, 2026Updated Jun 25, 2026
I'm trying to use this tool with the command PYTHONPATH="/mnt/beegfs_lfs3/training/riderapi/yangliulu/program/heretic/src:$PYTHONPATH" CUDA_VISIBLE_DEVICES=2,3,1,0 heretic --model /mnt/beegfs_lfs3/training/riderapi/yangliulu/model/Qwen3.6-27B I tried asking Claude to help resolve it, but its suggestions just kept getting wilder and further off track. Did I enter the command incorrectly?
Detected 4 CUDA device(s) (177.60 GB total VRAM)
CUDA Version: 12.8
Driver Version: 570.148.08
* CUDA 0: NVIDIA GeForce RTX 4090 X (44.40 GB)
* CUDA 1: NVIDIA GeForce RTX 4090 X (44.40 GB)
* CUDA 2: NVIDIA GeForce RTX 4090 X (44.40 GB)
* CUDA 3: NVIDIA GeForce RTX 4090 X (44.40 GB)
Loading model /mnt/beegfs_lfs3/training/riderapi/yangliulu/model/Qwen3.6-27B...
* Trying dtype auto...
Loading weights: 100%|██████████████████████████████████████████████████████████████| 1184/1184 [03:02<00:00, 6.48it/s]
* LoRA adapters initialized (target types: down_proj, o_proj, out_proj)
* Transformer model with 64 layers
* Abliterable components:
* attn.o_proj: 64 modules total
* mlp.down_proj: 64 modules total
Resident system RAM: 2.46 GB
Allocated GPU VRAM: 51.01 GB
Reserved GPU VRAM: 51.59 GB
Loading good prompts from mlabonne/harmless_alpaca...
README.md: 100%|███████████████████████████████████████████████████████████████████████| 388/388 [00:00<00:00, 1.14MB/s]
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
data/train-00000-of-00001.parquet: 100%|██████████████████████████████████████████████| 972k/972k [00:03<00:00, 292kB/s]
data/test-00000-of-00001.parquet: 100%|███████████████████████████████████████████████| 243k/243k [00:02<00:00, 101kB/s]
Generating train split: 100%|██████████████████████████████████████████| 25058/25058 [00:00<00:00, 419909.66 examples/s]
Generating test split: 100%|████████████████████████████████████████████| 6265/6265 [00:00<00:00, 1400485.77 examples/s]
* 400 prompts loaded
Loading bad prompts from mlabonne/harmful_behaviors...
README.md: 100%|███████████████████████████████████████████████████████████████████████| 381/381 [00:00<00:00, 1.63MB/s]
data/train-00000-of-00001.parquet: 100%|███████████████████████████████████████████| 15.0k/15.0k [00:00<00:00, 23.8kB/s]
data/test-00000-of-00001.parquet: 100%|████████████████████████████████████████████| 5.46k/5.46k [00:00<00:00, 7.57kB/s]
Generating train split: 100%|███████████████████████████████████████████████| 416/416 [00:00<00:00, 99562.37 examples/s]
Generating test split: 100%|████████████████████████████████████████████████| 104/104 [00:00<00:00, 39030.75 examples/s]
* 400 prompts loaded
Determining optimal batch size...
* Trying batch size 1... Ok (7 tokens/s)
* Trying batch size 2... Ok (14 tokens/s)
* Trying batch size 4... Ok (23 tokens/s)
* Trying batch size 8... Ok (40 tokens/s)
* Trying batch size 16... Ok (59 tokens/s)
* Trying batch size 32... Ok (81 tokens/s)
* Trying batch size 64... Ok (218 tokens/s)
* Trying batch size 128...
Ok (118 tokens/s)
* Chosen batch size: 64
Checking for common response prefix...
* None found
Loading good evaluation prompts from mlabonne/harmless_alpaca...
* 100 prompts loaded
* Obtaining first-token probability distributions...
Loading bad evaluation prompts from mlabonne/harmful_behaviors...
* 100 prompts loaded
* Counting model refusals...
* Initial refusals: 7/100
Calculating per-layer refusal directions...
* Obtaining residual mean for good prompts...
* Obtaining residual mean for bad prompts...
Running trial 1 of 200...
* Parameters:
* direction_index = 33.53
* attn.o_proj.max_weight = 1.42
* attn.o_proj.max_weight_position = 50.65
* attn.o_proj.min_weight = 0.10
* attn.o_proj.min_weight_distance = 16.63
* mlp.down_proj.max_weight = 1.42
* mlp.down_proj.max_weight_position = 43.96
* mlp.down_proj.min_weight = 0.47
* mlp.down_proj.min_weight_distance = 11.24
* Resetting model...
* Abliterating...
* Evaluating...
* Obtaining first-token probability distributions...
* KL divergence: 75.5001
* Counting model refusals...
* Refusals: 7/100
Elapsed time: 1m 56s
Estimated remaining time: 6h 24m
Resident system RAM: 2.89 GB
Allocated GPU VRAM: 51.01 GB
Reserved GPU VRAM: 61.21 GB
Running trial 2 of 200...
* Parameters:
* direction_index = 30.51
* attn.o_proj.max_weight = 1.13
* attn.o_proj.max_weight_position = 39.72
* attn.o_proj.min_weight = 0.95
* attn.o_proj.min_weight_distance = 9.01
* mlp.down_proj.max_weight = 0.83
* mlp.down_proj.max_weight_position = 47.27
* mlp.down_proj.min_weight = 0.24
* mlp.down_proj.min_weight_distance = 30.96
* Resetting model...
* Abliterating...
/usr/local/lib/python3.12/dist-packages/torch/_lowrank.py:173: UserWarning: torch.linalg.svd: During SVD computation with the selected cusolver driver, batches 0 failed to converge. A more accurate method will be used to compute the SVD as a fallback. Check doc at https://pytorch.org/docs/stable/generated/torch.linalg.svd.html (Triggered internally at /pytorch/aten/src/ATen/native/cuda/linalg/BatchLinearAlgebraLib.cpp:701.)
U, S, Vh = torch.linalg.svd(B, full_matrices=False)
[W 2026-06-03 08:40:07,020] Trial 1 failed with parameters: {'direction_scope': 'global', 'direction_index': 30.51295033937055, 'attn.o_proj.max_weight': 1.1328905616711376, 'attn.o_proj.max_weight_position': 39.722341130803464, 'attn.o_proj.min_weight': 0.839996270153127, 'attn.o_proj.min_weight_distance': 9.008620870631614, 'mlp.down_proj.max_weight': 0.8261386972769816, 'mlp.down_proj.max_weight_position': 47.274510713738046, 'mlp.down_proj.min_weight': 0.28876575830584117, 'mlp.down_proj.min_weight_distance': 30.957588075713897} because of the following error: _LinAlgError('linalg.svd: The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated singular values (error code: 10).').
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py", line 206, in _run_trial
value_or_values = func(trial)
^^^^^^^^^^^
File "/mnt/beegfs_lfs3/training/riderapi/yangliulu/program/heretic/src/heretic/main.py", line 594, in objective_wrapper
return objective(trial)
^^^^^^^^^^^^^^^^
File "/mnt/beegfs_lfs3/training/riderapi/yangliulu/program/heretic/src/heretic/main.py", line 569, in objective
model.abliterate(refusal_directions, direction_index, parameters)
File "/mnt/beegfs_lfs3/training/riderapi/yangliulu/program/heretic/src/heretic/model.py", line 539, in abliterate
U, S, Vh = torch.svd_lowrank(W, q=2 * r + 4, niter=6)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/_lowrank.py", line 146, in svd_lowrank
return _svd_lowrank(A, q=q, niter=niter, M=M)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/_lowrank.py", line 173, in _svd_lowrank
U, S, Vh = torch.linalg.svd(B, full_matrices=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch._C._LinAlgError: linalg.svd: The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated singular values (error code: 10).
[W 2026-06-03 08:40:07,029] Trial 1 failed with value None.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /usr/local/bin/heretic:7 in <module> │
│ │
│ 4 if __name__ == '__main__': │
│ 5 │ if sys.argv[0].endswith('.exe'): │
│ 6 │ │ sys.argv[0] = sys.argv[0][:-4] │
│ ❱ 7 │ sys.exit(main()) │
│ 8 │
│ │
│ /mnt/beegfs_lfs3/training/riderapi/yangliulu/program/heretic/src/heretic/main.py:1125 in main │
│ │
│ 1122 │ install() │
│ 1123 │ │
│ 1124 │ try: │
│ ❱ 1125 │ │ run() │
│ 1126 │ except BaseException as error: │
│ 1127 │ │ # Transformers appears to handle KeyboardInterrupt (or BaseException) │
│ 1128 │ │ # internally in some places, which can re-raise a different error in the handler │
│ │
│ /mnt/beegfs_lfs3/training/riderapi/yangliulu/program/heretic/src/heretic/main.py:626 in run │
│ │
│ 623 │ │ print("Resuming existing study.") │
│ 624 │ │
│ 625 │ try: │
│ ❱ 626 │ │ study.optimize( │
│ 627 │ │ │ objective_wrapper, │
│ 628 │ │ │ n_trials=settings.n_trials - count_completed_trials(), │
│ 629 │ │ ) │
│ │
│ /usr/local/lib/python3.12/dist-packages/optuna/study/study.py:490 in optimize │
│ │
│ 487 │ │ │ RuntimeError: │
│ 488 │ │ │ │ If nested invocation of this method occurs. │
│ 489 │ │ """ │
│ ❱ 490 │ │ _optimize( │
│ 491 │ │ │ study=self, │
│ 492 │ │ │ func=func, │
│ 493 │ │ │ n_trials=n_trials, │
│ │
│ /usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py:68 in _optimize │
│ │
│ 65 │ │
│ 66 │ try: │
│ 67 │ │ if n_jobs == 1: │
│ ❱ 68 │ │ │ _optimize_sequential( │
│ 69 │ │ │ │ study, │
│ 70 │ │ │ │ func, │
│ 71 │ │ │ │ n_trials, │
│ │
│ /usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py:165 in _optimize_sequential │
│ │
│ 162 │ │ │ │ break │
│ 163 │ │ │
│ 164 │ │ try: │
│ ❱ 165 │ │ │ frozen_trial_id = _run_trial(study, func, catch) │
│ 166 │ │ finally: │
│ 167 │ │ │ # The following line mitigates memory problems that can be occurred in some │
│ 168 │ │ │ # environments (e.g., services that use computing containers such as GitHub │
│ │
│ /usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py:263 in _run_trial │
│ │
│ 260 │ │ and func_err is not None │
│ 261 │ │ and not isinstance(func_err, catch) │
│ 262 │ ): │
│ ❱ 263 │ │ raise func_err │
│ 264 │ return trial._trial_id │
│ 265 │
│ 266 │
│ │
│ /usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py:206 in _run_trial │
│ │
│ 203 │ │
│ 204 │ with get_heartbeat_thread(trial._trial_id, study._storage): │
│ 205 │ │ try: │
│ ❱ 206 │ │ │ value_or_values = func(trial) │
│ 207 │ │ except exceptions.TrialPruned as e: │
│ 208 │ │ │ # TODO(mamu): Handle multi-objective cases. │
│ 209 │ │ │ state = TrialState.PRUNED │
│ │
│ /mnt/beegfs_lfs3/training/riderapi/yangliulu/program/heretic/src/heretic/main.py:594 in │
│ objective_wrapper │
│ │
│ 591 │ │
│ 592 │ def objective_wrapper(trial: Trial) -> tuple[float, float]: │
│ 593 │ │ try: │
│ ❱ 594 │ │ │ return objective(trial) │
│ 595 │ │ except KeyboardInterrupt: │
│ 596 │ │ │ # Stop the study gracefully on Ctrl+C. │
│ 597 │ │ │ trial.study.stop() │
│ │
│ /mnt/beegfs_lfs3/training/riderapi/yangliulu/program/heretic/src/heretic/main.py:569 in │
│ objective │
│ │
│ 566 │ │ print("* Resetting model...") │
│ 567 │ │ model.reset_model() │
│ 568 │ │ print("* Abliterating...") │
│ ❱ 569 │ │ model.abliterate(refusal_directions, direction_index, parameters) │
│ 570 │ │ print("* Evaluating...") │
│ 571 │ │ score, kl_divergence, refusals = evaluator.get_score() │
│ 572 │
│ │
│ /mnt/beegfs_lfs3/training/riderapi/yangliulu/program/heretic/src/heretic/model.py:539 in │
│ abliterate │
│ │
│ 536 │ │ │ │ │ │ W = W - W_org │
│ 537 │ │ │ │ │ │ # Use a low-rank SVD to get an approximation of the matrix. │
│ 538 │ │ │ │ │ │ r = self.peft_config.r │
│ ❱ 539 │ │ │ │ │ │ U, S, Vh = torch.svd_lowrank(W, q=2 * r + 4, niter=6) │
│ 540 │ │ │ │ │ │ # Truncate it to the part we want to store in the LoRA adapter. │
│ 541 │ │ │ │ │ │ # Note: svd_lowrank actually returns V, so transpose it to get V │
│ 542 │ │ │ │ │ │ U = U[:, :r] │
│ │
│ /usr/local/lib/python3.12/dist-packages/torch/_lowrank.py:146 in svd_lowrank │
│ │
│ 143 │ │ │ return handle_torch_function( │
│ 144 │ │ │ │ svd_lowrank, tensor_ops, A, q=q, niter=niter, M=M │
│ 145 │ │ │ ) │
│ ❱ 146 │ return _svd_lowrank(A, q=q, niter=niter, M=M) │
│ 147 │
│ 148 │
│ 149 def _svd_lowrank( │
│ │
│ /usr/local/lib/python3.12/dist-packages/torch/_lowrank.py:173 in _svd_lowrank │
│ │
│ 170 │ B = matmul(Q.mH, A) │
│ 171 │ iSource: p-e-w/heretic