用于超大型系统的统一内存 - 重物质化问题

Hi, I'm trying to compute extra large systems (more than 5.120 residues) and as advised in #31 I tried to use unified memory. I get this kind of error during the inference (details in out.txt): 2025-05-27 11:26:29.427895: W external/xla/xla/service/hlo_rematerialization.cc:3005] Can't reduce memory use below -20.61GiB (-22125470549 bytes) by rematerialization; only reduced to 28.35GiB (30443270400 bytes), down from 28.35GiB (30443270400 bytes) originally I have tried disabling rematerialization, and am waiting for the job to finish computing. This is the specifics of the node where this job is running, this is one of the usual nodes I use (sorry for bad formating): +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | | | MIG M. | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | MIG M. | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf Pwr:Usage/Cap | | | | Memory-Usage | | | | GPU-Util Compute M. | | | | Memory-Usage | | | | GPU Temp Perf P