Memory Error while loading Kinetic Dataset
Hi, I am trying to reproduce the results of the MViT model on the Kinetic dataset.
However, I encounter double free or corruption (!prev) and malloc(): invalid size (unsorted) errors while loading the Kinectic dataset
[06/01 12:11:45][INFO] kinetics.py: 88: Constructing Kinetics test... [06/01 12:11:46][INFO] kinetics.py: 152: Constructing kinetics dataloader (size: 386710 skip_rows 0) from ../tmp_slowfast/cv2_data/test.csv [06/01 12:11:46][INFO] test_net.py: 211: Testing model for 6043 iterations double free or corruption (!prev) malloc(): invalid size (unsorted) double free or corruption (!prev) Traceback (most recent call last): File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 986, in _try_get_data data = self._data_queue.get(timeout=timeout) File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/queue.py", line 179, in get self.not_empty.wait(remaining) File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/threading.py", line 306, in wait gotit = waiter.acquire(True, timeout) File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler _error_if_any_worker_fails() RuntimeError: DataLoader worker (pid 4035577) is killed by signal: Aborted.
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "tools/run_net.py", line 45, in main() File "tools/run_net.py", line 30, in main launch_job(cfg=cfg, init_method=args.init_method, func=test) File "/home/tduy/Workspace/git/newest_slowfast/slowfast/utils/misc.py", line 311, in launch_job func(cfg=cfg) File "/home/tduy/Workspace/git/newest_slowfast/tools/test_net.py", line 244, in test test_meter = perform_test(test_loader, model, test_meter, cfg, writer) File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/home/tduy/Workspace/git/newest_slowfast/tools/test_net.py", line 50, in perform_test for cur_iter, (inputs, labels, video_idx, time, meta) in enumerate( File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 517, in next data = self._next_data() File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1182, in _next_data idx, data = self._get_data() File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1138, in _get_data success, data = self._try_get_data() File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 999, in _try_get_data raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e RuntimeError: DataLoader worker (pid(s) 4035577) exited unexpectedly
It seems that the error is related to the buffer allocation in the decode function inside the kinetic.py file.
Does anybody know how to fix this?
Source: facebookresearch/SlowFast