Cudnn error when using SPADE

Hello, I am trying to use SPADE on a custom dataset. Here the command I run: python train.py --name myTest --dataset_mode custom --label_dir datasets/label_dir --image_dir datasets/image_dir --no_instance --gpu_ids 0,1 --batchSize 2

However, after the creation of the web directory, I get this error:

/pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [21,0,0] Assertion indexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [23,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [24,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [25,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [26,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [27,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [28,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [29,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [30,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [31,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [278,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [279,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [280,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [281,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [282,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [283,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [284,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [285,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [286,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [287,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]` failed. Traceback (most recent call last): File "train.py", line 40, in trainer.run_generator_one_step(data_i) File "/home/emanuele/repos/testSPADE/SPADE/trainers/pix2pix_trainer.py", line 35, in run_generator_one_step g_losses, generated = self.pix2pix_model(data, mode='generator') File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward outputs = self.parallel_apply(replicas, inputs, kwargs) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply raise output File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker output = module(*input, **kwargs) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 46, in forward input_semantics, real_image) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 137, in compute_generator_loss input_semantics, real_image, compute_kld_loss=self.opt.use_vae) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 196, in generate_fake fake_image = self.netG(input_semantics, z=z) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/emanuele/repos/testSPADE/SPADE/models/networks/generator.py", line 89, in forward x = self.fc(x) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward self.padding, self.dilation, self.groups) RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED terminate called after throwing an instance of 'c10::Error' what(): CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:564) frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f25a3999441 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so) frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f25a3998d7a in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so) frame #2: + 0x13652 (0x7f25a2f45652 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10_cuda.so) frame #3: c10::TensorImpl::release_resources() + 0x50 (0x7f25a3989ce0 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so) frame #4: + 0x30facb (0x7f2573acdacb in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch.so.1) frame #5: + 0x1420bb (0x7f25a3f270bb in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #6: + 0x3c30f4 (0x7f25a41a80f4 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #7: + 0x3c3141 (0x7f25a41a8141 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #24: __libc_start_main + 0xe7 (0x7f25a8297b97 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped) ` I am using python 3.6, CUDA 10.0, Torch 1.1.0, CuDNN 7.5.0 I have also tried on another machine with CUDA 9.0, Torch 1.0.0, CuDNN 7.4.1, but I got the same error.

Could you help me?