tf.raw.ops.ResourceSparseApplyAdadelta aborts after reporting InvalidArgument on malformed sparse gradient input
Author: DeLightorCreated Mar 23, 2026Updated Sep 17, 2026
Labelsstat:awaiting responsetype:bugcomp:opscomp:gpu2.21.0
Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
No
Source
source
TensorFlow version
tf 2.21.0
Custom code
Yes
OS platform and distribution
Ubuntu 24.04.3 LTS x86_64
Mobile device
No response
Python version
3.10.20
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
4 x NVIDIA GeForce RTX 2080 Ti, 11264 MiB each
Current behavior?
Current behavior
I verified this issue locally by running the original reproducer in the tf_test conda environment.
tf.raw_ops.ResourceSparseApplyAdadelta does not fail cleanly on malformed sparse gradient input. TensorFlow first reports a normal argument validation failure:
INVALID_ARGUMENT: var and grad must match in dimension 1
but then continues into a fatal internal CHECK failure and aborts the entire process:
F0000 tensor_shape.cc:362] Check failed: d < dims() (1 vs. 1)
The process terminates with Aborted (core dumped) / exit code 134.
This is a strong robustness bug because invalid user input should produce a recoverable exception only. After detecting an InvalidArgument, TensorFlow should not continue into an internal CHECK abort path.
Expected behavior
tf.raw_ops.ResourceSparseApplyAdadelta should stop after reporting the invalid input and return a regular TensorFlow exception such as InvalidArgumentError, instead of aborting the process.
Minimal reproducer
import tensorflow as tf
with tf.device('/GPU:0'):
var = tf.Variable(tf.random.uniform([10, 10], dtype=tf.float32))
accum = tf.Variable(tf.zeros([10, 10], dtype=tf.float32))
accum_update = tf.Variable(tf.zeros([10, 10], dtype=tf.float32))
lr = tf.constant(0.1, dtype=tf.float32)
rho = tf.constant(0.95, dtype=tf.float32)
epsilon = tf.constant(1e-7, dtype=tf.float32)
grad = tf.constant([[0.1], [0.2], [0.3]], dtype=tf.float32) # shape [3, 1]
indices = tf.constant([0, 2, 4], dtype=tf.int32)
tf.raw_ops.ResourceSparseApplyAdadelta(
var=var.handle,
accum=accum.handle,
accum_update=accum_update.handle,
lr=lr,
rho=rho,
epsilon=epsilon,
grad=grad,
indices=indices,
use_locking=False,
)
### Standalone code to reproduce the issue
```shell
Environment
Ubuntu 24.04.3 LTS x86_64
Python 3.10
TensorFlow 2.21.0
GPU-enabled TensorFlow runtime
Reproduced in local conda environment: tf_test
conda run --no-capture-output -n tf_test python repro.py
Relevant log output
(tf_test) user@user-Standard-PC-Q35-ICH9-2009:/home/lhj$ python /home/lhj/callChainBuild/src_tensorflow/output_tensorflow/targeted_mutation/generated_code/tf/tf_raw_ops_ResourceSparseApplyAdadelta/0.py
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1774273340.819282 3738650 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1774273340.974487 3738650 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1774273342.461486 3738650 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1774273343.513755 3738650 gpu_device.cc:2043] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 665 MB memory: -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
I0000 00:00:1774273343.515447 3738650 gpu_device.cc:2043] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 9450 MB memory: -> device: 1, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:02:00.0, compute capability: 7.5
I0000 00:00:1774273343.516891 3738650 gpu_device.cc:2043] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 9450 MB memory: -> device: 2, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:03:00.0, compute capability: 7.5
I0000 00:00:1774273343.518355 3738650 gpu_device.cc:2043] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 9450 MB memory: -> device: 3, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:04:00.0, compute capability: 7.5
W0000 00:00:1774273343.568847 3738650 local_rendezvous.cc:412] Local rendezvous is aborting with status: INVALID_ARGUMENT: var and grad must match in dimension 1
Error with standard inputs: {{function_node __wrapped__ResourceSparseApplyAdadelta_device_/job:localhost/replica:0/task:0/device:GPU:0}} var and grad must match in dimension 1 [Op:ResourceSparseApplyAdadelta] name:
F0000 00:00:1774273343.569195 3738650 tensor_shape.cc:362] Check failed: d < dims() (1 vs. 1)
*** Check failure stack trace: ***
@ 0x7d59a831d314 absl::lts_20250814::log_internal::LogMessage::SendToLog()
@ 0x7d59a831d296 absl::lts_20250814::log_internal::LogMessage::Flush()
@ 0x7d59a7f4915c tensorflow::TensorShapeBase<>::dim_size()
@ 0x7d598c64f459 tensorflow::SparseApplyAdadeltaOp<>::DoCompute()
@ 0x7d598c64f22a tensorflow::SparseApplyAdadeltaOp<>::Compute()
@ 0x7d59a83876bc tensorflow::BaseGPUDevice::Compute()
@ 0x7d59a5e0e778 tensorflow::(anonymous namespace)::SingleThreadedExecutorImpl::Run()
@ 0x7d59a5ddb99d tensorflow::FunctionLibraryRuntimeImpl::RunSync()
@ 0x7d59a5de6c26 tensorflow::ProcessFunctionLibraryRuntime::RunMultiDeviceSync()
@ 0x7d59a5decc60 tensorflow::ProcessFunctionLibraryRuntime::RunSync()
@ 0x7d599e9f119d tensorflow::KernelAndDeviceFunc::Run()
@ 0x7d599e9a00fb tensorflow::EagerKernelExecute()
@ 0x7d599e9a981e tensorflow::ExecuteNode::Run()
@ 0x7d599ef6f7b4 tensorflow::EagerExecutor::SyncExecute()
@ 0x7d599e99faa1 tensorflow::(anonymous namespace)::EagerLocalExecute()
@ 0x7d599e99d1d0 tensorflow::DoEagerExecute()
@ 0x7d599e9a0b2b tensorflow::EagerExecute()
@ 0x7d599e4ce247 tensorflow::EagerOperation::Execute()
@ 0x7d599e9edbb7 tensorflow::CustomDeviceOpHandler::Execute()
@ 0x7d599bb004e5 TFE_Execute
@ 0x7d596b89f910 TFE_Py_FastPathExecute_C()
@ 0x7d596b14bc53 pybind11::detail::argument_loader<>::call<>()
@ 0x7d596b14bb8c pybind11::cpp_function::initialize<>()::{lambda()#1}::__invoke()
@ 0x7d596b122fdd pybind11::cpp_function::dispatcher()
@ 0x5d2c7ae870c6 cfunction_call
Aborted (core dumped)
Source: tensorflow/tensorflow