Handling OpenSearch mapping exceptions are unreasonably heavy

I had a typical incident where someone sent an attribute as long (JSON number), and afterwards started sending Strings. A symptom of this was logs flooded with errors like

OpenSearchException[OpenSearch exception [type=mapper_parsing_exception, reason=failed to parse field [matches] of type [long] in document with id '752e2b45-fa90-11f0-b7fd-00505687ab33'. Preview of field's value: '314534|1']]; nested: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=For input string: "314534|1"]];

The correct way to fix the root cause is to tell OpenSearch map the attribute in question to string/keyword. However, I noticed something undesirable.

Handling this exception is insanely CPU heavy for the Graylog process. A cluster that can ordinarily process 10k+ EPS will start collapsing at couple hundred EPS causing this Exception. The mongodb and opensearch stay healthy, Graylog uses all available cores.

Expected Behavior

Graylog should probably cache the information the target index has a certain incompatible mapping, and bail out of the operation sooner. The time for this cache should be configurable. One extremely heavy operation once per minute sounds more reasonable.

Current Behavior

Commit to OpenSearch. Exception gets bounced back. Exception gets handled, saved to Mongo. The logging operation to server log file is probably heavy, too.

Possible Solution

Graylog should fail more gracefully, because now one misbehaving (sub-)system can impact the logging operations for everyone else. I'd add a caching type of feature and a status check for these operations.

A sysadmin is going to fix the root cause, and there'd be more time for them to work if the Graylog didn't crash under the load.

Steps to Reproduce (for bugs)

1. Have a high EPS cluster.
2. Cause a bunch of mapping Exceptions.

Context

I had to resolve an incident.

Your Environment

• Graylog Version: 7.0.3
• OpenSearch Version: 2.19
• MongoDB Version: 7.x
• Operating System: RHEL

Checklist

[no] This issue fix need to be backported. [no] Does this issue have security implications?