DNS resolver setup might shut down deployment on IBM iSeries 7.6 because of "LoadDnsConfig"

Context

We are deploying some Spring Boot application using project reactor pretty much with default setup. That project implements an API-gateway, so a web client needs to do REST-requests into some internal network, mostly localhost or 127.0.0.1. The important part here is that we are hosting as WAR-deployment under Tomcat 10.x on IBM iSeries and that Tomcat hosts other unrelated webapps as well. A few days ago that deployment stopped to work in the following way:

Whenever a request to the API-gateway was done, which involved any type of internal network request, the entire Tomcat received SIGTERM from the operating system level and was shut down. Authentication requests at the API-gateway itself worked, but any additional business level request always triggered the shutdown of the Tomcat. And it was really a graceful shutdown, we could see in the logs that after the SIGTERM was received, the webapps continued to work a bit until being undeployed by Tomcat. It as well was not related to any exception from within Tomcat and SIGTERM wasn't issued from within Tomcat, it really came form the outside directly sent tot the Tomcat JVM process.

Root cause

The root cause was ultimately found in Netty setting up its internal DNS resolver. It thinks that iSeries is some generic Unix and tries to handle it like that, e.g. considers /etc/resolv.conf being, available, which isn't the case by default. After not finding that, it executes LoadDnsConfig and that's where things become difficult, because at least in our setup Tomcat lacked the permissions to view the DNS config. While that seems to have been different in the past before 7.6, because we didn't trigger this error, with 7.6 things seems to have changed. Accessing that data requires *IOSYSCFG or function usage QIBM_IOSYSCFG_VIEW and after applying those permissions for Tomcat, things worked as before.

  DefaultDnsServerAddressStreamProvider.<clinit>
    -> DirContextUtils -> InitialDirContext (com.sun.jndi.dns.DnsContextFactory)
    -> sun.net.dns.ResolverConfiguration.open().nameservers()
    -> sun.net.dns.ResolverConfigurationImpl   (native)
    -> LoadDnsConfig                            <-- needs *IOSYSCFG / QIBM_IOSYSCFG_VIEW

The difference is where the data lives. On Linux and AIX it's a world-readable text file. On Windows and macOS it's an unprivileged API. On IBM i it's part of the TCP/IP configuration — what DSPTCPDMN / CHGTCPDMN operate on — held in system configuration objects rather than a file. Reading those is by definition an I/O-system-configuration operation, which is why it maps onto *IOSYSCFG and the finer-grained QIBM_IOSYSCFG_VIEW.

  ┌──────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────┐
  │   platform   │                                                path                                                 │ privileged? │
  ├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────┤
  │ macOS        │ native netty-resolver-dns-native-macos provider                                                     │ no          │
  ├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────┤
  │ Linux / AIX  │ /etc/resolv.conf parses → JNDI never reached                                                        │ no          │
  ├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────┤
  │ Windows      │ resolv.conf skipped by design → JNDI → ResolverConfiguration → IP Helper GetNetworkParams           │ no          │
  ├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────┤
  │ IBM i (PASE) │ looks like generic Unix → no usable /etc/resolv.conf → JNDI → ResolverConfiguration → LoadDnsConfig │ yes, on 7.6 │
  └──────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────┘

How to fix?

• there might be a file alternative like for /etc/hosts -> don't know currently
  • https://github.com/netty/netty/issues/16560
  • /QIBM/UserData/OS400/TCPIP/QTOCHOSTS
• we used 127.0.0.1 all the time, not DNS resolver needed at all -> lazier init for on-demand only would have prevented the issue
• Maybe make the JNDI-call optional at least on iSeries and fallback/error out instead?
  • we still don't know where the JVM-external SIGTERM came from -> VERY difficult to debug
  • an exception would have made the problem easier to find at least and most likely not taken down the entire Tomcat, only one webapp
• ...