在zk选主后,调canal admin获取config失败无重试,导致同步一直卡住
- I have searched the issues of this repository and believe that this is not a duplicate.
- I have checked the FAQ of this repository and believe that this is not a duplicate.
environment
- canal version:1.1.6
- mysql version
Issue Description
zk和canal admin在私有云,canal server和canal client在公有云,在私有云的交换机抖动重启,zk重新选主,但在canal server的方法handleDataDeleted抢到主,调canal admin拉取destination config失败时,直接抛异常停止了。导致zk上是主,但是目标destination没有start
Expected behaviour
期望是:在网络抖动恢复后,canal server能恢复订阅,destination能自动恢复start。应该在收到zk选主成功的回调中,拉取config失败就把zk抢到的节点删掉重新抢
Actual behaviour
canal client一直订阅失败,无法自动恢复
If there is an exception, please attach the exception trace: 2026-07-09 06:03:47.738 [pool-9-thread-1] ERROR c.a.o.canal.instance.manager.PlainCanalInstanceGenerator - generator instance failed. com.alibaba.otter.canal.common.CanalException: load manager config failed. Caused by: java.lang.RuntimeException: requestGet remote error, request : http://10.8.16.84:8089/api/v1/config/instance_polling/member_cowell_fund_0?md5= at com.alibaba.otter.canal.instance.manager.plain.HttpHelper.get(HttpHelper.java:100) ~[canal.instance.manager-1.1.6.jar:na] at com.alibaba.otter.canal.instance.manager.plain.PlainCanalConfigClient.doQuery(PlainCanalConfigClient.java:121) ~[canal.instance.manager-1.1.6.jar:na] at com.alibaba.otter.canal.instance.manager.plain.PlainCanalConfigClient.queryConfig(PlainCanalConfigClient.java:110) ~[canal.instance.manager-1.1.6.jar:na] at com.alibaba.otter.canal.instance.manager.plain.PlainCanalConfigClient.findInstance(PlainCanalConfigClient.java:88) ~[canal.instance.manager-1.1.6.jar:na] at com.alibaba.otter.canal.instance.manager.PlainCanalInstanceGenerator.generate(PlainCanalInstanceGenerator.java:40) ~[canal.instance.manager-1.1.6.jar:na] at com.alibaba.otter.canal.deployer.CanalController.lambda$initGlobalConfig$2(CanalController.java:374) [canal.deployer-1.1.6.jar:na] at com.alibaba.otter.canal.server.embedded.CanalServerWithEmbedded.lambda$start$0(CanalServerWithEmbedded.java:79) ~[canal.server-1.1.6.jar:na] at com.google.common.collect.MigrateMap$1.load(MigrateMap.java:23) ~[canal.common-1.1.6.jar:na] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) ~[guava-22.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) ~[guava-22.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) ~[guava-22.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) ~[guava-22.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[guava-22.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4151) ~[guava-22.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5140) ~[guava-22.0.jar:na] at com.google.common.collect.MigrateMap$MigrateConcurrentMap.get(MigrateMap.java:68) ~[canal.common-1.1.6.jar:na] at com.alibaba.otter.canal.server.embedded.CanalServerWithEmbedded.start(CanalServerWithEmbedded.java:129) ~[canal.server-1.1.6.jar:na] at com.alibaba.otter.canal.deployer.CanalController$1.processActiveEnter(CanalController.java:152) ~[canal.deployer-1.1.6.jar:na] at com.alibaba.otter.canal.common.zookeeper.running.ServerRunningMonitor.processActiveEnter(ServerRunningMonitor.java:242) ~[canal.common-1.1.6.jar:na] at com.alibaba.otter.canal.common.zookeeper.running.ServerRunningMonitor.initRunning(ServerRunningMonitor.java:145) ~[canal.common-1.1.6.jar:na] at com.alibaba.otter.canal.common.zookeeper.running.ServerRunningMonitor.access$600(ServerRunningMonitor.java:29) ~[canal.common-1.1.6.jar:na] at com.alibaba.otter.canal.common.zookeeper.running.ServerRunningMonitor$1.lambda$handleDataDeleted$0(ServerRunningMonitor.java:76) ~[canal.common-1.1.6.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_161] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_161] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_161] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_161] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161] Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 10.8.16.84:8089 [/10.8.16.84] failed: connect timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:143) ~[httpclient-4.5.1.jar:4.5.1] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.1.jar:4.5.1] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.1.jar:4.5.1] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.1.jar:4.5.1] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.1.jar:4.5.1] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.1.jar:4.5.1] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.1.jar:4.5.1] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.1.jar:4.5.1] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.1.jar:4.5.1] at com.alibaba.otter.canal.instance.manager.plain.HttpHelper.get(HttpHelper.java:90) ~[canal.instance.manager-1.1.6.jar:na] ... 28 common frames omitted Caused by: java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_161] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_161] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_161] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_161] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_161] at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_161] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.1.jar:4.5.1] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134) ~[httpclient-4.5.1.jar:4.5.1]
Just put your stack trace here!Source: alibaba/canal