[CHIA-4355] [Bug] chia says it fails to start daemon, but the process is running. chia can't connect to it.

What happened?

After upgrading a number of my linux (Ubuntu 20.04.4) systems to 1.3.4, I have a few harvesters that are failing to start up. When I try to start the harvester:

$ chia start harvester
Daemon not started yet
Starting daemon
Daemon not started yet
Failed to create the chia daemon
$

This reports this error in only a couple of seconds. I've noted that if I check the process table after this I see chia_daemon, and netstat shows a listener on port 55400. Maybe something isn't waiting long enough?

These same systems were working with 1.3.3, and other systems are working with 1.3.4, so I don't know what to look at.

The debug.log shows:

2022-05-01T20:51:47.333 harvester chia.rpc.rpc_server     : WARNING  Cannot connect to daemon at ws://localhost:55400
2022-05-01T20:51:49.529 harvester chia.rpc.rpc_server     : WARNING  Cannot connect to daemon at ws://localhost:55400
2022-05-01T20:51:51.719 harvester chia.rpc.rpc_server     : WARNING  Cannot connect to daemon at ws://localhost:55400

but the above are from an hour or two ago, and new invocations aren't producing more of the same.

The systems displaying this behavior are older systems, and all (3-5) of these like systems are displaying this same problem. I rebooted one completely and it's still misbehaving in this way.

Let me know what other diagnostics or information I can gather.

Version

1.3.4

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

2022-05-01T20:51:47.333 harvester chia.rpc.rpc_server     : WARNING  Cannot connect to daemon at ws://localhost:55400
2022-05-01T20:51:49.529 harvester chia.rpc.rpc_server     : WARNING  Cannot connect to daemon at ws://localhost:55400
2022-05-01T20:51:51.719 harvester chia.rpc.rpc_server     : WARNING  Cannot connect to daemon at ws://localhost:55400