[Data Corruption]File is truncated with padding-zeros after backend server restarted

Additional Information

Version of s3fs being used (s3fs --version)

v1.87, v1.91, v1.95

Details about issue

The issue description in short: s3fs is opening and reading file while S3 server is restarted, file could be truncated with padding zeroes, which causes data corruption.

I tracked the code, and found that the bug is in this stack: s3fs_open -> check_object_access -> get_object_attribute

Inside get_object_attribute, if an HEAD with 503 response (or other 5xx resp, or timeout err) would be returned because of backend being down for a while, then s3fs tries to send HEAD with filename plus "_$folder$". If the backend server recovers now, a 404 resp will be returned. Then s3fs consider the file not existing, and s3fs would call FdManager::Open with st.size=0, which will result in file being truncated.