[BUG] Logs sometimes missing using EFS and K8s on EKS

Mage version

0.9.72

Describe the bug

Sometimes logs are missing in the Mage UI after a pipeline run. I connected to the pod via kubectl and checked the underlying EFS filesystem under /home/src/mage_data/<project>/pipelines/<pipeline>/.logs/52/20240623T145411_204913.

I have made two observations:

• sometimes logs are missing completely
• sometimes the file is full of nullbyte characters -> see screenshots below

To reproduce

1. Setup mage on AWS EKS using the EFS filesystem to mount /home/src
2. Run a pipeline multiple times
3. Inspect logs

Expected behavior

I expect all logs to be available some time after the pipeline finished. Particularly for successful pipelines this should be the case since all blocks have been executed.

I wonder whether files are somehow not properly flushed or closed and thus a race condition arises on the EFS filesystem. Does somebody know where the logging stuff is implemented in the mage codebase and could quickly check pls?

Screenshots

Operating system

• K8s via EKS
• EFS

Additional context

No response