[BUG] Race condition with global state in process pool

Describe the bug Prompt flow appears to modify the global working directory and workers in the line execution process pool can get spawned with different working directories.

How To Reproduce the bug I created an example project to reproduce the bug here: https://github.com/bhanson-techempower/promptflow-concurrency-bug

Some runs will succeed and others will randomly fail with:

Flow path .../promptflow-concurrency-bug/sub_flow2/sub_flow3 does not exist.

Due to the worker for that particular node being spawned after the working directory has been changed.

With the example project the bug is reproduced for me almost every time.

In our production application we're seeing it about half the time when running a batch of 20 runs.

Expected behavior The flow executes successfully every time because prompt flow does not share extra global state between processes.

Running Information:

• Promptflow Package Version using pf -v:

{
  "promptflow": "1.12.0",
  "promptflow-core": "1.12.0",
  "promptflow-devkit": "1.12.0",
  "promptflow-tracing": "1.12.0"
}

Executable '.../promptflow-concurrency-bug/venv/bin/python'
Python (Darwin) 3.11.3 (main, May 25 2023, 12:42:30) [Clang 11.1.0 ]

• Operating System: macOS Sonoma 14.5

Additional context

• References of _change_working_dir
• References of getcwd
• This comment also stuck out as suspicious

We've worked around the issue by modifying the way we invoke the sub flows:

from pathlib import Path
flow_path = Path(__file__).parent / "sub_flow1"
flow = load_flow(flow_path)