Enable DXIL (SM 6.0+) on macOS via Wine

Feature Request: Enable DXIL (SM 6.0+) on macOS via Wine

Summary

DXIL shader compilation (Shader Model 6.0+) is currently disabled on macOS because Microsoft does not provide a native libdxcompiler.dylib. However, the same Wine-based IPC approach used for legacy HLSL via d3d4linux could be applied to enable DXIL on macOS.

Current State

Problem

Microsoft's DXC (DirectX Shader Compiler) releases only include:

• dxcompiler.dll for Windows
• libdxcompiler.so for Linux

There is no macOS binary (libdxcompiler.dylib). The current code in shaderc_dxil.cpp has a placeholder:

const char* dxcCompilerDllName =
#if BX_PLATFORM_WINDOWS
    "dxcompiler.dll"
#elif BX_PLATFORM_LINUX
    "libdxcompiler.so"
#else
    "dxcompiler???"  // <-- macOS falls here
#endif

Proposed Solution

Use the same Wine-based approach as d3d4linux to run dxcompiler.dll on macOS:

1. Create a dxc4linux bridge (similar to d3d4linux):

   • A Windows .exe that loads dxcompiler.dll and exposes DxcCreateInstance
   • Communicates with the native macOS process via pipes (IPC)
   • Runs under Wine on macOS

2. Add macOS-specific code path in shaderc_dxil.cpp:

   • On macOS, instead of bx::dlopen(), fork Wine and communicate via IPC
   • Same pattern as shaderc_hlsl.cpp uses for d3d4linux

3. Required DXC functions to bridge:

   • DxcCreateInstance → creates IDxcCompiler3, IDxcUtils, IDxcValidator
   • The rest uses COM interfaces over IPC

Implementation Notes

• Wine runs well on macOS (both Intel and Apple Silicon via Rosetta 2)
• The d3d4linux IPC pattern is proven to work for D3DCompile/D3DReflect
• DXC's COM interface is more complex than D3DCompiler, but still feasible
• Could potentially reuse the dxc4linux bridge on Linux as an alternative to requiring libdxcompiler.so to be installed

Benefits

• Full HLSL SM 6.0+ support on macOS for cross-compilation
• Consistent shader compilation across all desktop platforms
• No need to wait for Microsoft to release macOS binaries

References

• PR #3578: d3d4linux integration (HLSL SM 5.0 via Wine)
• https://github.com/killerdevildog/d3d4linux - Fork with bgfx fixes
• https://github.com/microsoft/DirectXShaderCompiler - DXC releases (Windows/Linux only)

If assigned I can have this done pretty quickly too. just let me know.