[Bug/UX] Dolphin: Mac + Python 3.11/3.12 installation and dtype pitfalls – Only pip works, uv/pdm/poetry fail due to numpy/pytorch/package quirks

Summary

After much testing, I want to document the only working path I found for Dolphin on Mac (Apple Silicon), with all the dependency, dtype, and torch quirks.
This may help users (and maybe even you, the maintainers) for future-proofing the install/usage docs.

Install (Working, but Only with pip)

mkdir dolphin-test-pip
cd dolphin-test-pip

conda create --name python311
conda activate python311
conda install -c conda-forge python==3.11

git clone https://github.com/ByteDance/Dolphin.git
cd Dolphin

pip install -r requirements.txt

brew install git-lfs
git lfs install
git clone https://huggingface.co/ByteDance/Dolphin ./hf_model

python demo_page_hf.py \
  --model_path ./hf_model \
  --input_path ~/Downloads/GwangJinKim_DataScientist_2025.pdf \
  --save_dir ./results

(Works, but only after patching for dtype and processor bugs)

• Had to grep for .from_pretrained, .half(), .to(self.device), and patch as follows:

  • Change all .half() to .float() in model loading.

  • Only call .float() on model, not processor/tokenizer.

  • Never cast indices (token IDs, attention masks) to float — keep them integer type.

  • Example fixes:

    self.model = VisionEncoderDecoderModel.from_pretrained(...).float()

    but:

    self.processor = AutoProcessor.from_pretrained(...)

    and:

    batch_prompt_ids = batch_prompt_inputs.input_ids.to(self.device) # not float32

If you get:

• Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.FloatTensor instead
  You have casted input_ids or similar indices to float. Fix as above.

What Does Not Work: uv / pdm / poetry (and why)

• uv (and pip’s strict new resolver) fails unless numpy constraints are modern and relaxed:
  • opencv-python==4.11.0.86 requires numpy >=1.26.0 on Python 3.12+.
  • The provided requirements.txt pins numpy==1.24.4, making the solver unsatisfiable.
• uv is stricter than pip — it refuses to fudge environment constraints; pip, being more forgiving, allows installs to proceed (sometimes in broken ways).

Example uv error

× No solution found when resolving dependencies...
opencv-python==4.11.0.86 depends on numpy>=1.26.0
Your project depends on numpy==1.24.4 and opencv-python==4.11.0.86, so requirements are unsatisfiable.

• After updating requirements.txt to numpy>=1.26.0, installation completes, but you may hit the torch dtype bug.

Why report this?

• The default requirements and code will not work out of the box for anyone using:
  • Mac (Apple Silicon or Intel) plus Python 3.12+ (or sometimes even 3.11)
  • Modern package managers (uv, pdm, poetry)
  • Default scripts: they mix dtype handling for model and processor, and float-cast indices
• These are the most common Python AI setup mistakes and trip up many advanced users.

Suggestion for Dolphin maintainers

• Update requirements.txt to allow numpy>=1.26.0 (or test for 2.x compatibility).
• Update all demo scripts:
  • Only cast model weights/tensors (not processors or indices) to float32.
  • Document Mac/Python compatibility (with pip versus strict resolvers).
• Maybe add a section to the README: If you hit dtype or index errors, here’s how to patch it.

Bonus: How I finally solved all dtype traps

Here are the bash one-liners and sed fixes I had to use:

# Replace .half() with .float()
find . -type f -name "*.py" -exec sed -i '' 's/\.half()/\.float()/g' {} +

# Add .float() after model from_pretrained, but not after processor/tokenizer from_pretrained
# Remove .float() from processor assignments
find . -type f -name "*.py" -exec sed -i '' -E '/processor *= *.*from_pretrained/ s/(\.float\(\)|\.to\([^)]+\))//g' {} +

# Make sure .to(self.device) on model is .to(self.device, dtype=torch.float32)
find . -type f -name "*.py" -exec sed -i '' 's/\.to(self.device)/\.to(self.device, dtype=torch.float32)/g' {} +

# Never use dtype=torch.float32 on input_ids, attention_mask, prompt_ids
find . -type f -name "*.py" -exec sed -i '' -E 's/(input_ids|attention_mask|prompt_ids)\.to\((self\.device|"cpu"|"cuda"|device), *dtype *= *torch\.float32 *\)/\1.to(\2)/g' {} +

Thanks

This project is impressive, but these little things make a huge difference for real-world adoption in data science and AI teams.

Happy to supply more details or a PR if desired.