#8186·flower

`fab-include` silently ignores a nested `pyproject.toml`, and rejects a nested `LICENSE`

Author: kaiser-dataCreated Sep 17, 2026Updated Sep 17, 2026

Environment

flwr==1.37.0, Python 3.12, macOS 15. Re-verified against 1.37.0 on 17 September 2026.

What happens

An app that ships a second app bundle inside itself (ours starts it on SuperGrid for a model call) cannot package that bundle, and the two failure modes contradict each other:

  • sub/pyproject.toml listed in fab-includebuild succeeds, the file is missing from the FAB, nothing is reported. The failure only appears at runtime, inside the installed app.
  • sub/LICENSE listed in fab-include, file present and tracked → hard error: Pattern in "fab-include" did not match any files: "sub/LICENSE".

So one nested file that cannot be included is silently dropped, and another fails the build loudly.

Minimal reproduction

bash
mkdir -p demo/demo demo/sub/pkg && cd demo
printf '' > demo/__init__.py
printf 'from flwr.serverapp import ServerApp\napp = ServerApp()\n' > demo/server.py
printf 'from flwr.clientapp import ClientApp\napp = ClientApp()\n' > demo/client.py
printf '' > sub/pkg/__init__.py
printf '[project]\nname = "sub"\nversion = "0.1.0"\n' > sub/pyproject.toml
echo "Apache License placeholder" > LICENSE
cat > pyproject.toml <<'TOML'
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "demo"
version = "0.1.0"
description = "FAB nesting repro"
license = { file = "LICENSE" }
requires-python = ">=3.11,<4.0"
dependencies = ["flwr>=1.35.0,<2.0"]

[tool.hatch.build.targets.wheel]
packages = ["demo"]

[tool.flwr.app]
publisher = "test"
fab-format-version = 1
flwr-version-target = "1.35.0"
fab-include = ["sub/**/*.py", "sub/pyproject.toml", "LICENSE"]

[tool.flwr.app.components]
serverapp = "demo.server:app"
clientapp = "demo.client:app"
TOML

flwr build
unzip -l *.fab | grep -E 'sub/|LICENSE|pyproject'

Observed:

 Successfully built test.demo.0-1-0.640bc7e4.fab
      587  pyproject.toml
       27  LICENSE
        0  sub/pkg/__init__.py

sub/pyproject.toml is absent and nothing was reported. Now add a nested licence:

bash
cp LICENSE sub/LICENSE
# fab-include = ["sub/**/*.py", "sub/pyproject.toml", "LICENSE", "sub/LICENSE"]
flwr build
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ Pattern in "fab-include" did not match any files: "sub/LICENSE". Correct the │
│ pattern or remove it.                                                        │
╰──────────────────────────────────────────────────────────────────────────────╯

No FAB is produced.

Why it behaves this way

  • common/constant.py:85FAB_EXCLUDE_PATTERNS contains FAB_CONFIG_FILE ("pyproject.toml") as a bare gitignore-style pattern, so it matches at any depth.
  • cli/build.py:417-419 — files removed by built-in excludes are explicitly treated as "expected removals and should not be flagged", which is why the user's include is ignored in silence.
  • common/constant.py:71-80FAB_INCLUDE_PATTERNS anchors the licence as /LICENSE, so a nested LICENSE is never a candidate file and the user pattern "matches nothing".

Expected

Either behaviour would be defensible on its own; the combination is what surprises. At minimum, a user fab-include pattern that is nullified by a built-in exclude should produce a warning at build time rather than a missing file at runtime.

Suggestions

  1. Warn (or fail) when a fab-include pattern matches files that the built-in excludes then remove.
  2. Anchor the built-in pyproject.toml exclude as /pyproject.toml, matching the /LICENSE convention, so nested manifests can be shipped deliberately.
  3. Longer term: a supported "sub-app" concept for an app that ships another app it may launch.

Impact and workaround

This cost us a demo cycle during a hackathon: our ServerApp ran from the installed FAB, found no bundle, and fell back to a rule-based path that looked identical from the outside.

Workaround: ship the nested manifest as agent/pyproject.fab.toml, copy the tree to a temp dir at runtime, rename it back, and copy the app-root LICENSE in beside it — soteria/llm.py.