pyflow

Pyflow

Simple is better than complex - The Zen of Python

Pyflow streamlines working with Python projects and files. Objectives: "Just works", and fast. It's an easy-to-use CLI app which manages python dependency and interpreter versions.

Example use, including setting up a project and switching Py versions:

If your project's already configured, the only command you need is pyflow, or pyflow myscript.py; setting up Python and its dependencies are automatic.

Goals: Make using and publishing Python projects as simple as possible. Actively managing Python environments shouldn't be required to use dependencies safely. We're attempting to fix each stumbling block in the Python workflow, so that it's as elegant as the language itself.

You don't need Python or any other tools installed to use Pyflow.

It runs standalone scripts in their own environments with no config, and project functions directly from the CLI.

It implements PEP 582 -- Python local packages directory and Pep 518 (pyproject.toml).

Install

• Pip - Run pip install pyflow.

• Standalone executables: Download the appropriate executable for your system, and add it to the PATH environment variable.

• If you have Rust installed - Run cargo install pyflow.

Quickstart

• (Optional) Run pyflow init in an existing project folder, or pyflow new projname to create a new project folder. init imports data from requirements.txt or Pipfile; new creates a folder with the basics.
• Run pyflow install requests etc to install packages. Alternatively, edit pyproject.toml directly.
• Run pyflow or pyflow myfile.py to run Python.

Quick-and-dirty start for quick-and-dirty scripts

Compliant with PEP 723, which allows specifying the Python interpreter, and dependency versions at the top of a script. Note: We previously used a different syntax for this, and still support that, but now support PEP 723 as well.

• Specify dependencies and a Python version in a comment at the top of the script. (See the example below)
• Run pyflow script myscript.py, where myscript.py is the name of your script. This will set up an isolated environment for this script, and install dependencies as required. This is a safe way to run one-off Python files that aren't attached to a project, but have dependencies.

Example, from that PEP:

…

toml [tool.pyflow] py_version = "3.12" name = "runcible" version = "0.3.1" authors = ["John Hackworth "]

[tool.pyflow.dependencies] numpy = "^1.16.4" diffeqpy = "1.1.0"

Example contents: PEP-621 style:
```toml
● [project]
  name = "runcible"
  version = "0.3.1"
  authors = [
      {name = "John Hackworth", email = "[email protected]"}
  ]
  requires-python = ">=3.12"
  dependencies = [
      "numpy>=1.16.4,=3.1` - Example with multiple dependencies, and specified versions
- `pyflow uninstall requests` - Remove one or more dependencies

### Running REPL and Python files in the environment:
- `pyflow` - Run a Python REPL
- `pyflow main.py` - Run a python file
- `pyflow ipython`, `pyflow black` etc - Run a CLI tool like `ipython`, or a project function
 For the former, this must have been installed by a dependency; for the latter, it's specified
under `[tool.pyflow]`, `scripts`
- `pyflow script myscript.py` - Run a one-off script, outside a project directory, with per-file
package management

### Building and publishing:
- `pyflow package` - Package for distribution (uses setuptools internally, and
builds both source and wheel.)
- `pyflow package --extras "test all"` - Package for distribution with extra features enabled,
as defined in `pyproject.toml`
- `pyflow publish` - Upload to PyPi (Repo specified in `pyproject.toml`. Uses `Twine` internally.)

### Misc:
- `pyflow list` - Display all installed packages and console scripts
- `pyflow new projname` - Create a directory containing the basics for a project:
a readme, pyproject.toml, .gitignore, and directory for code
- `pyflow init` - Create a `pyproject.toml` file in an existing project directory. Pull info from
`requirements.text` and `Pipfile` as required.
- `pyflow reset` - Remove the environment, and uninstall all packages
- `pyflow clear` - Clear the cache, of downloaded dependencies, Python installations, or script-
environments; it will ask you which ones you'd like to clear.
- `pyflow -V` - Get the current version of this tool
- `pyflow help` Get help, including a list of available commands

## How installation and locking work
Running `pyflow install` syncs the project's installed dependencies with those  specified in `pyproject.toml`. It generates `pyflow.lock`, which on subsequent runs, keeps dependencies each package a fixed version, as long as it continues to meet the constraints specified in `pyproject.toml`. Adding a
package name via the CLI, eg `pyflow install matplotlib` simply adds that requirement before proceeding. `pyflow.lock` isn't meant to be edited directly.

Each dependency listed in `pyproject.toml` is checked for a compatible match in `pyflow.lock` If a constraint is met by something in the lock file,
the version we'll sync will match that listed in the lock file. If not met, a new entry is added to the lock file, containing the highest version allowed by `pyproject.toml`. Once complete, packages are installed and removed in order to exactly meet those listed in the updated lock file.

This tool downloads and unpacks wheels from `pypi`, or builds wheels from source if none are available. It verifies the integrity of the downloaded file
 against that listed on `pypi` using `SHA256`, and the exact versions used are stored in a lock file.

When a dependency is removed from `pyproject.toml`, it, and its subdependencies not also required by other packages are removed from the `.venv` folder.

## How dependencies are resolved

Compatible versions of dependencies are determined using info from the [PyPi Warehouse](https://github.com/pypa/warehouse) (available versions, and hash info).

If all packages are either only specified once, or specified multiple times with the same newest-compatible version, we're done resolving, and ready to install and sync.

If a package is included more than once with different newest-compatible versions, but one of those newest-compatible is compatible with all requirements, we install that one. If not, we search all versions to find one that's compatible.

If still unable to find a version of a package that satisfies all requirements, we install multiple versions of it as-required, store them in separate directories, and modify their parents' imports as required.

Note that it may be possible to resolve dependencies in cases not listed above, instead of installing multiple versions. Ie we could try different combinations of top-level packages, check for resolutions, then vary children as-required down the hierarchy. We don't do this because  it's slow, has no guarantee of success, and involves installing older versions of packages.

## Not-yet-implemented
- Installing global CLI tools
- The lock file is missing some info like hashes
- Adding a dependency via the CLI with a specific version constraint, or extras.
- Install packages from a local `wheel` directly. In the meanwhile, you can use a `path`
dependency of the unpacked wheel.
- Dealing with multiple-installed-versions of a dependency that uses importlib
or dynamic imports
- Install Python on Mac

## Building and uploading your project to PyPi
In order to build and publish your project, additional info is needed in
`pyproject.toml`, that mimics what would be in `setup.py`. Example:

…

`package_url` is used to determine which package repository to upload to. If omitted,
`Pypi test` is used (`https://test.pypi.org/legacy/`).

Other items you can specify in `[tool.pyflow]`:
- `readme`: The readme filename, use this if it's named something other than `README.md`.
- `build`: A python script to execute building non-python extensions when running `pyflow package`.

## Building this from source
If you’d like to build from source, [download and install Rust]( https://www.rust-lang.org/tools/install),
clone the repo, and in the repo directory, run `cargo build --release`.

Ie on linux or Mac:
```bash
curl https://sh.rustup.rs -sSf | sh
git clone https://github.com/david-oconnor/pyflow.git
cd pyflow
cargo build --release

Updating

• If installed via Cargo, run cargo install pyflow --force.
• If installed via Pip, run pip install --upgrade pyflow.

Uninstalling

• If installed via Cargo, run cargo uninstall pyflow.
• If installed via Pip, run pip uninstall pyflow.
• If manually calling a binary, remove it.

Contributing

If you notice unexpected behavior or missing features, please post an issue, or submit a PR. If you see unexpected behavior, it's probably a bug! Post an issue listing the dependencies that did not install correctly.

Why not to use this

• It's adding another tool to an already complex field.
• Most of the features here are already provided by a range of existing packages, like the ones in the table above.
• The field of contributors is expected to be small, since it's written in a different language.
• Dependency managers like Pipenv and Poetry work well enough for many cases, have dedicated dev teams, and large userbases.
• Conda in particular handles many things this does quite well.

Python binary sources:

Repo binaries are downloaded from

• Windows: Python official Visual Studio package, by Steve Dower.
• Newer linux distros: Built on Ubuntu 18.04, using standard procedures.
• Older linux distros: Built on CentOS 7, using standard procedures.

Gotchas

• Make sure .venv is in your .gitignore file.
• You may need to set up IDEs to find packages in .venv. If using PyCharm: Settings → Project → Project Interpreter → ⚙ → Show All... → (Select the interpreter, ie (projname)/.venv/3.x/.venv/bin/python on Linux/Mac, or (projname)/.venv/3.x/Scripts/python on Windows) → Click the folder-tree icon at the bottom of the pop-out window → Click the + icon at the bottom of the new pop-out window → Navigate to and select (projname)/.venv/3.x/lib
• If using VsCode: Settings → search python extra paths → Edit in settings.json → Add or modify the line: "python.autoComplete.extraPaths": ["(projname)/.venv/3.7/lib"]

References

• PEP 582 - Python local packages directory
• PEP 518 - pyproject.toml
• Semantic versioning
• PEP 440 -- Version Identification and Dependency Specification
• Specifying dependencies in Cargo
• Predictable dependency management blog entry
• Blog on why Python dependencies are hard to determine