Python Environments

Page Contents

Virtual Environments

References:

  1. A non-magical introduction to Pip and Virtualenv for Python beginners by Jamie Matthews.
  2. Pipenv & Virtual Environments, Hitchhier's Guide to Python.
  3. Python Virtual Environments: A Primer
  4. Pipenv: Python Dev Workflow for Humans

Note: Recommend PipEnv - supersedes virtualenv - see next section!

VirtualEnv

# Install
pip install virtualenv

# Create project virtual env for Python 2
cd my_project_folder
virtualenv my_project # virtualenv my_project will create a folder in the current
                      # directory which will contain everything you need for this
                      # private environment

# Create for Python 3
python3 -m venv env # Now part of the distro and doesn't need virtualenv to be installed

# Use the venv
source my_project/bin/activate
# ... Your prompt should become prefixed with your environment name
# ... After activating, a new path to the Python interpretter in your
# ... environment folder takes precedence so it will be used. Same for
# ... the python package path lookup etc...

# Install packages as usual
pip install <package-name>

# When finished...
deactivate

# Delete a venv
rm -fr my_project

# Snapshot of current state:
pip freeze > requirements.txt

# Load snapshot
pip install -r requirements.txt
            

AutoEnv

On GitHub.

You can supplement VirtualEnv with something called AutoEnv. It is a little utility that will monitor your change-directory commands. When you cd into a directory that contains a .env file it will try to source that file. So, if inside that file you have the line source my_project/bin/activate, when you change into that directory, it will automatically activate your project for you. Happy days!

DirEnv

The author of AutoEnv actually recommends trying this above AutoEnv. TODO.

PyEnv

Install virtualenv: curl https://pyenv.run | bash

Add this to `~/.bashrc` and source:

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"            

PyEnv will try to build Python versions you install so add the following:
sudo apt update && sudo apt install -y \
    build-essential \
    checkinstall \
    zlib1g-dev  \
    libbz2-dev \
    libncurses5-dev \
    libncursesw5-dev \
    readline-common \
    libreadline-dev \
    libffi-dev \
    libssl-dev \
    liblzma-dev \
    tk \
    tk-dev \
    libsqlite3-dev

To install a Python version in PyEnv:

pyenv update
pyenv install 3.7:latest
pyenv install 3.8:latest

# Instruct Pyenv to first use system Python version, then versions
# 3.7 and 3.8.
pyenv global system 3.7.13 3.8.13

To create a virtual environment with PyEnv:

# Generally...
pyenv virtualenv <python-version> <environment-name>

# For example...
pyenv virtualenv 3.7.4 my_new_project

To activate a virtual environment its:

pyenv activate <environment-name>

To automatically use an environment when in a development directory do:

cd ~/my/dev/directory
pyenv local <my-dev-environment-name>

To upgrade PyEnv do

cd ~/.pyenv
git pull

Pipenv - Python Virtual Environments and Package Management In One

See Pipenv: A Guide to the New Python Packaging Tool. See also Why Python devs should use Pipenv.

Pipenv is a tool that aims to bring the best of all packaging worlds (bundler, composer, npm, cargo, yarn, etc.) to the Python world ...

... It automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your Pipfile as you install/uninstall packages ...

... The problems that Pipenv seeks to solve are multi-faceted:

  • You no longer need to use pip and virtualenv separately. They work together.
  • Managing a requirements.txt file can be problematic, so Pipenv uses Pipfile and Pipfile.lock to separate abstract dependency declarations from the last tested combination.
  • ...

... Pipfiles contain information for the dependencies of the project, and supersedes the requirements.txt file used in most Python projects.

Install using pip3 [--user] install pipenv and create a new virtualenv using pipenv --python 3[.minor].

Now, to add packages to your project use pipenv install <package>[~=maj.min[.patch]]

Like the original environments, it needs tobe actived: pipenv shell. Creates virtual environment if one doesn't exist. Note, pipenv creates all virtual environments in a default location.

To push to production, lock your environment: pipenv lock. Creates/updates the Pipfile.lock file (never edit this!). To install from your lock file use pipenv install --ignore-pipfile.

To leave a pipenv virtual environment just exit the current shell: pipenv starts a new shell session with the virtualenv path instead of changing the path in the current shell session.

Cheat Sheet

CommandMeaning
check Checks for PyUp Safety security vulnerabilities and against PEP 508 markers provided in Pipfile.
clean Uninstalls all packages not specified in Pipfile.lock.
graph Displays currently-installed dependency graph information.
install [--ignore-pipfile] Installs provided packages and adds them to Pipfile, or (if no packages are given), installs all packages from Pipfile. Use ignore option to install from lock file.
lock Generates Pipfile.lock
run Spawns a command installed into the virtualenv.
shell Spawns a shell within the virtualenv.
sync Installs all packages specified in Pipfile.lock.
update Runs lock, then sync.

Pipenv Competitors

There are a couple of competitors to pipenv that I've found at the time of writing (Aug 2021). The two that seem to pop up are Poetry and PDM. Pipenv is more mature than them, and Poetry is more mature than PDM.

Not sure that they give an amazing amount over pip + venv, especially given you have to install another tool and then maintain and keep that up to date. Perhaps for larger projects they are more useful or if you are publishing to PyPi?

Some rough notes comparing the various options:

venv + pip
    - Built in to Python 3
    - Pip freeze does the dependency freezing recursively so you can restore *full* dependencies
      uasing a requirements.txt from pip freeze - dont need pipenv for this!
    - Updating all package dependencies - no built in command but there are one
            liners - e.g. https://stackoverflow.com/a/3452888/1517244. However these
            will not figure out the correct set of dependencies between all required
            packages to ensure they all work together - need likes of pipenv/poetry
            for that I think.
        - No PyUp integration but easy to pip install util for this
    - Needs multiple commands IIRC: need to env\Scripts\activate on window
            then pip install, pip freeze, pip install -r requirements.txt and so on,
            but this is hardly a hardship lol

pipenv
    - venv and pip functionality under one command
    - Updating all packages to appropriate versions seems easier
    - Complains about caching speed
    - More mature than poetry
    - Can only spec one version of python - not able to do ranges.
    - liked built in hook for pyup - need more info about who curates DB driving it.

poetry
    - Like pipenv but newer with better cache performance
    - Can do python >= and < (pipenv can't)
    - Dependency management seems more stringent than pipenv
    - small pedantry with ansi terminals but as --no-ansi option
    - install outside of pip to avoid any global dependency damage - nice
        - No PyUp integration but easy to pip install util for this

PDM
    - Even faster in benchmarks - uses local packages - no venv at all!
    - uses same TOML file format and seems like it can coexist with poetry
    - **** Didn't work out the box for me on Windows :'( ****
    - Worked okay on WSL for me although I had to apt install an extra python package for
      virtual envs - python3-venv