Managing Python on macOS — The Clean Way

pyenv vs conda for an Effective Python Environment

A learning goal for me this year is Data Science and Machine Learning. While I had experience building financial models, this is my first time using Python. Right from the beginning, I had a difficult time figuring out a clean way to install and manage different versions of Python on my machine. I have to admit, this is mostly due to my perfectionism, which I’m trying hard to overcome (my other learning goal of the year). However, having many Python versions existing independently in different locations is a nightmare. I could sense those potential breakages staring at me in the dark. Not to mention I would lose control of where to install python modules and manage their versions of dependencies.

https://xkcd.com/1987

The Controversy: conda vs. pyenv

The easiest way to do it is through Anaconda. This is recommended by many data science bootcamps and companies on how to set up the environment. Anaconda is a package manager, an environment manager, and a Python distribution that contains a collection of many open-source packages, such as numpy, scikit-learn, scipy, and pandas. It is also the recommended way to install Jupyter Notebooks.

So why not? Bloatware. Many of the packages will be never used and they can be easily installed if and when needed. While conda offers Miniconda as a lightweight version (about 400 MB instead of over 3 GB for a full version of Anaconda), there are still some downsides using conda as a package manager alone.

Installing packages through conda, we are getting binaries and source code from anaconda’s servers, not the official packages from PyPi, which might or might not be up-to-date and feature-complete. We also need to decide using pip vs. conda when we install packages, and the same goes for specifying the dependencies. At that point, we’re managing two install streams, and for more advanced development, this can get messy.

After some research, I found pyenv — a simple, lightweight, Python version management. At a high level, pyenv intercepts Python commands using shim executables injected into the shell PATH, determines which Python version has been specified, and passes the commands along with the correct Python installation. Being said, we can install multiple versions of Python at once and quickly switch between different “activated” versions using pyenv.

pyenv-virtualenv and pyenv-virtualenvwrapper are also two plugins recommended to manage virtualenvs with pyenv. virtualenv is a very popular tool that creates isolated Python environments for Python libraries. pyenv-virtualenv is a plugin for pyenv, by the same author as pyenv, to allow us to use pyenv and virtualenv at the same time conveniently. virtualenvwrapper, on the other hand, is a set of extensions to virtualenv. It provides commands like mkvirtualenv, lssitepackages, and especially workon for switching between different virtualenv directories. This tool is especially useful if we want multiple virtualenv directories. pyenv-virtualenvwrapper is a plugin for pyenv to integrate virtualenvwrapper into pyenv.

Set Up a Python Development Environment

1. Install pyenv, pyenv-virtualenv and pyenv-virtualenvwrapper

If you haven’t done so, install Xcode Command Line Tools (xcode-select --install) and Homebrew. Then:

brew update
brew install pyenv pyenv-virtualenv pyenv-virtualenvwrapper

2. Install Python

First, install all the build dependencies for Python:

brew install openssl readline sqlite3 xz zlib

Install the latest Python version (3.8.3 as of this writing) or see all available versions with pyenv install --list:

pyenv install 3.8.3

2.2 Set Default Python Version

You can check which versions of Python you have installed by running pyenv versions. The * indicates which version is currently active (you could double-check using python -V). You can manage which version you’d like to use in your current session, globally, or on a per-project basis as well. To change the default Python version, you can use the pyenv global command:

pyenv global 3.8.3

To ensure your Python version is running properly, you can run the built-in test suite: python -m test. This will kick off lots of internal Python tests that will verify your installation. You can just kick back and watch the tests pass.

2.3 Uninstall Python

pyenv works by building Python from source; each installed version is nicely located in the pyenv root directory: ls ~/.pyenv/versions.

You could remove these versions directly (rm -rf ~/.pyenv/versions/3.8.3) or use the pyenv command:

pyenv uninstall 3.8.3

3. Add `pyenv init` to shell

In order for pyenv to work properly, you need to to set the pyenv shims to initialize by adding the following to your configuration file:

# bash: ~/.bash_profile or ~/.bashrc
# zsh: ~/.zshrc
# ksh: ~/.kshrc
# other: ~/.profile
if command -v pyenv 1>/dev/null 2>&1; then
  eval "$(pyenv init -)"
  eval "$(pyenv virtualenv-init -)"
fi

By default, macOS Catalina is using zsh and macOS Mojave is using bash.

4. Let pyenv play nice with Homebrew

It’s a known issue that pyenv doesn’t play nicely with Homebrew because it manages shims for the Python *-config executables. You will see the following warning message when running brew doctor.

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and which additional flags to use when
compiling and linking.
Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew-provided
script of the same name. We found the following "config" scripts:
  /Users/raycent/.pyenv/shims/python-config
  /Users/raycent/.pyenv/shims/python3-config
  /Users/raycent/.pyenv/shims/python3.8-config

While this is ignorable, any Python dependencies from Homebrew might be compiled against the Python version managed by pyenv. It’s recommended to use pyenv shell system, to ensure your current shell session is using the system Python (one that’s preinstalled on your OS), to install those dependencies via brew. After that, you can add an alias to your init file to suppress the warning:

# .bash_profile, .bashrc, .zshrc, .kshrc or .profile
alias brew='env PATH="${PATH//$(pyenv root)/shims:/}" brew'
if command -v pyenv 1>/dev/null 2>&1; then
  eval "$(pyenv init -)"
  eval "$(pyenv virtualenv-init -)"
fi

In conclusion, having one good version management solution for Python, like pyenv, is a great strategy for the long run. Anaconda is still good for new Python developers who want out-of-the-box solutions to bootstrap a project immediately. Also, it’s important to leave the system Python as the default because many parts of the system rely on the default Python being a specific version — another reason to customize your Python environment.