Python virtual environments

If you plan to do regular work on Astropy you should do your development in a Python virtual environment. Conceptually a virtual environment is a duplicate of the Python environment you normally work in, but sandboxed from your default Python environment in the sense that packages installed in the virtual environment do not affect your normal working environment in any way. This allows you to install, for example, a development version of Astropy and its dependencies without it conflicting with your day-to-day work with Astropy and other Python packages.

Note

“Default Python environment” here means whatever Python you are using when you log in; i.e. the default Python installation on your system, which is not in a Conda environment or virtualenv.

More specifically, in UNIX-like platforms it creates a parallel root “prefix” with its own bin/, lib/, etc. directories. When you activate the virtual environment it places this bin/ at the head of your $PATH environment variable.

This works similarly on Windows but the details depend on how you installed Python and whether or not you’re using Anaconda.

There are a few options for using virtual environments; the choice of method is dictated by the Python distribution you use:

  • If you use the Anaconda/Conda Python distribution you must use the conda command to make and manage your virtual environments.

  • If you do not use Anaconda you can use virtualenv and the conda-like helper commands provided by virtualenvwrapper; you can not use this with conda. As the name suggests, virtualenvwrapper is a wrapper around virtualenv.

  • A third, more recent option which is growing in popularity is pipenv which builds on top of virtualenv to provide project-specific Python environments and dependency management.

In both cases you will go through the same basic steps; the commands to accomplish each step are given for both conda and virtualenvwrapper:

Another well-maintained guide to Python virtualenvs (specifically pipenv and virtualenv, though it does not discuss conda) which has been translated into multiple languages is the Hitchhiker’s Guide to Python chapter on the subject.

Set up for virtual environments

  • virtualenvwrapper:

  • conda: No setup is necessary beyond installing the Anaconda Python distribution.

  • pipenv: Install the pipenv command using your default pip (the pip in the default Python environment):

    pip install --user pipenv
    

List virtual environments

You do not need to list the virtual environments you have created before using them…but sooner or later you will forget what environments you have defined and this is the easy way to find out.

  • virtualenvwrapper: workon
    • If this displays nothing you have no virtual environments

    • If this displays workon: command not found then you haven’t done the Set up for virtual environments; do that.

    • For more detailed information about installed environments use lsvirtualenv.

  • conda: conda info -e
    • you will always have at least one environment, called root

    • your active environment is indicated by a *

  • pipenv does not have a concept of listing virtualenvs; it instead automatically generates the virtualenv associated with a project directory (e.g. the Astropy source repository on your computer).

Create a new virtual environment

This needs to be done once for each virtual environment you want. There is one important choice you need to make when you create a virtual environment: which, if any, of the packages installed in your default Python environment do you want in your virtual environment?

Including them in your virtual environment doesn’t take much extra space–they are linked into the virtual environment instead of being copied. Within the virtual environment you can install new versions of packages like Numpy or Astropy that override the versions installed in your default Python environment.

The easiest way to get started is to include in your virtual environment the packages installed in your your default Python environment; the instructions below do that.

In everything that follows, ENV represents the name you give your virtual environment.

The name you choose cannot have spaces in it.

  • virtualenvwrapper:
    • Make an environment called ENV with all of the packages in your default Python environment:

      mkvirtualenv --system-site-packages ENV
      
    • Omit the option --system-site-packages to create an environment without the Python packages installed in your default Python environment.

    • Environments created with virtualenvwrapper always include pip and setuptools so that you can install packages within the virtual environment.

    • More details and examples are in the virtualenvwrapper command documentation.

  • conda:
    • Make an environment called ENV with all of the packages in your main Anaconda environment:

      conda create -n ENV anaconda
      
    • More details, and examples that start with none of the packages from your default Python environment, are in the documentation for the conda command and the guide on how to manage environments.

    • Next activate the environment ENV with:

      conda activate ENV
      
    • Your command-line prompt will contain ENV in parentheses by default.

    • If Astropy is installed in your ENV environment, you may need to uninstall it in order for the development version to install properly. You can do this with the following command:

      conda uninstall astropy
      
  • pipenv:
    • Make sure you are in the Astropy source directory. See Try the development version if you are unsure how to get the source code. After running git clone <your-astropy-fork> run cd astropy/ then:

      pipenv install -e .
      
    • This both creates the virtual environment for the project automatically, and also installs all of Astropy’s dependencies, and adds your Astropy repository as the version of Astropy to use in the environment.

    • You can activate the environment any time you’re in the top-level astropy/ directory (cloned from git) by running:

      pipenv shell
      

      This will open a new shell with the appropriate virtualenv enabled.

      You can also run individual commands from the virtualenv without activating it in the shell like:

      pipenv run python
      

Activate a virtual environment

To use a new virtual environment you may need to activate it; virtualenvwrapper will try to automatically activate your new environment when you create it. Activation does two things (either of which you could do manually, though it would be inconvenient):

  • Puts the bin directory for the virtual environment at the front of your $PATH.

  • Adds the name of the virtual environment to your command prompt. If you have successfully switched to a new environment called ENV your prompt should look something like this: (ENV)[~] $

The commands below allow you to switch between virtual environments in addition to activating new ones.

  • virtualenvwrapper: Activate the environment ENV with:

    workon ENV
    
  • conda: Activate the environment ENV with:

    conda activate ENV
    
  • pipenv: Activate the environment by changing into the project directory (i.e. the copy of the Astropy repository on your computer) and running:

    pipenv shell
    

Deactivate a virtual environment

At some point you may want to go back to your default Python environment. Do that with:

  • virtualenvwrapper: deactivate
    • Note that in virtualenvwrapper 4.1.1 the output of mkvirtualenv says you should use source deactivate; that does not seem to actually work.

  • conda: conda deactivate

  • pipenv: exit

    Note

    Unlike virtualenv and conda, pipenv does not manipulate environment variables in your current shell session. Instead it launches a subshell which is a copy of your previous shell, in which it can then change some environment variables. Therefore, any environment variables you change in the pipenv shell will be restored to their previous value (or lost entirely) when exit-ing the subshell.

Delete a virtual environment

In both virtualenvwrapper and conda you can simply delete the directory in which the ENV is located; both also provide commands to make that a bit easier. pipenv includes a command for deleting the virtual environment associated with the current directory:

  • virtualenvwrapper: rmvirtualenv ENV

  • conda: conda remove --all -n ENV

  • pipenv: pipenv --rm: As with other pipenv commands this is run from within the project directory.