Working with Python + Conda Containers

If you haven’t already created a Container to run the Python + Miniconda Service Image, it’s a good idea to do that before reading this guide.

Why We’ve Paired Python and Conda

Conda makes it easy to install Python packages with system dependencies. This matters, for example when you want to communicate with an external database on another server, or even just in another Container. Major Python clients for database communication (such as pymysql or psycopg) have some system dependencies. By pairing Python and Conda, we avoid the shortcomings of other options like:

pip, which will download almost any Python package you can imagine but only installs Python packages and dependencies.
apt, which usually lets you install system dependencies on Ubuntu-based systems but is not an option on Webslice Containers (because the system is designed to be immutable).

Conda (and Miniconda) can create isolated environments, with a critical feature for Webslice Containers: Python packages can be installed along with their system dependencies.

To install psycopg, for example, activate your environment then run:

conda install psycopg

This will get everything you need. (This is instead of running pip install psycopg and apt-get install pgsql-common.)

Conda also lets you create environments with a range of Python versions. Inside the Container, you can see the full list of options with conda search python, but officially, all stable releases that are at least in security or bugfix phases should be supported. Some older, unsupported versions might also be available. This gives you great flexibility by making it easy to update your applications to later Python releases.

Setting Up Conda

The first thing you’ll need to do in your Python + Miniconda Container is set up the Conda environment you want to use for your Python application.

Run the code below, replacing env_name with your actual environment name, and version with the Python version number (e.g. 3.11):

conda create -n env_name python=version`

When you create an environment you can specify multiple packages (and, optionally, their versions) that you want to include - not just Python (which will be automatically installed by Conda if it’s required).

Once an environment is active you can install more packages using conda install.

You can also create an environment with pre-defined characteristics using an environment.yml file. This is well-covered in Conda documentation.

Using Your Conda Environment

Activate your Conda environment using teh code below, where env_name matches the name you gave to the environment:

conda activate env_name

Then check that Python is alive and well with python --version, which ought to display the Python version you selected when you created the Container.

Using pip install

We do not generally recommended installing dependencies using pip install, but it is possible. Just remember that Conda won’t keep track of these, and any dependency that uses system packages to work (like the SQL examples above) will need to be installed via conda install instead.

conda-forge

By default, the conda-forge source is enabled. This allows a much wider range of community-maintained packages. To can change this, and add other sources, see Conda Configuration below.

Freezing Requirements

To list and export currently installed Conda dependencies, use the command:

conda list --export > requirements.txt

This will only check Conda dependencies, though, so if you’ve installed any dependencies via pip or any other source, remember to manage them separately.

To export the entire environment’s configuration, including pip dependencies, use:

conda env export > environment.yml

You can use this to build another Container with identical packages. You might want to manually update the file with version ranges (instead of fixed versions).

Conda Configuration

You can override the Conda environment defaults by:

Using conda config within the active environment, or
Via a .condarc file - in Conda’s documentation, see Using the .condarc conda configuration file.

To see the given defaults, use:

conda config --show-sources

This should show you the Container defaults, plus any user configs being sourced. We have locked down a few configuration options in order to maintain the integrity of the Container, and to ensure that your Conda environments continue to operate as expected if the Container is restarted. You can override anything else, including channels.

Directory Structure

As well as everything in the standard Webslice Container directory structure, Python + Miniconda Containers also have the /container/system/ directory. This supports two features:

Allowing SSH users to control Supervisor with supervisorctl.
Provide a separate read/write directory for Container-specific files (such as Conda environments), without cluttering up the /container/application/ directory.

Adding a Supervisor Process

It’s common to have Supervisor to take control of your process to ensure that it’s always running. There is a default template for this, commented out in /container/config/supervisor.conf, but there are a couple things to keep in mind:

Use source activate, not conda activate. This will activate the environment and make binaries, libraries and other modules available to the process. In comparison, conda activate fully integrates an environment with your prompt.
Ensure that your application is run by the www-data user, not your SSH user. Webslice Containers are built with directory and system permissions catered to www-data, so running your application as any other user will likely result in permission issues that stop your it from working as expected.

Cron Jobs

Webslice Containers come with Cron as a preferred scheduler for jobs like updating data, running mailers, etc.

While it’s possible to incorporate scheduled jobs into Python apps using modules like APScheduler, the system scheduler tends to be more forgiving on server resources. You’ll also avoid cluttering your application threads with additional work.

See our Cron Jobs documentation for more.