This document describes the current stable version of pytest_celery (1.0). For development docs, go here.

Introduction to pytest-celery

Release:

1.0

Date:

Apr 29, 2024

What is pytest-celery?

It is an essential pytest plugin designed for Celery application developers. It enables dynamic orchestration of Celery environments for testing tasks in isolated conditions, leveraging Docker & pytest-docker-tools for environment simulation.

The plugin is designed to be highly configurable, and can be used to test a wide range of Celery architectures. It encapsulate the complexity of setting up a Celery environment, and provides a simple interface to access the Celery architecture components in a test case.

The plugin functions as the backbone of Celery’s testing infrastructure, and is used to test Celery itself. The pytest-celery plugin is considered the official Celery testing infrastructure, internally and externally.

What do I need?

Version Requirements

pytest-celery version 1.0 runs on

• Python ❨3.8, 3.9, 3.10, 3.11, 3.12❩

The pytest-celery plugin is Celery-agnostic, and should work with virtually any version of Celery.

Warning

Currently, due to limited resources, we are unable to officially support Microsoft Windows. Please don’t open any issues related to that platform.

The pytest-celery plugin is using docker containers to define a Celery setup for a test. This means the testing environment must have a working Docker installation. The plugin provides built-in containers, but it is expected to be configured with the architecture and requirements of the target project.

Get Started

Sometimes jumping into the deep water is the best way to learn how to swim. A good place to start are the Examples. They can demonstrates basic common use cases and provide a good starting point for understanding how to use the plugin.

If you wish a more structured approach, start with the First Steps with pytest-celery section.

Other useful use cases are the smoke tests of the Celery repository. The Celery smoke tests are the official production environment for pytest-celery, and may be insightful for understanding how to use the plugin from a production perspective.

Note

The Celery smoke tests were firstly introduced in Celery v5.4.0’s development cycle. The tests requirements have defined the MVP scope for version 1.0.0 of pytest-celery. They use most of the plugin’s features, and vary in complexity and use cases.

It is recommended to familiarize yourself with Celery & pytest-celery before diving deep into the smoke tests.

Essential Resources

The pytest-celery plugin is using core pytest features to encapsulate the complexity of setting up a dockerized Celery environment into a simple interface. It is highly recommended to familiarize yourself with the following resources to get the most out of the plugin.

• Fixtures Reference: Detailed guide to pytest fixtures. Extremely useful for understanding how to use the plugin effectively.

• Pytest Parametrization: Guide for pytest parametrization. Useful for understanding how the Celery architecture matrix is generated.

• pytest_docker_tools: The engine that powers up the dockerized environment. Useful for understanding how the plugin manages the docker containers.

Additional resources can be found in the Useful Resources section.

pytest-celery is…

• Simple

  The plugin provides a single entry point to the test case and makes sure everything is configured according to the selected architecture and requirements.

  By default, all of the supported architecture components are added to a matrix of all possible combinations. Pytest will generate a test case for each combination, and will run it in an isolated environment.

  This allows separation of concerns, and makes it simple to access different architectures in a single test case, for example:

  def test_hello_world(celery_setup: CeleryTestSetup):
      assert celery_setup.ready()
  

  This code will generate test cases for all possible combinations of the supported brokers and backends, using the latest version of Celery. Under the context of the test, each combination will be available as a celery_setup fixture, with access to all of the required components, and will run in an isolated environment.

• Flexible

  The plugin is highly configurable, and can be used to test a wide range of Celery architectures. It can be configured to use a specific version of Celery, or to use a specific version of a broker or backend. It can also be configured to use a custom broker or backend, or to use a custom Celery application.

  For basic usage, the plugin provides default components that can be configured and extended.

  For more advanced use cases, the plugin uses the pytest fixtures mechanism to allow injecting custom components into the environment and build a custom Celery architecture for your project.

  For example, see the rabbitmq_management example, which demonstrates how to replace the default broker matrix with a single RabbitMQ Management broker.

• Fast

  The plugin is designed to run tests in parallel using isolated environments. It supports the pytest-xdist plugin to run tests in parallel, and scales well with available resources to improve the overall test suite performance.

• Annotated

  The codebase is fully annotated with type hints, and is tested with mypy to ensure type safety across the board, allowing a better development experience.

It supports

Workers Latest Celery version. Custom worker. Brokers RabbitMQ. Redis. Custom broker.

Backends Redis. Memcached. Custom backend. Clusters Worker clusters. Broker clusters. Backend clusters.

Features

Architecture Injection By default, a set of predefined components is used to build the Celery architecture for each test. Each built-in component can be either configured or completely replaced with a custom implementation. Architecture Injection can be done at different layers, and can be used to replace only specific elements of the architecture pipeline, or to replace the entire pipeline altogether. Docker Based The plugin uses docker containers to build the Celery architecture for each test. This means that the plugin is not limited to specific versions, and can be used to test potentially any Celery setup. It uses the pytest-docker-tools plugin to manage the docker containers which is useful for accessing the docker containers in the test case during the test run, and assert on their state with high granularity. Batteries Included The plugin provides a set of built-in components that can be used to test ideas quickly. You can start with the default settings, and gradually modify the configurations to fine tune the test environment. By focusing on the test case, you can quickly iterate and test ideas, without wasting time on the overhead of setting up different environment manually. Code Generation One of the challenges in testing production Celery applications is the need to inject testing infrastructure into the Celery worker container at runtime. The plugin provides a Code Generation mechanism that can be used to inject code into the Celery worker container at runtime according to the test case. This opens the door to a wide range of testing scenarios, and allows higher level of control over the tested Celery application.

Isolated Environments Each test case is executed in an isolated environment. This means that each test case has its own Celery architecture, and is not affected by other test cases. Tests may run in parallel and take care of tearing down themselves when they are done, regardless of the test result. Tests as first class citizens The plugin is designed to enhance testing capabilities by treating tests as first-class citizens. It uses advanced mechanisms to encapsulate the complexity of setting up a Celery environment, thus allowing the developer to focus on the test case itself and leave the hard lifting to the plugin. Extensible The plugin is designed to be extensible to fit a wide range of use cases and provides a set of built-in components that can be extended to fit more advanced use cases. It’s based on the S.O.L.I.D principles and provides APIs for developing high quality tests suites. It combines the sophistication of the pytest fixtures mechanism with OOP principles to create separation of concerns between each layer of the infrastructure and its elements, which allow higher level of granularity and control when extending the plugin.

Quick Jump

I want to ⟶

Start the official tutorial. To see demo examples. To see production examples. To learn more about pytest.

To learn more about Celery. To create a standalone Celery bug report with pytest-celery. To copy a simple project for bootstrapping. To see an example that uses different celery workers at the same time.

Jump to ⟶

FAQ Vendors Built-in Celery Worker Vendor Class Test Containers Test Nodes

Test Clusters Test Setup Manipulating the Environment Default Fixtures Celery Application Disabling the Backend

Using the built-in components Components Layers Celery Worker RabbitMQ Broker Redis Broker Redis Backend

Memcached Backend Hello, World! Code Generation Architecture Injection Contributing API Documentation

Installation

The pytest-celery plugin can be easily installed via the Python Package Index (PyPI) using pip.

Installing the pytest-celery package

To install the latest version of pytest-celery, run the following command:

pip install -U pytest-celery

This command installs pytest-celery along with its required dependencies.

This will include:

• Latest version of celery.

• RabbitMQ broker via kombu, installed as a dependency of Celery.

Installing pytest-celery vendors

The plugin detects which vendor dependencies are installed in the test environment to configure the default configurations automatically. This means that by just installing the matching dependencies, the plugin will allow extending the default configurations, up to the supported built-in Vendors.

Warning

If you don’t install any vendor (e.g. no extras and no manual installation), the plugin will result in an empty Test Setup Matrix and might not be fully functional.

To install the vendors, you may either install all of the dependencies manually, or use the following extras:

• all: Installs all vendors.

• redis: Installs Redis vendor, providing broker and result backend components.

• memcached: Installs Memcached vendor, providing a result backend component.

The following extra is installed by default:

• rabbitmq: Installs RabbitMQ vendor, providing a broker component.

To install pytest-celery with the built-in Vendors, replace <extra> with the name of the vendor.

pip install -U "pytest-celery[<extra>]"

RabbitMQ & Redis combo

pip install -U "pytest-celery[redis]"

This will configure the plugin to generate all possible setups using only RabbitMQ and Redis vendors.

All vendors

pip install -U "pytest-celery[all]"

This will configure the plugin to generate all possible setups.

This approach allows you to tailor the installation to your project’s specific needs by including only the necessary optional vendors.

Tip

See Advanced Installation for more advanced installation options.