This document describes the current stable version of pytest_celery (1.0). For development docs, go here.

How to release a new version

Release:

1.0

Date:

Apr 29, 2024

The following guide will describe the steps to release a new version of the pytest-celery plugin. It will explain how does the CI/CD pipeline work and how to trigger a new release.

CI/CD Pipeline

Continuous Integration

New in version 1.0.0.

The CI platform is based on GitHub Actions and it is triggered on every push to the repository, and on every pull request, according to the changes made.

The configuration files for the CI pipeline are located in the .github/workflows directory of the repository.

CI Tests

Unit, Integration and Smoke tests.

.github/workflows/python-package.yml

# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: pytest-celery

on:
  push:
    branches: ["main"]
    paths:
      - ".github/workflows/python-package.yml"
      - 'src/pytest_celery/vendors/worker/**'
      - "**.py"
      - "**.txt"
      - "**.toml"
      - "tox.ini"
      - 'Dockerfile'
      - "poetry.lock"
  pull_request:
    paths:
      - ".github/workflows/python-package.yml"
      - 'src/pytest_celery/vendors/worker/**'
      - "**.py"
      - "**.txt"
      - "**.toml"
      - "tox.ini"
      - 'Dockerfile'
      - "poetry.lock"

permissions:
  contents: read # to fetch code (actions/checkout)

jobs:
  Unit:
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Fetch Docker Images
        run: |
          docker pull redis:latest
          docker pull rabbitmq:latest
          docker pull memcached:latest

      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'

      - name: Install Poetry
        if: startsWith(matrix.os, 'ubuntu-')
        uses: snok/install-poetry@v1.3.4

      - name: Install tox
        run: |
          poetry config virtualenvs.create false
          poetry install --only ci

      - name: Run tox for "${{ matrix.python-version }}-unit"
        timeout-minutes: 5
        run: |
          tox --verbose --verbose -e "${{ matrix.python-version }}-unit"

      - name: Upload coverage reports to Codecov
        uses: codecov/codecov-action@v4
        with:
          fail_ci_if_error: true # optional (default = false)
          token: ${{ secrets.CODECOV_TOKEN }}
          verbose: true # optional (default = false)

  Integration:
    needs:
      - Unit
    if: needs.Unit.result == 'success'
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Fetch Docker Images
        run: |
          docker pull redis:latest
          docker pull rabbitmq:latest
          docker pull memcached:latest

      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'

      - name: Install Poetry
        if: startsWith(matrix.os, 'ubuntu-')
        uses: snok/install-poetry@v1.3.4

      - name: Install tox
        run: |
          poetry config virtualenvs.create false
          poetry install --only ci

      - name: Run tox for "${{ matrix.python-version }}-integration"
        timeout-minutes: 15
        run: |
          tox --verbose --verbose -e "${{ matrix.python-version }}-integration" -- -n auto --reruns 3 --rerun-except AssertionError

  Smoke:
    needs:
      - Integration
    if: needs.Integration.result == 'success'
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Fetch Docker Images
        run: |
          docker pull redis:latest
          docker pull rabbitmq:latest
          docker pull memcached:latest

      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'

      - name: Install Poetry
        if: startsWith(matrix.os, 'ubuntu-')
        uses: snok/install-poetry@v1.3.4

      - name: Install tox
        run: |
          poetry config virtualenvs.create false
          poetry install --only ci

      - name: Run tox for "${{ matrix.python-version }}-smoke"
        timeout-minutes: 30
        run: |
          tox --verbose --verbose -e "${{ matrix.python-version }}-smoke" -- -n auto --reruns 3 --rerun-except AssertionError

Parallel Tests

These are the parallel and xdist tox environments. The purpose of this CI pipeline is to make sure that the plugin is compatible with parallel running, both in terms of supporting pytest-xdist and functionally (i.e. that the plugin does not break when running in parallel).

.github/workflows/python-package.yml

# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: parallel-support

on:
  push:
    branches: ["main"]
    paths:
      - ".github/workflows/parallel-support.yml"
      - 'src/pytest_celery/vendors/worker/**'
      - "**.py"
      - "**.txt"
      - "**.toml"
      - "tox.ini"
      - 'Dockerfile'
      - "poetry.lock"
  pull_request:
    paths:
      - ".github/workflows/parallel-support.yml"
      - 'src/pytest_celery/vendors/worker/**'
      - "**.py"
      - "**.txt"
      - "**.toml"
      - "tox.ini"
      - 'Dockerfile'
      - "poetry.lock"

permissions:
  contents: read # to fetch code (actions/checkout)

jobs:
  xdist:
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Fetch Docker Images
        run: |
          docker pull redis:latest
          docker pull rabbitmq:latest
          docker pull memcached:latest

      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'

      - name: Install Poetry
        if: startsWith(matrix.os, 'ubuntu-')
        uses: snok/install-poetry@v1.3.4

      - name: Install tox
        run: |
          poetry config virtualenvs.create false
          poetry install --only ci

      - name: Run tox for all environments in parallel
        timeout-minutes: 30
        run: |
          tox -e xdist

  parallel:
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Fetch Docker Images
        run: |
          docker pull redis:latest
          docker pull rabbitmq:latest
          docker pull memcached:latest

      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'

      - name: Install Poetry
        if: startsWith(matrix.os, 'ubuntu-')
        uses: snok/install-poetry@v1.3.4

      - name: Install tox
        run: |
          poetry config virtualenvs.create false
          poetry install --only ci

      - name: Run tox for all environments in parallel
        timeout-minutes: 30
        run: |
          tox -e parallel

Linting

Standard linting checks.

.github/workflows/linting.yml

name: Linter

on: [pull_request]

jobs:
  check:
    name: ${{ matrix.check }} check
    runs-on: ubuntu-latest
    strategy:
      matrix:
        check: [lint, mypy]
    steps:
      - name: Checkout branch
        uses: actions/checkout@v4

      - name: Install apt packages
        run: |
          sudo apt update

      - name: Set up Python 3.12
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install Poetry
        uses: snok/install-poetry@v1.3.4

      - name: Install CI dependencies
        run: |
          poetry config virtualenvs.create false
          poetry install --only ci

      - name: Run check
        run: tox -e ${{ matrix.check }}

Examples

The official plugin examples are tested as part of the standard CI pipeline.

.github/workflows/examples.yml

# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: examples

on:
  push:
    branches: ["main"]
    paths:
      - ".github/workflows/examples.yml"
      - 'src/pytest_celery/vendors/worker/**'
      - "**.py"
      - "**.txt"
      - "**.toml"
      - "tox.ini"
      - 'Dockerfile'
      - "poetry.lock"
  pull_request:
    paths:
      - ".github/workflows/examples.yml"
      - 'src/pytest_celery/vendors/worker/**'
      - "**.py"
      - "**.txt"
      - "**.toml"
      - "tox.ini"
      - 'Dockerfile'
      - "poetry.lock"

permissions:
  contents: read # to fetch code (actions/checkout)

jobs:
  celery_bug_report:
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'
      - name: Install dependencies
        working-directory: examples
        run: |
          python -m pip install --upgrade pip pytest-cov
          pip install -U "pytest-celery[all]@git+https://github.com/celery/pytest-celery.git"

      - name: Run tests
        working-directory: examples
        timeout-minutes: 5
        run: |
          pytest -xsv celery_bug_report.py --no-cov

  myworker:
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'
      - name: Install dependencies
        working-directory: examples/myworker
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run tests
        working-directory: examples/myworker
        timeout-minutes: 10
        run: |
          pytest -xsv tests --reruns 3 --rerun-except AssertionError

  range:
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'
      - name: Install dependencies
        working-directory: examples/range
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run tests
        working-directory: examples/range
        timeout-minutes: 30
        run: |
          pytest -xsv tests --reruns 3 --rerun-except AssertionError

  rabbitmq_management:
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'
      - name: Install dependencies
        working-directory: examples/rabbitmq_management
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run tests
        working-directory: examples/rabbitmq_management
        timeout-minutes: 10
        run: |
          pytest -xsv tests --reruns 3 --rerun-except AssertionError

  django:
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'
      - name: Install dependencies
        working-directory: examples/django
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run Migrations
        working-directory: examples/django
        run: |
          ./manage.py migrate

      - name: Run tests
        working-directory: examples/django
        timeout-minutes: 10
        run: |
          export DJANGO_SETTINGS_MODULE=proj.settings
          pytest -xsv tests --reruns 3 --rerun-except AssertionError

  myutils:
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'
      - name: Install dependencies
        working-directory: examples/myutils
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run tests
        working-directory: examples/myutils
        timeout-minutes: 10
        run: |
          pytest -xsv tests --reruns 3 --rerun-except AssertionError

  worker_pool:
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'
      - name: Install dependencies
        working-directory: examples/worker_pool
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run tests
        working-directory: examples/worker_pool
        timeout-minutes: 10
        run: |
          pytest -xsv tests --reruns 3 --rerun-except AssertionError

  hybrid_setup:
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'
      - name: Install dependencies
        working-directory: examples/hybrid_setup
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run tests
        working-directory: examples/hybrid_setup
        timeout-minutes: 10
        run: |
          pytest -xsv tests --reruns 3 --rerun-except AssertionError

  vhost:
    runs-on: ${{ matrix.os }}

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.12"]
        os: ["ubuntu-latest"]

    steps:
      - name: Install apt packages
        if: startsWith(matrix.os, 'ubuntu-')
        run: |
          sudo apt update
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'
          cache-dependency-path: '**/setup.py'
      - name: Install dependencies
        working-directory: examples/vhost
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run tests
        working-directory: examples/vhost
        timeout-minutes: 10
        run: |
          pytest -xsv tests --reruns 3 --rerun-except AssertionError

Docker

This pipeline is used to to make sure the provided Dockerfiles from the plugin are built successfully.

.github/workflows/docker.yml

name: Docker

on:
    pull_request:
      branches: [ 'main']
      paths:
        - '.github/workflows/docker.yml'
        - 'src/pytest_celery/vendors/worker/**'
        - "**.py"
        - "**.txt"
        - "**.toml"
        - "tox.ini"
        - 'Dockerfile'
        - "poetry.lock"
    push:
      branches: [ 'main']
      paths:
        - '.github/workflows/docker.yml'
        - 'src/pytest_celery/vendors/worker/**'
        - "**.py"
        - "**.txt"
        - "**.toml"
        - "tox.ini"
        - 'Dockerfile'
        - "poetry.lock"


jobs:
  build-worker:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
    - uses: actions/checkout@v4
    - name: Build Celery Worker
      run: cd src/pytest_celery/vendors/worker && docker build -t pytest-celery-worker .

Continuous Deployment

New in version 1.0.0.

The CD is configured to deploy a new release to the PyPI package index.

The following release workflow is triggered automatically when a new released is tagged and published on GitHub.

.github/workflows/deploy.yml

name: Deploy to PyPI

on:
  release:
    types: [published]

jobs:
  deploy:
    runs-on: ubuntu-latest
    env:
      POETRY_VIRTUALENVS_CREATE: "false"
    steps:
      - uses: actions/checkout@v4

      - name: Install poetry
        run: |
          pipx install poetry
          pipx inject poetry poetry-bumpversion

      - name: Build
        run: |
          poetry version ${{ github.ref_name }}
          poetry build

      - name: Publish
        run: |
          poetry config pypi-token.pypi ${{ secrets.PYPI_API_TOKEN }}
          poetry publish

Note

The poetry version command should say there’s nothing to change, because this should have been done in the PR that prepared the release.

Release Steps

1. Celery Tests

The plugin is used as the official testing infrastructure for Celery. Every new release requires manually testing that the new version works as expected with the Celery test suite.

To run the Celery test suite with the new version of the plugin, modify the Celery test environment as follows:

test.txt

Comment out the pytest-celery installation line.

tox.ini

Add -e "../pytest-celery[all]" to the deps list.

[testenv]
...
deps=
    -e "../pytest-celery[all]"
    -r{toxinidir}/requirements/test.txt
...

And then execute with tox:

tox -e 3.12-smoke -- -n auto

This will run the Celery test suite with the new version of the plugin in edit mode, allowing you to test the new version before releasing it and tweaking it if necessary to debug any issues.

Tip

Use the following snippet to run all of the tests with the new version of the plugin:

Pull RabbitMQ & Redis images for the integration tests:

docker run -d -p 6379:6379 --name redis redis:latest
docker run -d -p 5672:5672 -p 15672:15672 --name rabbitmq rabbitmq:management

Unit Tests:

tox -e 3.12-unit

Integration Tests:

docker start rabbitmq redis
tox -e 3.12-integration-rabbitmq_redis
docker stop rabbitmq redis

Unit & Integration & Smoke Tests:

tox -e 3.12-unit && docker start rabbitmq redis && tox -e 3.12-integration-rabbitmq_redis && docker stop rabbitmq redis && tox -e 3.12-smoke -- -n auto

Warning

The instructions above assume you have the pytest-celery and celery repositories cloned in the same root directory.

2. Release PR

To make a new release, you need to create a new PR with one of these titles.

• Official Release: Prepare for release: vX.Y.Z

• Pre-release: Prepare for (pre) release: vX.Y.Z

The PR should contain the following changes:

• Version bump using poetry version command.

• Changelog update.

This PR will be used as a double check for the CI to make sure everything passes successfully before releasing the new version. Once this PR is merged, the last step is to release a version on GitHub. This will trigger the CD pipeline to deploy the new release to PyPI automatically.

SemVer

If you’re not sure how to number the version, consult the SemVer documentation.

3. Post-release

After the release is done, you should update the official Celery to use the new version of the plugin, in the same test.txt that you modified earlier.

Future Releases

Releases should be planned in the official Milestones of the repository. Each milestone should include in its description what is planned for the release and when is the expected release date.