Django Celery Background Tasks

Introduction

Web requests should be short-lived; generating PDFs, sending thousands of emails, or calling external APIs causes timeouts. Celery runs async tasks in worker processes separate from Django. Redis or RabbitMQ manages the message queue as broker.

This guide covers connecting Celery to Django, defining tasks, periodic jobs, and production monitoring.

Architecture: Broker, Backend, and Worker

The Django app enqueues tasks (apply_async / delay). The broker holds messages. Workers pick messages and run task functions. An optional result backend (Redis/DB) stores outcomes.

Broker: Redis (fast) or RabbitMQ (reliable)
Worker: celery -A proj worker -l info
Beat: scheduled task scheduler
Flower: worker monitoring UI

Django Integration

celery.py creates the Celery app instance and loads Django settings. Import proj.celery in __init__.py so Celery loads when Django starts.

# proj/celery.py
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
app = Celery('proj')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

# settings.py
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/1'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'

Defining and Using Tasks

@shared_task distributes tasks across apps. bind=True enables self.retry(). Design idempotent tasks: running twice must not corrupt data.

from celery import shared_task
from django.core.mail import send_mail

@shared_task(bind=True, max_retries=3, default_retry_delay=60)
def send_welcome_email(self, user_id):
    from accounts.models import User
    user = User.objects.get(pk=user_id)
    try:
        send_mail(
            'Welcome',
            f'Hello {user.username}',
            'noreply@example.com',
            [user.email],
        )
    except Exception as exc:
        raise self.retry(exc=exc)

Do not do heavy work in views; enqueue with delay() and return 202/redirect immediately.

Periodic Tasks (Celery Beat)

django-celery-beat stores schedules in the database and manages them from admin. Supports crontab and interval schedules.

# settings.py
INSTALLED_APPS += ['django_celery_beat']

CELERY_BEAT_SCHEDULE = {
    'cleanup-sessions': {
        'task': 'core.tasks.cleanup_expired_sessions',
        'schedule': crontab(hour=3, minute=0),
    },
}

Error Handling and Retry

autoretry_for, retry_backoff, and retry_jitter provide exponential backoff on transient errors. Dead letter queues or failed task admin improve operational visibility.

Use max_retries to prevent infinite loops
Task timeout: soft_time_limit and time_limit
Chord and group for parallel workflows

@shared_task(
    autoretry_for=(RequestException,),
    retry_backoff=True,
    retry_jitter=True,
    max_retries=5,
)
def fetch_external_api(url):
    return requests.get(url, timeout=10).json()

Production and Scaling

Tune worker count (-c concurrency) for CPU vs I/O profile. Isolate critical jobs with separate queues (high_priority, default). Manage worker processes with Supervisor or systemd.

Configure Redis persistence and memory limits
Metrics via Flower or Prometheus exporter
Task idempotency and transaction.on_commit pattern
Test beat + worker together in staging

Conclusion

Celery lets Django apps carry background load without hurting user experience. With proper retry, monitoring, and idempotent design you build a reliable task infrastructure in production.

transaction.on_commit(lambda: task.delay()) reduces partial-save risk
Send IDs instead of large payloads
Prefer Celery 5+ JSON serializer for security

Data Usage Notice