How to Implement Celery and RabbitMQ With Django Framework

Reading Time: 4 minutes

In this blog, we are going to learn about Celery and Rabbitmq for performing tasks asynchronously as well as we will Implement Celery and RabbitMQ With Django Framework.

Prerequisite:

  • Basic understanding of Django framework

What is Celery?

Celery is a task queue for executing work outside a Python web application Http request-respond cycle. Celery can be defined as a task or process manager. It can execute tasks based upon demand or periodically using prioritization and execute tasks when resources are available.

Task Queue?

A task queue input is a unit of work called a task. Dedicated worker processes constantly monitor task queues for new work to perform.

Why Celery?

When the user accesses a certain URL of your application the Web browser sends a request to your server. In the case of Django, the application Django receives this request and performs some tasks based on the request. Usually, it involves executing queries in the database, processing data, etc. Django takes some time to process this request and perform a task so during that time user has to wait until Django sends a response.

We can reduce the time by working around this issue using the cache, optimizing database queries, and so on. But there are some cases where there’s no other option to perform heavy work or tasks. A report page, export of a big amount of data, and video/image processing are a few examples of cases where you may want to use Celery.

The main motive is to respond to the user as quickly as possible, send the time-consuming tasks to the queue so as to be executed in the background, and always keep the backend ready to respond to new requests.

Installation

$ pip install celery

RabbitMQ

Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task the client adds a message to the queue, and the broker then delivers that message to a worker.

So RabbitMQ is a message-queueing software also known as a message broker or queue manager. It is software where queues are defined to which applications connect in order to transfer a message or messages.

Install RabbitMQ

$ sudo apt install rabbitmq-server

Enable RabbitMQ Server:

$ sudo systemctl enable rabbitmq-server

Start the RabbitMQ Server:

$ sudo systemctl start rabbitmq-server

We have installed celery and RabbitMQ, now we are good to go and implement both with Django.

First, create a Django project using the below command:

$ django-admin startproject <poject_name>

Under this project create a Django app:

$ python3 manage.py startapp <app_name>

Consider the following Django project named demo with an app name app1:

├── app1
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── demo
│   ├── asgi.py
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-38.pyc
│   │   └── settings.cpython-38.pyc
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── manage.py

So in order to use celery in our Django project first, we must define an instance of celery or an instance of celery library.

let’s create a new file named celery.py inside the Django project to define an instance of celery.

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'demo.settings')
app = Celery('demo')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

First, we set the default DJANGO_SETTINGS_MODULE environment variable for the celery command-line program:

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'demo.settings')
app = Celery('demo')

We also add the Django settings module as a configuration source for Celery. This means that you don’t have to use multiple configuration files, and instead configure Celery directly from the Django settings, but you can also separate them if wanted.

app.config_from_object('django.conf:settings', namespace='CELERY')

Next, a common practice for reusable apps is to define all tasks in a separate tasks.py module, and Celery does have a way to auto-discover these modules:

app.autodiscover_tasks()

With the line above Celery will automatically discover tasks from all of your installed apps, following the tasks.py convention.

Now we will create a file called tasks.py inside the Django application that is app1.

from celery import shared_task

@shared_task()
def prime(num):
    flag = False

    if(num > 1):
        for i in range(2, num):
            if (num % i) == 0:
                flag = True
                break
    if flag:
        print(num, "is not a prime number")
    else:
        print(num, "is a prime number")

I have created a task that will check whether a number is prime or not.

Now we will start the celery worker using the below command:

$ celery -A demo worker -l info

Now we will use the Django shell to execute our prime function.

$ python3 manage.py shell

Import the prime function:

$ from app1.tasks import prime

Celery provides delay and apply method to call task so we will use the delay method to call task.

prime.delay(5)

Now go to your terminal where celery is running you will see the output.

Conclusion:

So far in this blog, we have seen how to install celery and RabbitMQ as well as how to implement celery and RabbitMQ with Django. I hope you liked it. Please let me know if any changes are required and give your valuable feedback in the comment section.

References:

https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html

Written by 

Jubair Ahmad is a Software Consultant (DevOps)at Knoldus.Inc.He loves learning new technology and also have interest in playing cricket.

Leave a Reply