In Django, when you run a time-intensive and resource-intensive task, you may feel a performance delay, because python is single-threaded and it runs the code in sequence. Sometimes the API can return a timeout error. We can run the time-intensive tasks asynchronously to overcome the delay and timeout issues. To run the asynchronous tasks we are going to use python celery in Django.
Where the celery can be used?
Celery (asynchronous tasks) can be used for various purposes, some of the common examples are listed below.
Sending Bulk Emails
In this tutorial, we are going to integrate celery with an existing Django application for running video processing tasks asynchronously.
My Django project is located in the following path
So in this tutorial, I’ll be referring to the following paths and directories.
Django 4.0.2 or above
If you want to create a Django
If you like to run periodic tasks on the existing Django project, then navigate to the Django project directory.
Install the celery package using the pip package manager.
Verify the installed celery version by executing the following command.
It’ll prompt the version details as shown in the following screenshot.
Install Redis using the apt package manager.
Start redis-server using the following command.
Enable redis-server. This command helps to start the redis server automatically as soon as the server is rebooted.
Check the status of the redis server, by running the below command.
If redis is running successfully, it will return a PONG message.
A sample screenshot for the redis-cli ping command is shown below.
Now it's time to integrate celery with our Django project.
Navigate to the Django core app directory.
Create a file named celery.py with the following snippet.
In the above code, we just created the Celery instance called app, and to use Celery within our project we simply import this celery instance.
A sample celery.py file can be found in the following git repository file.
Add the below code in the __init.py__ file present in the current directory.
A sample __init__.py file can be found in the following git repository file.
Now navigate to the project root directory.
And run the following command to test the above configurations in the Django app directory.
An example execution screenshot is given below for your reference.
Now we are going to create the tasks which are going to be run asynchronously.
Tasks can be created within any app of the project under the filename tasks.py (so that celery can auto-discover them)
Navigate to the Django project directory and create a Django app named multimedia.
Add the app name
"multimedia" to the settings.py file as shown in the following example screenshot.
We are now going to create an API with the functionality of downloading a video from the given URL and creating a subclip from that video.
We need to install a video processing library named moviepy, to do that run the following command in the terminal.
Create a file named
tasks.py and add the following snippet.
An example of tasks.py can be found in the following git repo.
Now open the
views.py file and add the following code. It defines a POST Method API that will get an URL, the time duration to clip a video, and the new video file name.
An example of views.py can be found in the following git repo.
Open the urls.py in the current directory (multimedia app) and add the following code.
Open the urls.py file in the Django core app directory and append
path('', include('multimedia.urls')), as in the screenshot following.
Now run the Django Application in the terminal by running the following command.
Open the Postman Application and enter the request URL and choose POST Method as the request method, in the body choose JSON and add the following data into the body.
An example screenshot has been shown below for your reference.
Now hit the Send button to send the request to the locally running Django Application.
Wait for the request to return the response, and you’ll see the response duration as in the following example screenshot.
Now we are going to transform the function
clip_path() in tasks.py as a celery task by importing the decorator
@shared_task and defining the function
clip_path() using the imported decorator.
Open the tasks.py file and change to code as shown in the below snippet.
An example tasks.py file can be found in the following git repo.
Open the views.py file and change the line where we call the
clilp_path as in the following code.
Open a new terminal and navigate to the Django project directory.
Start the celery worker by running the following command in the terminal.
Now you can see the celery worker picking up the defined task clip_video as shown in the below reference screenshot.
Now open the Postman Application and send the request to the clip_video API.
In the meantime, you can see the task
multimedia.tasks.clip_video is invoked and processing the video cropping as shown in the below reference screenshot.
Now open the Postman Application and check the request duration. you can see that the API called the video processing function asynchronously and returned the result.
We can save the result of the celery to our database for various purposes. To do that we need a library named django-celery-results.
Install the Django celery results by running the command.
django_celery_results to the
INSTALLED_APPS in our project’s
Run the migrations command to create the Celery database.
Now that you can see the tables for
django_celery_results have been created.
An example screenshot is given below for your reference.
Appending the following line to the settings.py file.
CELERY_RESULT_BACKEND = 'django-db'
In order to start the celery automatically after the server reboot, we are going to configure the celery under the supervisor service.
On Ubuntu, we can install supervisor as a Debian package, to do so run the following command.
Navigate to the supervisor configuration directory and create a file named celery.conf.
Add the following line to the
A sample configuration file celery.conf can be found in the following git repository file.
To fetch the latest celery configuration changes added to the supervisor, please execute the below command.
To make the supervisor update the celery configuration, run the following command.
To view the supervisor processes status run the following command.
If the supervisor process is already started, please run the following command to start the process.
Helpful Commands for your reference:
sudo supervisorctl start process_name
sudo supervisorctl stop process_name
sudo supervisorctl restart process_name