Asynchronous Tasks in Django with Redis and Celery

This study shows you why the celery line is so relevant and how celery can be used in your Django app in conjunction with Redis. We build a small image management framework to display a particular application that summarizes the photos the user submits.

This tutorial offers an overview of why the celery message queue is useful in combination with celery use in a Django program. I will create a minimalist image processing tool that creates thumbnails of images provided by users to illustrate implementation details.

Background on Celery and Redis Message Queues

Celery is a Python based task queuing software package that allows asynchronous workloads powered by messages generated in application code (Django in this example) for Celery’s task queue. Celery can also be used to perform repeatable, time-limited tasks, but not the subject of this post.

In combination with a storage solution also called a message broker, celery is frequently used. Redis is an efficient memory with a key-value datastore. A popular message broker used with celery. In particular.

In the Python / Django world, Celery is the most used to manage this kind of tasks. Instagram uses it for example to feed its users’ feeds.

So we have a set of tasks to process, sometimes in significant volume. The objective is to distribute them among multiple workers, possibly running on different machines. The workers constantly monitor the Task Queue to get new tasks to process.

Concretely, in Celery, the communication between the client who creates the tasks and the workers is done through a Message Broker, for example Redis (“key-value in-memory store”).

Celery is a powerful library capable of handling the distribution and processing of millions of tasks per day. For smaller projects, it may be interesting to look at lighter alternatives such as Huye, the self-proclaimed “little task queue”.

Open chat
Powered by