Import data from Excel File Using Celery, Pandas using Django Signals

[Updated 29th May, 2023] Today I’m gonna share an interesting tutorial with you. We will import data from an excel file and send it to the background task which will be executed by celery. Also to read the data from excel, we will use Pandas. Everything will be configured in Django. Using Django signals we will invoke the celery tasks. Btw we need redis too ! So it will help you learn couple of technologies at once! if you found this article useful, please clap and share and share your comments below !

For example you have a phonebook application in which you want to insert bulk data from excel. You can use third party packages but the problems of using third party packages are not very low in numbers. Anyway look at the models.py

We have a simple PhoneBook model in which we will insert data. So to do that we need another model to upload the file PhoneBookImport. But how to use Celery ? We need to configure it first. You can find here https://medium.com/swlh/asynchronous-task-with-django-celery-redis-and-production-using-supervisor-ef920725da03

pip install celery
pip install redis
pip install pandas 
pip install openpyxl
#setup redis on local machine 
# Add this to settings 
CELERY_BROKER_URL  = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Asia/Dhaka'

After installing celery and redis , Add this on __init__.py on main project directory

from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app
__all__ = ('celery_app',)

then create a celery.py file in your project root directory where settings.py file resides

Here “customuser” was our demo project name and change it to your project name app = Celery(‘projectname’)

Now create a tasks.py file in your app directory in our case phonebook/tasks.py

Here we imported some modules related to celery i.e shared_task which binds the tasks with celery and let celery knows that it is a task and using sleep method we just delay the execution of the tasks. In line 7 we initialized the celery app using the projectname. import_phonebook() method has an decorator @shared_task which binds it to a celery task. So if the task is called by celery, it will execute all the methods which are bined to @shared_task decorator.
The method import_phonebook(duration,id) receives two parameter , one is duration and another one is the id of the file being uploaded. We will pass the id using django signals ! So what is django signals ?

Basically whenever we do something with the model i.e try to save or after saving data, we can execute some operations. Signal is basically does the same thing. django.db.models.signals.post_save and django.db.models.signals.pre_save . Here pre_save singnals fires before saving data into the model. and post_save signals fires after the data is save into the model. For example we can use post_save signal to initiate user profile right after registration or use pre_save signals to add slug field data.

Anyway, create a signals.py file in app directory i.e phonebook/signals.py

At line 7 we initialized the method with the model PhoneBookImport on pre_save signal. So whenever we save any data to the PhoneBookImport model, the method insert_records(sender, instance, created,*args, **kwargs) is called and we get the intance and at line 9 we only call the task import_phonebook.delay(3,instance.id) with a delay of 3 seconds and the intance.id of the record of PhoneBookImport Model.

Now if you try to upload the file like this

Nothing will work right? Because we need to configure the signals to the app!
Edit the apps.py [phonebook/apps.py]of the app phonebook and add the following lines like this

Just add lines 7 and 8 where we tell the app’s configuration class then use the signals [signals.py]

Now we need to run celery by using the following command

celery -A projectname worker -l info -P threads

If you now try to upload now, then the celery will receive the task, and execute based on the delay duration.

Celery will execute the tasks like this

After finishing the tasks, it will change the status as Done.

So here are the records imported

I kept things simpler and didn’t add much logic in it. We can use any kinds of calculation, validation , feature engineering etc. using Pandas before importing.
Don’t forget to follow me on Medium !

Explanation of Pandas code

In line 16 we read the file being upload using pd.read_excel() method.
As panda converts the object of line 16 into a dataframe, we need to convert it to a dictionary records so that we can iterate through it. We just looped over the results and inserted. We can put any try except logic as we want here.

If you wanna learn Django management command to import all country languages at once ,then read this
https://medium.com/@mahmudtopu3/import-all-country-languages-using-django-management-command-fc2c7725981f