Hola, Fellow Developers 👋

In this post I am going to write how to implement jobs scheduling in Python.

I was recently working on a Flask app where I encountered with a task where I need to poll the Azure message queue to read messages after each 2 hours and use this messages for further processing. Since, this was just a simple task of polling which need to implemented so I thought I need to look for simpler solution for this purpose.

I banged my head around different stack overflow solutions, tech blogs related to problems. I found that I need to implement threading for this, make use of Celery workers, and blah blah 🙄. I tried some sample solution on stack overflow based on threading but that doesn't seems to work for me 😴

Let's dive further to see the solution that worked for me and various other possible solutions I found.

Table of Contents

1. What is meant by scheduling of job?
2. Use cases of scheduling
3. My first simplest solution
4. Implementation of solution that worked
5. Other possible solutions

1. What is meant by scheduling of job?

In simplest term, scheduling of job means that we want a task to be performed at scheduled time or regularly at a given item. Also, jobs can be scheduled to be executed after a given definite interval of 5 mins, 60 mins or so.

2. Use cases of scheduling

There can be several use cases of polling as per the requirements of the application which is being developed.

1. Read the database for any data at scheduled time everyday.
2. Check any of the message queue for new messages(Azure Message Queue or AWS Queue).
3. Logging application status at definite interval or at scheduled time.

3. My first simplest solution

The first solution I thought of for this problem is to use time.sleep() in an infinite loop. Later, I found polling library available in Python which can be used for this purpose. Below is the sample code I used with polling library:

polling2.poll(
    lambda: webhook_queue.receive_message() == 'Stop', 
    step=5,
    poll_forever=True)

This code stops polling when 'Stop' message is received. It's scheduled to poll forever at an interval of 5 seconds(used for test). After, running this code it was working fine. I thought yeah this solution worked 😀

But what after running the server ?

After, running the server I saw that the flask app is not fulfilling any other requests and is just busy with polling. I realized that this solution is not gonna work. I started looking off to other solution and the solution I found was around Celery workers implementation. I thought this would be my last choice if I couldn't find any other simpler solution.

And wonder I found it :P

4. Implementation of solution that worked

After searching a lot I found that there's a APScheduler library available in Python which can used for scheduling of jobs. I thought of giving it a try and it worked for me :) Below is my implementation by making use of APScheduler python library.

#set the background scheduler for doing the scheduled task
schedule_task = BackgroundScheduler(daemon=True)

#add the job to be completed to the scheduler
schedule_task.add_job(polling_task, 'interval', minutes=120)

#start the task scheduler
schedule_task.start()

I have passed the polling_task function which needs to be executed after every 2 hours as a job to scheduler. Make sure you install the APScheduler in your app env and import BackgroundScheduler for this solution to work for you too.

Let's see other possible solution I found while searching the solution to this problem.

4. Other possible solutions

I have found few other possible solution that would work for this problem. Though, I haven't tested it so read the docs and see if it fulfill your needs.

1. Making use of Celery workers to add background scheduled task.
2. Use of Flask APscheduler

That's all for this blog. I hope this blog will save your precious time in case you meet with similar problems to be solved!!!

Follow me on Twitter

Connect with me on Linkedin