Background Jobs and Workers

A background worker is a program that does slow jobs after the app server has answered the request. The user gets a quick answer while the slow job finishes afterwards.

An upload answered at step one

A photo upload arrives at the app server. The server answers accepted at step 1 and leaves the resize job waiting in the queue.

queue = []
step = 1
queue.append("resize-photo")
print("Step", step, "- answer sent: accepted")
print("Step", step, "- jobs in queue:", queue)

The same work inside the request path

Without a background worker, the resize runs before the answer. The answer cannot leave until step 4 in this timeline.

print("Step 1 - upload received")
print("Step 2 - resizing the photo")
print("Step 3 - resizing the photo")
print("Step 4 - answer sent")

The worker finishes the job at step four

The worker takes the resize job from the queue after the answer. The queue is empty when the job finishes at step 4.

queue = ["resize-photo"]
print("Step 2 - worker starts:", queue[0])
print("Step 3 - worker resizes the photo")
queue.pop(0)
print("Step 4 - job done, queue length:", len(queue))

Example

queue = []
step = 1
queue.append("resize-photo")
print("Step", step, "- answer sent: accepted")
print("Step", step, "- jobs in queue:", queue)
print("Step 2 - worker starts:", queue[0])
print("Step 3 - worker resizes the photo")
queue.pop(0)
print("Step 4 - job done, queue length:", len(queue))

Key points

Background worker: a program that finishes slow jobs after the answer has been sent.
Request path: the steps that happen while the user waits for an answer.
Accepted answer: a quick response that confirms a request arrived and its job is queued.
Use background workers to move slow work like photo resizing off the request path.

In this example, the answer leaves at step 1 while the resize job is still in the queue. The worker finishes that job at step 4, so slow work stays off the request path and the user is not kept waiting.

Note

the result of the job appears later, after the job finishes.

Try it Yourself

Edit the code, then press Run.

Running is simulated

queue = []
step = 1
queue.append("resize-photo")
print("Step", step, "- answer sent: accepted")
print("Step", step, "- jobs in queue:", queue)
print("Step 2 - worker starts:", queue[0])
print("Step 3 - worker resizes the photo")
queue.pop(0)
print("Step 4 - job done, queue length:", len(queue))

Exercise

Why move a photo resize job to a background worker?

So the photo never gets resized at all. So the queue can hold the job forever. So the app server can answer at once and the resize can finish later. So the user waits for the slowest step.