Running Celery in Production

This article explains , how to use celery in production and also follows best practices of celery.

Hello all. Today we will see how to run celery in production (daemon mode).

In local system we used to run with command like this

    celery -A celery_app worker -l INFO

Why daemon mode?

     In production we need to run celery in daemon mode and celery need to be started automatically whenever system reboots.

we are going to use  celeryd to run celery in production.

What we are gonna do here?

• Install celeryd package
• Simple celery app
• Run celery in daemon mode

Installation

   $ sudo apt-get install python-celery celeryd

Prerequistes:

• RabbitMQ (see my previous article here)
• Celery ( pip install celery)
• Python(version 2)

Sample celery app

save this code as  testapp.py

from __future__ import absolute_import
from celery import Celery
import sys
import time


celery_url = "amqp://%s:%s@%s//" % ("guest", "guest", "0.0.0.0")

celery_app = Celery("testapp", broker=celery_url,
 backend='rpc://')


@celery_app.task()
def add(x,y):
 print "hi from task add", x, y
 return x+y


def main(x,y):
 """
 This is main function gets called initially while running
 """
 print "Let's add %s and %s" % (x, y)
 res = add.apply_async(args=[x, y])
 time.sleep(3)
 if res.ready():
 print "Result from task %s" % res.result

 

if __name__ == '__main__':
 if len(sys.argv) < 3:
 print "Usage: python testapp.py 1 2"
 sys.exit(0)
 x = int(sys.argv[1])
 y = int(sys.argv[2])
 main(x,y)

Configure celeryd in /etc/default/celeryd

If you have installed  celeryd.  it created daemon script in  /etc/default/celeryd.

change/add these lines in  /etc/default/celeryd

ENABLED="true" # enable daemon mode
CELERYD_CHDIR="/home/bala/personal/python/celerytest/main"
# change above line to the directory which contains our testapp.py
CELERYD_OPTS="--app=testapp --loglevel=info --time-limit=300 --concurrency=8"

Now restart celeryd to reflect config changes

   $ sudo service celeryd restart

Note: change CELERYD_CHDIR to your directory which has our testapp.py

Run celery in daemon mode

Now to run open two tabs in terminal

Start celeryd by

  sudo service celeryd start

which results like this

celery multi v3.1.6 (Cipater)
> Starting nodes...
 > w1@bala: OK

# you can check the logs in /var/log/celery/w1.log

Run python file by python testapp.py 2 3

Output 

Let's add 2 and 3
Result from task 5

You can find complete source code in my repository here

That's it . Thanks for reading . Happy coding !!!

Keywords: Rabbitmq Celery