Celery
Celery is an asynchronous task queue/job queue based on distributed message passing.
http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html
Slackware 14 installation
- easy_install celery
- pip install celery
Check installation:
Test app with Redis
Run redis:
Toggle line numbers
1 $redis-server &
2 $redis-cli
3 redis 127.0.0.1:6379> quit
App:
Toggle line numbers
1 $celery -A celeryTest worker --loglevel=info
Run task
Docker compose example
app.py
Toggle line numbers
1 import time
2 import redis
3 from flask import Flask,jsonify
4 from celery import Celery
5
6 celery = Celery()
7 celery.config_from_object('celeryconfig')
8
9 """
10 docker-compose build
11 docker-compose run
12 docker exec -it celerytest_web_1 sh
13 docker-compose buid
14 docker-compose restart
15
16 docker-compose stop
17 docker system prune -a
18 docker-compose up
19 docker exec -it celerytest_web_1 sh
20 pip freeze
21 """
22
23 app = Flask(__name__)
24 cache = redis.Redis(host='redis', port=6379)
25
26 def get_hit_count():
27 retries = 5
28 while True:
29 try:
30 return cache.incr('hits')
31 except redis.exceptions.ConnectionError as exc:
32 if retries == 0:
33 raise exc
34 retries -= 1
35 time.sleep(0.5)
36
37 @app.route('/')
38 def hello():
39 count = get_hit_count()
40 return 'Hello World!?= I have been seen {} times.\n'.format(count)
41
42 @app.route('/add/<int:op1>/<int:op2>')
43 def add(op1=None,op2=None):
44 """
45 http://localhost:5000/add/2/8
46 """
47 result=(celery.send_task('celery_worker.add', (op1,op2)))
48 return jsonify( add_res=result.get())
Dockerfile-celery
FROM python:3.7-alpine WORKDIR /code RUN apk add --no-cache gcc musl-dev linux-headers COPY requirements.txt requirements.txt RUN pip install -r requirements.txt EXPOSE 5000 COPY . . CMD ["celery","-A","celery_worker","worker","--loglevel=info"]
Dockerfile
FROM python:3.7-alpine WORKDIR /code ENV FLASK_APP=app.py ENV FLASK_RUN_HOST=0.0.0.0 RUN apk add --no-cache gcc musl-dev linux-headers COPY requirements.txt requirements.txt RUN pip install -r requirements.txt EXPOSE 5000 COPY . . CMD ["flask", "run"]
docker-compose.yml
celery_client.py
Toggle line numbers
1 from celery import Celery
2
3 def callback(taskid):
4 print("callback taskid %s"%(taskid))
5 print("callback ready %d task id %s"%(result.get() , result.task_id ))
6
7 celery = Celery()
8 celery.config_from_object('celeryconfig')
9 result=(celery.send_task('celery_worker.add', (2,2)))
10
11 result.on_ready.then(callback)
12 print(result.get())
celery_worker.py
requirements.txt
flask redis celery