python - Celery worker not reconnecting on network change/IP Change -
i deployed celery tasks need performed @ workplace. these tasks huge , bought few high-spec machines performing these. before detail issue, let me brief i've deployed:
- rabbitmq broker on remote server
- producer pushes tasks on remote server
- workers @ 3 machines deployed @ workplace
now, when started whole process smooth tested , process great!
the problem
unfortunately, forgot consult network guy fixed ip address, , per our location, not have fixed ip address our isp. celery workers upon network disconnect freeze , nothing. when network running, because ip address changed, , connection broker not being recreated or worker not retrying connection. have tried configuration broker_connection_max_retries = 0
, broker_heartbeat = 10
. had no option post out here , experts on matter!
ps: cannot restart workers manually everytime network changes ip address kill -9
Comments
Post a Comment