The flare project celery uses the pits encountered in redis sentinel

flask project celery Error reported in Can’t reconnect until invalid transaction is rolled back

Found to be celery There is one place in the procedure of db.session.commit() No exception was handled , add to try except InvalidRequestError and db.session.rollback() After the repair . It's been SQLAlchemyError.

celery The database connection pool size limit in and the connection is not released

QueuePool limit of size 10 overflow 10 reached, connection timed out
The database connection used after query has not been released .
Solution ： Automatically release the connection .
Reference resources ：https://stackoverflow.com/questions/24956894/sql-alchemy-queuepool-limit-overflow

@app.teardown_appcontext
def shutdown_session(exception=None):
    db.session.remove()

In addition, increase the size of the database connection pool .

celery+flask+redis+sentinel How to configure and initialize with password

Set directly at the beginning REDIS_URL = "sentinel://:[email protected]:26379/0" Report errors ：

ValueError: Couldn't import 'app.celery': Redis URL must specify one of the following schemes (redis://, rediss://, unix://)

Thought it was celery, But this is actually initialization redis-client What happened .
take redis_client Initialization of from Flask-redis Change it to redis.sentinel

# redis_client = FlaskRedis(app)
redis_client = Sentinel([('localhost', 26379)], socket_timeout=1, sentinel_kwargs={
    'password': 'password'})
print(redis_client.discover_master("mymaster"))
#  Example of print results : 
# ('10.1.1.1', 6379)
# redis Use examples 

That's it redis, Then solve it later celery Of broker and result backend Configuration of
(mymaster Is the default primary cluster ）

BROKER_URL = "sentinel://:[email protected]:26379/1"
BROKER_TRANSPORT_OPTIONS = {
    
    'master_name': 'mymaster',
    'socket_timeout': 1,
    'sentinel_kwargs': {
    'password': 'password'}
}
CELERY_RESULT_BACKEND = "sentinel://:[email protected]:26379/2"
CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS = BROKER_TRANSPORT_OPTIONS