I have a question with respect to the setup i have, i currently use an active active setup with a tcp load balancer in google cloud. Request can be routed to any instance from the load balancer.
But with this setup, I seem to have issues with blocking commands that run from multiple servers, I use these commands on queues. When I push data to the queue, and when two servers have blockingpop commands on the two active keydb setup(the load balancer round robin) the first server gets data from queue, when this data was replicated to second active keydb instance, the second server also gets this data from queue. This is making my system process data from queue twice.
Now mitigation would be to have only one master and another as hot backup for keydb. Is there any other solution that i am missing?
With the current implementation you are correct about the behavior and the only workaround is the hot spare as you mentioned.
That said I think blpop should be atomic with respect to Active Replication so I consider this to be potentially a bug. I need to think about it for a bit before attempting changes here.
Thanks for the reply. Will be using the current hot spare method till a new solution is found
I just discovered this issue as well. I posted a test project that demonstrates it.
I’ve started work on this feature. If an Active Replica unblocks a client as a result of blpop it will inform the other replica not to unblock a client. This should ensure that blocking clients are only handled once with a two server active replication setup. For multimaster the same issue will exist until we implement proper CRDT support.