Failed to complete the message with the lock token 'xxxxx'. The lock on the message has been lost

{tocify} $title={Table of Contents}

Error/Issue:

Recently found that a few of the logic app instances were failing with following error:


Failed to complete the message with the lock token



"Failed to complete the message with the lock token 'xxxxx'. The lock on the message has been lost"








Why it happened

 
If you see the duration of the failed instance in the above image, it is over 5 minutes and that's the reason.

Does that mean that a logic app can't run over 5 minutes?

No it's not like that.

The workflow gets triggered by a message in service bus queue and in Peek lock mode.

In Peeklock mode if the message is not processed by the client, it’s lock is released and retry on the message is performed. Once all retry is exhausted/time to live in queue is reached – message is moved to deadletter queue.

When the message is received in ReceiveAndDelete mode, the message is automatically removed from the queue. 


When the message is received in PeekLock mode and if the message is not completed or if the message lock is expired, the message will be automatically available for other receivers.


Thus the message received in PeekLock mode should be removed from the queue by calling Complete() method.


And while performing this complete action, the above mentioned occur and reason is the service bus lock duration.


Default lock duration for the Azure Service Bus transport is set to 30 seconds. Maximum lock duration allowed by the broker service is 5 minutes.


Thus in a scenario, where steps before marking queue message complete took over 5 minutes, the lock was released on that message and thus not available for complete action.


What to do

To handle this scenario, we can use Defer Message feature of Service bus queue.

The idea is to defer the message after you receive(Using PeekLock mode) and after finishing all the steps in workflow, get the deferred message and mark it as complete(so that it can be removed from queue).

When we defer a message, we are basically keeping that message aside in the same queue however it is not available to others for processing.

Also note that defer messages never automatically get moved to Deadletter queue.

But how do we ensure that when we get the deferred message - it is the same which was deferred?

For that we need to use the sequence number property, use the same sequence number of the message while fetching the deferred message.


So to address the issue, added below step after receiving the message

Defer the message in a queue


Use triggerBody()?[LockToken] property against Lock token of the message

After all the processing steps, get the deferred message by providing the sequence number - triggerBody()?[SequenceNumber]

Get the deferred message


Finally, mark the above fetched deferred message as complete by using the LockToken and SequenceNumber from above step

complete the message in a queue

That's it, after saving it tested the workflow and now there was no error even after the duration went above 5 minutes


Successful run









Post a Comment

If you have any suggestions or questions or want to share something then please drop a comment

Previous Post Next Post