Home → Problems →

Delay in hooks reverts order of operations

• Edit

Stéphane Géraud

21 Aug, 2020 02:48 PM

We had two requests sent by cheddar hooks in the wrong order.

User: "[email blocked]"
In rows 8,9 in https://www.getcheddar.com/admin/activity/hooked-events/orderBy/cre... we see:

2020-07-26 16:38:42 subscriptionReactivated 200 OK
2020-07-26 16:34:38 subscriptionCanceled 200 OK

But our backend received:

• cancelation hook at 2020-07-26 16:51:23
• reactivation hook at 2020-07-26 16:50:36

This causes that we mark a customer as inactive instead of active.

What produced those delays of 12 and 17 minutes?
Is it possible to send requests in the same order as hooks?
If not, a timeout for the delay a hook is sent after the activity started can help too.

1. Support Staff 1 Posted by Mike Trotzke on 23 Aug, 2020 07:45 PM

   

   Thanks for the report. We'll dig a little deeper and let you know what we find.
   Mike

   • Edit

2. 2 Posted by Sebastien Migno... on 28 Aug, 2020 06:25 PM

   

   Hi Mike.
   Is there any new?

   • Edit

3. 3 Posted by Sebastien Migno... on 03 Sep, 2020 12:02 PM

   

   Hello.

   • Edit

4. Support Staff 4 Posted by Mike Trotzke on 03 Sep, 2020 02:03 PM

   

   Sebastian,

   Sorry for the delay. This question has sparked some internal discussion from multiple team members. We’re working on some updates to our documentation to make this more clear.

   The order in which hooks are received is not guaranteed. The nature of the hook retry system means that individual hooks could fail and be retried later. If this occurs, we don’t intend to block subsequent hooks. They are delivered as usual by design.

   Hooks themselves contain information as to the date and time of the event they represent (activityDatetime). In your example, there is also an explicit subscription cancelation and create time. Best practice when using hooks to manage authorization (in conjunction with an at login verification) is to cache the create and cancelation dates as opposed to just a binary flag. This way you can manage the handling of hook timing and retries without a single failure causing a product wide lock on hooks (or the resulting catch-up load spike).

   On our end, hook processing is queued across multiple load balanced servers. This also can affect timing of delivery. During peak processing times (like when we are running batches of recurring transactions), it isn’t unusual for there to be a few minutes delivery delay— another reason to use data sent instead of hook execution timing.

   We recently migrated to a new hosting platform and made upgrades to our hook queuing system to improve performance and reliability. We’re investigating any changes there that may impact hook delivery order. However, it’s important to note that regardless of any changes we make, retries will continue to mean that it is best practice to leverage timing information contained in the hook to manage hook timing.

   Thanks for the question. It has prompted us to update our knowledge base to be more explicit regarding hook timing as it relates to idempotence.

   Mike

   • Edit

5. Mike Trotzke closed this discussion on 03 Sep, 2020 05:36 PM.