Home → Problems →

Is add-item-quantity atomic?

• Edit

david.ells

25 Oct, 2011 09:06 PM

From some recent logs we just looked over, we found an odd discrepancy between what we logged and the usage recorded in Cheddar. The only explanation I have for it is that add-item-quantity is not atomic, which is to say, if add-item-quantity is called twice on the same account at nearly the same time, it may only result in a single increase of the usage number. Can you confirm this behavior?

1. Support Staff 1 Posted by Marc Guyer on 26 Oct, 2011 03:47 PM

   

   add-item-quantity is additive and all calls to it are honored even if they occur at the same time.

   Related: the latest call to set-item-quantity always wins.

   • Edit

2. 2 Posted by david.ells on 26 Oct, 2011 04:15 PM

   

   Is there anyway we can access a detailed history of add-item-quantity calls / results in order to audit / reconcile our usage tracking versus the one recorded in Cheddar?

   • Edit

3. 3 Posted by david.ells on 26 Oct, 2011 05:08 PM

   

   Actually, I think I have validated my thought here. I believe you should go back and check out the code. I don't think add-item-quantity is additive even in the face of near instantaneous updates. In fact, it seems to do quite poorly with lots of concurrent requests. I've attached a very short bash script which will demonstrate the problem. It makes (practically) about 50 concurrent "curl" requests on the add-item-quantity call. In my tests, only around 15 are actually ever reflected in cheddar.

   I thought at first it might have been an error with how I was trying these concurrent curl commands, but testing against a local debug page, every request came in fine. I can also repeat this behavior using our java wrapper library. Thinking it might have something to do with our java code, I decided to try and simplify down to a simple bash script, and found the results agreed. Can you explain why I'm seeing this behavior? Can you confirm the same behavior in your environment? Thanks!

   • do_add_item.sh 536 Bytes
   • Edit

4. Support Staff 4 Posted by Marc Guyer on 27 Oct, 2011 03:47 PM

   

   Thanks very much for looking into this further and for the bash script. That was a big help.

   Using your script, we were able to test this and find the problem. Our existing tests never ran any concurrent requests. Our own usage of the add-item-quantity call would not have exposed this problem since the requests are queued and run serially.

   You learn something new every day...

   We've released a patch which solves the problem almost 100%. However, we suspected that we might have a problem with the very first 2 (or more) requests running currently causing one or more of them to be overwritten. The first item quantity setting causes an insert. After several attempts, we were able to get it to fail and record only 49 adds instead of the expected 50. We're still looking into that. It seems to happen about 10-20% of the time.

   • Edit

5. 5 Posted by david.ells on 27 Oct, 2011 04:12 PM

   

   Marc,

   Good deal, thanks for taking a second look. Our next option here was to do something similar and queue up the requests or implement some locking in order to call the cheddar API, but it's nice to avoid the artificial code there and let Cheddar handle the concurrency, since we don't have to lock anywhere else along that path...

   I think the lingering issue you mentioned is fine for our use. Because our usage numbers number in the 100's and 1000's, we can tolerate the occasional loss of a single record, and don't much expect it to come up actually.

   I don't much like having to prove my question is valid and really worth checking out, but I do appreciate the speedy patch release. Thanks for that.

   • Edit

6. Support Staff 6 Posted by Marc Guyer on 04 Nov, 2011 01:13 PM

   

   We've also fixed the sporadic loss of the very first call to add-item-quantity for a customer. Thanks again for bringing this to our attention!

   • Edit

7. Marc Guyer closed this discussion on 04 Nov, 2011 01:13 PM.