CellarTracker Main Site
Register for Forum | Login | My Profile | Member List | Search

Caching resiliency

 
View related threads: (in this forum | in all forums)

Logged in as: Guest
Users viewing this topic: none
  Printable Version
All Forums >> [Cellar Talk] >> Release Notes >> Caching resiliency Page: [1]
Login
Message << Older Topic   Newer Topic >>
Caching resiliency - 5/24/2011 11:01:00 PM   
Eric

 

Posts: 17326
Joined: 10/10/2003
From: Seattle, WA
Status: offline
Over the past 6 weeks people have been running into sporadic glitches that were causing wines to seemingly multiple in their cellar. The underlying data was safe and solid, but there some issues maintaining certain caches that the site depends upon. This has actually been a lifelong issue for the site but started to become considerably more prevalent starting on April 3rd due to some changes I had made on behalf of overall site performance and concurrency.

A few threads from the support forum:

http://www.cellartracker.com/forum/tm.asp?m=145097
http://www.cellartracker.com/forum/tm.asp?m=141504
http://www.cellartracker.com/forum/tm.asp?m=139850
http://www.cellartracker.com/forum/tm.asp?m=138834

Over the past day I have made some rather deep changes to try and prevent this from happening and to make the site more resilient. I don't think it is 100%, but it should be dramatically better. I will continue to look at this over the next few days until it is hopefully 100%.


_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker
Post #: 1
RE: Caching resiliency - 5/26/2011 9:47:02 AM   
Eric

 

Posts: 17326
Joined: 10/10/2003
From: Seattle, WA
Status: offline
I am rather delighted to say that these changes seem to be working quite well. I did find and iron out two other related issues, and I have one more preventative change to make today.

Of course deadlocks and race conditions can and still will happen all the time. Where before you might see an error message and then resultant weird/glitchy and inconsistent data (e.g. a consumed bottle still appears to be in inventory), now in the error case I use exception handle (specifically the rather elegant and simply try/catch mechanism in SQL 2008) to queue up a background task to go and fix the cache in question. There is a background task that runs these fixup jobs every 10 seconds, so it is possible that you might hit a glitch (e.g. when adding or drinking a bottle) and not realize it. Maybe the drinking task will complete, but the bottle will still seem to be there. However by the time you refresh everything will already be fixed up. To put this in perspective, out of roughly 35,000 additions and removals in a given day people seemed to be hitting about a dozen glitches per day in total. The only except is on Sunday night when I run a very aggressive maintenance job that last week triggered 150 errors in one hour. Anyway, now, even in the error case, the worst thing you will see is a little brief latency/inconsistency that will fix itself up automatically.


_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker

(in reply to Eric)
Post #: 2
RE: Caching resiliency - 5/26/2011 1:12:57 PM   
Eric

 

Posts: 17326
Joined: 10/10/2003
From: Seattle, WA
Status: offline
OK, I actually implemented three separate competitive changes (constraints) that should make it impossible to get row duplication for some critical caches. No errors are showing up in the logs any more, and the exception handling seems to be working smoothly. At some point in the next month with a future maintenance update (I will be deploying a 3rd backend host server) I will take a late night opportunity to regenerate all the major caches from scratch during a planned overnight update. However for now I have written and run dozens of consistency checking queries. Things are looking good.

Scratch a nasty one off the todo list. I am very pleased with myself which is a very rare state of being for me.


_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker

(in reply to Eric)
Post #: 3
RE: Caching resiliency - 5/26/2011 7:28:52 PM   
brigcampbell

 

Posts: 2483
Joined: 2/16/2009
From: Mission Viejo, CA
Status: offline
Nice, its the feeling of accomplishment and piece of mind knowing the holes on the code aren't constantly blowing holes in the application.

<slap on the back>

_____________________________

Sincerely,

-brig campbell

(in reply to Eric)
Post #: 4
Page:   [1]
All Forums >> [Cellar Talk] >> Release Notes >> Caching resiliency Page: [1]
Jump to:





New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts


Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI

0.094