SERVICE ADVISORY: Planned Outage (Tuesday, 3/3 @ 10pm PST)

CellarTracker Main Site

SERVICE ADVISORY: Planned Outage (Tuesday, 3/3 @ 10pm PST)

View related threads: (in this forum | in all forums)

Logged in as: Guest

Users viewing this topic: none

Printable Version

All Forums >> [Cellar Talk] >> Release Notes >> SERVICE ADVISORY: Planned Outage (Tuesday, 3/3 @ 10pm PST)

Page: [1]

Message

<< Older Topic Newer Topic >>

SERVICE ADVISORY: Planned Outage (Tuesday, 3/3 @ 10pm PST) - 2/28/2009 4:40:11 PM

Eric

Posts: 17316
Joined: 10/10/2003
From: Seattle, WA
Status: offline

PLANNED OUTAGE: Tuesday, 3/3 10pm PST

On this coming Tuesday, 3/3 starting at 10pm PST there will be a planned outage while I migrate the site to new servers and a new IP address. The planned duration is about 2 hours, but even though I have scripted and tested the move 3 times I could still easily hit snags. So it is best to assume that this will be an overnight outage lasting until the morning on Wednesday, 3/4.

One aspect of the site-update is that I will be issuing a worldwide DNS change to advertise the new IP addresses. Depending upon your ISP and whether they refresh DNS according to posted TTL values you may find that typing www.cellartracker.com gets an error from your browser instead of actually taking you to the new site. Conceptually this is like getting a new phone number, except when everything works no one has to change their speed dial, as it updates automatically. Well sometimes the speed dial takes longer to update than it should. So, with that in mind, if you have problems on Wedneday, 3/4, the new site will also be available at http://new.cellartracker.com to help people work around this issue. (Please note that a 'test' version of the site is already up and running at that URL, but please do not bother adding/editing data there as it is just a SNAPSHOT and will be blown away during the switchover.)

Once the move is complete it should be invisible to end users. No one should need to update any favorites. No functionality should be changing as part of this. This is just about me moving to new hardware, upgraded software and a new secure, private physical space that is no longer shared with other hardware. If you will need to access your data that evening I highly recommend you follow the instructions here to download a copy now: http://www.cellartracker.com/ow.asp?ExcelDownload

And for those curious, the updates include a pair of new HP dual quad-core servers, a new HP SAN, VMWare with high availability and upgrades to SQL 2008 and Windows 2008 all running 64-bit editions. And LOADS more RAM. Plus multiple overlapping backup schemes that, in the event of future catastropic hardware failure (I hope not but you have to plan for it) close to 0 downtime and minimal data loss. And eventually I will shell out even more $$ for a 3rd server and the crazy cool VMotion from VMWare. But first things first.

I also will probably do an email out with this information on Monday, but I recommend you save a cpy if you are reading this. BTW, my email is hosted entirely separately so you should be able to email me throughout in case you have questions or issues.

< Message edited by Eric -- 2/28/2009 4:42:56 PM >

_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker

Post #: 1

RE: SERVICE ADVISORY: Planned Outage (Tuesday, 3/3 @ 10... - 3/1/2009 4:51:57 PM

rjonas

Posts: 371
Joined: 11/5/2006
From: Newcastle, WA
Status: offline

Eric -

As an HP employee, I'd like to THANK YOU for your business.

Best, Ron

_____________________________

I'm not as think as you drunk I am!

(in reply to Eric)

Post #: 2

RE: SERVICE ADVISORY: Planned Outage (Tuesday, 3/3 @ 10... - 3/1/2009 8:28:03 PM

Eric

Posts: 17316
Joined: 10/10/2003
From: Seattle, WA
Status: offline

Hey Ron, I have been a happy Dell guy for the past 5 years, but I felt like going with HP this time. Time for a change I guess. It is serious hardware for sure!

_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker

(in reply to rjonas)

Post #: 3

VMotion - 3/3/2009 1:50:45 PM

Snookboy

Posts: 1
Joined: 3/3/2009
Status: offline

VMotion is the bomb. Gotta shell out some serious $ for it, but it is sweet. In any case, this sounds like a pretty serious upgrade to the system as is. The VMotion will be an easy addition later.

(in reply to Eric)

Post #: 4

RE: VMotion - 3/3/2009 2:18:57 PM

Eric

Posts: 17316
Joined: 10/10/2003
From: Seattle, WA
Status: offline

Yes, from what I can tell VMotion is a dream. I have the core of what I need already with High Availability. So if a physical server melts down VMWare is already set up to migrate the image to the other physical box and reboot the virtual server(s). The images are sitting on a SAN with RAID10, two sets of HBA's, two separate switches, two power supplies, two extra hot spare drives, battery backed cache, so the SAN has a lot of redundancy built in. So basically a catastrophic failure of one of the 2 servers should just look like server(s) rebooting. In our testing (just yanking power from one of the servers) everything fired up and was running on the one remaining server in about 5 minutes. Now with VMotion there would be zero downtime apparently and not even a reboot. Sounds too magically cool to believe... It's only about another $12K above what I am already about to be running and apparently just a license upgrade and even a free 30 day trial to see it in action. My plan down the line is to add a third physical server and VMotion for the ultimate coolness and ease of maintenance. All in good time though. Plenty of cool (invisible) changes coming tonight. I am just praying that SQL runs well enough in a virtual environment for my needs--it's a somewhat controversial topic out there in cyberspace. I will know pretty quickly once I throw the real load at the new site...

The calm before the storm...

< Message edited by Eric -- 3/3/2009 2:27:58 PM >

_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker

(in reply to Snookboy)

Post #: 5

RE: VMotion - 3/3/2009 9:46:43 PM

Eric

Posts: 17316
Joined: 10/10/2003
From: Seattle, WA
Status: offline

T-15 minutes until the migration starts...

_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker

(in reply to Eric)

Post #: 6

RE: VMotion - 3/4/2009 1:10:29 AM

superbarre

Posts: 209
Joined: 8/1/2008
From: Leuven, Belgium
Status: offline

and we're up!!!!!

(in reply to Eric)

Post #: 7

RE: VMotion - 3/4/2009 1:13:28 AM

Eric

Posts: 17316
Joined: 10/10/2003
From: Seattle, WA
Status: offline

Yep!

There will be some DNS weirdness for a few hours. You can browse on http://new.cellartracker.com/ for a little while and over the next few hours http://www.cellartracker.com should sort itself out to come to the same place. (So in other words, if you have favorites don't bother fixing them all up. They will work in a few hours or end of day at worst.)

_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker

(in reply to superbarre)

Post #: 8

RE: VMotion - 3/4/2009 4:23:40 AM

burnhard

Posts: 18
Joined: 6/9/2008
Status: offline

I'm not sure if you noticed it: There seems to be a problem with label image uploadson the new server. The message says this:

Microsoft VBScript runtime error '800a01ad' ActiveX component can't create object /lbl_up.asp, line 16
Seems like the ActiveX component for the uploads is missing - it hasn't made the move to the new server yet...

(in reply to Eric)

Post #: 9

RE: VMotion - 3/4/2009 8:42:18 AM

Eric

Posts: 17316
Joined: 10/10/2003
From: Seattle, WA
Status: offline

quote:

ORIGINAL: burnhard

I'm not sure if you noticed it: There seems to be a problem with label image uploadson the new server. The message says this:

Microsoft VBScript runtime error '800a01ad' ActiveX component can't create object /lbl_up.asp, line 16
Seems like the ActiveX component for the uploads is missing - it hasn't made the move to the new server yet...

It did make the move, but I must be missing something. I will get on that as soon as the coffee settles in...

_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker

(in reply to burnhard)

Post #: 10

RE: VMotion - 3/4/2009 10:30:56 AM

brigcampbell

Posts: 2483
Joined: 2/16/2009
From: Mission Viejo, CA
Status: offline

http://en.wikipedia.org/wiki/ES7000

And if you need hotswap and 96 cores running windows. it's good to know they're headroom for CT.

_____________________________

Sincerely,

-brig campbell

(in reply to Eric)

Post #: 11

RE: VMotion - 3/4/2009 10:34:01 AM

Eric

Posts: 17316
Joined: 10/10/2003
From: Seattle, WA
Status: offline

That ES7000 gets an OMFG, that is so cool! I wish I had $500K to toss on a server, because I would be cuddling and caressing that one!

_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker

(in reply to brigcampbell)

Post #: 12

RE: VMotion - 3/4/2009 11:02:42 AM

Eric

Posts: 17316
Joined: 10/10/2003
From: Seattle, WA
Status: offline

quote:

ORIGINAL: burnhard

I'm not sure if you noticed it: There seems to be a problem with label image uploadson the new server. The message says this:

Microsoft VBScript runtime error '800a01ad' ActiveX component can't create object /lbl_up.asp, line 16
Seems like the ActiveX component for the uploads is missing - it hasn't made the move to the new server yet...

OK, this is fixed.

_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker

(in reply to burnhard)

Post #: 13

RE: SERVICE ADVISORY: Planned Outage (Tuesday, 3/3 @ 10... - 3/4/2009 7:56:23 PM

wallstreet

Posts: 465
Joined: 10/27/2008
From: Florida
Status: offline

Hi Eric,

FYI, if you are using enterprise edition of VMware, Vmotion is included. I hope you are managing it through Virtual Center. A very cool management console. Is your storage an EVA4400? Great box. Good choice on servers. I am glad you didn't go DELL, they are pieces of sh*t compared to HP and even IBM. Look into Visioncore for backup of Vmware or even VCB which I believe is included as well. Now if you want to get fancy, take a look at SRM for DR purposes :

You really should consider a 3rd server when the funds allow. Thats best practices for a min. Vmware server farm. I am sure you used P2V to move it from your physical machine to the virtual machine. Piece of cake most likely for you.

Glad it went well. Did you end up going ISCSI ?

Al

_____________________________

"One Man Gathers, What Another Man Spills"...Robert Hunter 1968

----
Wine, Watches And Women....not necessarily in that order

(in reply to rjonas)

Post #: 14

RE: SERVICE ADVISORY: Planned Outage (Tuesday, 3/3 @ 10... - 3/4/2009 11:14:04 PM

Eric

Posts: 17316
Joined: 10/10/2003
From: Seattle, WA
Status: offline

Yes iSCSI. No P2V. Just built up clean, new images and then migrated backups. I used this as an opportunity to move from 32-bit Windows 2003 and SQL 2005 to 64-bit Windows 2008 and SQL 2008. Playing it a little dangerous by changing so many variables at once, but I am quite glad. The SAN is HP StorageWorks Modular Smart Array 2012i Dual Controller with QLogic SANblade QLE4060C. Not sure yet which online backuo product. Definitely want to move to a 3rd server and VMotion. My VMWare license is Infrastructure Standard High Availability Acceleration Kit. So not enterprise. It is another 12K for the current 4 sockets to move up to the big leagues.

_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker

(in reply to wallstreet)

Post #: 15

RE: SERVICE ADVISORY: Planned Outage (Tuesday, 3/3 @ 10... - 3/5/2009 4:32:44 AM

superbarre

Posts: 209
Joined: 8/1/2008
From: Leuven, Belgium
Status: offline

Eric,

congratz on a job well done!

Bart

(in reply to Eric)

Post #: 16

RE: SERVICE ADVISORY: Planned Outage (Tuesday, 3/3 @ 10... - 7/6/2011 6:23:10 PM

Eric

Posts: 17316
Joined: 10/10/2003
From: Seattle, WA
Status: offline

quote:

ORIGINAL: wallstreet

Hi Eric,

FYI, if you are using enterprise edition of VMware, Vmotion is included. I hope you are managing it through Virtual Center. A very cool management console. Is your storage an EVA4400? Great box. Good choice on servers. I am glad you didn't go DELL, they are pieces of sh*t compared to HP and even IBM. Look into Visioncore for backup of Vmware or even VCB which I believe is included as well. Now if you want to get fancy, take a look at SRM for DR purposes :

You really should consider a 3rd server when the funds allow. Thats best practices for a min. Vmware server farm. I am sure you used P2V to move it from your physical machine to the virtual machine. Piece of cake most likely for you.

Glad it went well. Did you end up going ISCSI ?

Al

BTW, over the past few months we have added a third server, beefed all up to 64GB of RAM, moved to ESX 4.1, upgraded iSCSI switches so we can use jumbo frames. Man this is a dream now. I just have a few more software and SQL replication changes to spread out my load, and I will be a happy camper for the first time in about 7 months with regard to perf/scale. It feels a LOT better to have a 3rd host in place.

_____________________________

Cheers!
-Eric LeVine

http://twitter.com/cellartracker
http://facebook.com/cellartracker

(in reply to wallstreet)

Post #: 17