Re: GroupStudy Server crash

Im going to donate as well, gs helped me a lot!

Jeferson

On 12/05/2011, at 20:12, "Daniel Kutchin" <daniel_at_kutchin.com> wrote:

> Supported
>
> I have just donated to Paul at Paypal
>
> Login to paypal |
> Choose the register "Send Money" |
> Enter the e-mail pborghese_at_groupstudy.com and the amount/currency |
> click continue
> confirm payment
> Then you will see the reassuring "Daniel Kutchin, you have sent $XXX,00 USD
> to Artisan Networking Inc."
>
> -
> Daniel
>
>
> -----Original Message-----
> From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On Behalf Of
> Elliott Reyes
> Sent: Freitag, 13. Mai 2011 00:35
> To: Haroon
> Cc: ccieagent; Paul Borghese; ccielab_at_groupstudy.com
> Subject: Re: GroupStudy Server crash
>
> We should take up a collection to help paul out
>
> On Thu, May 12, 2011 at 3:32 PM, Haroon <itguy.pro_at_gmail.com> wrote:
>
>> Paul,
>>
>> Thats some ordeal... sorry to hear!
>>
>> Your backup strategy is great.... I also use rsync to backup over
>> 110GB to remote servers twice a day but my stuff (linux server, mysql
>> DBs, etc.) is not on vm.
>>
>> ESXi 3.5 comes with very little tools for recovery.
>>
>> Have you tried enabling SSH on esxi??? That may give you more control
>> and go under the hood as far as vmware is concerned...
>>
>> As far as your datacenters, what datacenter do you use in Dallas?
>> Softlayer?
>> Instead of copying 200GB to your house (residential cable/dsl?), I can
>> recommend a vendor that I use to backup data right in dallas
>> datacenter using rsync.
>>
>> Please keep us posted.
>>
>> Thanks,
>>
>> Haroon
>>
>>
>> On Thu, May 12, 2011 at 2:00 PM, ccieagent <ccieagent_at_verizon.net> wrote:
>>
>>> Paul,
>>> Good to see you got it working again. I was beginning to wonder if
>>> the pressure of our joining GroupStudy and the CCIE Flyer was going
>>> to
>> happen!
>>> LOL
>>> Sorry to hear about your ordeal. Talk to you soon.
>>>
>>> -----Original Message-----
>>> From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On Behalf
>>> Of Paul Borghese
>>> Sent: Thursday, May 12, 2011 4:40 PM
>>> To: ccielab_at_groupstudy.com
>>> Subject: GroupStudy Server crash
>>>
>>> On Monday one of the GroupStudy servers in Atlanta had a
>>> catastrophic
>> disk
>>> failure. This brought down the entire site and every mailing list.
> The
>>> good news is this list is impacted only minimally from the failure
>>> and should (may?) be back to normal shortly. I apologize for any
>>> inconvenience.
>>>
>>> Since this is a technical list I will go into the details for those
>>> that are interested. Please comment if you have any suggestions.
>>> On Monday I decided it had been too long since I backed up the
>>> Atlanta server, which
>>> is now running as a VM on a VMWare ESXi 3.5 server. At one time I had
>> an
>>> elaborate backup mechanism where every evening critical database
>>> file
>> were
>>> copied to a directory then rsync ed to a linux server at my house.
>>> But over the last two years we moved a number of times and the
>>> remote server is still packed away in a storage unit.
>>>
>>> I decided since it was a VM I could simply scp the .vmdk file to my
>> house,
>>> thus creating a perfect backup. The problem was the VM contained
>>> multiple 200 GB files. I noticed part of the storage size was caused
>>> because a snapshot had been taken on the GroupStudy server. To reduce
>>> the size of the backup, I decided to delete the snapshot. This
>>> action should have merged the snapshot with the primary. But
>>> instead it removed all .vmdk disks and left me with a .vmdk file
>>> that was simply 26k
>> (instead
>>> of 200+ GB)!! Oops. The GroupStudy disk was totally destroyed.
>>>
>>> If the VMWare server was running on another OS, I could have simply
>>> gone
>> in
>>> with an undelete program and tried to recover the files. But VM
>>> ESXi
>> runs
>>> a
>>> propriety locked-down OS with very few tools. I hate to say this,
>>> but my last backup of the server was made two years ago, just before I
>>> moved (yea I know but hey it is a hobby). I called a number of
>> people
>>> asking for advice. I would like to thank in particular Darby Weaver
>>> who found a VMWare guy who was quite knowledgeable. But even he was
> stumped.
>>>
>>> I decided to take it slow and not do anything that may prevent a
>> recovery.
>>> In the evenings (I still have a day job :-) ) I started reading
>> everything
>>> I could find about VM ESXi. I have now read more VMWare
>>> knowledge base articles then I care to admit. I am thinking about
>> seeing
>>> what certifications VMWare offers and simply taking the test.
>>>
>>> VMWare does offer support, for $300/call, which I really did not
>>> want to spend. But it was obvious I was not getting anywhere trying to
> figure it
>>> out myself. So I reluctantly plunked down my credit card. In my
>>> research I found VM did offer an undelete program for the ESX platform.
>> I
>>> was hopeful my support request would, at minimum, give me access to
>>> the undelete program for ESXi. Or maybe some internal use only
>>> magic VMWare has in their back pocket. But in all honestly, they
>>> were not much help.
>>> The VMWare tech support is not that great. Cisco TAC will escalate
> your
>>> problem until you are fixed. VMWare support seems to be for people
> that
>>> don t know how to read manuals. In all fairness, it could be that my
>>> $300 support call is not going to the same people that handle high
> paying
>>> corporate outages. VMWare suggested I call a data recovery service.
>> The
>>> data recovery service said this happens all the time and they could
>>> recover the data for $3-$5k. I simply do not want to spend that kind
> of
>>> money.
>>>
>>> The VMWare server contains two 1 GB disks, a primary and extra disk.
>>> The original GroupStudy VM was running on the primary. I used the
>>> two-year-old backup disks to create a new GroupStudy VM on the extra
>> disk,
>>> thus preserving the primary to the best of my ability. Of course
>>> the backups were created before migrating to VMWare, so none of the
>>> kernel drivers worked out of the box.
>>>
>>> After fixing the kernel and initrd boot files, the GroupStudy
>>> website has been restored literally to the date of the Obama
>>> Inauguration. So
>> welcome
>>> back to January 2009 (quick buy Apple Stock and gold!). With regards
>>> to the CCIE List, this actually has less impact then you would think.
>> The
>>> actual mailing list is running off a Linux server in Dallas, and has
>>> been unaffected. What we lost was two years of archives. But I may
>>> be able
>> to
>>> get them back as the Dallas server has copies of the archives in a
>>> MySQL DB.
>>> If they are complete, I can simply write a Perl script to extract
>>> the archives to a format the website can use.
>>>
>>> Bu there is other lists that are affected more and frankly being a
>>> techie
>> I
>>> hate to give up. We know the data is most likely still on the disk. We
>>> just need to find it. I feel with the disk, a hex editor, and some
>>> voodoo I could recover the data I needed. Frankly I only need one of
> the
>>> backup files that was created daily, not the entire disk. The problem
>> is
>>> the Primary disk is where the VMWare OS is located, so I can t simply
>>> remove it. And it currently resides in a data center in Atlanta, a 10
>>> hour drive from my house!
>>>
>>> So now I am trying to MacGyver my way to the disk. The extra disk
>>> contains about 400 GB of free space. It turns out VMWare does offer
>>> disk dump and gzip on the ESXi platform. I am disk dumping the
>>> entire primary hard drive to the extra drive, using gzip to compress
>>> the data. I am praying for a much better then 2:1 compression
>>> ratio! If that works I
>> will
>>> download the dd file and restore to another 1 TB hard drive, thus
>>> creating a copy of the primary drive. Then I need to figure out the
>>> VMWare partition tables and vmdk disk formats. If the 400GB of free
>>> space is not enough, I am considering mounting an Amazon EC2 NFS
>>> server
>> on
>>> the VMWare file system and trying again. I also called the VMWare
>> support
>>> engineer (that poor guy) and asked him to send me any documentation
>>> he
>> can
>>> find about the VMFS and VMDK file structures. I also found an open
>> source
>>> VMFS driver (http://code.google.com/p/vmfs/) that may be of use.
>>>
>>> So if you have any suggestions, please send them to me! No matter
>>> how
>> bad
>>> it got, I kept on thinking at least I am not Sony!
>>>
>>> Paul Borghese
>>>
>>>
>>> Blogs and organic groups at http://www.ccie.net
>>>
>>> ____________________________________________________________________
>>> ___ Subscription information may be found at:
>>> http://www.groupstudy.com/list/CCIELab.html
>>>
>>>
>>> Blogs and organic groups at http://www.ccie.net
>>>
>>> ____________________________________________________________________
>>> ___ Subscription information may be found at:
>>> http://www.groupstudy.com/list/CCIELab.html
>>
>>
>> Blogs and organic groups at http://www.ccie.net
>>
>> ______________________________________________________________________
>> _ Subscription information may be found at:
>> http://www.groupstudy.com/list/CCIELab.html
>
>
> Blogs and organic groups at http://www.ccie.net
>
> _______________________________________________________________________
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html
>
>
> Blogs and organic groups at http://www.ccie.net
>
> _______________________________________________________________________
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html

Blogs and organic groups at http://www.ccie.net
Received on Thu May 12 2011 - 22:19:56 ART