Amazon S3 worth it?

DJ-S1

Interesting article calculating whether using S3 for backup is cheaper than having your own backup server.

DoctorIt

OK, but do you need to be a computer geek to use it? I read on and found that another company, Altexa, is using Amazon S3 to offer backup services to individuals and small businesses. So that leads me to believe S3 isn't as simple as uploading and downloading files... or is it?

Maybe not this year while I'm still studying, but heck, I'd pay $20/month to ensure that I had a backup, and one that I could access from anywhere. Another bonus, It'd sure make it easier for people like me who sometimes work at home, lab, office, while travelling, etc.

mwgrice

DoctorIt wrote:

OK, but do you need to be a computer geek to use it? I read on and found that another company, Altexa, is using Amazon S3 to offer backup services to individuals and small businesses. So that leads me to believe S3 isn't as simple as uploading and downloading files... or is it?

Maybe not this year while I'm still studying, but heck, I'd pay $20/month to ensure that I had a backup, and one that I could access from anywhere. Another bonus, It'd sure make it easier for people like me who sometimes work at home, lab, office, while travelling, etc.

I don't know about uploading, but it looks like the default download protocol is http. So you can do it with a web browser. I suspect it's reasonably simple, although what you consider simple and what I consider simple might not be the same. What it looks like Altexa is offering is ease of use and automation. Most of the backup clients currently available to home users either won't support S3, or would require additional configuration. Altexa is betting that it will be much easier for the average home or small business user to back up their data with Altexa.

DJ-S1

DoctorIt wrote:

OK, but do you need to be a computer geek to use it? I read on and found that another company, Altexa, is using Amazon S3 to offer backup services to individuals and small businesses. So that leads me to believe S3 isn't as simple as uploading and downloading files... or is it?

Maybe not this year while I'm still studying, but heck, I'd pay $20/month to ensure that I had a backup, and one that I could access from anywhere. Another bonus, It'd sure make it easier for people like me who sometimes work at home, lab, office, while travelling, etc.

Yeah, I don't know what the UI is. I just stumbled on the article and since we had been discussing S3 earlier I thought it was interesting. The guy even used a utility to determine how much power his raid array was consuming - and I used to think I was a geek!

sirsloop

Those numbers work out well if you just use that backup machine for nothing other than storage. Lots of people run one main machine with raid arrays in them for redundancy. My one and only machine I use at home has like 1.2TB worth of hard drive space. The article mentioned spending $700 on drives and another $700 on a mobo and ram. For this type of application you get youself a old clunker P3 and toss in a $70 PCI SATA Promise Fastrack controller, two $100 320GB WD 16MB cache drives mirroed... on sale now at newegg.com. You can probably get the p3 clunker for $50 bucks on craigslist, format the drive, and have youself a backup machine for $320.

Even more realistically, people will simply buy an external USB/Firewire storage unit. For a couple hundred bucks you get a tiny little box that has two 500GB drives mirrored, connected to a low wattage power supply. There are a number of companies out there that have 500Gb storage in the ~$300 price point.

Lets not forget... you have to upload all of your stuff to Amazon. That costs money and takes A LONG TIME!! While sending stuff off-site is good security against lightning, fires, floods, ect... its slow and a PITFA!!! They took for granted that you have (and pay for) a high speed internet connection. There are a lot of people, especially out in rural areas that are still using dialup. Lets see how long it would take to backup that 8GB microdrive you just filled up. You would be better off sending the microdrive to amazon USPS and having THEM upload it! HAHA!

I wouldn't know the resources needed... or worked out the costs, but it may be cheaper for smugmug to just setup a few monster machines in a colo somewhere and do their own backups. Rent a rack and pay per megabit per month.

bwg

DJ-S1 wrote:

The guy even used a utility to determine how much power his raid array was consuming - and I used to think I was a geek!

I met Jeremy at Yahoo this past weekend. Super nice guy. Onethumb says he is the MySQL expert of all experts. Wrote the book on it (literally).

Mike Lane

DoctorIt wrote:

OK, but do you need to be a computer geek to use it? I read on and found that another company, Altexa, is using Amazon S3 to offer backup services to individuals and small businesses. So that leads me to believe S3 isn't as simple as uploading and downloading files... or is it?

Maybe not this year while I'm still studying, but heck, I'd pay $20/month to ensure that I had a backup, and one that I could access from anywhere. Another bonus, It'd sure make it easier for people like me who sometimes work at home, lab, office, while travelling, etc.

www.jungledisk.com That'll help make S3 act like an external drive on your computer. Piece of cake.

sirsloop

Even with basic optimum online (probably the fastest home cable internet service) that grants me 15/1.5MBps (OOL Boost is like 30/2 I think)... thats still horrific write speeds. Lets say I fill up my 8GB microdrive, and backup the files at 1.5Mbps.

Files are 10MB each
~200KBps transfer speed (8 bits in a byte... round to 200 to make the numbers easy)
5sec=1MB
50sec per photo
40,000 seconds for the whole disk.
666 minutes
11.1 hours

Thats just for data backup... not even including time it takes to SendToSmugmug! DOUBLE that!

Andy

Mike Lane wrote:

www.jungledisk.com That'll help make S3 act like an external drive on your computer. Piece of cake.

Indeed.

Jungle Disk + Amazon S3

I used it for all of my RAW files. It's *wonderful*

pathfinder

Andy,

About Jungle Disk. This sounds very interesting, but I would like to know more about the particulars.

How many Gigabytes total, are you storing, and what is it costing you? Jungledisk shows 20Gb, but that would be way too small for my files - more like 500Gb.

Are billied by the month or year? Pay by check or automatic charge to your credit card. ( Hard to kill automatic deduction sometimes. Remember AOL??)

How much up and downloading of files are you doing in terms of bandwidth?

Do you think this is better than buying a TerraStation 1.2 Tb Raid drive - Raid 6 preferably??

Andy

pathfinder wrote:

Andy,

About Jungle Disk. This sounds very interesting, but I would like to know more about the particulars.

Remember, JD is just the front end, they do nothing wrt the storage or money.

How many Gigabytes total, are you storing, and what is it costing you?

I have loads of Gb up there. It's .20 ONE TIME charge to upload per gig. Then it's .15 per gigabyte per month

Are billied by the month or year? Pay by check or automatic charge to your credit card. ( Hard to kill automatic deduction sometimes. Remember AOL??)

Monthly, to my same credit card on my Amazon account. There's no long-term fee or commitment. If you stop using, you stop paying. If you use less, you pay less.

How much up and downloading of files are you doing in terms of bandwidth?

Once a month now, a few Gigs I guess.

Do you think this is better than buying a TerraStation 1.2 Tb Raid drive - Raid 6 preferably??

I do, I don't like hardware. I like having my stuff available to me anywhere, I can grab a file from my S3 account anytime, anywhere I might be.

More here:
http://www.amazon.com/gp/browse.html?node=16427261

BeachBill

bigwebguy wrote:

I met Jeremy at Yahoo this past weekend. Super nice guy. Onethumb says he is the MySQL expert of all experts. Wrote the book on it (literally).

Yes I agree Jeremy is a nice guy. We worked together as co-technical editors on the book "MySQL and Perl for the Web", written by Paul Dubois. Jeremy then went on to author the book "High Performance MySQL".

Andy

Jungle Disk for Storage of Files

pathfinder wrote:

Andy,

About Jungle Disk. This sounds very interesting, but I would like to know more about the particulars.

How many Gigabytes total, are you storing, and what is it costing you? Jungledisk shows 20Gb, but that would be way too small for my files - more like 500Gb.

Are billied by the month or year? Pay by check or automatic charge to your credit card. ( Hard to kill automatic deduction sometimes. Remember AOL??)

How much up and downloading of files are you doing in terms of bandwidth?

Do you think this is better than buying a TerraStation 1.2 Tb Raid drive - Raid 6 preferably??

Another thing I really, really like - if I'm away from my studio, and just have my laptop, I can access any of my files I like, so long as I can access the interpipes...



They just added some Automated Backup which I'm now investigating, too

sirsloop

I prefer to have the second machine running. Its currently setup with 1TB raid 5 array just for photo storage, another single 500gb non-business drive, and a 250gb OS drive (lol, yeah 5 drives) and have around 380GB used in the raid array. I use the machine for batch processing (free's up my main workstation from hours of raw processing of thousands of images from the weekend), storage, music playback over my home stereo, video guitar lessons in my living room, tab viewing on my projector, movie viewing, etc etc etc. So you pay cash for just a service, or you can pay cash for an asset that you have in your home for many uses. Factor in a couple UPS's too. I'm a network engineer at my day job so its second nature really. Oh ya... my internet connection may be 30/5Mbps, but my local network is sync 1Gbps. Moving files at hard drive speed

SloYerRoll

sirsloop wrote:

I prefer to have the second machine running.

This is obviously a preference thing. So I'm not trying to talk you into one way or anohter. There is something to be said for geographic redundancy though.
Example: I set up a friends machine w/ every possible safeguard I could imagine. RAID, UPS, on and on...
What happens? His kid brother takes an electromagnet out of some kids build it yourself science kit and proceeds to clamp that thing onto the side of his machine.....
DOH!!!!!

It's a lol now. But it was a disaster at the time.

Just adding to this already interesting thread.

-Jon

scottV

S3 + jungledisk is awesome. My files go from camera to external hd and then to S3. I don't shoot a ton but I back up my new files weekly and it takes about a day to transfer everything. It is super simple, if you are worried about needing to be a geek to get it to work then wait till you try and set up a second box and raid, talk about geektothemax. My next purchase will be one of these: www.drobo.com to have local redundancy but still easy to manage, and will still upload to S3.. for the low low price it's foolish not to include S3 in your backup strategy in case of major disaster at home.

Mike Lane

f00sion wrote:

S3 + jungledisk is awesome. My files go from camera to external hd and then to S3. I don't shoot a ton but I back up my new files weekly and it takes about a day to transfer everything. It is super simple, if you are worried about needing to be a geek to get it to work then wait till you try and set up a second box and raid, talk about geektothemax. My next purchase will be one of these: www.drobo.com to have local redundancy but still easy to manage, and will still upload to S3.. for the low low price it's foolish not to include S3 in your backup strategy in case of major disaster at home.

My savings are going to to towards a drobo as well. . Drobo + dreamhost (with a smidge of MacFusion) + Airport extreme (which will automatically mount my drobo) = a superb backup solution. S3 is great too, but I've already got DH with tons of extra room

sirsloop

Geographic backup is nice... which is why I push jpg's up to smugmug. Granted all my files are not backed up there, but its great for $150/yr!

SloYerRoll

Thank you Scott and Mike for exposing me to Drobo!
Wow that thing is cool!
I'm not at the point where any of my data is truly critical (the small ammount I have is on my own raid array of (2) 2GB thumb drives... lol)
When I do have critical data though. This will definately be a solid investment!

-Jon

SloYerRoll

Andy wrote:

Another thing I really, really like - if I'm away from my studio, and just have my laptop, I can access any of my files I like, so long as I can access the interpipes=

From the small amount I've read on jungle disk. It takes the files and sends them to S3. S3 then encrypts and does all their huge DB stuff.

Question:
Can you search using keywords or metadata through JD (or another app)? If not, I assume that a solid folder heirarchy is vital for sucessful archiving.

I spend allot of time updating metadata to ensure I can readily find shots when I look for them. Going back to the hammer chisel way of file archiving doesn't sound appealing to me.

Am I missing something here?

Thanks,
-Jon

LiquidAir

I did a quick calculation rate I am generating new data to back up comparing it to my DSL upload speed. To back up to S3 (or any internet service) would require 6-7 hours a day of upload time on average. For me that just isn't practical. Without forking out for a high end broadband service, I don't think I can resonably use S3. I am looking at a local dedicated NAS box instead.

Andy

LiquidAir wrote:

I did a quick calculation rate I am generating new data to back up comparing it to my DSL upload speed. To back up to S3 (or any internet service) would require 6-7 hours a day of upload time on average. For me that just isn't practical. Without forking out for a high end broadband service, I don't think I can resonably use S3. I am looking at a local dedicated NAS box instead.

mine just happens when i sleep, no bother at all

LiquidAir

Andy wrote:

mine just happens when i sleep, no bother at all

Do you have days where you shoot more than you can back up?

I am looking at the Hammer Storage myshare server. File server + print server + 1TB of storage (2x500GB) for just over $400. It supports RAID 1 mirroring and it can back itself up to a USB drive. They are supposed to have a version with 2x1TB drives available soon.

SloYerRoll

Quote:
<TABLE cellSpacing=0 cellPadding=4 width="100%" border=0><TBODY><TR><TD class=alt2 style="BORDER-RIGHT: 1px inset; BORDER-TOP: 1px inset; BORDER-LEFT: 1px inset; BORDER-BOTTOM: 1px inset">Originally Posted by Andy
Another thing I really, really like - if I'm away from my studio, and just have my laptop, I can access any of my files I like, so long as I can access the interpipes=
</TD></TR></TBODY></TABLE>
From the small amount I've read on jungle disk. It takes the files and sends them to S3. S3 then encrypts and does all their huge DB stuff.

Question:
Can you search using keywords or metadata through JD (or another app)? If not, I assume that a solid folder heirarchy is vital for sucessful archiving.

I spend allot of time updating metadata to ensure I can readily find shots when I look for them. Going back to the hammer chisel way of file archiving doesn't sound appealing to me.

Am I missing something here?

Thanks,
-Jon

scottV

jungledisk makes your amazon storage show up as a drive on your computer, so you could presumably load those files into whatever asset manager you are using... but I think it would be slow and you are also charged per 1000 requests that you send to s3, it's pennies but I guess it could add up with lots of files and depending on how often your software lists them... so I would say no to searching across metadata, but it would be easy enough to try, free to signup, no recurring billing if you aren't using it.

I think S3 would work best as a backup and not necessarily archive, so you would still keep your files locally and just use S3 for fault tolerance, that way you really don't need to search across the repository... using it to archive may be more difficult to manage, then you also have to deal with downloading the files if you need them, not exactly quick access like a local usb drive can provide.

SloYerRoll

f00sion wrote:

so I would say no to searching across metadata

Thanks Scott,
I figured as much. It's still a fantastic deal and I'll definately be signing up when the need arises.

All the best,
-Jon

Andy

LiquidAir wrote:

Do you have days where you shoot more than you can back up?

I am looking at the Hammer Storage myshare server. File server + print server + 1TB of storage (2x500GB) for just over $400. It supports RAID 1 mirroring and it can back itself up to a USB drive. They are supposed to have a version with 2x1TB drives available soon.

Nope, never. It's really been working out great.

Pindy

The Mac FTP app Transmit (www.panic.com, the best IMO) has S3 ability now, so you can use it just like you would any FTP transfer. The best part is that Transmit has great folder synchronization features, so you can update the remote files with only new stuff. Super cool.