Interested in RAW File Backups on SmugMug?
DJ-S1
Registered Users Posts: 2,303 Major grins
***** MODERATOR NOTE***** there's another, related discussion, here:
http://www.dgrin.com/showthread.php?t=38082
Don, any thought to using S3 to store raw files for a fee? Obviously any pro can purchase space on S3 themselves, but keeping it all through the Smuggy interface and perhaps hotlinking the jpegs though to the raw files would be way cool. Raw file storage seems to be a frequent request from your pro subscribers.
Just to be clear, I don't mean you would have the raw files stored locally on smug servers - only the same data you currently allow. The extra fee would be to pay S3 to store the raw files and also to provide the uploading/linking-to-the-jpg functionality.
Just a thought -
http://www.dgrin.com/showthread.php?t=38082
Don, any thought to using S3 to store raw files for a fee? Obviously any pro can purchase space on S3 themselves, but keeping it all through the Smuggy interface and perhaps hotlinking the jpegs though to the raw files would be way cool. Raw file storage seems to be a frequent request from your pro subscribers.
Just to be clear, I don't mean you would have the raw files stored locally on smug servers - only the same data you currently allow. The extra fee would be to pay S3 to store the raw files and also to provide the uploading/linking-to-the-jpg functionality.
Just a thought -
0
Comments
Hey DJ, this may not be the solution that you're hoping for, but you can in fact interact with Amazon S3 on your own (at very reasonable prices) by using a utility like www.jungledisk.com. Jungle Disk is free and you only pay for the amazon service itself (which only costs you for what you actually use). The coolest thing is that you can set it up so it acts just like a drive on your computer so you can do automatic updates and everything.
http://photos.mikelanestudios.com/
This is just my ramblings, but since Smugmug services pros and pros want this and are willing to pay for it, it seems like it could work. OR...I could be just blowing smoke.
It's an interesting idea. I think I like it.
The one gotcha is that most of the requests for RAW I've heard want to actually upload the RAW and have us generate the JPGs from it - which is an animal with completely different spots. That's much harder.
If I thought there was enough interest for a 'Store RAW files for extra fee' feature, I'd probably build it. Any takers?
Oh, and by the way, our Amazon S3 story is close to making the front page on digg -which would be great exposure for us. If you haven't dugg it already, please do. You can do it right from my blog. Thanks!
Don
I would be interested in the raw storage.
http://photos.mikelanestudios.com/
http://www.chrislaudermilkphoto.com/
AFAIK, we don't have any repeatable, persistent uploading problems any longer. If I'm wrong, though, I'd love to know about it. (Though we probably need a new thread, rather than hijacking this one).
Don
I do think the uploading interface will have to be different depending on implementation. I would think that you would need a way to associate jpg's with RAW files at upload time. Unless you were just planning on doing a filename association. Or no association at all.
When I first heard about Amazon S3, I was pretty excited. I have a local scheduled backup system running at home, but this of course doesn't protect my RAWs, JPGs, home videos, documents, etc. from fire, flood, theft, or total system meltdown. So, I've been looking for a (cheap) personal off-site backup solution for quite a while. I know S3 isn't intended for out-of-the-box consumer use, but it didn't take long for JungleDisk to show up on the scene, enabling just that.
At first, the numbers on S3 looked pretty good - $0.15 per GB per month for static storage, or about $15/mo for my current 100GB (and growing) base of data. But then I got to thinking... For less than $100, I can buy a 250GB external USB drive. Assuming that the drive will last for 5 years (60 months), that's less than $0.07/GB/mo. So, if I could find some company to just host this drive I've purchased, there's a ton of pricing room between the two options. (I'm leaving bandwidth charges out of the equation here, assuming they'd be comparable and unimportant, given the fact that we're talking about backup.)
So here are my two questions...
- If this analysis holds true for my measly amount of data, how is "leasing" storage from Amazon more cost-effective than purchased storage for a data-centric company with vast storage needs like SmugMug? Please understand that I'm not questioning SmugMug's business logic. There's just obviously something I'm missing.
- Does anyone know of a company that will host owned drives for a monthly fee? I've seen a few of the personal server colocation providers, but I don't need anything that power-hungry (and thus expensive). All I need is a large networked drive that's not in my home.
:
One more thought... If Google's rumored GDrive or M$'s Live Drive hit the scene, personal offsite backup might be getting super-easy (and super-cheap) for everyone. In that environment, I wouldn't see RAW file storage as a promising growth area for SmugMug.Thanks for reading. I look forward to hearing some other opinions.
-Greg
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
There are more costs involved with hosting storage than just the actual cost of the storage media itself. Just off the top of my head:
- power
- cooling
- rackspace
- backups
- redundancy
- cabling
- replacement of old hardware
- monitoring
- the actual machines the drives go in
and i'm sure there is a bunch i'm missing. Now $0.15/GB doesnt seem all that badOK, I'll concede those points.
However, I found an error in my earlier maths. $100 for 250GB allocated over 60 months comes to 0.67 cents per month, not 6.7 cents per month. So, there's a factor of >20 between the two monthly costs ($0.15/GB/mo vs $0.0067/GB/mo). Maybe we're truly in an age where the cost of the drive is insignificant compared with the cost of running and maintaining that drive.
I'll keep holding out for "hard drive colocation" or GDrive, though.
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
bigwebguy made some compelling points. Let me add a few more:
- hard drives fail often. Since you can't have everything live on a single drive, in case of failure, you have to do multiple drives (RAID1, RAID5, etc). We buy "enterprise" grade drives that have already been heavily pre-tested by our vendor, and they still fail remarkably often.
- servers fail often. Since you can't have "Photo not found" showing up for your customers, this means you need multiple servers so you can continue to provide service when a server fails. Of course, this server's hard drives fail often, so you need to have multiple drives (RAID1, RAID5, etc) in this server, too.
- Up front capital costs suck. You'll use the storage over a year (at least, if you don't renew), but we're paying for it up front. Spreading the cost over the same period of usage helps with our cash flow immensely, and enables us to buy more servers, or hire more customer service reps, or whatever with that same cash that's no longer tied up in hunks of metal in a datacenter.
- Taxes suck. The US Gov't does something stupid with taxes, too, where we get taxed on the hardware we buy. It's totally stupid - if I buy $1,000,000 worth of drives, the government only considers 20% ($200,000) to be expenses. The other $800,000 is treated *exactly* like profit and I have to pay 33% of that in taxes. So my $1,000,000 in drives actually cost me $1,264,000. I know it sounds insane, but it's the truth.
I could keep going, but I think you're getting the picture. Add in all the cooling, energy, replacement, monitoring, etc costs that bwg mentioned, and you've got a complex situation where the raw cost of the disk isn't indicative of the Total Cost of Ownership (TCO).
Don
I use an external drive with 1 button backup software... just plug it in, hit the button, it figures out the delta and gives me a full backup. Then I just keep the drive at my work. So if my house burns down, or my computer is stolen, I have my backup. Have a routine to bring the drive home to do the backup, say once a week, and you should be covered.
My other paranoid partial backup's include....
How are you sure that amazon can actually keep this up? For the exact same reasons you outlined, I just don't see how it would work. I'm not a business person, just a computer nerd (CS grad student studying imaging), so I'm asking because I'm curious if I'm missing something obvious of nonobvious. It would seem to me that amazon has to make money off of this to answer to share holders. You've outlined how it costs more than 15 cents per gigabyte to manage that much data, so how can you be sure smugmug (and therefore my photos) are safe using this (other than your own copies of them). I guess they could be using some sort of complex solution with many levels of data storage (Tape->Disk->fast server disks) but if stuff is on tape, it cant be served so fast, so there has to be servers storing it on disk.
Anyway, ive begun to ramble, but i guess my question can be seen. I hope i didnt come off too critical, im more just curious.
-r
Don,
Thanks for your explanation. I'm a design engineer in the chemicals business, and I'm accustomed to the old "six tenths rule" - scale a process by factor x, and the cost scales by x^0.6 (roughly speaking, of course). It would seem that chemical engineering logic doesn't quite apply to scaling always-avaliable-storage. Your points on server/drive redundancy requirements, upfront capital, and taxes make perfect sense. Good thing you're running your business instead of me.
-Greg
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
The work swap thing is a good idea. I have a nightly backup within my own PC, and that would be complimented well by your technique. The good thing for now is that all my data will fit on one drive. Once I start needing multiple disks to hold my data, things might get more compicated.
Thanks,
Greg
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
Sounds like a great option. If I understand it correctly, you might include TIFs as well.
Referral: ( wXtCbmTTvmJSE )
I only want archive of RAW files. No viewing or anything else, just archive and restore. I'd pay >$100/yr for this. I also want something that can be automated so that I don't have to "remember" to backup my RAW images (perhaps the 3rd party uploaders like StarExplorer that I use would implement this part for you if you had the right APIs).
Today, I make a copy of all images on a 2nd hard drive in another computer and have a batch copy process that runs at night to automate it. The problems with that approach are that it's in the same location (so not protected from fire/theft/flood) and I regularly have to buy/manage more disk space and the batch automation process is cobbled together and subject to some problems (e.g. not industrial strength).
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
However, it's mainly for backing up stuff on your PC, so it probably would not work so well as primary offsite storage of files that you don't still have copies of locally.
(I have no connection to the Carbonite company itlsef - just a trial user who has not even plunked down any money for it yet).
Matt
Hmmm. Unlimited backup storage for $5/mo. It sounds too good to be true and I'm sure they'd lose money on me. I think I'll have to try it for a subset of RAW files and see how real it is and meanwhile read more about the company to see if they seem like they'll be around in a while.
Thanks for pointing them out.
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
Oops - thanks for pointing that out- I edited my original to fix it.
Very interesting and thanks for sharing. I just shared the following info with my family and thought some of that e-mail might be of interest here as well. I have no affiliation with the company, etc.
For the average user, this looks like a good option as it is even easier to implement off-site backup than JungleDisk.
http://www.carbonite.com
Carbonite will automatically back up your PC over the Internet for $5 per month
What is Carbonite and who is it for?
Carbonite is Backup for Everyone -- a simple, safe online backup service for casual home PC users, students, home office users, road warriors, etc. For five dollars a month, Carbonite will backup all the data on your PC (digital photos, music, office documents, and other valuable data) whenever your PC is connected to the Internet. Carbonite is very easy to setup and use, requiring little or no ongoing user involvement. Our service is always on and continuously protecting data on your PC whenever it is connected to the Internet
Here’s a blog posting, which include a post from a Carbonite representative that clearly describes the site’s purpose
http://thebogles.com/blog/2006/06/carbonite-online-backup/
There is a free 3-month subscription, which is available from this link
http://www.carbonite.com/goodday/signup.aspx?utm_content=radiobutton&contenttag=radiobutton
This service IS NOT targeted to the power user as there are some obvious limitations:
1. “Carbonite does not currently back up files that are larger than 2GB in size, removable hard drives (e.g., USB drives) or mapped network drives.”
2. The $5 is to back up one PC only. Each PC in the house would require a subscription and/or a way to move data to a central computer that is subscribed to the Carbonite service.
Hmmm. After reading the blog postings you pointed to, it appears that unlimited is NOT unlimited. One blog entry describes how Carbonite said they would shut down his account if he didn't reduce his pending uploads to less than 100GB. So, for this example, unlimited appears to mean 100GB. This seems like false advertising to me. Which part of "unlimited" do they not understand? If it's not unlimited, they shouldn't use that word. It's still probably a good deal for 100GB, but this deceptive advertising makes me wonder if I can trust the company.
Their terms of service are very vague about this. They say that you can't use more than their average user in the terms. This is a bit strange because it's hard to have an average without a lot of users being over the average.
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
Not saying they wouldn't flag an account as their T0S is still somewhat vague. I also think that their service has some slowness built in (approx. 2-GB per day upload speed if I remember correctly) and that will have a natural tendency to limit the size of accounts. Still looks like it would be worth looking into, at least for 3 months
You'r right. Here's the current Terms of Use document. They have removed the bit about average use. They do reserve the right to refuse service to "abusers" without any definition of what that might be. I've pasted in the relevant section from their terms of use below. I understand that the company wants to reserve the right to refuse service to anyone they want, but given their earlier stance on unlimited, I'm not ready to trust them quite yet.
I did notice that their CEO is named "David Friend". Maybe he's a long lost relative of mine
Here's Carbonite's current termination policy:
Termination and Fair Use Policy
YOUR USE OF CARBONITE PRODUCTS IS SUBJECT TO CARBONITE’S "TERMINATION AND FAIR USE POLICY.” THIS POLICY IS INTENDED TO ALLOW CARBONITE TO DENY SERVICE TO ABUSERS. USERS WHO ARE DEEMED TO BE "ABUSERS", IN CARBONITE'S SOLE DISCRETION, MAY BE NOTIFIED PRIOR TO SUSPENSION OR TERMINATION OF THEIR ACCOUNTS, HOWEVER, CARBONITE RESERVES THE RIGHT TO TERMINATE OR SUSPEND SUCH ACCOUNTS WITHOUT PRIOR NOTICE IN THE EVENT OF A POLICY VIOLATION. ANY FAILURE BY CARBONITE TO ENFORCE THIS POLICY WILL NOT PRECLUDE US FROM ENFORCING IT AT ANYTIME IN THE FUTURE, WHETHER FOR PAST OR CURRENT VIOLATIONS.
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
Sweet. Thanks for the referral, Mike. I put it together in about 90 seconds (first had to enable S3 in my Amazon AWS account)... and now I'm happily uploading about 100gigs of RAW files
Portfolio • Workshops • Facebook • Twitter
Just to be clear - my math suggests it'd have to *a lot* more than $100/year. Possibly even $100/month to do this and not lose money.
How many GBs of RAW do you have, for example? If it's 100GB, that would easily cost us more than $360/year in costs, leaving no profit for us.
For a RAW offering, we'd almost definitely have to move away from "unlimited" and into something metered like "100GB = $50/month". Note that I haven't calculated our real actual costs, I'm just ballparking it at this point.
Is there still interest, even at these price points?
One alternative is that we could store the RAWs in only one location (Amazon OR SmugMug, rather than both), which would save at least 50% of the cost. But then, of course, the reliability theoretically takes a hit. Since this is a backup option, I'd think the reliability is key.
AFAICT, the big benefit to doing this at SmugMug is that it's tied to your JPEGs so you can easily search and retrieve stuff just like you can with the rest of your photos - only in RAW. That seems pretty key to me.
FYI, if you store stuff at Amazon yourself using JungleDisk or something else, it'll cost you $15/month for 100GB. So our costs will be higher than that, depending on what people end up wanting. Is it worth it for SmugMug integration?
I wouldn't trust anyone in the industry who charges less than $15/month (Carbonite's $5 is ludicrous) since I'm an expert in this field and Amazon is far and away the cheapest and least likely to go out of business doing it. Store photos somewhere cheaper at your own peril.
(Can one of the mods please fork this thread into a RAW discussion thread? Thanks!)
Don
I personally do have a lot of interest in storing RAW files linked to jpegs. Certainly there is no need for smugmug to convert them to jpegs since we do apply correction to raw files. That actually means that correction setting should be stored together with raw files.
At the same time current possible price and upload speed do not seem good. But I think smugmug should move into this area while technology improves.
I think it is hard to an average user to estimate risks of loosing his/her photos. What I mean, it seems to me that currently it is much more likely that I will loose my data, not because of earthquake/terrorist act/fire in several locations simultaneously, but because smugmug goes bankrupt. Amazon seems much more reliably since I can see their balance sheet.
No.
No.
No.