smugoumug backup

whizzerwheel

If...dog forbid, smug mug servers crashed...what should I have done to protect all the work on my site? Do you offer some kind of backup service?:dunno

nipprdog

whizzerwheel wrote:

If...dog forbid, smug mug servers crashed...what should I have done to protect all the work on my site? Do you offer some kind of backup service?

:eat

denisegoldberg

Have you seen this? http://www.smugmug.com/photos/photo-video-backup/

--- Denise

mcgil

A precision regarding the DVD backup service, it looks to be momentarily not available : http://www.smugmug.com/help/backups

An extra layer of safety for you could be to keep a local version of your photos, be it on a NAS in your home or a series of DVD in your bank deposit box

whizzerwheel

So...what you're saying is...smugmug already backs up our photos and if they had a crash...amazon has copies. So, not to worry.
Otherwise we could keep copies on our own home storage units. Of course, cd's and the like will be obsolete like the floppy disc and 8 track soon. Can't find an 8 track player to play that Bachman Turner Overdrive tape I got. Damn...

mcgil

Exactly ... it's hard to do more secure and affordable. Your photos will reside in several data centers so you'd have to have a pretty major catastrophe to lose them. Now it never hurts to have an extra copy in a Swiss bank

Now if you're thinking long term storage, say if you're concerned with your kids having access to your photos in 50 years, I honestly don't know the best way to proceed.
Will the computers in 50 years still be able to open jpg ?
If JPG disappears as a format, will SmugMug migrate your photos for you to something new ?
The prints will still be around, so you could also follow what museums and archival places do. They use some custom films/folders to preserve prints, for example check the products sold by lightimpressionsdirect.com

jfriend

backblaze.com
I keep my own backups at BackBlaze.com. I think several other Smugmuggers use them too. I also use them to backup other household documents, all my RAW files, and all the other documents that Smumug doesn't take, etc...

Unlimited storage, pretty inexpensive (~$50/yr), fully automatic off-site backup.

BigRed

whizzerwheel wrote:

If...dog forbid, smug mug servers crashed...what should I have done to protect all the work on my site? Do you offer some kind of backup service?

If you mean the site customizations, did you know that you can request a copy of your code via email, and then just save it locally? The Email it! button is right next to the Update button on the Advanced Site-wide Customizations page. I haven't figured out a way, however, to backup the settings -- for example, Easy Customizer & individual Gallery settings (especially important for HTML-only pages).

SamirD

As they always say, if it's important, back it up. Multiple places and methods is best.

But even doing it yourself has issues. I have two sets of external drives with all my images on them. Each drive in a set is an exact copy of the other, compared bit for bit. About every year, these drives fill up, so I get another set. This year, I don't have cash, so I'm trying to re-use some of the older drives, the data of which I copied to my most recent set when I bought them a year ago. So before doing mass deletes, I use a little freeware program called xcomp to compare all the files from the old drives to the current ones. In theory, these should be exactly the same since they were copied just a year ago and xcomp'd then. What I found surprised me.

One of my newer drives was different on 3 files. No errors, no clicking, nothing indicating a failure of the disk itself. I then proceeded to xcomp these files with all the other copies of the 3 files on the other drives. I discovered that one of my only 1 year old drives has some files that are not like the others.

But how could this be? The files were the same when they were originally copied and xcomp'd! Well, there are two types of hard drive read errors. Recoverable and unrecoverable. Unrecoverable are the really bad ones and mean the drive is dying, but recoverable ones are the dark ninjas that cut your data to pieces silently. Luckily, photos and videos consist of information that doesn't change noticeably if a bit or two is different. But if these files were important excel spreadsheets or something that requires every bit to be correct, I would have serious problems.

So what I've learned is that a local mirrored backup isn't enough because one of the mirrors might show you a different reflection one day. The solution is three mirrors, where the third lets you know which one is incorrect by "best two out of three". This is a cost effective way of getting data integrity. RAID 4,5,6 based disk systems are another way to do it, but not many RAID systems take into account the drive controller as a point of failure, which can be more likely than a drive failure in SCSI RAID systems. A controller failure will cost you more downtime than a drive failure ever will.

And so here comes my question--what is SmugMug's policy on data integrity? Knowing that photos and videos can take the loss of a bit or two without issue, does the Smugmug storage system use this fact in the design? Or is it made to have 100% data integrity? In other words, if my two drives say that a file that was once the same is now different, can I confidently use the original on SmugMug as the tie breaker?

I was planning to use SM as my third mirror, and I've actually already been doing so for new images. But I still have a lot left to upload. And if I can't depend on the data on SM, I'll have to re-think this entire strategy. :cry

SamirD

SamirD wrote:

And so here comes my question--what is SmugMug's policy on data integrity? Knowing that photos and videos can take the loss of a bit or two without issue, does the Smugmug storage system use this fact in the design? Or is it made to have 100% data integrity? In other words, if my two drives say that a file that was once the same is now different, can I confidently use the original on SmugMug as the tie breaker?

Bump for an answer to these questions.

phototristan

SamirD wrote:

Bump for an answer to these questions.

We don't alter your original uploaded images so if you download them, they will be the same image files. We also have redundant backups in multiple locations so as long as your account is active, you can download your images back to your computer at any time.

SamirD

phototristan wrote:

We don't alter your original uploaded images so if you download them, they will be the same image files. We also have redundant backups in multiple locations so as long as your account is active, you can download your images back to your computer at any time.

Thank you for the reply Tristan, but I know all this. What I'm really asking is, how confident can I be that the image uploaded will be the EXACT same if I re-download it? For example, if I uploaded a million files and then immediately downloaded all of them off SM and compared them bit-by-bit with the original files I uploaded, would they be the exact same? (Assume there were no errors in transmission via the Internet.)

phototristan

SamirD wrote:

Thank you for the reply Tristan, but I know all this. What I'm really asking is, how confident can I be that the image uploaded will be the EXACT same if I re-download it? For example, if I uploaded a million files and then immediately downloaded all of them off SM and compared them bit-by-bit with the original files I uploaded, would they be the exact same? (Assume there were no errors in transmission via the Internet.)

You can be very confident.

It would be the same if you copy the file to another computer or another hard drive. I'm not saying some data corruption can't occur during the transfer process, however that's rare.

SamirD

phototristan wrote:

You can be very confident.

It would be the same if you copy the file to another computer or another hard drive. I'm not saying some data corruption can't occur during the transfer process, however that's rare.

Thank you again for the quick reply. This is basically what I expected since you guys have a lot riding on your storage system.

But just to clarify, when you refer to the "transfer process" and the possibility for data corruption, you're talking about the Internet transfer and not an internal process, correct? (This is my final question, I promise. )

phototristan

SamirD wrote:

Thank you again for the quick reply. This is basically what I expected since you guys have a lot riding on your storage system.

But just to clarify, when you refer to the "transfer process" and the possibility for data corruption, you're talking about the Internet transfer and not an internal process, correct? (This is my final question, I promise. )

Right. Your computer could also corrupt a file upon upload. Again, very rare but it could happen.

SamirD

phototristan wrote:

Right. Your computer could also corrupt a file upon upload. Again, very rare but it could happen.

Thank you very much for the clarifications. It allows me to sleep a lot better at night knowing how safe my photos are with SM. :ivar

SamirD

phototristan wrote:

You can be very confident.

And after downloading and doing bit-by-bit comparisons on all my photos from 2006 and the 4th quarter of 2005 (over 10,000 images/80gb+), I'm extremely confident. Each file on SM is as perfect as the copies on my hard drives, bit-by-bit.

Now, only 7 more years of data to download and compare for me to complete my new data integrity system.

SamirD

phototristan wrote:

You can be very confident.

My confidence has waned today.

Here's what has happened. I rented a Canon 50D. I'm shooting raw+jpg, both at highest resolutions. I'm uploading the jpgs. When I download the jpgs, some of the file sizes are different.

When I copy them off the card and compare them to my archive drives, they're the same. When I upload them to SM and then download using AF or owner save, the file size is off by 5k.

I've tried deleting the file, replacing the file, both using SM's web tools and the result is the same.

Here's a link to the two files:
http://newpics.huntsvillecarscene.com/Archive/2010/10040201/11699640_34TJ8#829586892_gnXYr
http://newpics.huntsvillecarscene.com/Archive/2010/10040201/11699640_34TJ8#829586559_L3Lzz

The first should be 6489kb. The second 4450kb. When downloading, the first is 6485 and the second 4455.

This puts a real hiccup in my data integrity system since I want to compare my archives on SM with the originals before deleting the originals. I have an event coming up in 48hrs and need to compare about 10gb, but keep running into this problem. I need to know if this a problem on SM's back end and what's going on. Help!

jfriend

SamirD wrote:

My confidence has waned today.

Here's what has happened. I rented a Canon 50D. I'm shooting raw+jpg, both at highest resolutions. I'm uploading the jpgs. When I download the jpgs, some of the file sizes are different.

When I copy them off the card and compare them to my archive drives, they're the same. When I upload them to SM and then download using AF or owner save, the file size is off by 5k.

I've tried deleting the file, replacing the file, both using SM's web tools and the result is the same.

Here's a link to the two files:
http://newpics.huntsvillecarscene.com/Archive/2010/10040201/11699640_34TJ8#829586892_gnXYr
http://newpics.huntsvillecarscene.com/Archive/2010/10040201/11699640_34TJ8#829586559_L3Lzz

The first should be 6489kb. The second 4450kb. When downloading, the first is 6485 and the second 4455.

This puts a real hiccup in my data integrity system since I want to compare my archives on SM with the originals before deleting the originals. I have an event coming up in 48hrs and need to compare about 10gb, but keep running into this problem. I need to know if this a problem on SM's back end and what's going on. Help!

Were the files you uploaded sRGB? That's the only case I know of where Smugmug modifies your original besides the image modification tools (cropping, color, etc...).

SamirD

I just did a binary compare on the 7965 set using FC thinking that maybe there's just some padding or an offset of data that makes them different.

What I found looks kinda strange--I see patterns of bits that have been 'rearranged'. Check out the series of four 01s on the left column. They're in the right column, but in a different order, broken up by the 02s that in the left column follow sequentially. As you scan down the left column, there is a 05 followed by a series of 03s before three 05s. On the right, the 03s have all been put together as have the 05s.

00006BDD: 06 05
00006BDE: 05 04
00006BDF: 05 04
00006BE1: 04 05
00006BE5: 01 FF
00006BE6: 01 DB
00006BE7: 01 00
00006BE8: 01 43
00006BEC: 02 01
00006BEF: 02 01
00006BF0: 05 02
00006BF1: 03 01
00006BF2: 03 01
00006BF3: 03 02
00006BF5: 05 03
00006BF6: 05 03
00006BF7: 05 03
00006C26: FF 05
00006C27: C0 05
00006C28: 00 05
00006C29: 11 05
00006C2A: 08 FF
00006C2B: 0C C0

SamirD

jfriend wrote:

Were the files you uploaded sRGB? That's the only case I know of where Smugmug modifies your original besides the image modification tools (cropping, color, etc...).

Thank you jfriend! Let me check the camera...dammit...yes. The camera is set to sRGB. The other option is Adobe RGB. Is there any reason that some of the files are comparing okay while others aren't? Which option should I use on the camera--adobe or s?

jfriend

SamirD wrote:

Thank you jfriend! Let me check the camera...dammit...yes. The camera is set to sRGB. The other option is Adobe RGB. Is there any reason that some of the files are comparing okay while others aren't? Which option should I use on the camera--adobe or s?

I can't tell you what to use on the camera because that depends upon your workflow, but you should ONLY upload sRGB images to Smugmug. If you aren't color-profile-savvy, then shoot in sRGB.

If you upload an AdobeRGB to Smugmug (or other profile), Smugmug will convert the original to sRGB (thus changing the original).

SamirD

jfriend wrote:

If you upload an AdobeRGB to Smugmug (or other profile), Smugmug will convert the original to sRGB (thus changing the original).

Well, then the compare errors don't make sense since everything is in sRGB. I just checked the color space on 7965 off the original file to make sure--sRGB. What else do you think is going on besides a genuine back-end bug/process?

jfriend

SamirD wrote:

Well, then the compare errors don't make sense since everything is in sRGB. I just checked the color space on 7965 off the original file to make sure--sRGB. What else do you think is going on besides a genuine back-end bug/process?

I don't know. I think someone from Smugmug will have to help you sort it out.

I use BackBlaze for my backup because it takes all sorts of files, not just image files and never modifies anything. It also backs up everything automatically in the background so I don't have to remember to do anything.

SamirD

jfriend wrote:

I don't know. I think someone from Smugmug will have to help you sort it out.

Thank you for all your help John. I was sure hoping it was just a color space issue.

jfriend wrote:

I use BackBlaze for my backup because it takes all sorts of files, not just image files and never modifies anything. It also backs up everything automatically in the background so I don't have to remember to do anything.

Yeah, a lot of these solutions just don't work for me. I don't use a single computer and am more of a 'portable computing' guy so stuff is always slightly different with drive letters and such. Maybe one day I'll go back to a single computer again, but not likely.

phototristan

The files are bit-perfect despite what you are seeing. Any time you transfer any file from a computer to a server or even server to server you can see this type of thing if you analyze the files to the extend you are doing. This doesn't mean the files are degraded in any way.

Do they not open for you or look or perform any different? If yes, then there is a problem. If not, this is really not something to worry about. It's natural.

Best regards,

SamirD

phototristan wrote:

The files are bit-perfect despite what you are seeing. Any time you transfer any file from a computer to a server or even server to server you can see this type of thing if you analyze the files to the extend you are doing. This doesn't mean the files are degraded in any way.

I haven't run into this even once in over 300,000 files I've uploaded,downloaded, and compared in this method with my other 4 cameras. Why is the Canon's files having issues?

phototristan wrote:

Do they not open for you or look or perform any different? If yes, then there is a problem. If not, this is really not something to worry about. It's natural.

They won't look different because photo and video files have so much information that small changes won't be noticeable to the human eye. But they are different from the original, and telling me not to worry about it sends up a red flag in my book. It's not natural--it's data corruption.

If I spent hours of time to create an image in Photoshop, I want every one of those bits making to Bay Photo or EZprints. I don't want them distorted or changed in any way except via the corrections that I've applied via SM or set for SM to apply.

For now, I've worked around this issue by keeping three local copies of the images and not relying on the images on SM as a 'true' copy. But if I start seeing issues like this with my other cameras that have been perfect for years, then something is amiss at SM and I'll need a better explanation.

jfriend

phototristan wrote:

The files are bit-perfect despite what you are seeing. Any time you transfer any file from a computer to a server or even server to server you can see this type of thing if you analyze the files to the extend you are doing. This doesn't mean the files are degraded in any way.

Do they not open for you or look or perform any different? If yes, then there is a problem. If not, this is really not something to worry about. It's natural.

Best regards,

How can you say that they're bit perfect, when they don't contain the same bits. Either the files are the same or the files are not - there is no middle ground. He's doing a bit-by-bit compare and finding them different.

Are you saying that his bit compare is flawed or are you saying that Smugmug is modifying the images? It has to one or the other. This is not a question about whether they are perceived the same. It is a question about whether Smugmug is modifying them in ANY way.

phototristan

SamirD wrote:

I haven't run into this even once in over 300,000 files I've uploaded,downloaded, and compared in this method with my other 4 cameras. Why is the Canon's files having issues?
They won't look different because photo and video files have so much information that small changes won't be noticeable to the human eye. But they are different from the original, and telling me not to worry about it sends up a red flag in my book. It's not natural--it's data corruption.

If I spent hours of time to create an image in Photoshop, I want every one of those bits making to Bay Photo or EZprints. I don't want them distorted or changed in any way except via the corrections that I've applied via SM or set for SM to apply.

For now, I've worked around this issue by keeping three local copies of the images and not relying on the images on SM as a 'true' copy. But if I start seeing issues like this with my other cameras that have been perfect for years, then something is amiss at SM and I'll need a better explanation.

I'm sorry but I beg to differ. This is not data corruption. It's like when files on a hard disk become fragmented. This is a totally normal situation and the nature of the way bits get written to the hard disk through normal usage. That data may be fragmented however that does not instantly mean they are corrupt.

jfriend

phototristan wrote:

I'm sorry but I beg to differ. This is not data corruption. It's like when files on a hard disk become fragmented. This is a totally normal situation and the nature of the way bits get written to the hard disk through normal usage. That data may be fragmented however that does not instantly mean they are corrupt.

Uhhh - please go talk to someone else at Smugmug about his because this is absolutely not true. If one is only comparing the true number of bits in the file (as per the filesize of the file and not looking at whole clusters or anything like that) then ANY change in bits is file corruption or modification - plain and simple.

When I transfer 1,030,546 bytes from one place to another, those 1,030,546 should be EXACTLY the same at the destination. If they are not, then the data has been corrupted. If I copy it to one disk, then from there to another, then from there to another, those 1,030,546 bytes should still be exactly the same.

phototristan

SamirD wrote:

I haven't run into this even once in over 300,000 files I've uploaded,downloaded, and compared in this method with my other 4 cameras. Why is the Canon's files having issues?
They won't look different because photo and video files have so much information that small changes won't be noticeable to the human eye. But they are different from the original, and telling me not to worry about it sends up a red flag in my book. It's not natural--it's data corruption.

If I spent hours of time to create an image in Photoshop, I want every one of those bits making to Bay Photo or EZprints. I don't want them distorted or changed in any way except via the corrections that I've applied via SM or set for SM to apply.

For now, I've worked around this issue by keeping three local copies of the images and not relying on the images on SM as a 'true' copy. But if I start seeing issues like this with my other cameras that have been perfect for years, then something is amiss at SM and I'll need a better explanation.

Please create a ticket (or update your already created ticket in our helpdesk system) and upload us here, one of your original images: http://dropbox.yousendit.com/SmugMug

In the ticket/email, please also link us to the same image on SmugMug. I will escalate to an engineer so they can take a look.