Do you use Amazon S3?

greenpea

If you use Amazon S3 for backing up all those raw and psd files, what client do you use? Anyone using anything other than jungle disk or the s3 firefox organizer? I'm wondering if anyone is using different client that they really like.

Andy

I use Jungle Disk, no troubles thus far.

Was gonna give S3Fox a try though, too, have heard good things about that.

denisegoldberg

I'm using JungleDisk, no problems so far - but S3 Firefox Organizer looks interesting. I may give that a try too.

--- Denise

greenpea

The problem is jungle disk writes the uploaded files in its own format and as such only jungle disk can only read files that uploaded in its format, this means files uploaded with s3fox can't be seen or downloaded by jungle disk. Note this is what is happening if jungle disk encryption is turned off.

Now s3fox can see the files that jungle disk uploaded, but they are unusable by s3fox once they are downloaded.

Jungle disk seems much slower than s3fox, but s3fox is buggy (throws script errors when uploaded large numbers of files).

So I'm wondering is there a 3rd option?

bwg

greenpea wrote:

The problem is jungle disk writes the uploaded files in its own format and as such only jungle disk can only read files that uploaded in its format, this means files uploaded with s3fox can't be seen or downloaded by jungle disk. Note this is what is happening if jungle disk encryption is turned off.

Now s3fox can see the files that jungle disk uploaded, but they are unusable by s3fox once they are downloaded.

Jungle disk seems much slower than s3fox, but s3fox is buggy (throws script errors when uploaded large numbers of files).

So I'm wondering is there a 3rd option?

I hate that about JungleDisk. The newest version of Transmit (for the mac) has S3 support and I just sync folders periodically.

DavidTO

bigwebguy wrote:

I hate that about JungleDisk. The newest version of Transmit (for the mac) has S3 support and I just sync folders periodically.

That is really good news.

greenpea

bigwebguy wrote:

I hate that about JungleDisk. The newest version of Transmit (for the mac) has S3 support and I just sync folders periodically.

I'll have to try transmit.

I should note that jungle disk does provide the source code for encoding and decoding their file format, but the code is in Microsoft's .NET C#.

EDIT: I just checked out transit, this app is pretty cool. Looks like the way to go. It doesn't encode the files, its fast, and its not buggy.

billanders

greenpea wrote:

If you use Amazon S3 for backing up all those raw and psd files, what client do you use? Anyone using anything other than jungle disk or the s3 firefox organizer? I'm wondering if anyone is using different client that they really like.

For the past two months I've been using "Bucket Explorer". It's got pro's and con's. They've been incrementally building it since I've started using it and it's getting much better. With it I've been able to upload ~115GB of photos.

The upside is that I've been able to use the trial version to upload all of my RAW + PSD's from 2004-present. The downside is that the $29.95 price (after the trial expires) allows upgrades only for six months, then you have to pay again. As far as file transfer software goes, that's pretty pricey. But I don't begrudge the developers the right to charge what they think is fair.

I have S3 Fox running down in my toolbar, but haven't used it to transfer many files to my bucket. I think I'll use it to maintain my buckets once the Bucket Explorer trial is done.

greenpea

billanders wrote:

I have S3 Fox running down in my toolbar, but haven't used it to transfer many files to my bucket. I think I'll use it to maintain my buckets once the Bucket Explorer trial is done.

S3 Fox is okay, but it appears to create some additional files with each upload (not sure what there for). Also, on a mac, S3 Fox kept crashing on me or locking up.

So far, transmit is the best tool I've found (mac only though), and transmit is also $29.95.

DavidTO

bigwebguy wrote:

I hate that about JungleDisk. The newest version of Transmit (for the mac) has S3 support and I just sync folders periodically.

OK, so I'm on Transmit, doing my first upload (haven't tried the sync function yet), but I gotta say: this is one well-written app, from what I can see. I'm liking it.

Also, there's an automator action for syncing. I've set it up, but haven't done anything with it yet, since I'm on the initial upload, and I don't wanna mess that up. I'll try it when the upload is complete.... sometime next year!

DavidTO

My son unplugged our internet connection

So, since the upload aborted, I tried the sync function, and it's not working, I think because the automatic server time offset detection isn't working. Any idea how to figure the S3 server time offset manually?

greenpea

DavidTO wrote:

My son unplugged our internet connection

So, since the upload aborted, I tried the sync function, and it's not working, I think because the automatic server time offset detection isn't working. Any idea how to figure the S3 server time offset manually?

The only thing I've tried so far is the upload mirror sync with only "Deterimine server time offset automatically" selected, and everything seemed to work fine. However my sync was pretty simple, it was only adding files that were missing.

mrcoons

For me I'd say that S3Fox is at least twice as fast as JungleDisk.

rdlugosz

I use Jungle Disk to handle the backups for some of my smaller web clients, however I find that the S3 pricing model doesn't stack up when you're storing large amounts of data. You may want to try a service like Mozy or Carbonite... check out this article on my blog regarding my photo backup strategy.

greenpea

rdlugosz wrote:

I use Jungle Disk to handle the backups for some of my smaller web clients, however I find that the S3 pricing model doesn't stack up when you're storing large amounts of data. You may want to try a service like Mozy or Carbonite... check out this article on my blog regarding my photo backup strategy.

I looked at both carbonite and mozy, the problem was they both seem to be painfully slow on the upload and fairly slow on the download, and mozy only has beta support for macs. :cry

Upload and download on S3 seems to be only limited by your connection speed.

rdlugosz

greenpea wrote:

I looked at both carbonite and mozy, the problem was they both seem to be painfully slow on the upload and fairly slow on the download, and mozy only has beta support for macs. :cry

Upload and download on S3 seems to be only limited by your connection speed.

That may be true for download speeds, but upload is able to keep pace with my (kind of average) 768kbps cable modem uploaad rate. I'm not sure how big a factor the download speed really is - I hope to *never*, ever use it!

For me the only reason for the off site backup is just that - a failsafe to turn to in the event of a complete catastrophy in my office that takes out both my PC and the backup.

If you're looking for an online disk that you'll be using in remote locations then maybe something like Jungle+S3 is the right thing for you, but at the same time I think you'd want to have some type of web-based file browsing... you may not want to install software everywhere you want to view a file.

greenpea

rdlugosz wrote:

That may be true for download speeds, but upload is able to keep pace with my (kind of average) 768kbps cable modem uploaad rate. I'm not sure how big a factor the download speed really is - I hope to *never*, ever use it!

For me the only reason for the off site backup is just that - a failsafe to turn to in the event of a complete catastrophy in my office that takes out both my PC and the backup.

If you're looking for an online disk that you'll be using in remote locations then maybe something like Jungle+S3 is the right thing for you, but at the same time I think you'd want to have some type of web-based file browsing... you may not want to install software everywhere you want to view a file.

I am using S3 as more of an online disk, rather than just a back up. My jpegs go on smugmug, my raw files, psd, and video files go on S3.

As for slow speeds, I read this article that said carbonite will let you upload 2GB a day, that would mean 2 months to do my initial back up my entire harddrive on my laptop. As for mozy, I seem to recall reading somewhere that mozy was also slow, but now I can't find that link.

javierplumey

greenpea wrote:

I am using S3 as more of an online disk, rather than just a back up. My jpegs go on smugmug, my raw files, psd, and video files go on S3.

As for slow speeds, I read this article that said carbonite will let you upload 2GB a day, that would mean 2 months to do my initial back up my entire harddrive on my laptop. As for mozy, I seem to recall reading somewhere that mozy was also slow, but now I can't find that link.

Well I am trying Mozy right now and it is indeed REALLY slow. Plus I can't use Vonage when Moxy is uploading for some reason. Their support is slow but I'm going to finish this backup and then see how the incrementals go.

I used to use a Dreamhost account with 200 GB storage and a couple of simple rsync scripts and it worked great. I could upload as fast as my connection would allow and I synced up every day. I had my entire 26 GB iPhoto library online in no time at all.

Restoring was easy. My hard drive crashed and I simply ran the rsync the other way and I got my files back in less than two days.

DId you notice all the past tense there? That's because Dreamhost changed their terms of service that doesn't allow me to use it as a file backup.

I was really happy with my old solution, but now I am looking for a new one. I't was hard to beat 200+GB of space and nearly unlimited transfer for only $5 a month.

rdlugosz

javierplumey wrote:

Well I am trying Mozy right now and it is indeed REALLY slow. Plus I can't use Vonage when Moxy is uploading for some reason. Their support is slow but I'm going to finish this backup and then see how the incrementals go.

I'd guess that it comes down to your maximum upload speed on your broadband provider. Mine tops out at 768kbps - which is hella slow - but it eventually gets done. Mozy will saturate that upstream limit for me, but that doesn't necessarily mean that they'll go higher... I just can't test it!

As for the Vonage, it is probably because it's fighting for bandwidth (going up) with the Mozy client. I have a similar situation with my VOIP phone (through ViaTalk) - I fixed it by giving the MAC address of my VOIP Adapter higher priority on the firewall/router than anything else. This way, anything the VOIP needs gets priority & the calls aren't interrupted.

javierplumey

rdlugosz wrote:

I'd guess that it comes down to your maximum upload speed on your broadband provider. Mine tops out at 768kbps - which is hella slow - but it eventually gets done. Mozy will saturate that upstream limit for me, but that doesn't necessarily mean that they'll go higher... I just can't test it!

As for the Vonage, it is probably because it's fighting for bandwidth (going up) with the Mozy client. I have a similar situation with my VOIP phone (through ViaTalk) - I fixed it by giving the MAC address of my VOIP Adapter higher priority on the firewall/router than anything else. This way, anything the VOIP needs gets priority & the calls aren't interrupted.

I get consistent 2.5 Mbps on speed tests and when I upload through rsync I used to get around 300KB/s upload speeds and I had no issues with Vonage. With Mozy, the speed is much slower and I can't run Vonage at all.
I should be done by the end of the week and then I'll see how the incrementals go at night.

I'm researching using S3 but I haven't found a client I like yet.

AbeFroman

How to create sub-folders within S3 buckets using Transmit?
Hi there,

I thought about starting a new thread, but I wanted to try first in this thread in case anyone knew the answer to my Transmit/S3 question:

I am having trouble creating a sub-folder within a S3 bucket. When I am in Transmit and in my bucket, I right-click and select "New Folder..." and I provide a folder name. Then Transmit gives me the error message:

"Could not create folder named "". You must provide the Content-Length HTTP header."

I am able to create buckets on my S3 account and drop files into them just fine, it's just when I try to create a sub-folder that I get the error message above. Anyone know what I am doing wrong and how to move forward?

Thank you!

greenpea

AbeFroman wrote:

Hi there,

I thought about starting a new thread, but I wanted to try first in this thread in case anyone knew the answer to my Transmit/S3 question:

I am having trouble creating a sub-folder within a S3 bucket. When I am in Transmit and in my bucket, I right-click and select "New Folder..." and I provide a folder name. Then Transmit gives me the error message:

"Could not create folder named "<folder name>". You must provide the Content-Length HTTP header."

I am able to create buckets on my S3 account and drop files into them just fine, it's just when I try to create a sub-folder that I get the error message above. Anyone know what I am doing wrong and how to move forward?

Thank you!

I do use Transmit with S3, but generally I copy full folders from my hard drive to S3. However for the sake of testing this issue, I just tried on my MacBook Pro to create a new folder inside one of my buckets. It created a folder without any problem.

What version of Transmit are you using? I have Version 3.6.4.

AbeFroman

greenpea wrote:

I do use Transmit with S3, but generally I copy full folders from my hard drive to S3. However for the sake of testing this issue, I just tried on my MacBook Pro to create a new folder inside one of my buckets. It created a folder without any problem.

What version of Transmit are you using? I have Version 3.6.4.

Thanks for the reply. Actually, I neglected to mention that I tried to copy entire folders over as well and got the same error message as when I tried to create an empty folder in the bucket. Very odd, huh?

Sorry, forgot to mention the version. I have version 3.6.4 as well on a Mac running 10.3.9 (which is the minimum requirement). I sent an email to Panic yesterday but so far no replies. I hope something happens soon as I love the software and would like to buy it and use it.

greenpea

AbeFroman wrote:

Thanks for the reply. Actually, I neglected to mention that I tried to copy entire folders over as well and got the same error message as when I tried to create an empty folder in the bucket. Very odd, huh?

Sorry, forgot to mention the version. I have version 3.6.4 as well on a Mac running 10.3.9 (which is the minimum requirement). I sent an email to Panic yesterday but so far no replies. I hope something happens soon as I love the software and would like to buy it and use it.

Sorry, I forgot to meniton the other difference is that my Mac is running 10.5.2.

In the short term there is always the FF plugin S3Fox that Andy mentioned at the beginning of this thread. Just don't try and use S3Fox for uploading huge numbers of files.

AbeFroman

greenpea wrote:

In the short term there is always the FF plugin S3Fox that Andy mentioned at the beginning of this thread. Just don't try and use S3Fox for uploading huge numbers of files.

Thanks for the suggestion and the continued help. I like the idea of using S3Fox as a workaround since I only need to create a few sub-folders on S3 before populating them with my files. Unfortunately, when I used S3Fox to create a folder (for example "travel"), Transmit recognizes the folder as "travel_$folder$" and doesn't let me drill down a level to drop files into it. Maybe it's because I am a S3 newbie, but this doesn't sound right at all. Is this a common problem/situation?

The original error message I posted about sounded like a software bug with Transmit at first, but since I seem to be the only one with the problem maybe it's because I am using an older OS or something? Argh....

greenpea

AbeFroman wrote:

Thanks for the suggestion and the continued help. I like the idea of using S3Fox as a workaround since I only need to create a few sub-folders on S3 before populating them with my files. Unfortunately, when I used S3Fox to create a folder (for example "travel"), Transmit recognizes the folder as "travel_$folder$" and doesn't let me drill down a level to drop files into it. Maybe it's because I am a S3 newbie, but this doesn't sound right at all. Is this a common problem/situation?

The original error message I posted about sounded like a software bug with Transmit at first, but since I seem to be the only one with the problem maybe it's because I am using an older OS or something? Argh....

S3Fox does create $folder$ files in my bucket (that I only see when using Transmit), but I just delete them later with Transmit. However in my case S3Fox also created a folder that Transmit could read.

Hopefully the people at panic.com can help, otherwise I can tell you OS 10.5 is pretty cool

One other option if your tech savvy. Amazon has sample applications using serveral different technologies. You could always try one of those.

AbeFroman

greenpea wrote:

S3Fox does create $folder$ files in my bucket (that I only see when using Transmit), but I just delete them later with Transmit. However in my case S3Fox also created a folder that Transmit could read.

Hopefully the people at panic.com can help, otherwise I can tell you OS 10.5 is pretty cool

One other option if your tech savvy. Amazon has sample applications using serveral different technologies. You could always try one of those.

Very odd, indeed. It must be my OS version. You are having a much different experience than me and that's the major variable between us. Hopefully Panic will write back soon. Don't they want my money? We'll see. I might check out those Amazon apps you mentioned. Either way, thanks very much for your time and suggestions. Much appreciated.