And at some point with each, I had to re-upload something that fell through the cracks.
I've only briefly played with Star*Explorer, but from what I read here on the forums, Nikolai built the tool specifically to avoid having to re-upload stuff. My understanding is that it has all kinds of retry/timeout settings and queueing to make sure that you can just "set it and forget it" and the next morning everything will have been uploaded.
Did you contact him about your issues? I'd be surprised if he didn't respond immediately. He's a very attentive developer.
I've only briefly played with Star*Explorer, but from what I read here on the forums, Nikolai built the tool specifically to avoid having to re-upload stuff. My understanding is that it has all kinds of retry/timeout settings and queueing to make sure that you can just "set it and forget it" and the next morning everything will have been uploaded.
Did you contact him about your issues? I'd be surprised if he didn't respond immediately. He's a very attentive developer.
This is why I use StarExplorer. It even has a persistent queue (on disk) so even if you have a power outage in the middle of an upload and your computer goes dark, it will know what it has and hasn't uploaded when you start it up again. And, you can upload to multiple galleries in one unattended session.
If there's really a hard error on Smugmug's end of things or in your network connection, it may eventually run out of retry attempts on a few images. If that's the case, when it's all done, it will just have the images that didn't upload sitting in it's queue and you can just hit Upload again and they will go (assuming the transmission problem is now gone). I have never been left trying to figure out which images uploaded successfully and which did not in StarExplorer which is a real pain in many other uploaders when doing large uploads across many galleries.
If you want robust uploads, I'd highly recommend StarExplorer. I've been using it for years and Nikolai has been very responsive.
I'm not sure I'm completely following you here. Is there a feature in Livedrive that allows the files to be directly transferred to SM?
Well, the only place Google found somebody talking about Livedrive integration with Smugmug is uhm... DrDavid's Twitter feed, where just yesterday he was complaining about Livedrive's unexplained outage that apparently went on for hours:
So DrDavid, can you tell us more about how you use Livedrive? I see they have integration with FaceBook photo albums, so I have an idea of how it might work with SmugMug.
But $55/year is a lot to pay just to be able to FTP your files somewhere and then drag them to SmugMug.
An extra place to store your backups is always nice, but from web.archive.org, it looks like this company's only been around since 2006, and in 2007 they were still in beta. So... future prospects for your data are hazy, please ask again in 5 years. :-}
I've only briefly played with Star*Explorer, but from what I read here on the forums, Nikolai built the tool specifically to avoid having to re-upload stuff. My understanding is that it has all kinds of retry/timeout settings and queueing to make sure that you can just "set it and forget it" and the next morning everything will have been uploaded.
Did you contact him about your issues? I'd be surprised if he didn't respond immediately. He's a very attentive developer.
I've used Star Explorer and even bought a pro copy, which I stopped using after having more issues than with the standard web uploaders when trying to do multi-threaded uploads. I think it was a request I had years ago, but to be honest I haven't checked if the new version has it or not. It also wasn't a 'light' application, so I had to have it installed on a computer to use it. This is a problem for me since sometimes all I have is my hard drive and 'a' computer of some sort. I do need to revisit SE if it has multi-threaded upload capabilities now.
I really wish it was more robust. I found it to be no better than any of the other third party or built-in uploaders in terms of working every time.
But $55/year is a lot to pay just to be able to FTP your files somewhere and then drag them to SmugMug.
The FTP process would save me enough time to warrant getting their $130/yr unlimited plan if the upload to SM would be a very quick process. Waiting twice defeats the purpose of the FTP transfer.
Well, the only place Google found somebody talking about Livedrive integration with Smugmug is uhm... DrDavid's Twitter feed, where just yesterday he was complaining about Livedrive's unexplained outage that apparently went on for hours:
Well, what I disliked was the lack of communication. They did finally write me an email apologizing for the downtime; and it was (as I later found) on their website at status.livedrive.com... BUT, it was communicated VERY poorly.
Here's the readers-digest version though: the service works well, and has good uptime. But, since they're in the UK, they're "night" troubleshooting was smack-dab in the middle of my day. This doesn't happen often though.
They really are unlimited. I have a few hundred GB's uploaded on their servers right now. I get **VERY** good throughput (like, coming close to maxing my 20Mbps connection--consistently!) I actually use SmartFTP to upload to them using 15 streams. Also, their desktop software allows 8 streams max. So, it does a good job of maximizing your connection and reducing overhead.
Synchronizing between multiple computers is painless, and conflict resolution is actually quite good. I've only had a few times when it just couldn't figure out the conflict automatically and solve it. If it doesn't work, it just pops a message asking to keep both, keep remote or local copy.
Hope that helps?
David
p.s. If you do sign up, please use this link: http://tr.im/livedrive Yes, I do get a referral bonus if you use it You could also use my referral code: UU32MC3YM7WGUX3VG2SY to use it, on the page where you're asked where you heard about livedrive, choose "other" and enter that code. Thanks!
I don't think you'll be seeing big sites like SmugMug, Flickr, Facebook, etc., supporting FTP, unfortunately. Google just dropped support of it for Blogger.
We are working on some network improvements to streamline uploads, however, which would apply to all uploaders and plugins.
I'm sorry I don't have a better answer for you with FTP.
I don't think you'll be seeing big sites like SmugMug, Flickr, Facebook, etc., supporting FTP, unfortunately. Google just dropped support of it for Blogger.
We are working on some network improvements to streamline uploads, however, which would apply to all uploaders and plugins.
I'm sorry I don't have a better answer for you with FTP.
OK so, why not then? Personally I don't really care that much but since adding it would make a whole lot of FTP geeks happy, there has to be a real reason.
OK so, why not then? Personally I don't really care that much but since adding it would make a whole lot of FTP geeks happy, there has to be a real reason.
I've never heard of ISPs restricting FTP. I manage FTP site for our company, and we have many clients downloading and uploading files for us by FTP. Some do have problems, mostly with their own NAT routers or firewalls, but most of the problems are easily overcome by using FTP passive mode (default in practically everything newer than Windows native FTP). All browsers support ftp: urls. So FTP is far from dead, as that post pictures it. FTP is more efficient than HTTP, a lot more convenient for uploading multiple files (the problem we're trying to address here!), and sFTP is a lot more secure than HTTP and even HTTPS (allow us to upload public ssh key through Control Panel for sFTP connections, if you want to be geeky).
By the way. There's a reason why we're here rather than at Flickr. Maybe SM should try to differentiate itself more, rather than copying others' limitations.
I read in a different thread that there's some technical limitation with the way Smugmug stores the files that makes it incompatible with FTP.
I read that too, and I don't doubt it's true. But that being said, Exposure Manager has a method by which they have implemented FTP.
Theirs seems to work by having an interim storage locally that transfers the uploaded files to their system. Think of an FTP server at SM that runs a shell script to upload to a gallery based on the FTP upload. It would be lightning quick within the data center. And it would only happen after you click a button or indicate that the FTP upload is complete. The interface and method at EM works really well, and I would like to see SM implement it here. Check out the details on the system in EM's FAQ: http://help.exposuremanager.com/kb/?CategoryID=35
It's also important to note that EM still has a web-based uploader. SM can do the same. There's no reason to eliminate it as it works so well for so many.
This was such a debilitating problem for me back in the day that I actually signed on with EM and tried the FTP system. But then SM added videos (which I was searching for a solution for), so I just stayed with SM. And I'm glad I did as the uploaders have gotten much, much better. It's only when I've got a batch of oh...100gb...that this problem becomes a problem again.
I don't think you'll be seeing big sites like SmugMug, Flickr, Facebook, etc., supporting FTP, unfortunately. Google just dropped support of it for Blogger.
We are working on some network improvements to streamline uploads, however, which would apply to all uploaders and plugins.
I'm sorry I don't have a better answer for you with FTP.
Thank you for the reply Baldy. The uploaders have gotten much, much better in the years I've been with SM. But that being said, I still see a better way to do it, and I think EM has it right now. I'd like to see a similar system for SM. Maybe if enough people want it...
Sounds like pretty nasty issues to secure ftp uploads appropriately and make them reliable.
I don't think it's any more insecure than the current http methods.
The current SM upload login is secure via https, which sFTP can also do. But in both scenarios, the data (a bunch of random 0s and 1s that when ony perfectly put together make a photo) is transferred in the wide open (as far as I can tell).
I think that FTP definitely isn't for everyone. In fact, I think only 10% of pros will use it. But I for one am willing to pay extra for it.
Hell, I would jump on a plane tomorrow if I could sit somewhere with a high-speed connection to SM's datacenter and do this transfer. I've literally spent a week on this. And this is on three different computers with three different cable modems, simultaneously! I slept at 1:45am this morning and was up at 6:45a! There have been times I've been losing my sanity.
I know it's more difficult to support FTP since it was never designed to operate in this web 2.0, dynamically-generated, fancy, pretty Internet. FTP is a brute workhorse, and can only be dressed up so much to go to the ball. But FTP's strength is that it gets the job done--and that's something all of us can love, even if it isn't pretty.
I started playing with my EM account again. I think I'm going to attempt to upload everything that I'm trying to upload here over there and see how much time it takes. Of course, that just means queuing everything in my FTP client, telling it to use multiple connections, and completely walking away.
I don't think it's any more insecure than the current http methods.
The current SM upload login is secure via https, which sFTP can also do. But in both scenarios, the data (a bunch of random 0s and 1s that when ony perfectly put together make a photo) is transferred in the wide open (as far as I can tell).
I think that FTP definitely isn't for everyone. In fact, I think only 10% of pros will use it. But I for one am willing to pay extra for it.
Hell, I would jump on a plane tomorrow if I could sit somewhere with a high-speed connection to SM's datacenter and do this transfer. I've literally spent a week on this. And this is on three different computers with three different cable modems, simultaneously! I slept at 1:45am this morning and was up at 6:45a! There have been times I've been losing my sanity.
I know it's more difficult to support FTP since it was never designed to operate in this web 2.0, dynamically-generated, fancy, pretty Internet. FTP is a brute workhorse, and can only be dressed up so much to go to the ball. But FTP's strength is that it gets the job done--and that's something all of us can love, even if it isn't pretty.
I started playing with my EM account again. I think I'm going to attempt to upload everything that I'm trying to upload here over there and see how much time it takes. Of course, that just means queuing everything in my FTP client, telling it to use multiple connections, and completely walking away.
Did you read the referenced article because I don't think you responded to the main points in that article? I, for one, put some faith into Google's understanding of large scale data movement and their experience with blogger. Google doesn't give up because something's hard. They give up when there are better ways to solve the problem. They rewrote their ftp infrastructure a couple times and still ended up with 10% failures. That's horrible. Then, due to data structure changes, they were left with more rewrites.
As I've said before, in my 5+ years experience with Smugmug, the best way I've found to reliably move a lot of photos into Smugmug is with StarExplorer. Not only does it have a reliable disk-based queue and keeps track of what was successfully uploaded or not so you know what got there and what didn't, but you can also upload to lots of different galleries in one unattended upload so you don't have to dump zillions of images into one gallery. I never understood why it didn't work for you, but it's been my best way to solve the problem.
If you're really just trying to do online backup of a lot of images, then I'd suggest BackBlaze. It has a background uploader that just works in the background forever. I've put more than 500GB online through BackBlaze. It's less than $50 year for unlimited online backup and a totally automated system.
The uploaders have gotten much, much better in the years I've been with SM.
This is true. But the demands have gotten much larger too, with videos and SmugVault. And FTP clients have been improving as well.
I've been using SmugBrowser for uploads, and while it is better than it used to be, using it for videos is a big PITA (with it starting 6 simultaneous uploads of large videos, killing bandwidth for anything else, and needing to restart everything from beginning after any hick-up).
As I've said before, in my 5+ years experience with Smugmug, the best way I've found to reliably move a lot of photos into Smugmug is with StarExplorer.
+1 on this.
If you're really just trying to do online backup of a lot of images, then I'd suggest BackBlaze. It has a background uploader that just works in the background forever. I've put more than 500GB online through BackBlaze. It's less than $50 year for unlimited online backup and a totally automated system.
Did you read the referenced article because I don't think you responded to the main points in that article? I, for one, put some faith into Google's understanding of large scale data movement and their experience with blogger. Google doesn't give up because something's hard. They give up when there are better ways to solve the problem. They rewrote their ftp infrastructure a couple times and still ended up with 10% failures. That's horrible.
I don't know how much they tried to integrate FTP into their infrastructure -- I'm not using Blogger much. The failures must have been in that integration area, because for just transferring files from here to there FTP is extremely reliable (the protocol itself, and there are many very mature clients, for any platform). This would also explain their constant need to rewrite their end. With the way SamirD and I have proposed, with a temporary holding area and an on-demand import, it requires almost no integration with the rest of SM back-end. I also would be interested to know what the failure rate is with SM uploaders -- I wouldn't be surprised if it was more than 10%.
As I've said before, in my 5+ years experience with Smugmug, the best way I've found to reliably move a lot of photos into Smugmug is with StarExplorer.
+1 on this.
For all the praises, SE still can't resume an interrupted video upload where it stopped (a limitation of SM API, according to the author -- but who cares). FTP can.
And SE is still an external application, closed source, developed by a 3rd party, that needs full access to your SM account. Not to mention it costs extra $$$.
For all the praises, SE still can't resume an interrupted video upload where it stopped (a limitation of SM API, according to the author -- but who cares). FTP can.
And SE is still an external application, closed source, developed by a 3rd party, that needs full access to your SM account. Not to mention it costs extra $$$.
Anything that uploads is going to need access to your account. Uploads are restricted to the site owner so that has to be authenticated.
I'm sure it would be way, way eaiser for Smugmug to fix resumable video uploads using their current protocols than to implement ftp from scratch. If they are really serious about video uploads, they should support incremental, restartable uploads - that should be a no brainer any time you're asking customers to upload files of 100's of MBs that can take hours to upload.
Samir said he was happy to pay for a better uploading solution so $$ shouldn't be an issue here. Closed source. I don't understand that in this instance. Everything you use at Smugmug is closed source. If Smugmug drops something you're unhappy with or won't fix something you're unhappy with, there's nothing you can do about it. Any ftp client you used would be developed by a third party too. I'm not getting the resistance. Sure, I understand you have a fondness for ftp, but there are other solutions and they work fine. I, for one, would rather Smugmug put their resources into other issues that will benefit far more users.
Comments
I've only briefly played with Star*Explorer, but from what I read here on the forums, Nikolai built the tool specifically to avoid having to re-upload stuff. My understanding is that it has all kinds of retry/timeout settings and queueing to make sure that you can just "set it and forget it" and the next morning everything will have been uploaded.
Did you contact him about your issues? I'd be surprised if he didn't respond immediately. He's a very attentive developer.
If there's really a hard error on Smugmug's end of things or in your network connection, it may eventually run out of retry attempts on a few images. If that's the case, when it's all done, it will just have the images that didn't upload sitting in it's queue and you can just hit Upload again and they will go (assuming the transmission problem is now gone). I have never been left trying to figure out which images uploaded successfully and which did not in StarExplorer which is a real pain in many other uploaders when doing large uploads across many galleries.
If you want robust uploads, I'd highly recommend StarExplorer. I've been using it for years and Nikolai has been very responsive.
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
Well, the only place Google found somebody talking about Livedrive integration with Smugmug is uhm... DrDavid's Twitter feed, where just yesterday he was complaining about Livedrive's unexplained outage that apparently went on for hours:
http://twitter.com/wolfsnap
So that doesn't bode well. :-/
So DrDavid, can you tell us more about how you use Livedrive? I see they have integration with FaceBook photo albums, so I have an idea of how it might work with SmugMug.
But $55/year is a lot to pay just to be able to FTP your files somewhere and then drag them to SmugMug.
An extra place to store your backups is always nice, but from web.archive.org, it looks like this company's only been around since 2006, and in 2007 they were still in beta. So... future prospects for your data are hazy, please ask again in 5 years. :-}
I really wish it was more robust.
Want faster uploading? Vote for FTP!
Want faster uploading? Vote for FTP!
Here's the readers-digest version though: the service works well, and has good uptime. But, since they're in the UK, they're "night" troubleshooting was smack-dab in the middle of my day. This doesn't happen often though.
They really are unlimited. I have a few hundred GB's uploaded on their servers right now. I get **VERY** good throughput (like, coming close to maxing my 20Mbps connection--consistently!) I actually use SmartFTP to upload to them using 15 streams. Also, their desktop software allows 8 streams max. So, it does a good job of maximizing your connection and reducing overhead.
Synchronizing between multiple computers is painless, and conflict resolution is actually quite good. I've only had a few times when it just couldn't figure out the conflict automatically and solve it. If it doesn't work, it just pops a message asking to keep both, keep remote or local copy.
Hope that helps?
David
p.s. If you do sign up, please use this link: http://tr.im/livedrive Yes, I do get a referral bonus if you use it
Twitter: @WolfSnap
Facebook: http://www.facebook.com/WolfSnapDesigns
SmugMug & Wordpress Customization - WolfSnap.com | Custom Domains
I did, and it upped our rank from 40 to 39. :cry
Malte
Want faster uploading? Vote for FTP!
Mike
Website
Sports Sample Photos
Sportsshooter.com/ctproctor
Four Seam Images
Want faster uploading? Vote for FTP!
http://smugmug.uservoice.com/forums/17723-smugmug/suggestions/294159-ftp-uploading?ref=title
Want faster uploading? Vote for FTP!
I read in a different thread that there's some technical limitation with the way Smugmug stores the files that makes it incompatible with FTP.
Malte
BS.
We are working on some network improvements to streamline uploads, however, which would apply to all uploaders and plugins.
I'm sorry I don't have a better answer for you with FTP.
OK so, why not then? Personally I don't really care that much but since adding it would make a whole lot of FTP geeks happy, there has to be a real reason.
Malte
Edit: I mean, other than "Google is dropping it".
GreyLeaf PhotoGraphy
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
Theirs seems to work by having an interim storage locally that transfers the uploaded files to their system. Think of an FTP server at SM that runs a shell script to upload to a gallery based on the FTP upload. It would be lightning quick within the data center. And it would only happen after you click a button or indicate that the FTP upload is complete. The interface and method at EM works really well, and I would like to see SM implement it here. Check out the details on the system in EM's FAQ:
http://help.exposuremanager.com/kb/?CategoryID=35
It's also important to note that EM still has a web-based uploader. SM can do the same. There's no reason to eliminate it as it works so well for so many.
This was such a debilitating problem for me back in the day that I actually signed on with EM and tried the FTP system. But then SM added videos (which I was searching for a solution for), so I just stayed with SM. And I'm glad I did as the uploaders have gotten much, much better. It's only when I've got a batch of oh...100gb...that this problem becomes a problem again.
Want faster uploading? Vote for FTP!
Want faster uploading? Vote for FTP!
The current SM upload login is secure via https, which sFTP can also do. But in both scenarios, the data (a bunch of random 0s and 1s that when ony perfectly put together make a photo) is transferred in the wide open (as far as I can tell).
I think that FTP definitely isn't for everyone. In fact, I think only 10% of pros will use it. But I for one am willing to pay extra for it.
Hell, I would jump on a plane tomorrow if I could sit somewhere with a high-speed connection to SM's datacenter and do this transfer. I've literally spent a week on this. And this is on three different computers with three different cable modems, simultaneously! I slept at 1:45am this morning and was up at 6:45a! There have been times I've been losing my sanity.
I know it's more difficult to support FTP since it was never designed to operate in this web 2.0, dynamically-generated, fancy, pretty Internet. FTP is a brute workhorse, and can only be dressed up so much to go to the ball. But FTP's strength is that it gets the job done--and that's something all of us can love, even if it isn't pretty.
I started playing with my EM account again. I think I'm going to attempt to upload everything that I'm trying to upload here over there and see how much time it takes. Of course, that just means queuing everything in my FTP client, telling it to use multiple connections, and completely walking away.
Want faster uploading? Vote for FTP!
As I've said before, in my 5+ years experience with Smugmug, the best way I've found to reliably move a lot of photos into Smugmug is with StarExplorer. Not only does it have a reliable disk-based queue and keeps track of what was successfully uploaded or not so you know what got there and what didn't, but you can also upload to lots of different galleries in one unattended upload so you don't have to dump zillions of images into one gallery. I never understood why it didn't work for you, but it's been my best way to solve the problem.
If you're really just trying to do online backup of a lot of images, then I'd suggest BackBlaze. It has a background uploader that just works in the background forever. I've put more than 500GB online through BackBlaze. It's less than $50 year for unlimited online backup and a totally automated system.
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
I've been using SmugBrowser for uploads, and while it is better than it used to be, using it for videos is a big PITA (with it starting 6 simultaneous uploads of large videos, killing bandwidth for anything else, and needing to restart everything from beginning after any hick-up).
+1 on this, too.
GreyLeaf PhotoGraphy
For all the praises, SE still can't resume an interrupted video upload where it stopped (a limitation of SM API, according to the author -- but who cares). FTP can.
And SE is still an external application, closed source, developed by a 3rd party, that needs full access to your SM account. Not to mention it costs extra $$$.
I'm sure it would be way, way eaiser for Smugmug to fix resumable video uploads using their current protocols than to implement ftp from scratch. If they are really serious about video uploads, they should support incremental, restartable uploads - that should be a no brainer any time you're asking customers to upload files of 100's of MBs that can take hours to upload.
Samir said he was happy to pay for a better uploading solution so $$ shouldn't be an issue here. Closed source. I don't understand that in this instance. Everything you use at Smugmug is closed source. If Smugmug drops something you're unhappy with or won't fix something you're unhappy with, there's nothing you can do about it. Any ftp client you used would be developed by a third party too. I'm not getting the resistance. Sure, I understand you have a fondness for ftp, but there are other solutions and they work fine. I, for one, would rather Smugmug put their resources into other issues that will benefit far more users.
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question