FTP, gripes...

olegos

I don't have time to read through the hundreds of pages of "Feature Requests", "Dealbreaker", etc. threads that might deal with this, and searching for ftp doesn't work ("The following words are either very common, too long, or too short and were not included in your search : ftp"), and as far as I'm concerned it should have been implemented ages ago... So I'm just starting a new thread.

Is FTP upload going to be implemented any time soon?

Frankly, SmugMug's refusal to use an open protocol designed for the purpose, much requested by the users, is very puzzling.

And don't tell me that it's difficult to implement. SM managed to implement uploads by e-mail -- a protocol not designed for large file transfers; doing the same with ftp should be trivial, as has been re-hashed in this forum multiple times.

[Now that AGAIN my SmugBrowser is broken, wondering AGAIN why I have to keep wasting my time on a problem that's had great solutions available for ages, that SM is refusing to make use of]

Andy

Hi, I used the Google Site search that I built into Dgrin, to search for FTP

Here's the link:
http://www.google.com/cse?q=ftp+upload&sa=Search&cx=015490029931808806336%3Awyeua6yybvs

This thread was posted by you back in March:
http://www.dgrin.com/showthread.php?t=94584

I'm sorry I have no new news to tell you about.

Andy

What uploader do you use? I routinely upload hundreds/thousands of photos every day, and I don't have any issues - I use the same uploaders and internets that you have available - nothing special...

Perhaps if you can tell us more about he issues you're having maybe we can point to some relief - we'd LOVE to help!

olegos

Andy, thanks for the search tip. Obviously I knew about the thread that I started, but I wanted to know if there's been any change since then. You've answered this question too.

So, what's the current answer for why FTP isn't implemented at SM? As far as I can tell, all technical reasons that were brought up in that thread have been countered.

Look, SM is wasting their own resources, and those of countless other people -- customers and developers that develop apps to do a fraction of what a good FTP client can do, wrt file transfers -- by refusing to support FTP. As you can see from your search, FTP is getting asked about all the time.

olegos

Andy wrote:

Perhaps if you can tell us more about he issues you're having maybe we can point to some relief - we'd LOVE to help!

I usually use SmugBrowser, but can't now: http://www.dgrin.com/showthread.php?t=47484&page=54

And even when it works, it isn't 100% doing what I want:
- When a long video upload is interrupted, it has to be restarted from the beginning,
- If it crashes or I need to reboot my PC, it's a pain to figure out what's been transfered and what hasn't,
- I'd like to be able to upload from command line. I have a kludgy solution for photos, but nothing at all for videos.
- SmugBrowser is BUGGY.

jfriend

olegos wrote:

I usually use SmugBrowser, but can't now: http://www.dgrin.com/showthread.php?t=47484&page=54

And even when it works, it isn't 100% doing what I want:
- When a long video upload is interrupted, it has to be restarted from the beginning,
- If it crashes or I need to reboot my PC, it's a pain to figure out what's been transfered and what hasn't,
- I'd like to be able to upload from command line. I have a kludgy solution for photos, but nothing at all for videos.
- SmugBrowser is BUGGY.

I use StarExplorer. I don't think it will restart a video upload in the middle, but it will reliably keep track of what it has and has not uploaded in a persistent queue that will even survive a computer reboot (the queue is stored on disk). It is not uncommon for me to upload 600 images over night and then find one or two still left in the queue that didn't upload successfully, sometimes because of network conditions or a hiccup on Smugmug's side and sometimes because there's something that needs to be addressed on the image itself (like it's too large). I never have to sort out which images uploaded successfully or not because StarExplorer records that for me automatically.

If I were to look at Smugmug's overall offering, I'd have to say that the single most glaring weakness in their core functionality is upload tools. The fact that they don't offer a reliable, persistent uploader that doesn't have the issues you mention is mind blowing. Uploading should be their bread and butter and something they are first class at. Instead it's quite a weak point. The only reason I don't regularly complain a lot is because I use a third party uploader (that I paid for) that solves some of these issues.

Andy

jfriend wrote:

Uploading should be their bread and butter and something they are first class at. Instead it's quite a weak point.

We have improved uploading a billion percent in 2008 - due largely to a ton of work by Baldy. The amount of issues we've had at the help desk and on Dgrin from people who are having trouble uploading has gone to a statistically low, low number. It's no longer the #1 issue on the help desk week after week. It's not even in the top 5 or 10 any more. That's how reliable and good and easy to use our uploading has become.

What you and Olegos are telling me though is that there's something missing for advanced users, I guess. But then again, I upload thousands of files per day and just use the Simple Uploader, or MacDaddy and have no trouble

Baldy

jfriend wrote:

If I were to look at Smugmug's overall offering, I'd have to say that the single most glaring weakness in their core functionality is upload tools. The fact that they don't offer a reliable, persistent uploader that doesn't have the issues you mention is mind blowing. Uploading should be their bread and butter and something they are first class at. Instead it's quite a weak point. The only reason I don't regularly complain a lot is because I use a third party uploader (that I paid for) that solves some of these issues.

Tell me if I'm wrong, but I think our strength and weakness is we offer many uploaders.

The strength is you usually get to choose one that fits your workflow—a plugin to Lightroom, for example. And a secondary strength is if one doesn't work for some reason, you can use another.

The weaknesses are not every uploader has retry and resume in the middle, depending on the developer, and if you get one that's buggy you assume they all are. Plus it's more expensive and harder for us to support so many.

The simple uploader, for example, does retry and resume many times, it's ActiveX for IE and Java for the rest so it's compatibility is unmatched, and that's why it's the standard uploader for Facebook, Mpix, etc.

We get quite a lot of customers who come to us from other prominent sites that have good uploaders but for some reason they don't work on a particular computer (interference with Norton or whatever), and then they have no choice, so they come to us because we do have alternatives.

While we love Nik and Star Explorer, we have to be careful about how we represent it, not as a panacea. It's a great uploader but like all other uploaders it doesn't guarantee success, just like MacDaddy doesn't on the Mac, which we spend a lot of resources developing ourselves.

jfriend

Andy wrote:

We have improved uploading a billion percent in 2008 - due largely to a ton of work by Baldy. The amount of issues we've had at the help desk and on Dgrin from people who are having trouble uploading has gone to a statistically low, low number. It's no longer the #1 issue on the help desk week after week. It's not even in the top 5 or 10 any more. That's how reliable and good and easy to use our uploading has become.

What you and Olegos are telling me though is that there's something missing for advanced users, I guess. But then again, I upload thousands of files per day and just use the Simple Uploader, or MacDaddy and have no trouble

Are these tools reliable when the connection is iffy or when Smugmug drops for 2 minutes in the middle of a large upload? That's when we know that the tools are really designed robustly. You could upload thousands of images successfully with a very poorly designed app if there were never any connection issues.

But, the real world of home internet connections is that there are often connection issues. My DSL connection has hiccups multiple times a day. I've seen Smugmug have issues on the upload side before too. When I start an 800 image upload before I go to bed and come back in the morning and find that it stopped with an error, didn't complete and doesn't tell me a thing about which files were successfully uploaded, that is not a robust tool and does not make me a happy camper. It's not a world class uploader.

Perhaps you don't realize it, but it is possible to reliably (and without user intervention) upload images even when the network conditions are going up and down. It takes a lot more code than just writing to a TCP socket and stopping when there's an error. It takes persistent queues, retry logic, restart logic and when it still can't get something to upload, it takes a UI that keeps that image in a list so the user can decide what to do with it without having to manually figure out which of the 800 images had issues. All of that becomes even more important when dealing with video files because starting over from the beginning is really painful.

Art Scott

You may want to give Nitro Desk a try.....but I do not upload video and am not usre if it will.................

Andy

jfriend wrote:

Are these tools reliable

Hey John - I'm just saying that we improved it a billion percent. A lot. It used to be the #1 issue that people wrote us about There are always folks with dropped connections and such - but I'm so glad these issues number in the very few / week. There's ALWAYS room to improve but you make it sound like we've been sitting on our hands and not paying attention to this stuff, when in fact, we have.

We love to be pushed and pushed hard, keep it up :jfriend

jfriend

Baldy wrote:

Tell me if I'm wrong, but I think our strength and weakness is we offer many uploaders.

The strength is you usually get to choose one that fits your workflow—a plugin to Lightroom, for example. And a secondary strength is if one doesn't work for some reason, you can use another.

The weaknesses are not every uploader has retry and resume in the middle, depending on the developer, and if you get one that's buggy you assume they all are. Plus it's more expensive and harder for us to support so many.

The simple uploader, for example, does retry and resume many times, it's ActiveX for IE and Java for the rest so it's compatibility is unmatched, and that's why it's the standard uploader for Facebook, Mpix, etc.

We get quite a lot of customers who come to us from other prominent sites who have good uploaders but for some reason they don't work on a particular computer (interference with Norton or whatever), and then they have no choice, so they come to us because we do have alternatives.

While we love Nik and Star Explorer, we have to be careful about how we represent it, not as a panacea. It's a great uploader but like all other uploaders it doesn't guarantee success, just like MacDaddy doesn't on the Mac, which we spend a lot of resources developing ourselves.

If offering and supporting many uploaders is keeping you from doing one world class one that works on Windows and is incredibly robust, then I'd question if that is really the right strategy.

And I think the OP has a good point. A lot of these issues of reliable upload have already been addressed in FTP servers and clients yet you aren't leveraging any of that on either the client or server side. And supporting a robust flavor of FTP would likely lead to more robust client options in the future since there's so much code floating around that's already been written to do that.

I'm not suggesting StarExplorer is god's gift to uploaders. But, it does do realiability really well with it's persistent queues and automatic retry logic and it lets me do one continuous unattended upload across many galleries which is something I often have to do for my sports teams. And, when an image just won't upload for some reason, it keeps it just sitting there in the queue so I can either manually try one more time or I can see exactly which image it is that has a problem.

The last Smugmug uploader I used just quit with an error and left me to go find the upload log in the control panel (something hardly any Smugmug users know about) to try to sort out what did and didn't get uploaded and why. Try that part way through an 800 image upload that stopped sometime in the middle of the night and was supposed to be done that morning, but now you have to somehow figure out which images didn't go. That was just horrendous to me. Maybe the current offerings are better than that, but that's what it was like when I left.

Baldy

jfriend wrote:

If offering and supporting many uploaders is keeping you from doing one world class one that works on Windows and is incredibly robust, then I'd question if that is really the right strategy.

We might be able to beat the Simple uploader for robustness, I'm not sure. Facebook and Mpix and a ton of other sites haven't been able to and it's why it's the only uploader they offer, but we're willing to try.

Where we may be able to beat it is with features. I just wouldn't hold my breath too hard because a lot of great companies have taken swings with this bat and were humbled, as we have been.

olegos

Baldy wrote:

Tell me if I'm wrong, but I think our strength and weakness is we offer many uploaders.

The strength is you usually get to choose one that fits your workflow—a plugin to Lightroom, for example. And a secondary strength is if one doesn't work for some reason, you can use another.

Baldy, the situation with uploaders is analogous to SM not supporting standard http and general purpose web browsers, but instead developing their own proprietary API, and a couple of applications using that API. Developers could develop their own apps for browsing, there'd be several to choose from, they could be customized to make viewing photos and galleries better, maybe even do a more secure login, some people would still be coming to your from all those other services supporting only http, etc. Do you think that's a winning strategy?

Why should I spend my time (and money) to investigate a number of proprietary apps that have no use outside of SM, that may or may not satisfy my needs, may or may not be supported in the future, etc., when I already have a tool (a number of them, actually) that I know how to use, that reliably does what I need, that's available on all possible platforms?

Andy, I totally agree with you that uploaders have improved tremendously. By the way, Filezilla did too in the last couple of years. IMO, you will never catch up with a mature FTP client as far as upload robustness, nor do I think you should be spending your time on that.

SamirD

jfriend wrote:

If I were to look at Smugmug's overall offering, I'd have to say that the single most glaring weakness in their core functionality is upload tools. The fact that they don't offer a reliable, persistent uploader that doesn't have the issues you mention is mind blowing. Uploading should be their bread and butter and something they are first class at. Instead it's quite a weak point. The only reason I don't regularly complain a lot is because I use a third party uploader (that I paid for) that solves some of these issues.

After reading this, I realized I spend more time on my uploading process than on my galleries.

FTP is an awesome way to transfer files. After all, it stands for File Transfer Protocol. The multi-thread upload capability of most FTP clients would blow away my current process. But I don't see it being implemented. I'm sure there's a good reason for this, but it does remove a very good tool.

I use the simple uploader under IE because it does multi-image uploads. This does a better job of saturating the bandwidth of three cable modems I have. But I usually have to set up at least two upload sessions to completely max out the bandwidth.

Then comes the verification process. After experiencing problems in the past, I verify that the upload was done successfully. If not, I spend the time to find out which ones uploaded and which ones did not. On a 400+ picture upload, this has taken me up to three hours to figure out and upload.

This process is much better than anything I've had in the past for my workflow. But if something more robust for uploading can be implemented, half of my work is eliminated.

If an image never makes it to SM, the user will never realize the full potential of SM. It all starts with uploading. This should not be a weak point in my opinion.

jfriend

SamirD wrote:

Then comes the verification process. After experiencing problems in the past, I verify that the upload was done successfully. If not, I spend the time to find out which ones uploaded and which ones did not. On a 400+ picture upload, this has taken me up to three hours to figure out and upload.

Samir, that's a nice illustration of one of the big weaknesses. Three hours to make sure all your images get to the right place when there was a hiccup in the process? That certainly illustrates a failure of the tools. And, the more you upload, the more likely you are to hit this just due to the time you spend uploading.

If you're a pro, those three hours could suck most or all of the profit out of a job right there or you miss a deadline because it didn't complete when it was supposed to. If you're a non-computer-savvy amateur, you probably don't recover properly and some images just never get to where they were supposed to be. If you're a computer-savvy amateur, you end up just dreading all the extra work to try to recover and you're a lot less likely to repeat that process again because it can be such a pain.

Andy

SamirD wrote:

After reading this, I realized I spend more time on my uploading process than on my galleries.

When you've uploadedd 125,080 photos, that's understandable

cabbey

sigh. It would be so awesome if the guys at Apple would just let folks use their development tools to build for windows like we know they do (itunes, safari, etc). Then we could offer a build of MacDaddy for you windows folks and everyone could live happily.

DrDavid

cabbey wrote:

sigh. It would be so awesome if the guys at Apple would just let folks use their development tools to build for windows like we know they do (itunes, safari, etc). Then we could offer a build of MacDaddy for you windows folks and everyone could live happily.

Like that would *ever* happen....

But, FTP would be a nice feature. Surely it couldn't be THAT hard to do, could it?

David

mlee

I really really like the idea of FTP over the other methods I've been using. I've had several problems with the simple uploader and finally broke down and bought SE - and it works decent enough.

I think FTP could work similarly to how it works at EM. After you upload something into your FTP account, it just shows up in your 'control panel' interface as a folder and that folder is then ready for you to process it however you want (i.e. add the photos to a gallery).

SamirD

Andy wrote:

When you've uploadedd 125,080 photos, that's understandable

That's all?

SamirD

You know, I just thought about the process I had to go through on some of those uploads getting to that 100k photo mark. Recently, I re-started a project to upload all of my archives to SM as a fail-safe backup. Originally when I started this project, I gave up because of the problems of having to find missing files in uploads. This was before I discovered the IE simple uploader was using multiple threads and the improvements to the process because of this. I was originally starting three different upload sessions on two different computers under firefox. But it would be so nice if I could just set up an upload and walk away.

I've estimated it will take about two weeks to upload my archive. And yet, it's been almost two years that have passed and I haven't finished the upload. Looking back at the time span, I'd say the upload issue has been fundamental in the delay of my project.

I've tried sendtosmugmug, and StarExplorer, as well as komodo drop. And at some point with each, I had to re-upload something that fell through the cracks. (Komodo drop did really well, and I have yet to experiment with it fully, so there may be hope.)

I really don't think bad uploads or re-uploads are avoidable. But, I do think a smarter upload tool would help with large batches, and mixed batches of video/photo. (I have videos that are coming out 200-300mb from my camera that I just set to upload and then sleep. It's not too fun to wake up in the morning to find they didn't make it for some reason.)

All in all, SM is great. I really can't complain. I just want to see the best get better.

SamirD

With the advent of the newest uploading tool available for SM, my mind came back to this subject of FTP. I would like to see a FTP to SM 'bridge' software of some sort that could be run and then any FTP client used. This would allow the user to pick their interface and possibly bring some more robustness to the whole process.

SamirD

After spending almost 6 hours with every imaginable uploader for SM, I am offering $500 to a developer who makes the FTP to SM bridge a reality.

DrDavid

SamirD wrote:

After spending almost 6 hours with every imaginable uploader for SM, I am offering $500 to a developer who makes the FTP to SM bridge a reality.

Livedrive ( http://tr.im/livedrive ) has a link to SmugMug. You move all your files to the livedrive L:\ drive on your computer, and you can just drag them into SmugMug from the website (or not). It's quite slick

David

afarber

Just another "me toooo!" for ftp uploads!

HamSmuggler

afarber wrote:

Just another "me toooo!" for ftp uploads!

FTP would help me too, but there are other solutions like SendTo Smugmug. Quite honestly, the Lightroom Plugin has largely ameliorated my need for an FTP uploader since everything I have is RAW. The plugin allows you to export your edited RAW files directly to Smugmug via a temporary folder of JPG's. The temporary JPGs are deleted once the export is complete, which saves a ton of space and hassle. You may want to look into it if you're using Lightroom here: http://regex.info/blog/lightroom-goodies/smugmug.

HamSmuggler

SamirD wrote:

After spending almost 6 hours with every imaginable uploader for SM, I am offering $500 to a developer who makes the FTP to SM bridge a reality.

It would probably be very difficult to do this without collaboration with the smugmug in-house developers. I wonder if you could put that $$ towards a Mozy account or other dedicated online storage company who probably supports FTP (and other file types besides JPG, such as RAW). That's what I do. I think Smugmug is connected with Amazon S3 which is another file storage solution similar to Mozy (though it is more expensive, which is why I went with Mozy).

SamirD

DrDavid wrote:

Livedrive ( http://tr.im/livedrive ) has a link to SmugMug. You move all your files to the livedrive L:\ drive on your computer, and you can just drag them into SmugMug from the website (or not). It's quite slick

David

I'm not sure I'm completely following you here. Is there a feature in Livedrive that allows the files to be directly transferred to SM?

SamirD

HamSmuggler wrote:

It would probably be very difficult to do this without collaboration with the smugmug in-house developers.

Not really. There is the SM API. That's what all the developers of the 3rd party uploads have used.

HamSmuggler wrote:

I wonder if you could put that $$ towards a Mozy account or other dedicated online storage company who probably supports FTP (and other file types besides JPG, such as RAW). That's what I do. I think Smugmug is connected with Amazon S3 which is another file storage solution similar to Mozy (though it is more expensive, which is why I went with Mozy).

I'm not worried about storage of files, I just want them to get into my galleries faster, securely, and without the hassle of quirky uploaders.

It seems like this company developed one of the retired uploaders (drag and drop) http://www.thinfile.com/. I'm going to contact them about the development of a FTP to SM bridge.

SamirD

Email sent to the company:

I am a customer of Smugmug.com. It looks like they actually use one of your uploaders for their 'drag and drop' uploader. While this does work, my preference is to upload files using FTP. Smugmug does not offer FTP nor is working on offering it as an upload option.

So what I am looking for is a piece of software that is a 'bridge' between FTP and Smugmug. This way, I can use a standard FTP client to interface and upload files to Smugmug. I'm not the only one interested in this software, as expressed by the replies in this Smugmug support discussion thread:
http://www.dgrin.com/showthread.php?t=114560

Smugmug has a publically available API which other developers have used to create 3rd party upload tools. One of the most popular is StarExplorer (www.starexplorer.com). As you can see, at over 700 copies at $15 each, this is a sizable amount of money. A reliable FTP to SM bridge software would be the most timeless and reliable uploader ever made, and the only one to support FTP. The only time you would need to update it is when the Smugmug API changed. The user interface and everything else would be dependent on the maker of the FTP client software. And since the FTP protocol isn't rapidly changing, there's little chance of required updates due to this.

If this project interests you, my ideas on how it would work can be found in the thread below. You are welcome to use any of these ideas or implementations without any compensation to me. I simply want to save time by using the end product.
http://www.dgrin.com/showthread.php?t=136363

Thank you and I look forward to your reply.