ftp please please please

olegos

After wasting time AGAIN trying to upload a bunch of photos, running into SmugBrowser bugs (will explain below), etc., I repeat for the N-th time my plea of FTP support.

PLEASE. It's a trivial thing for you to implement, and it solves sooo many problems right away:
- uploading over slow and unreliable connections
- resumes
- command line uploads
- wide range of clients for all possible platforms
etc.

Implementing it the way uploads by mail are implemented is trivial. I'm sure running an FTP server is well within the technical expertise of the SM team, and dropping all uploaded photos (and videos!) into "FTP Uploads" gallery, the way the mail server does, is nothing.

Please at least acknowledge this request (unlike the previous N-1 times). [Really, at this point I'm almost expecting "Won't do for religious reasons" -- this seems like such a no-brainer]


About the aforementioned SmugBrowser bug. So I return from vacation, and decide to upload photos directly from SD card. Stick the card into a card reader in my new all-in-one printer, open SmugBrowser, and start dropping photos into it. Mostly in order. But, they get added to the list in a somewhat shuffled order, and on the top of that, SMB decides to skip some files as it uploads them -- so 15 upload, 5 get skipped, 10 more upload, 7 skipped... Getting to the end of the list, it just stops, and no selecting "Upload" or "Upload All" can get the skipped photos uploaded. So I get to get a pen out, write down the numbers of the skipped ones (quite a list, exacerbated by the scrambling that was done when the photos were added), restart SMB, carefully select the files on the list in Explorer, and drop them into SMB again. All this after I just played with a new version of FileZilla (for transferring some files at work) which is fantastic.

Baldy

Hi Olegos,

Ouch, we're really sorry to disappoint on this score. We have a lot of issues with ftp, unfortunately...

One is that everyone who uses it has to have an account on the ftp server, which creates a nest of administrative and security issues. We don't want an avenue of potential exploits to your SmugMug account.

Also, uploaders programmed to our api can get sophisticated and real-time feedback from our servers for more advanced error checking (file too big? wrong format?) and retry. We can check each file as it arrives and take action on it immediately depending on what we see and feed it back to the sending application.

I hope this helps.

Thanks,
Chris

DrDavid

You need to check out StarExplorer.com; it's IMHO, the best uploader Smugmug has. It *should* be standard issue on SM. It does cost a few bucks, but considering what it does, it's really not a lot.

Top benefit: it can handle THOUSANDS of images at a time, can upload to multiple galleries in one go, and VERIFIES the upload. ALSO, if a file fails, it keeps a record and can retry it. Computer crashes? No worries--it writes a log to the computer as it goes. Just rerun the program and it picks up where it left off.

Highly recommended!

David

jfriend

DrDavid wrote:

You need to check out StarExplorer.com; it's IMHO, the best uploader Smugmug has. It *should* be standard issue on SM. It does cost a few bucks, but considering what it does, it's really not a lot.

Top benefit: it can handle THOUSANDS of images at a time, can upload to multiple galleries in one go, and VERIFIES the upload. ALSO, if a file fails, it keeps a record and can retry it. Computer crashes? No worries--it writes a log to the computer as it goes. Just rerun the program and it picks up where it left off.

Highly recommended!

David

Yep, this is all true about StarExplorer. I regularly upload hundreds of images (school and sporting events). I've had power failures, crashes, internet outages Smugmug hiccups and Smugmug maintenance all during long uploads and StarExplorer has always kept track of what was successfully uploaded and what needs a retry. It retries automatically (you set the number of retries) and if the outage is something longer, it just keeps the items in a permanent upload queue waiting for you to hit upload to finish off the last ones. You NEVER have to wonder which ones got successfully uploaded and which ones you have to find and do again.

In addition to that, I use it to create 20 galleries at once, all with the same settings.

Baldy

DrDavid wrote:

You need to check out StarExplorer.com; it's IMHO, the best uploader Smugmug has. It *should* be standard issue on SM.

We just changed the help landing pages so that if you have a pro account, you get a different set of frequently asked questions. One of the top questions is, "Do you have a power uploader for pros?" and it links to StarExplorer.

We oughta do the same on the uploader page.

DrDavid

Baldy wrote:

We just changed the help landing pages so that if you have a pro account, you get a different set of frequently asked questions. One of the top questions is, "Do you have a power uploader for pros?" and it links to StarExplorer.

We oughta do the same on the uploader page.

That's cool S*E really is the bees-knees

David

Souwalker

Baldy wrote:

We just changed the help landing pages so that if you have a pro account, you get a different set of frequently asked questions. One of the top questions is, "Do you have a power uploader for pros?" and it links to StarExplorer.

We oughta do the same on the uploader page.

Si is StsrExplorer only for Pr accout holders or standard as well?
Pat

DrDavid

Souwalker wrote:

Si is StsrExplorer only for Pr accout holders or standard as well?
Pat

Check out his site: http://www.StarExplorer.com

There's three different versions to choose from. All have the same amazing upload functions

David

Nikolai

Souwalker wrote:

So is StsrExplorer only for Pr accout holders or standard as well?
Pat

Pat,
Any Star*Explorer (S*E) edition (including free 30 day trial) works with any level of Smugmug (SM) subscription. But just like SM subscription levels get more features as you advance, so do S*E editions. However, all the upload features are included into the most basic one.
David, John, Chris: thank you!

PeterLyons

WAAAH! It doesn't work on Mac. :cry

Nikolai

PeterLyons wrote:

WAAAH! It doesn't work on Mac. :cry

Sorry, it was never supposed to... At least directly. However, if you happen to have Parallels, VMWare or BootCamp, you should be fine!

DrDavid

Nikolai wrote:

Sorry, it was never supposed to... At least directly. However, if you happen to have Parallels, VMWare or BootCamp, you should be fine!

You know.. You'd probably make a lot of money if you could convert it to work on mac too... Just my two cents worth..

David

Nikolai

DrDavid wrote:

You know.. You'd probably make a lot of money if you could convert it to work on mac too... Just my two cents worth..

David

David, I really appreciate the sentiment , but I honestly doubt it.
It's purely ROI/business issue.
It will take me years to learn Mac OS / Mac developer tools to the degree I know Windows/WinAPI/Delphi/Visual Studio. I'm not even talking about all the $$$ I'd have to spend on mac hard- and software. So even if ALL SM mac users would miraculously join in for an S*E shopping spree, I would hardly break even...
I'm sure there are plenty talented Mac developers with enough free time on their hands who can make S*E-like tool in a heart beat... Or at least in several months if they put their back into it like I did...

cmason

S*E works great in VMWare Fusion, I simply fire it up, point it to the folder of images ON MY MAC, and it uploads dutifully.

rainforest1155

PeterLyons wrote:

WAAAH! It doesn't work on Mac. :cry

Speaking of which. I just tried to run S*E on Mac using Wine. I have to click 2 times on 'continue application' and then I get that:

(click on it for a bigger version)

The actual screen shows up, but clicking continue again doesn't bring me any further. Top right shows what DLL's I tried to add to help getting around the problem, but the actual error message in the bottom left doesn't seem to have changed after adding all these DLL's.

Any comments, Nik? Would it somehow be possible to get this working in Wine or are chances zero?

Sebastian

Nikolai

rainforest1155 wrote:

Speaking of which. I just tried to run S*E on Mac using Wine. I have to click 2 times on 'continue application' and then I get that:

(click on it for a bigger version)

The actual screen shows up, but clicking continue again doesn't bring me any further. Top right shows what DLL's I tried to add to help getting around the problem, but the actual error message in the bottom left doesn't seem to have changed after adding all these DLL's.

Any comments, Nik? Would it somehow be possible to get this working in Wine or are chances zero?

Sebastian

Sebastian,
sorry about that.
I don't know about Wine (hence I didn' mention it:-). I suppose it's a mac-written emulator for Windows. I also suppose it's pretty old and as such doesn't provide 100% WinAPI support. In any case, please send the the bug report next time, so I can at least look at the call stack and the environment context.

DavidTO

Nikolai wrote:

Sebastian,
sorry about that.
I don't know about Wine (hence I didn' mention it:-). I suppose it's a mac-written emulator for Windows. I also suppose it's pretty old and as such doesn't provide 100% WinAPI support. In any case, please send the the bug report next time, so I can at least look at the call stack and the environment context.

Nik, you may remember that I was trying out a demo of a commercial product to get your app to run on my Mac without installing windows. Anyway, that was a commercial version of wine.

I never got it to work, either.

Nikolai

DavidTO wrote:

Nik, you may remember that I was trying out a demo of a commercial product to get your app to run on my Mac without installing windows. Anyway, that was a commercial version of wine.

I never got it to work, either.

Hey David, welcome back!
I guess S*E needs Windows in some way.. In any case, without a call stack (which is a part of S*E bug report) I can't do much about it...

DavidTO

Nikolai wrote:

Hey David, welcome back!
I guess S*E needs Windows in some way.. In any case, without a call stack (which is a part of S*E bug report) I can't do much about it...

Well, just to be clear, I couldn't get that app to work with anything, so I wouldn't necessarily blame it on some aspect of S*E, more just the whole WINE thing, or more likely, me.

Nikolai

DavidTO wrote:

Well, just to be clear, I couldn't get that app to work with anything, so I wouldn't necessarily blame it on some aspect of S*E, more just the whole WINE thing, or more likely, me.

IIRC, Wine was first designed for like Window 3.x, which at the time had only about 1,000+ API entries (if not 600+). I think nowadays if someone wants to emulate any modern version of Windows they pretty much have to have...Windows. Too many idiosynchrasies to try to comply with, and *any* single point of failure will, essntially, kill the whole thing. Like, in Sebastian's snapshot it looks like it's about some security token - definitely stuff 2-3 layers deeper than I am working with....

olegos

Baldy wrote:

Hi Olegos,

Ouch, we're really sorry to disappoint on this score. We have a lot of issues with ftp, unfortunately...

One is that everyone who uses it has to have an account on the ftp server, which creates a nest of administrative and security issues. We don't want an avenue of potential exploits to your SmugMug account.

Also, uploaders programmed to our api can get sophisticated and real-time feedback from our servers for more advanced error checking (file too big? wrong format?) and retry. We can check each file as it arrives and take action on it immediately depending on what we see and feed it back to the sending application.

You figured out how to handle authentication with the mail server, didn't you. Doing something similar with ftp server should be no big deal. Hell, even anonymous ftp upload to user/mail-pwd/ directory, and a cron job on the server taking the files from there and passing to whatever mechanism takes it from the mail server should work (not saying that's what you should do, knowing the internals of your system I'm sure you can come up with a better way...) Also... The beauty of using a standard thing like FTP is that your problems most likely have already been solved. Take a look at Pure-FTPd for example. Supports custom authentication methods among many other useful features:

Custom authentication methods can easily be added. Pure-FTPd supports external authentication modules, and writing a new backend can be as simple as a few lines of shell script.

Error handling -- just accept everything, and if you have a problem with it, log the error in the upload log.

Security -- I have a much bigger problem with installing some third-party closed-source relatively not widely used executable on my computer, than with whatever you're likely to do. I mean, I trust you guys to get authentication within your own back-end right much more than I trust an executable from some web site not to come with a key logger or a virus, which by the way would compromise not just SM account but a lot more. (No offence to S*E developers, I'm sure they're doing a great job, I'm just speaking in general terms.)

By the way. After fighting for 2 days with SmugBrowser to upload my photos, and getting through maybe half of them, I put FileZilla to work to ftp the rest to a well-connected Linux host, and the upload was done completely unattendedly in about 4 hours. I then spent about 20 min hacking FileUpload to make it work on that host, then with a shell script uploaded the photos to SmugMug (does anyone know of a command-line uploading tool that can handle videos?). Can you see why I'd like to be able to just FTP into SM directly?

Nikolai

olegos wrote:

Security -- I have a much bigger problem with installing some third-party closed-source relatively not widely used executable on my computer, than with whatever you're likely to do. I mean, I trust you guys to get authentication within your own back-end right much more than I trust an executable from some web site not to come with a key logger or a virus, which by the way would compromise not just SM account but a lot more. (No offence to S*E developers, I'm sure they're doing a great job, I'm just speaking in general terms.)

No matter what local client you are going to use you will have to provide the password to it in one way or another. S*E authentication module was developed according to SM guidelines (and with a fairly close communication with Don, you can check the history right here at Dgrin). Password is not stored or logged and it's only sent via https (good luck hacking that one;-) - unlike FTP, which sends it barely encoded, if at all (yes, I'm aware of FTPS, but not all ftp clients support it, and it's even more hassle to support it on the server side); or a "shell script", which would store it in a plain text file. So much for security...

As to the installing - it's pretty much downloading an SFX rar and unpacking it.

FWIW, S*E has been on the market for 3+ years already, several months longer than Pure-FTP Never had any security complains (except that it's hard to hack ).

DrDavid

Nikolai wrote:

FWIW, S*E has been on the market for 3+ years already, several months longer than Pure-FTP Never had any security complains (except that it's hard to hack ).

Yeah Nik.. No kidding.. I couldn't pirate it by making a fake key--and had to actually pay for it!

David

Nikolai

DrDavid wrote:

Yeah Nik.. No kidding.. I couldn't pirate it by making a fake key--and had to actually pay for it!

David

:-)
It's my favorite security model. Costs me next to nothing, breaking is hard, and if I get a hint of a break-in - all the existing copies are disabled the next day or so, and I simply update the key, making a "kool hacka'" to start from sratch all over again...

chuckinsocal

Earlier this evening I tripped over a PS plug in to upload to Flickr directly from PS. Maybe womeone could write one for SmugMug? Just a thought, that's all.

Nikolai

chuckinsocal wrote:

Earlier this evening I tripped over a PS plug in to upload to Flickr directly from PS. Maybe womeone could write one for SmugMug? Just a thought, that's all.

There is one already. Check SM help. It's not much better than a standard SM uploader, though (only IMHO).
I mean, *any* uploader is OK for a dozen files. The problems usually begin when you start counting in hundreds and thousands...

devbobo

olegos wrote:

Security -- I have a much bigger problem with installing some third-party closed-source relatively not widely used executable on my computer, than with whatever you're likely to do. I mean, I trust you guys to get authentication within your own back-end right much more than I trust an executable from some web site not to come with a key logger or a virus, which by the way would compromise not just SM account but a lot more. (No offence to S*E developers, I'm sure they're doing a great job, I'm just speaking in general terms.)

olegos, we have just implemented OAuth (not officially annouced yet, while we work on documentation), which we are hoping that 3rd party developers will embrace. It is a little be more cumbursome for developers initially but provides more security (and piece of mind) for users.

olegos

Nikolai wrote:

No matter what local client you are going to use you will have to provide the password to it in one way or another. S*E authentication module was developed according to SM guidelines (and with a fairly close communication with Don, you can check the history right here at Dgrin).

As I said, I'm not accusing you of anything, and much much more likely than not it's not an issue (currently anyway). But it pretty much just has to be blind trust to you and your security practices. I'm not very comfortable with that (again, not you personally, but closed-source relatively obscure software in general; and paid software even more so, as it needs to include some code to prevent cracking and copying -- witness Sony with their rootkits (I realize they weren't just protecting the software, but still)). I always prefer open-source generic solutions to proprietary closed source ones.

FWIW, S*E has been on the market for 3+ years already, several months longer than Pure-FTP.

But Pure-FTPd is open-source.

olegos

Just to clarify, I'm not saying that security concerns are a reason for anyone not to use S*E. If it was clear to me that S*E would do everything I need, I would probably give it a try (because I've seen enough good testimonials, and public statements by the author that haven't been challenged -- otherwise if it were just a list of features on the web site, I'd skip anyway).

I was just replying to Baldy where he suggests that security is a reason for SM not to implement FTP, while in the same post endorsing use of third-party closed-source application. To me, this results in a net decrease of security for average user.

Nikolai

Speaking about ftp.. this is just in, from my most recent customer:

I learned about Star Explorer on the forum at SmugMug, where I was hoping to learn that FTP was an option for uploading photos -- no joy on that, but
Star Explorer actually turns out to be even better than FTP, with the right click and Send to SmugMug function. I've been uploading like crazy all
weekend, and this program has taken the headache out of it. I was able to select several hundred photos at a shot and just walk away for the day --
any photos that had a problem uploading stayed in the window, and I was easily able to reinitiate the upload for them, no worries with figuring out
which photos were missing and having to relocate them. I am overjoyed to have found SmugMug, and having Star Explorer for uploading makes the whole thing completely perfect -- thanks so much!

:ivar

Other than that - Oleg, I understand your concerns, yet I have a very different viewpoint on the whole opensource issue (although that leads us way off this topic, so I'm not gonna go there...:-).

olegos

Nikolai wrote:

Speaking about ftp.. this is just in, from my most recent customer: ...

So when a long video fails to upload, and you restart, does it start from the beginning again, or from where it stopped?

Some not too long time ago, with my usually speedy cable connection, there were very periodic interruptions of ~30-40 seconds, breaking connections with lots of traffic (low traffic connections, such as ssh, would survive). It was annoying, but not so much that I would bother calling a technician to come in and troubleshoot it. In that situation, a large file upload would never complete if it had to restart from the beginning every time. But as most places where I would upload (or download, for that matter) large files support ftp (or sftp) with restarts, SmugMug being a notable exception, it wasn't a big problem.