You Want $500? Make me a FTP to SM bridge

SamirD · July 8, 2009

After the most recent change in SM's uploaders, my primary uploading station can't function. And I know this problem will get worse with time. I can't upgrade computers every time SM upgrades uploaders.

So here's my offering--develop a way to have a standard FTP client connect to a 'bridge' software that connects to SM for uploading, and I'll give you $500.

Make communications on duplicates, retries, etc relay to the FTP client so it can be handled the standard FTP way. Make it multi-threaded so when you have multi-wan connections or multiple t1 channels or the like, the user can set the number of threads to saturate bandwidth. Make it lean, like a single executable. Make it portable so no install is needed.

Basically it is a local FTP server on one side at 127.0.0.1, and a SM uploader on the other. It has to communicate the SM gallery and existing files to the FTP server to pass onto the FTP client. I'm sure there should be a way to do this because many of the other uploaders can enumerate an account tree. The user can then any FTP client they want, and the user gets the interface they want.

I'm no programmer or I'd do it myself. I think many of the other users that wish there was FTP for SM will like your product. So setup a blog page for it and let the user registrations pour in. I'll be your first. :thumb

Any questions, feel free to ask.

mbrady · July 14, 2009

For all of what you're asking, I don't think $500 is enough to get many programmer's attention. Especially ones that are actually capable of doing all that.

SamirD · July 14, 2009

It's not enough to make a product just for me. I think it is enough 'motivation' for someone to start a project like this. It would become the one uploader that always works across all platforms--that's powerful.

jfriend · July 15, 2009

SamirD wrote:

After the most recent change in SM's uploaders, my primary uploading station can't function. And I know this problem will get worse with time. I can't upgrade computers every time SM upgrades uploaders.

So here's my offering--develop a way to have a standard FTP client connect to a 'bridge' software that connects to SM for uploading, and I'll give you $500.

Make communications on duplicates, retries, etc relay to the FTP client so it can be handled the standard FTP way. Make it multi-threaded so when you have multi-wan connections or multiple t1 channels or the like, the user can set the number of threads to saturate bandwidth. Make it lean, like a single executable. Make it portable so no install is needed.

Basically it is a local FTP server on one side at 127.0.0.1, and a SM uploader on the other. It has to communicate the SM gallery and existing files to the FTP server to pass onto the FTP client. I'm sure there should be a way to do this because many of the other uploaders can enumerate an account tree. The user can then any FTP client they want, and the user gets the interface they want.

I'm no programmer or I'd do it myself. I think many of the other users that wish there was FTP for SM will like your product. So setup a blog page for it and let the user registrations pour in. I'll be your first.

Any questions, feel free to ask.

Seriously, have you tried StarExplorer? It's not FTP, but it is a lean mean, single executable and does pretty much everything else you asked for (I'm not sure about the saturating bandwidth with multiple network paths). It's very robust, supports uploads to multiple galleries and costs way, way less than $500. I've uploaded more than 20GB with it.

http://www.starexplorer.com

FYI, an FTP gateway would require someone to host a server and then pay for all your upload bandwidth (twice, once when you ftp to them and a second time when they pass it on to Smugmug). When a whole bunch of direct to Smugmug APIs exist that would have none of these costs, it's hard to see how this would be a good business proposition for anyone.

SamirD · July 15, 2009

I bought SE a long time ago and stopped using it because it was taking forever to load--20-30 minutes. It was my uploader of choice until one day, I just tried the simple one while waiting for SE to load and just started using that. I used send to SM for a while and I've experimented with Komododrop.

I need to revisit SE again, but I don't have a clue where my registration info is.

Why would someone need to host an FTP server? All we need is something to take the incoming/outgoing API requests and basically 'translate' them to FTP locally. A FTP/SM applet could do that. Then someone just points their FTP client to the 127.0.0.1 loopback address.

jfriend · July 15, 2009

SamirD wrote:

I bought SE a long time ago and stopped using it because it was taking forever to load--20-30 minutes. It was my uploader of choice until one day, I just tried the simple one while waiting for SE to load and just started using that. I used send to SM for a while and I've experimented with Komododrop.

I need to revisit SE again, but I don't have a clue where my registration info is.

Why would someone need to host an FTP server? All we need is something to take the incoming/outgoing API requests and basically 'translate' them to FTP locally. A FTP/SM applet could do that. Then someone just points their FTP client to the 127.0.0.1 loopback address.

OK, I guess one could do it that way. You said this solution would instantly get multi-platform support which led me to believe you were talking about a service rather than a client-side app which is only multi-platform when someone ports it to every platform (same as all other solutions).

What you are asking for still seems odd to me. You want FTP, yet you're asking for a local client translator app that likely wouldn't preserve any of the benefits of FTP. Because other upload protocols don't work the same as FTP, this translator app would probably just have to cache the entire upload file locally, then just use Smugmug's API to upload it. So, at that point, you just have another client upload app without any UI to manage it. You're dreaming if you think you'd maintain all the benefits of FTP when there's a translator to another protocol in the way. Retransmissions, for example, wouldn't be done at the FTP level. You'd be at the mercy of how Smugmug's protocol works and how the client app that talks that protocol handles it. You'd have no feedback UI on how the actual upload was going and no way to control all the upload parameters. You can keep looking or asking for it, but it doesn't make sense to me.

Your $500 might be more usefully applied in a conversation with Nikolai if StarExplorer doesn't yet meet your needs.

Nikolai · July 15, 2009

John, thanks for the pointer!

Samir, S*E is a bit faster these days. In fact, a lot faster.
And if you lost your license, I'll gladly resend it, simply shoot me an email.

As to the FTP to SM bridge... I was actually thinking about it. Of course everything John just said about such a "translator" (bridge/gateway/etc) is very true. It will definitely be a tad slower (although not by much if done right), since it would go like this:
HDD --> FTP client --> local, network (LAN) or remote (WAN) FTP server (= the piece you want) --> { internet } --> SM API.
All these instead of
HDD --> local client (browser, S*E, etc) --> {internet} --> SM API.

The extra steps inevitably take some time to go through. Plus you'll be fighting FTP protocol (with its arbitrary folder nesting as opposed to SM very strict cat - _{- album hierarchy), which is a rarely a good idea.

Anyway, if this is what you really want to have - shoot me an email, we can discuss it.}

jtames · July 16, 2009

Try KomodoDrop
Try KomodoDrop

SamirD · July 16, 2009

jfriend wrote:

OK, I guess one could do it that way. You said this solution would instantly get multi-platform support which led me to believe you were talking about a service rather than a client-side app which is only multi-platform when someone ports it to every platform (same as all other solutions).

What you are asking for still seems odd to me. You want FTP, yet you're asking for a local client translator app that likely wouldn't preserve any of the benefits of FTP. Because other upload protocols don't work the same as FTP, this translator app would probably just have to cache the entire upload file locally, then just use Smugmug's API to upload it. So, at that point, you just have another client upload app without any UI to manage it. You're dreaming if you think you'd maintain all the benefits of FTP when there's a translator to another protocol in the way. Retransmissions, for example, wouldn't be done at the FTP level. You'd be at the mercy of how Smugmug's protocol works and how the client app that talks that protocol handles it. You'd have no feedback UI on how the actual upload was going and no way to control all the upload parameters. You can keep looking or asking for it, but it doesn't make sense to me.

Your $500 might be more usefully applied in a conversation with Nikolai if StarExplorer doesn't yet meet your needs.

Thank you for the feedback. I'm not a developer, so it's great to get some feedback from someone who is.

I think 'translator' is definitely the best description of what I'm looking for. Something that will translate on a one-by-one basis to FTP. Would it be possible for the API to translate errors to FTP vs handling them internally? I think that's the key to this whole program.

I thought about this as a solution as well--a command line uploader that uploads one file at a time, and that's all. Like wget in reverse. This would be very light and could be called from external UIs and wrappers.

SamirD · July 16, 2009

Nikolai wrote:

John, thanks for the pointer!
Samir, S*E is a bit faster these days. In fact, a lot faster.
And if you lost your license, I'll gladly resend it, simply shoot me an email.

As to the FTP to SM bridge... I was actually thinking about it. Of course everything John just said about such a "translator" (bridge/gateway/etc) is very true. It will definitely be a tad slower (although not by much if done right), since it would go like this:
HDD --> FTP client --> local, network (LAN) or remote (WAN) FTP server (= the piece you want) --> { internet } --> SM API.
All these instead of
HDD --> local client (browser, S*E, etc) --> {internet} --> SM API.

The extra steps inevitably take some time to go through. Plus you'll be fighting FTP protocol (with its arbitrary folder nesting as opposed to SM very strict cat - _{- album hierarchy), which is a rarely a good idea.

Anyway, if this is what you really want to have - shoot me an email, we can discuss it.}

Thank you for the feedback Nikolai. I'll shoot you an email about SE. I do need to give it a shot again after all these years. Then maybe I can just forget FTP, lol.

SamirD · July 16, 2009

jtames wrote:

Try KomodoDrop

Tried it. Failed.

It had real issues with the videos where it would have to read the whole file before transmitting it. This is also what the new 'simple' uploader does, but Komodo also had problems with file errors and retrys and the multi-thread didn't work all too well.

This was all on one test system, so I could try another. I liked it because it's a very light application and has multi-thread uploading.

darryl · July 17, 2009

SamirD wrote:

I thought about this as a solution as well--a command line uploader that uploads one file at a time, and that's all. Like wget in reverse. This would be very light and could be called from external UIs and wrappers.

Shell script: http://braindump.dk/tech/2008/02/10/updated-smugupsh-script/
Perl: http://jeremy.qux.net/projects/sm-fileupload/
Python: http://blog.fullvalence.com/2008/12/01/smugmug-uploader/
Java: http://sinopop.net/smugjcu/

mike.strock · July 17, 2009

SamirD wrote:

Tried it. Failed.

Samir - As others have said, Star*Explorer is the way to go. If it doesn't work, Nik can fix it to make it work, that much I'm sure of, it's a great program and I use it quite frequently.

Mike.

SamirD · July 18, 2009

darryl wrote:

Shell script: http://braindump.dk/tech/2008/02/10/updated-smugupsh-script/
Perl: http://jeremy.qux.net/projects/sm-fileupload/
Python: http://blog.fullvalence.com/2008/12/01/smugmug-uploader/
Java: http://sinopop.net/smugjcu/

Great links! Thank you.

iamback · August 2, 2009

Cross-platform!

darryl wrote:

Shell script: http://braindump.dk/tech/2008/02/10/updated-smugupsh-script/
Perl: http://jeremy.qux.net/projects/sm-fileupload/
Python: http://blog.fullvalence.com/2008/12/01/smugmug-uploader/
Java: http://sinopop.net/smugjcu/

Thanks! I love KomodoDrop, which is what I'm using most of the time, but I'm not spending all of my time in Windows any more, so cross-platform capability is very important for any tools I use!

SamirD · August 4, 2009

smugjcu is AWESOME!! I've just played with the interface so far, but it looks extremely promising.

You Want $500? Make me a FTP to SM bridge

Comments