Options

FTP, gripes...

1246

Comments

  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited February 22, 2010
    timk519 wrote:
    Maybe SM should just buy SE's code and integrate it in with their platform.
    I don't think it would be possible to integrate since SE is a stand-alone application and the simple uploader is Java. I guess they could re-write SE in Java, but SE also has quirks like the delay between sending each file. That alone wasted so much of my bandwidth that I had to stop using it (even besides the crashing issues).

    I would love to see a complete java app written that does multi-gallery uploading and queuing, but debugging that would be a nightmare with all the various platforms. But why re-invent the wheel? Use FTP for the transport mechanism and re-work the back-end to work with ftp. Problem solved forever. FTP isn't going anywhere and is only going to get better with time. thumb.gif
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited March 7, 2010
    It's time to bump this again to garner more support for FTP. It's 5am and I've wasted at least hour trying to get the simple uploader to queue a batch of 10gb. I could have been sleeping.

    Why does the simple uploader read every byte of every video file? What a waste of time. You know how long 10gb takes to read off usb drives? Even at 20mb/sec, that's over 8 minutes! To just queue the files, not even start the upload.

    FTP would have bought me over an hour of extra sleep tonight. And I have to be back up and out the door in 5 hours. The extra hour of sleep would have been good for me.

    Maybe I should just use this thread to log the number of hours lost due to the lack of FTP. Starting now, that's 1 hour.
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    NikolaiNikolai Registered Users Posts: 19,035 Major grins
    edited March 7, 2010
    SamirD wrote:
    ... SE also has quirks like the delay between sending each file...

    Samir,
    I think SE has changed a bit since we last spoke, you may want to take a look at the recent version.
    It now sports (up to) 7 threads, each with its own connection. I don't know if it would saturate 3 fast connections, but on my 20/20 Fios there is no delays between files anymore.
    Just FYI...
    Nikolai
    "May the f/stop be with you!"
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited March 7, 2010
    Nikolai wrote:
    Samir,
    I think SE has changed a bit since we last spoke, you may want to take a look at the recent version.
    It now sports (up to) 7 threads, each with its own connection. I don't know if it would saturate 3 fast connections, but on my 20/20 Fios there is no delays between files anymore.
    Just FYI...
    Nikolai
    Thank you very much Nikolai! thumb.gif I'll take a look at it asap.
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    afarberafarber Registered Users Posts: 267 Major grins
    edited March 10, 2010
    SamirD wrote:
    Thank you very much Nikolai! thumb.gif I'll take a look at it asap.

    Doesn't help us Mac users :cry
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited March 11, 2010
    afarber wrote:
    Doesn't help us Mac users :cry
    Oh, I still believe in FTP to be the ultimate solution. Especially if I still have the problem of having to wait 30-45mins for SE to parse my list of galleries. That became a dealbreaker for me in the past.
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    NikolaiNikolai Registered Users Posts: 19,035 Major grins
    edited March 12, 2010
    SamirD wrote:
    Especially if I still have the problem of having to wait 30-45mins for SE to parse my list of galleries. That became a dealbreaker for me in the past.
    That *was* in the past. I switched from RPC-XML to JSON, the speed improvement is at least in an order of magnitude, if not more. deal.gif
    Besides, if you use something like the Dropbox as a centralized (even if "cloudized":-) location, you would never have to sync albums/categories ever again. Think about it...
    "May the f/stop be with you!"
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited March 12, 2010
    Nikolai wrote:
    That *was* in the past. I switched from RPC-XML to JSON, the speed improvement is at least in an order of magnitude, if not more. deal.gif
    Besides, if you use something like the Dropbox as a centralized (even if "cloudized":-) location, you would never have to sync albums/categories ever again. Think about it...
    Cool. Great to hear. I still haven't had a chance to take a look at it, but I'm looking forward to it even more. Dropbox sounds neat. I did some quick reading on it, and while it's not for me, it sure beats a lot of the other Internet based backup concepts out there.
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited April 13, 2010
    Another bump for FTP. While the newer versions of SE and the new SmugLoader are great advances in uploading capability, FTP still wins. I've lost about 24hrs this weekend to uploading issues. I've had the same data on my Exposure Manager account now for 2 days thanks to their FTP implementation. clap.gif

    Vote now to make FTP uploading a reality:
    http://smugmug.uservoice.com/forums/17723-smugmug/suggestions/294159-ftp-uploading
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    echo465echo465 Registered Users Posts: 9 Beginner grinner
    edited May 2, 2010
    +3
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited June 13, 2010
    Another hour lost to the lack of FTP. And I need sleep! Grrr...

    Vote now to make FTP uploading a reality:
    http://smugmug.uservoice.com/forums/17723-smugmug/suggestions/294159-ftp-uploading
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited August 15, 2010
    And the new uploader foils my uploads again.

    Let's make FTP a reality. Vote here:
    http://smugmug.uservoice.com/forums/17723-smugmug/suggestions/294159-ftp-uploading
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited September 20, 2010
    I lost some more time to the current uploader. :cry FTP is the real solution. Vote for it here:
    http://smugmug.uservoice.com/forums/17723-smugmug/suggestions/294159-ftp-uploading
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    NikolaiNikolai Registered Users Posts: 19,035 Major grins
    edited September 20, 2010
    SamirD wrote: »
    I lost some more time to the current uploader. :cry FTP is the real solution. Vote for it here:
    http://smugmug.uservoice.com/forums/17723-smugmug/suggestions/294159-ftp-uploading

    Samir, I'm still wondering how come S*E's 100 threads are not enough for you to saturate your bandwidth? I'm on 35/35 Mbps Fios, and it only takes about 4..6, maaaybe 10 threads to saturate the upload. So I'm running 25 total and 10 uploading. With all 100 threads allowing to upload we're talking about saturating ~0.5 Gbps upload channel. Not unheard of, but not ubiquitous (or cheap:-) either. What kinda of connection are you using, if you don't mind my asking? headscratch.gif
    "May the f/stop be with you!"
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited September 20, 2010
    Nikolai wrote: »
    Samir, I'm still wondering how come S*E's 100 threads are not enough for you to saturate your bandwidth? I'm on 35/35 Mbps Fios, and it only takes about 4..6, maaaybe 10 threads to saturate the upload. So I'm running 25 total and 10 uploading. With all 100 threads allowing to upload we're talking about saturating ~0.5 Gbps upload channel. Not unheard of, but not ubiquitous (or cheap:-) either. What kinda of connection are you using, if you don't mind my asking? headscratch.gif
    I think I tried it, but I still ran into the same problem I ran into from day one--it will just stop uploading on its own after a while. ne_nau.gif And this happens on several different computers with different versions of xp (home/pro/sp1/sp2/sp3) and on even different Internet connection speeds and types. I just can't figure it out, and unfortunately don't have time to try to diagnose it. I'm behind schedule almost 48hrs now. :cry

    I have 3 Knology 25/5 connections connected via a Cisco rv016 for a total of 75/15 available to the network. My current configuration is to use multiple sessions of the simple uploader and manually break up the upload to saturate the bandwidth. I need 3-4 instances to saturate one pipe, so I usually need 9-12 uploader sessions to saturate everything. The problem is, once the uploader was changed, I now have a problem with it randomly crashing on large uploads. I'm thinking this is due to videos, but I can't be sure.

    FTP would eliminate these problems.
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    darryldarryl Registered Users Posts: 997 Major grins
    edited September 20, 2010
    While nobody likes having to pay for Star*Explorer (except Nik and his satisfied users), I'm happy the option is there. I'm not uploading anywhere near the volume that you guys are, but Samir, it seems that if you added up all the hours struggling with failed downloads with SmugMug's uploaders, maybe it would be worth your time to sit down with Nik and work-out exactly what your issues are with S*E. I'm sure he has debug versions of the tool, logging, etc.

    Because as somebody who's used to waiting a while for features SmugMug has committed to implementing, I wouldn't hold my breath waiting for one that they have expressly said they're not interested in at this time. :-/
  • Options
    darryldarryl Registered Users Posts: 997 Major grins
    edited September 20, 2010
    SamirD wrote: »
    <-- more than 2000 galleries right now. :D

    And I had to move a bunch of them from one sub-category to another. Do you know how much of a pain that is with 50 or so galleries? And then deleting another 50 galleries, well, I wrote some html to help me do that.

    Pros run into these situations. And sometimes we need pro tools. I usually don't touch more that a few galleries in a work session, but for my archive project, I was working on hundreds.

    Oh man, bulk moving galleries in SmugMug. What a nightmare.

    A few years back a company named Nitrodesk had an app that let you do this. It was pretty clunky and slow, but it does handle the particular task of moving galleries into different categories or subcategories pretty well.

    Unfortunately they discontinued development and presumably support of the product, but I was lucky enough to get a license before they killed it.

    You can still download the installer here: http://www.tucows.com/preview/520908

    Don't know if it'll work without a license though.
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited September 20, 2010
    darryl wrote: »
    While nobody likes having to pay for Star*Explorer (except Nik and his satisfied users), I'm happy the option is there. I'm not uploading anywhere near the volume that you guys are, but Samir, it seems that if you added up all the hours struggling with failed downloads with SmugMug's uploaders, maybe it would be worth your time to sit down with Nik and work-out exactly what your issues are with S*E. I'm sure he has debug versions of the tool, logging, etc.

    Because as somebody who's used to waiting a while for features SmugMug has committed to implementing, I wouldn't hold my breath waiting for one that they have expressly said they're not interested in at this time. :-/
    I actually didn't mind paying for my copy of SE. Nikolai has made a great product and it seems to work well for everyone but me. Figures, as that's the story of my life. :cry

    The problem with all those added up hours is that they always come at the wrong time. The only reason I'm even posting here is because I have to wait for my uploading computer to reboot, again. This is the 5th reboot in the last few hours because the uploader keeps hanging on multiple sessions. :cry I can't get these final 2gb uploaded in any fast way, and I can't wait 3-4hrs to have them up because then I have to go through the gallery and spend an hour or so making sure everything got there okay and finding and eliminating the dupes that happen usually with this many uploader crashes. Nikolai and I worked extensively during my active use of SE, but we never could fix the problem. :cry

    And that's what the votes are for. I thought it was only me that thought FTP was the way to go. It's almost in the top 10 requested things and the average vote is over 2. This means it's important to the users of SM. And SM is very aware of what their users want. I've seen things they said weren't on the drawing board get released so I'm going to continue to push for FTP. thumb.gif
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited September 20, 2010
    I remember Nitrodesk. Thank you for the link. If I ever run into the situation again, I'll know what to look for. thumb.gif
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    darryldarryl Registered Users Posts: 997 Major grins
    edited September 20, 2010
    SamirD wrote: »
    I actually didn't mind paying for my copy of SE. Nikolai has made a great product and it seems to work well for everyone but me. Figures, as that's the story of my life. :cry

    The problem with all those added up hours is that they always come at the wrong time. The only reason I'm even posting here is because I have to wait for my uploading computer to reboot, again. This is the 5th reboot in the last few hours because the uploader keeps hanging on multiple sessions. :cry I can't get these final 2gb uploaded in any fast way, and I can't wait 3-4hrs to have them up because then I have to go through the gallery and spend an hour or so making sure everything got there okay and finding and eliminating the dupes that happen usually with this many uploader crashes. Nikolai and I worked extensively during my active use of SE, but we never could fix the problem. :cry

    And that's what the votes are for. I thought it was only me that thought FTP was the way to go. It's almost in the top 10 requested things and the average vote is over 2. This means it's important to the users of SM. And SM is very aware of what their users want. I've seen things they said weren't on the drawing board get released so I'm going to continue to push for FTP. thumb.gif

    Got it. Are you still somehow bonding 3 cable modems? What are you using to do that? I wonder if that might be part of the issue.
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited September 20, 2010
    darryl wrote: »
    Got it. Are you still somehow bonding 3 cable modems? What are you using to do that? I wonder if that might be part of the issue.
    It's actually not bonding, but smart load balancing. The router is a Cisco/Linksys rv016. It's a multi-wan router capable of handling up to 7 different wan connections of varying speeds. I have three of these wan connections connected to the three 25/5 cable modems.

    The LAN side is pretty much standard Linksys configuration. Each system gets a single IP like normal. The way it works is when a WAN request is made, the router checks the load on the various WAN connections in its pool and sends the request to the least busy one. When uploading, each file initiates its own upload session, so if one upload session is started and if it saturates the entire bandwidth of a WAN connection, then a second upload session would automatically use the next WAN connection, and so on until they are all saturated. And once a file is finished that connection is free, so it drops back into the pool. Unfortunately, it takes about 3-4 upload sessions to completely saturate the upload bandwidth, so I need 9-12 sessions to get the 15mb upload speeds.

    I've used this configuration for better part of a decade now and with all the various uploaders available on SM and other places. I have no problems getting 15mb to Exposure Manager--they use FTP. I simply use an ftp client, set the threads to 12, start the upload, and walk away. I can only dream of this on SM. I would have saved at least 10hrs in the last 24. I was even waking up every two hours last night to check on the upload, and found it stopped at 4am.
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    Art ScottArt Scott Registered Users Posts: 8,959 Major grins
    edited September 20, 2010
    SamirD wrote: »
    I remember Nitrodesk. Thank you for the link. If I ever run into the situation again, I'll know what to look for. thumb.gif

    Nitro Desk is gone..............there may still be an old link that you can upload with...but I had ND and loved it and then they quit supporting it and mine just sorta died one day during an upload......

    I would love to see us able to use FTP...for one thing it is useful for so many things...such as when I outscource a wedding I can just send it to my processor...and they just send back.........
    "Genuine Fractals was, is and will always be the best solution for enlarging digital photos." ....Vincent Versace ... ... COPYRIGHT YOUR WORK ONLINE ... ... My Website

  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited September 20, 2010
    Art Scott wrote: »
    I would love to see us able to use FTP...
    Vote here! thumb.gif
    http://smugmug.uservoice.com/forums/17723-smugmug/suggestions/294159-ftp-uploading
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    NikolaiNikolai Registered Users Posts: 19,035 Major grins
    edited September 20, 2010
    SamirD wrote: »
    I think I tried it, but I still ran into the same problem I ran into from day one--it will just stop uploading on its own after a while. ne_nau.gif And this happens on several different computers with different versions of xp (home/pro/sp1/sp2/sp3) and on even different Internet connection speeds and types. I just can't figure it out, and unfortunately don't have time to try to diagnose it. I'm behind schedule almost 48hrs now. :cry

    I have 3 Knology 25/5 connections connected via a Cisco rv016 for a total of 75/15 available to the network. My current configuration is to use multiple sessions of the simple uploader and manually break up the upload to saturate the bandwidth. I need 3-4 instances to saturate one pipe, so I usually need 9-12 uploader sessions to saturate everything. The problem is, once the uploader was changed, I now have a problem with it randomly crashing on large uploads. I'm thinking this is due to videos, but I can't be sure.

    FTP would eliminate these problems.

    Samir, thank you.
    Well, as I have mentioned above, 15Mbps is really not that much for S*E to deal with. And I have dealt with 30-60-100Gb uploads as a part of my Mass Upload service. I'm not saying I never encounter any hiccups, but I had it run for days non-stop.
    Once you're out of your 48hrs snag, give me buzz me and maybe I'll be able to get you on RDP/VNC and debug your issue. Seriously, you're wasting days of your life over what can be a 30 minute find-and-fix session...rolleyes1.gif
    "May the f/stop be with you!"
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited September 20, 2010
    Nikolai wrote: »
    Samir, thank you.
    Well, as I have mentioned above, 15Mbps is really not that much for S*E to deal with. And I have dealt with 30-60-100Gb uploads as a part of my Mass Upload service. I'm not saying I never encounter any hiccups, but I had it run for days non-stop.
    Once you're out of your 48hrs snag, give me buzz me and maybe I'll be able to get you on RDP/VNC and debug your issue. Seriously, you're wasting days of your life over what can be a 30 minute find-and-fix session...rolleyes1.gif
    Thank you for the reply Nikolai. Thank you for the offer on diagnosing what's wrong. We will have to do that, but only after I get out of the peak season, which ends in Oct-Nov. Then I'll have plenty of time to go through the various scenarios.

    And you're right, it's literally hours and days of my life. I'm still waiting on some uploads from earlier today. I would've loved to have dropped them into SE Sunday night and not have spent the entire last 24hrs babysitting uploads. thumb.gif
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    timk519timk519 Registered Users Posts: 831 Major grins
    edited September 20, 2010
    SamirD wrote: »
    And this happens on several different computers with different versions of xp (home/pro/sp1/sp2/sp3) and on even different Internet connection speeds and types.
    Have you tried a Windows 7 system?

    The other issue could be your router - I've got the RV04 Dual WAN router, and it's thrown some strange 'policy violation' errors where it would block packets when I've run some bit-torrents - perhaps that's contribution to the issue. I'd suggest setting it up to email policy violations somewhere - particularly if it's configured to fend off DOS attacks, because if it's doing things like this:

    RGFW-IN: BLOCK-RULES (TCP 192.168.1.102:2997->74.14.119.206:28524 on ixp0)
    Sat Aug 28 13:32:12 2010
    RGFW-IN: BLOCK-RULES (TCP 192.75.213.4:1667->209.183.136.63:53 on ppp0)
    Sat Aug 28 13:32:00 2010

    that may be where your problem lies.
    • Save $5 off your first year's SmugMug image hosting with coupon code hccesQbqNBJbc
  • Options
    Art ScottArt Scott Registered Users Posts: 8,959 Major grins
    edited September 21, 2010
    "Genuine Fractals was, is and will always be the best solution for enlarging digital photos." ....Vincent Versace ... ... COPYRIGHT YOUR WORK ONLINE ... ... My Website

  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited September 21, 2010
    timk519 wrote: »
    Have you tried a Windows 7 system?

    The other issue could be your router - I've got the RV04 Dual WAN router, and it's thrown some strange 'policy violation' errors where it would block packets when I've run some bit-torrents - perhaps that's contribution to the issue. I'd suggest setting it up to email policy violations somewhere - particularly if it's configured to fend off DOS attacks, because if it's doing things like this:

    RGFW-IN: BLOCK-RULES (TCP 192.168.1.102:2997->74.14.119.206:28524 on ixp0)
    Sat Aug 28 13:32:12 2010
    RGFW-IN: BLOCK-RULES (TCP 192.75.213.4:1667->209.183.136.63:53 on ppp0)
    Sat Aug 28 13:32:00 2010

    that may be where your problem lies.
    Nope, don't have a win7 system and can't afford one just to upload photos.

    I've worked extensively with the rv016 and rv082. I've had to turn off most of the firewall functions due to my ISP hammering over 100 packets/second of ICMP requests after they switched to the new Arris c4 on their carrier side equipment. This would cause my rv016 to disable a pipe almost all the time thinking that it's under attack. I found a solution to this issue by putting other routers in front of the rv016 on two of the WAN connections. One one is connected directly as I use it for gateway to gateway vpn and had to leave it this way. There's no functional difference between the directly connected one and the other two, other than if you look at the stats, the direct one is receiving a lot of packets that the other two aren't.

    The crashing problem I run into seems to be more of a system/js issue as it seems this newer version of the uploader chokes on certain file sizes or number of files. And I don't have time to diagnose it. There's photos to process.
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
  • Options
    NikolaiNikolai Registered Users Posts: 19,035 Major grins
    edited September 21, 2010
    SamirD wrote: »
    The crashing problem I run into seems to be more of a system/js issue as it seems this newer version of the uploader chokes on certain file sizes or number of files. And I don't have time to diagnose it. There's photos to process.

    It's called a viscious circle, my friend. deal.gif

    Think of a person who decided to cross the continental US from coast to coast, but since his car had a minor trouble instead of spending a day or two fixing it he went to proceed on foot... rolleyes1.gif

    Anyway, it's your life...ne_nau.gif Given my (outside, yet extensive) knowledge of how SM operates, FTP will not happen no matter how many votes you'd get. Just like S*E will not work no Macs natively. There are technical, business, and logistical barriers that prevent both from happening. You're fooling yourserlf and others if you think otherwise, and wasting day after day on what seems to be a minor technical problem... :cry
    How much does *your* time cost? Think of all the days you have already spent watching the uploads and add all those endless days and sleepless nights you will... I looks like it would cover upgrading a few machines to Window7x64 and maybe even upgrading your routers.. mwink.gif
    "May the f/stop be with you!"
  • Options
    SamirDSamirD Registered Users Posts: 3,474 Major grins
    edited September 21, 2010
    I do the best I can with the very meager resources that I have. People are constantly amazed that the work I do just being one person. I don't expect you or anyone else to understand unless they are put in my shoes.

    I find myself working on fires. Put one out, and try to resume what real work needs to get done, and then there's another fire. The uploader situation is one of those fires that is flaring up every so often, as this thread reflects, but it's not hot enough to give any priority on a permanent fix as of now--there's hotter fires I'm fighting. In the winter time, the fires will be less and I can hope to work with you on making SE work for me.

    I don't agree with you on FTP never being a reality. In the almost 5 years that I've been on SM, I've seen things I never thought would be a reality like video. And since I'm not the only one that likes FTP, I'll keep promoting what I think is the ultimate solution whenever I'm in between system reboots.
    Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
    Want faster uploading? Vote for FTP!
Sign In or Register to comment.