The infamous Lightroom "wrong format()" error

kodachi614

Lightroom 3, Jeffrey Friedl's SmugMug plugin, ahhhhnd, you guessed it, getting intermittent "wrong format()" errors when trying to update photos. I'm finding that if I click Publish enough times, eventually it'll get to the end, but it's really depressing to have to manually babysit this thing, even for the only-a-few-hundred-photos I have online right now. As I roll more out... ouch.

Googling around, I see a number of old threads acknowledging a problem, then saying it's fixed, but not much recent... I dunno, is it just me? What _is_ the current state of the world with respect to this issue?

(For SmugMug's info: I'm a Power trial user. I'm likin' what I see of SmugMug, with this _very_ notable exception... )

More details:
Lightroom 3, MacOS X 10.6.4 on a MacBook Pro i5
yeah, I'm pushing JPG in sRGB
yeah, the plugin log indicates that the files are _way_ under 12MB
(yeah, I have enhanced logging turned on in the plugin)
yeah, new photos seem to work fine, it's just updating photos
yeah, I'm changing metadata (accidentally stripped the date taken at first! :P )
yeah, I tried changing the export resolution and republishing to see if it was really metadata-changing-with-no-change-in-the-file; didn't help
yeah, I'm seeing intermittent timeout errors too, but the plugin seems able to ride through those

Any information appreciated -- thanks guys!

kodachi614

Some more information
After awhile of clicking "Publish" and having it get one or two photos in before bailing, I went the Draconian route: deleted all the photos from Smugmug, told Lightroom to mark 'em all for republishing, and tried again. Ran two galleries in parallel with nary a hitch.

This would certainly tend to imply that there is still a significant bug around uploading. <cough>

I noticed a couple of other interesting things, though:

1. During the upload that's meant to be updating things, the my Mac is neither hammering its CPU nor hammering the network, implying that it's waiting for SmugMug(*1).
2. During the upload that's meant to be updating things, I get a lot of timeout errors.
3. During the upload with nothing on SmugMug, it goes much faster. No timeouts. Much more consistent use of the network -- generally it looks like the remote end is being as quick as one might expect.
4. Looking in the debug logs from Jeffrey's plugin, I see situations where it waited over a minute for a response before getting one.

Put all that together, read between a few lines, and I start feeling that there's some expensive thing that SmugMug does on receiving an image that runs asynchronously when adding a new image, but is forced to be synchronous when updating an image.(*2) Hey SmugMug, any comment? I'm resigning myself to deleting before republishing, but, ew. That just ain't right!




(*1: For the record, yes, I do indeed have the kind of experience in computing that this statement implies, please let's not get sidetracked by whether or not I can tell if my local systems are doing anything. )

(*2: My gut feel is that the expensive thing is MD5ing the image: on update, you'd probably want the MD5 in order to determine whether you have to regenerate scaled and sharpened images. But I can't think of any reason you'd treat an update any differently from an initial upload there – a properly authenticated client is telling you to replace the image either way; they shouldn't have to wait for you to decide what that entails.)

rccoleman

I'm getting the same thing repeatedly to the point that I, too, just delete the files from Smugmug and start over. It's really, really annoying and only happens during a "replace" operation. If there's something that Jeffrey needs to do, I'm sure that he'd be happy to help. His current position is that it's in Smugmug's camp, though.

For what it's worth, I made changes to about 10 files today and started a "publish" to both Zenfolio and to Smugmug at the same time using Jeffrey's plugins. The Zenfolio plugin pegged the network while it was uploading the files and completed successfully without any hiccups. The Smugmug upload took about four times as long, with one timeout message in the middle. Watching my router blink away, it looks like there are long stretches of time where nothing much is happening, consistent with kodachi614's observations. At least this time I didn't get any "Wrong Format()" messages that abort the upload, but it's clear to me who "wins" here.

Edit: I just edited a single photo and uploaded to both services. They were both neck-and-neck until I got the "finished" message for Zenfolio and Smugmug sat around for 30 seconds or so before giving me a timeout. It tried again to upload the whole file and then gave me the "Wrong Format()" message. Sigh.

I wonder if the long delays are in some way due to the "Smugmungous" image option that I have enabled. Could it be generating the larger image in preparation for replacing it during the upload process? It looks like Zenfolio takes the images as fast as my network can provide them.

Rob

Andy

rccoleman wrote: »

but it's clear to me who "wins" here.

DId you try the new SmugMug Publish in LR3?

rccoleman

Andy wrote: »

DId you try the new SmugMug Publish in LR3?

Yes, I have, and I apologize for the snarky remark. It was very frustrating last night.

In terms of uploading files for the first time, both Smugmug and Zenfolio are about the same speed (pretty zippy). When replacing files, either last night at home, or tonight at my office, Zenfolio maintains that same level of performance, but the Smugmug update *drags*. I'm not sure what's going on in the background, but it's taking *forever* to update the same 10 files on Smugmug using a fast corporate network. In the end, one of the Smugmug updates failed and two timed out (based on a dialog at the end of the "publish").

To put some numbers to it, I re-ran the test and timed it. Updating 10 files to Zenfolio took 4 minutes and completed successfully. For Smugmug, using the native Lightroom 3.2 Publish Service, it took 9 minutes and two of the images timed out.

The Smugmug gallery that I'm using to test is "Test" in my account, for what it's worth.

Rob

rccoleman

In another test today, 5 of the 10 images timed out using the native client in Lightroom 3.2 on a reliable, corporate network. Do you agree that this is an issue?

Thanks,
Rob

Andy

Thanks for the info, our developer is looking into some things with regard to the replace & upload feature.

e-Scape Photo

Have not been able to focus any time on this for a week or so and, on the advice of the smug support heroes I tried going back top the simple uploader. What I noticed was that the upload proceeded extremely slowly - taking most of an overnight to load some 50 images. I know my home ISP upload speed isn't great but even so.... and then I noticed going back into the upload log on my sumg site that it shows the images as having loaded in somewhere between 20 and 40-50s each - but the time stamp on each log entry is anything from 5-20 mins apart.... sounds to me like a queueing issue at smug??

I will write this all down in response to the smuggy heroes on my ticket number but wondered if anyone else had noticed this. Next stop will be to my ISP but LR to facebook integration works sweetly (yes I know the image sizes are a lot smaller but even so) so I am kind of suspecting something on the smug end... reinforced by my total failure to get the jfriedl plugin to work with wrong format errors

nzsnapper

SmuginProForAperture also has issues with replace

Andy wrote: »

Thanks for the info, our developer is looking into some things with regard to the replace & upload feature.

I have had a few of my users report this same error when performing a replace - upload always seems to be fine. Sometimes I see a timeout (I have increased the timeout value which may help) and sometimes the Wrong Format error.

It would be really nice to confirm that this is a SmugMug problem and not something that I am doing in the plugin.

Cheers
Richard

kodachi614

Andy wrote: »

DId you try the new SmugMug Publish in LR3?

Hi Andy, sorry for taking so long to answer here. I, too, wasn't able to devote much time to this for some time. At this point, though, I've upgraded to Lightroom 3.2 and tried the plugin that ships with Lightroom.

The short answer is that neither plugin does what it should.

Both will upload to an empty gallery. However, the ships-with-Lightroom plugin (does this have a name? I'm going to call it SwLR) seems to leave all the photos "Marked for Republishing" rather than "Published". Jeffrey Friedl's plugin (which I'll call JF) ends up with everything marked "Published", as one would expect.

Neither can update photos in situ. In general, JF gives timeouts and wrong format errors, where SwLR usually just fails silently (it occasionally logs timeouts too).

Your support ticket #105864 has a lot more. While the Support Heroes have been polite and professional, and are clearly trying to help, I have to confess that I'm getting frustrated because it seems to me that we're past the point where the developers need to be directly involved to sort this out.

kodachi614

kodachi614 wrote: »

...it seems to me that we're past the point where the developers need to be directly involved to sort this out.

Actually Lindy's latest reply may be starting down the right path. I'll let you know in the morning when the Draconian re-exports finish and I can make a better test.

Andy

kodachi614 wrote: »

because it seems to me that we're past the point where the developers need to be directly involved to sort this out.

And he is. I'll be sure he sees your update, thanks.

kodachi614

Andy wrote: »

And [the developer] is [involved]. I'll be sure he sees your update, thanks.

Great!

So the latest goes like this. Lindy the Support Hero pointed out:

If the images are in your library, you can replace them through the uploader. You would have needed to add them through it to begin with, though. It won't recognize photos that were uploaded from a different application. So, if they were uploaded using Freidl's plugin, you won't be able to replace them using the LR3.2 uploader.

I'd read about this, of course, when starting to set up SwLR. Didn't think much of since I wasn't trying to update photos that I'd updated from JF, or at least I didn't think I was. After Lindy pointed it out again, I thought harder about it and realized that I could come up with a couple of scenarios where the plugin might be getting confused about what it owned and what it didn't.

To rule that out I went the very Draconian route: I deleted all of my non-smart galleries from SmugMug, deleted all the galleries from SwLR, and then completely wiped the settings for JF, so it's not even in the picture any more. Then I set up a couple of galleries using SwLR and hit Publish.

Doing both galleries in parallel, I never got a fully-succesful publish. I hit Publish, it grinds away for awhile, and eventually pops up a dialog warning me of errors. Sometimes the errors include timeouts, sometimes just "couldn't upload", sometimes both. No further information seems to be available. When it's done, all the photos are still "Modified Photos to Re-Publish". Hit it again, exactly the same behavior.

I just tried doing only one of the two galleries, and aha, that actually got to the end with no errors, but all the photos are still "Modified Photos to Re-Publish." Repeating that gives me the same behavior. Next step will be deleting everything again, and combining the two galleries into one to see if the number of photos matters.

Obviously this whole thing raises several questions.

1. Is publishing multiple galleries simultaneously supposed to work?
2. Where is there any information about what fails when an upload fails?
3. Why are all the photos left as "modified" instead of "published"?

As it stands, I may have a way to at least start using the thing, by dumping everything into a single gallery on SmugMug, using Smart Galleries for people to look at, and suffering with Lightroom re-exporting several hundred images every time I want to add a new one. Definitely a step in the right direction, but equally definitely not where I want to end up.

Also realize that I'm an exceptional case here: I care nothing for deleting everything in SmugMug and starting over, at this point, because I'm new to SmugMug, Lightroom is my one true master image repository, and I haven't published gallery URLs to the world. So what process exactly is SmugMug expecting an established SmugMugger to follow to start using the SmugMug Lightroom plugin?

kodachi614

I've done many more tests over the weekend, including (at the suggestion of the Support Heroes) basically resetting the whole plugin and starting over from scratch.

At this point I have one gallery with 496 photos. The very first time I tried to publish, two photos had a local error (the underlying image had been moved and not relinked) so that failed. Publishing never succeeded after that. Trying to republish was just godawful slow, and invariably failed with indeterminate errors -- mostly "upload timed out" or "upload failed", but one "incomplete file received." Publishing twice in a row yielded disjoint sets of error files, and the plugin gives no useful information about these errors.

After fixing the local errors, I deleted the gallery - again - before recreating it and publishing again. It ran straight through, no errors at all, and left every single one of those 496 photos in "Modified Photos to Re-Publish". I don't see any point in trying it again until I hear something substantive from SmugMug's developers. I have never, under any circumstances, seen the plugin you guys are shipping actually work. Full stop, end of story. (Yes, the images do get to SmugMug, but the point of the Lightroom plugin is that it should end up with Lightroom in a correct state too. This one ends up in a state where you can endlessly publish and never finish, with each attempt taking hours and throwing random errors while doing utterly pointless work.)

Andy, in candor, all of this experience leaves me feeling that SmugMug has a serious unaddressed problem with updating images, that the Lightroom plugin you guys ship is really beta quality, and that the wonderfully-responsive Support Heroes just aren't getting backstopped by anyone in development. These are big disappointments. What can I do to help get them fixed? At the least I'd expect that development would be really interested in getting extra info from a 100% reproducible failure case actually happening in the field...

kodachi614

Whoa.
OK, so I owe SmugMug a big apology for part of this.

I decided to try publishing to the hard drive and using the SmugMug uploader. Imagine my surprise when publishing to my hard drive resulted in every photo marked "Modified Photos to Re-Publish." After some looking, DGrin's "The SmugMug and Lightroom 3 Publish Services Thread" (http://www.dgrin.com/showthread.php?t=176963&page=8) led me to try unchecking "Write keywords as Lightroom hierarchy" – and it worked, both to my hard drive and to SmugMug.

So Andy and the rest of SmugMug: my apologies for blaming you there. That's clearly not your bug.

On the other hand, it is definitely something that the Support Heroes need to be trained up on. I'm going to write to them as well, and then move to "The SmugMug and Lightroom 3 Publish Services Thread" for further discussion of the abysmal SmugMug update performance, since that thread seems more active.

End result:

1. "Write keywords as Lightroom hierarchy" considered harmful.
2. I remain convinced that SmugMug has a serious problem with updating previously-published content.
3. As long as Lightroom doesn't think that all my photos always need republishing, I can probably make things work.

Erick L

I keep getting the same "wrong format()" error for one particular file. I've tried three uploaders with no luck. I've cleared cache, reboot and made a new jpeg file with a new name and it just won't accept it. It's under the file size limit.

Andy

Erick L wrote: »

I keep getting the same "wrong format()" error for one particular file. I've tried three uploaders with no luck. I've cleared cache, reboot and made a new jpeg file with a new name and it just won't accept it. It's under the file size limit.

Hi Erick, can you put it here http://dropbox.yousendit.com/SmugMug and put ATTN: Andy ? I'll have a look.

brianb

I'm seeing similar problems as kodacahi was in his original post (JF plugin). I reduced the number of photos I had on smugmug to about 14,000, before it would finally sync (about 26 hours), but now as I go through and delete & modify images, its completely hit or miss as to if I get the "wrong format" error or not (I tried various things like removing router, wifi, etc; no change). Given the number of photos, I'd obviously prefer not to delete everything and start over, but I wouldn't mind testing out ideas with a couple new/existing galleries.

Is there any additional info I can provide or tests to run that might help the people looking into it? I try to run through some of the things he described in his post to see if its exactly the same as well (I'm in IT, so this type of stuff doesn't phase me).

Thanks,
Brian

brianb

A quick update - I had about 25 images that needed republished (with JF smugmug plugin). Straight "republish" seems to be at least 75% failure rate (per image). I'd be lucky if after 3 or 4 tries it was able to upload one. I copied all of those into another collection, removed them from the publish service collection (so they'd be deleted). Publish, successfully deleted them from smugmug. I then re-added the same photos back into the publish service collection, published, and they all uploaded successfully, no errors at all. So its definitely something with replacing existing photos.

Bicicletapc

Great! I am looking at LR over ACR and now I am faced with the issue of wrong format.

Whats the status of this problem?

Thanks
Paul