The part about this tale that still confuses me is why, if it was about metadata, the export, import-jpg, publish worked. Unless when that was done you deleted metadata in the export? ONe would think even if this was some kind of invisible character inline, that the export would still contain it if the caption was there at all (or conversely if export doesn't then the publish's internal export would not have it either).
It's a shame we can't figure out what did it, and give a reproducible case to Adobe. Or maybe even Smugmug -- if we knew exactly what did it, they might also see, understanding the internals better, what is happening. Maybe their own code is calling the Adobe code with invalid structure BECAUSE of this (not just a transitive passing of it through).
No, I did not delete the metadata in the export, import process.
Its even weirder...
Some of my problem photos had been previously published in other galleries with no problem. If I make minor changes, without copy and paste of the metatdata as I described earlier, I can continue to publish the photo in the original gallery, but not in the problem gallery or any other new gallery in which I add the photo.
I suspect it may have something to do with the formatting of the metadata in the caption and possible invisible characters. I am trying to ensure that the displayed caption does not overflow into a new line without my control and so am putting new lines in the caption.
Here are some cut and pastes from a couple problem photos from the caption box in the LR library:
Aquarelle sur Papier Arches Watercolor on Arches Paper
76 x 56 cm 30"x22"
and
Aquarelle sur Papier Arches/Watercolor on Arches Paper
56 x 76 cm / 22"x30"
and
Aquarelle sur Papier Strathmore Watercolor on Strathmore Paper
28,5 x22,5 cm/12x9"
But why, the remove metadata, publish and cut and paste process described in my previous posts (which would presumably retain the invisible characters) works and why its problem for some galleries and not others is a mystery.
Smugmug has been strangely silent recently, both on this forum and the Helpline. It would seem to me they should be able to investigate the metadata issue as they have a copy of my LR catalog and I can supply the problem photos if necessary.
Here is a summary of what I had communicated to SmugMug help:
_ the issue seems to revolve around images uploaded to multiple galleries
Initially several images failed when uploading from LR to Smugmug.
Photos that have been previously successfully uploaded in other galleries can be uploaded again in these other galleries. For example I made a minor change in a problem photo. I could upload it in the original gallery, but not the new (see above) or the test gallery. I also tried adding photos (not necessarily the problem photos) that have already been uploaded to other galleries. Same problem. So, effectively I am blocked. I cannot upload any photos in existing galleries to any new or other existing galleries if they have not previously been uploaded.
So there seems to be some issue about photos that have been previously uploaded being uploaded again to a different gallery.
I also tried to add a new photo – ie one that exists nowhere in any other gallery...that worked. Then I added it to a second gallery. That worked as well.
I also tried to remove a photo from an existing gallery, republish (that worked) and then add it back. That failed. So there seems to a memory of the fact that a photo as been added to a gallery, even when it is removed.
SmugMug has been silent on this thread since we have indicated that the problem seems to be associated with upload of certain metadata ( as opposed to hardware). There is something clearly amiss with the metadata that LR and/or SmugMug does not like and it would seem to me SmugMug should be in a position to at least inform us what the should be avoided in the metadata and whether there is a problem with an invisible character and what that character is. The solution mentioned above is painful at best. I have uploaded a sample problem photo to David at help at his request, but there has been no response after one week.
@frisson, I am not part of Smugmug, but am still curious about this... did you or anyone ever identify WHAT characters cause the issue, or find a photo from which it can be reproduced by a third party? (I'm happy to try if so). It's good that we understand it is metadata in caption, but it would be so much better to be able to say something like "if there is a tab in the metadata it dies".
No one (including SmugMug) has identified what characters cause the issue. In fact I do not think we can say definitively whether any characters cause the issue, since copy and paste from the LR caption field into the previously uploaded image without captions seems to solve the issue. Presumably copy and paste copies all characters including invisible characters.
I have already posted the caption above, which you can try, but I suspect it will upload just fine, given the sequence defined above. I think to really check someone needs to try with a photo that includes metadata and edits, which required the catalog. Both have been uploaded to SmugMug.
Another mystery is why all the problem photos are jpg and raw files were converted and uploaded just fine.
@frisson said:
Presumably copy and paste copies all characters including invisible characters.
I don't know if that's true or not, the programs involved (LR in this case) get control over how they interpret the data pasted in.
I have already posted the caption above, which you can try, but I suspect it will upload just fine, given the sequence defined above. I think to really check someone needs to try with a photo that includes metadata and edits, which required the catalog. Both have been uploaded to SmugMug.
Not necessarily, but if someone with a still-failing image wants to try, do this. Actually two possibilities.
One is to export the failing image (alone) as a catalog. This should give you a tiny, separate catalog and that one image (I can't recall where it physically puts the image, or if it leaves it in place, but that will be obvious after export). Open that new catalog, set up publishing, see what happens. If it fails, one could share a tiny catalog and one image.
Another related approach is to save-metadata, then go to that folder on disk and you will see two files (e.g. XXX.NEF and XXX.XMP, this presumes it was an original camera raw). Copy these two elsewhere. Create a brand new catalog and import them. They come in with the edits preserved and the metadata preserved (at least to the extent that the process of writing and reading it can -- if it is some kind of garbage character this may be the equivalent of the copy/paste). Then set up publishing and publish and see what happens. A nice side effect of this approach is the XMP file is just a text file, and you can look at the data with a binary editor to see if the caption contains anything unusual. But my guess is this will NOT work, just as copy/paste fixes it.
A final approach is to get someone who knows SQL and have them look in the catalog and see the actual contents, in binary, of the caption. Sadly to have a third party do this you would need to share the catalog with them (though not the images) and the catalog contains a lot of private information including credentials to smugmug, so this is a bit more problematic.
Again, I don't work for Smugmug or Adobe, I just am a software developer and debugging these things intrigues me.
Sorry, I've been following along but since Ferguson has been doing such an excellent job asking great questions to help narrow things down, I've been waiting to see if any further insights would be made (full details on why David and I have been distracted below, if you care). If it gets to the point where y'all have identified all you can, and need David or an Adobe engineer to review files, we can try to facilitate that.
Some additional details / context:
I wish we could devote our full attention to every bug that we find but unfortunately we have to prioritize issues the impact a larger number of people. Because the number of occurrences of this is so small, we've made the tough call to keep our attention focused on other projects. Luckily it sounds like we've found a workaround for this, should it occur again in the future, it appears there's a way to recover (although cumbersome). That's another reason why we've been fairly silent lately.
I do my absolute best to provide as much help as I can on here, but I currently serve 2 main roles here at SmugMug: I manage our entire Product Team and serve as the Product Manager for our apps (which includes Lightroom, and thus my interest in this topic). Since I'm also a passionate photographer, I don't get much sleep (up late editing photos!). David and I have been heads down on 2 other big projects (for example, we're about to launch Search in our iOS app) and have been trying to stay focused on those. David is also about to head out for a 2-week holiday and we wanted to make sure he completes his project before he leaves, so we've sadly had to take our attention away from troubleshooting this issue. That also means it'll be a few weeks before David can look at any catalog files. If y'all have investigated further and found all the bits that can be found, I'll try to find some time for David to poke at the catalog files with the additional insight that you've uncovered.
I know that's not the answer you wanted to hear but I'm hoping that at least my honesty buys some time?
Comments
The part about this tale that still confuses me is why, if it was about metadata, the export, import-jpg, publish worked. Unless when that was done you deleted metadata in the export? ONe would think even if this was some kind of invisible character inline, that the export would still contain it if the caption was there at all (or conversely if export doesn't then the publish's internal export would not have it either).
It's a shame we can't figure out what did it, and give a reproducible case to Adobe. Or maybe even Smugmug -- if we knew exactly what did it, they might also see, understanding the internals better, what is happening. Maybe their own code is calling the Adobe code with invalid structure BECAUSE of this (not just a transitive passing of it through).
No, I did not delete the metadata in the export, import process.
Its even weirder...
Some of my problem photos had been previously published in other galleries with no problem. If I make minor changes, without copy and paste of the metatdata as I described earlier, I can continue to publish the photo in the original gallery, but not in the problem gallery or any other new gallery in which I add the photo.
I suspect it may have something to do with the formatting of the metadata in the caption and possible invisible characters. I am trying to ensure that the displayed caption does not overflow into a new line without my control and so am putting new lines in the caption.
Here are some cut and pastes from a couple problem photos from the caption box in the LR library:
Aquarelle sur Papier Arches Watercolor on Arches Paper
76 x 56 cm 30"x22"
and
Aquarelle sur Papier Arches/Watercolor on Arches Paper
56 x 76 cm / 22"x30"
and
Aquarelle sur Papier Strathmore Watercolor on Strathmore Paper
28,5 x22,5 cm/12x9"
But why, the remove metadata, publish and cut and paste process described in my previous posts (which would presumably retain the invisible characters) works and why its problem for some galleries and not others is a mystery.
Smugmug has been strangely silent recently, both on this forum and the Helpline. It would seem to me they should be able to investigate the metadata issue as they have a copy of my LR catalog and I can supply the problem photos if necessary.
Here is a summary of what I had communicated to SmugMug help:
_ the issue seems to revolve around images uploaded to multiple galleries
Initially several images failed when uploading from LR to Smugmug.
Photos that have been previously successfully uploaded in other galleries can be uploaded again in these other galleries. For example I made a minor change in a problem photo. I could upload it in the original gallery, but not the new (see above) or the test gallery. I also tried adding photos (not necessarily the problem photos) that have already been uploaded to other galleries. Same problem. So, effectively I am blocked. I cannot upload any photos in existing galleries to any new or other existing galleries if they have not previously been uploaded.
So there seems to be some issue about photos that have been previously uploaded being uploaded again to a different gallery.
I also tried to add a new photo – ie one that exists nowhere in any other gallery...that worked. Then I added it to a second gallery. That worked as well.
@leftquark,
SmugMug has been silent on this thread since we have indicated that the problem seems to be associated with upload of certain metadata ( as opposed to hardware). There is something clearly amiss with the metadata that LR and/or SmugMug does not like and it would seem to me SmugMug should be in a position to at least inform us what the should be avoided in the metadata and whether there is a problem with an invisible character and what that character is. The solution mentioned above is painful at best. I have uploaded a sample problem photo to David at help at his request, but there has been no response after one week.
@frisson, I am not part of Smugmug, but am still curious about this... did you or anyone ever identify WHAT characters cause the issue, or find a photo from which it can be reproduced by a third party? (I'm happy to try if so). It's good that we understand it is metadata in caption, but it would be so much better to be able to say something like "if there is a tab in the metadata it dies".
No one (including SmugMug) has identified what characters cause the issue. In fact I do not think we can say definitively whether any characters cause the issue, since copy and paste from the LR caption field into the previously uploaded image without captions seems to solve the issue. Presumably copy and paste copies all characters including invisible characters.
I have already posted the caption above, which you can try, but I suspect it will upload just fine, given the sequence defined above. I think to really check someone needs to try with a photo that includes metadata and edits, which required the catalog. Both have been uploaded to SmugMug.
Another mystery is why all the problem photos are jpg and raw files were converted and uploaded just fine.
I don't know if that's true or not, the programs involved (LR in this case) get control over how they interpret the data pasted in.
Not necessarily, but if someone with a still-failing image wants to try, do this. Actually two possibilities.
One is to export the failing image (alone) as a catalog. This should give you a tiny, separate catalog and that one image (I can't recall where it physically puts the image, or if it leaves it in place, but that will be obvious after export). Open that new catalog, set up publishing, see what happens. If it fails, one could share a tiny catalog and one image.
Another related approach is to save-metadata, then go to that folder on disk and you will see two files (e.g. XXX.NEF and XXX.XMP, this presumes it was an original camera raw). Copy these two elsewhere. Create a brand new catalog and import them. They come in with the edits preserved and the metadata preserved (at least to the extent that the process of writing and reading it can -- if it is some kind of garbage character this may be the equivalent of the copy/paste). Then set up publishing and publish and see what happens. A nice side effect of this approach is the XMP file is just a text file, and you can look at the data with a binary editor to see if the caption contains anything unusual. But my guess is this will NOT work, just as copy/paste fixes it.
A final approach is to get someone who knows SQL and have them look in the catalog and see the actual contents, in binary, of the caption. Sadly to have a third party do this you would need to share the catalog with them (though not the images) and the catalog contains a lot of private information including credentials to smugmug, so this is a bit more problematic.
Again, I don't work for Smugmug or Adobe, I just am a software developer and debugging these things intrigues me.
Sorry, I've been following along but since Ferguson has been doing such an excellent job asking great questions to help narrow things down, I've been waiting to see if any further insights would be made (full details on why David and I have been distracted below, if you care). If it gets to the point where y'all have identified all you can, and need David or an Adobe engineer to review files, we can try to facilitate that.
Some additional details / context:
I wish we could devote our full attention to every bug that we find but unfortunately we have to prioritize issues the impact a larger number of people. Because the number of occurrences of this is so small, we've made the tough call to keep our attention focused on other projects. Luckily it sounds like we've found a workaround for this, should it occur again in the future, it appears there's a way to recover (although cumbersome). That's another reason why we've been fairly silent lately.
I do my absolute best to provide as much help as I can on here, but I currently serve 2 main roles here at SmugMug: I manage our entire Product Team and serve as the Product Manager for our apps (which includes Lightroom, and thus my interest in this topic). Since I'm also a passionate photographer, I don't get much sleep (up late editing photos!). David and I have been heads down on 2 other big projects (for example, we're about to launch Search in our iOS app) and have been trying to stay focused on those. David is also about to head out for a 2-week holiday and we wanted to make sure he completes his project before he leaves, so we've sadly had to take our attention away from troubleshooting this issue. That also means it'll be a few weeks before David can look at any catalog files. If y'all have investigated further and found all the bits that can be found, I'll try to find some time for David to poke at the catalog files with the additional insight that you've uncovered.
I know that's not the answer you wanted to hear but I'm hoping that at least my honesty buys some time?
Former SmugMug Product Team
aaron AT aaronmphotography DOT com
Website: http://www.aaronmphotography.com
My SmugMug CSS Customizations website: http://www.aaronmphotography.com/Customizations