Sort by Date Taken Not Working
Darter02
Registered Users Posts: 947 Major grins
I uploaded some new images for a client today. I keep their galleries organized by Date Taken, newest first. They do NOT appear that way once I hit "done." Instead they are totally random. Please fix this. It's been a few weeks. I just haven't had time to post about it before today.
Thanks,
Thanks,
0
Comments
customize > gallery settings > appearance
My Website index | My Blog
My Website index | My Blog
And please confirm it is not a smart gallery -- others have reported problems with date-taken in smart galleries.
So the correct dates are there according to the organizer but not according to the actual displayed images.
Yeah, I had that also, and I can tell you where mine came from.
There is an EXIF field DateTimeOriginal, which comes from the camera, and is rarely changed by any software.
There is also an IPTC field called "Date Created". I have historically ignored this field, but (after a lot of digging) I found that I would periodically copy metadata from one shot to another after I coded things like title, map coordinates, keywords, etc. I did this copy in lightroom where it is fast and easy, as opposed to typing all the same things in again.
Well, when I did the copy it will not copy DateTimeOriginal, but it does copy Date Created if you do not turn that off explicitly. So I was copying one shot's Date Taken (in IPTC) all through the others.
On Smugmug this used to not matter, as best I can tell -- it used DateTimeOriginal (as, in my opinion, it should). But in a recent round of changes it APPEARS that they started taking the IPTC field in preference. Leftquark thinks otherwise and mentioned in another thread he will check into it when time permits.
But is it possible you, somewhere in your workflow, have done this?
Can you look at the actual photos (download the original if you need to) and see if the IPTC field Date Created is where this is coming from?
Bottom line is Smugmug changed their algorithm for figuring ou tthe Date Taken recently, looking in sequence in various places, and that could easily account for you seeing different behavior now when you do not think you changed anything.
SmugMug, PLEASE take a look at this and set it back to the way it used to work. I can't even begin to imagine redoing thousands of photos just to fix the date taken according to the IPTC field.
Well... you may want to anyway. I think Smugmug needs to fix this, but I also know I need to fix my files, as the issue may show up some day somewhere else. It's on my list to do, either via some SQL magic or when I get time I'll upgrade my Data Explorer plugin (one of JF's, but he insists on getting paid for each LR major version) and see if it will mass-fix it (I really just want to delete it). There's also a data transporter plugin floating around that might do it en mass.
Now not sure if mass fixes like this will trigger the images for re-upload or not.
But just saying -- yes, SM should look at this, but you may also want to look at some mass fixit programs, as the data is actually wrong (if indeed you have the same thing as I did). It might show up unexpectedly somewhere else one day.
Seriously, it WAS working here for years. Now it is not. I'd like to see it returned to working condition. That is not too much to ask.
I'm not mad. Just frustrated.
I'll figure out (if I can) how to fix mine, and update this. The nice thing about a computer is that, largely, number of things to fix does not matter if it can be done by a program. It's just computer time, and your computer is probably not working full time anyway.
But... have you checked that is the problem?
Mine was caused by copying metadata with the "Sync" function, not templates. While the template does have Date Created available in it, if you created them by hand, it seems very unlikely you would have entered something in there.
Check some of your photos that are out of sequence and in lightroom in the metadata panel put up "IPTC" as the choice, and scroll down to "Image" and "Date Created". Make sure it's the wrong date showing on Smugmug before assuming my problem is also yours. You may have yet a different issue with their recent changes.
I'm sure we're not the only one to have this happen. Just wait as more folks begin to update old galleries only to find they're now all messed up. We can't be the ONLY TWO photographers who use File Info exif templates...
JF's Data Explorer plugin has a special feature for comparing dates to see if they match -- unfortunately it includes date modified in the comparison, and date modified is set by many of the transfer programs. It also includes Date Digitized instead of the IPTC Date Created. - Won't work.
John Beardsworth's ListView will let you pull all the data to excel, but won't let you update the Date Created in the write-back features.
His Search and Replace actually can solve the problem -- sort of. It will let you unconditionally copy all the data from Date Time Original into the IPTC Date Time field. That will fix the issue I think, but has an issue (more in a second). It's $23 Euro.
LR Transporter (from Arctic Whiteness) will do the same as the search and replace above, also unconditionally. It's donationware (minimum 3.50 Euro)
The problem with both of these is that the format for the IPTC date/time field is more complex than the EXIF DateTimeOriginal - it can include the time zone, and I've got a ton of shots with different formats, e.g. 03/01/2016 17:50:20 PM vs 2016-03-01T17:50:20 vs 2016-03-01T17:50:20-04:00 (the latter including the time zone) and sometimes with fractional seconds. So I am not sure if just blindly copying the DateTimeOriginal into the field is really legitimate.
Just to make it a bit more complicated, the Lightroom IPTC field will show these more complex dates, but if you actually try to type a value into it, it will only accept a date -- I can't find a format which works where it does not remove the time portion of the field entirely. Which is a bit weird, as I think it's Adobe that is putting it in in this format in some cases, if you import with Lightroom.
After-the-fact-edit: If you enter the field as YYYY-MM-DDTHH:MM:SS.ss-HH:MM (where the last portions are optional) it will take, you cannot use a MM/DD/YYYY HH:MM:SS PM format which is what the capture date/time comes in as from these programs.
And... if you make a change and they are already equal, it will mark the image to be updated (but to be fair, it updates only the metadata, so it goes really fast -- but even though it updates only metadata on Smugmug and is fast it does NOT update Date Created because, well, they have a "feature" that replacements will not update Date/times).
I think, maybe, you could use Listview to pull a complete list of mismatched fields, then filter in excel, and then upload via Transporter. That might let you correct only specific cases, and I guess might also let you handle time zones and such to be more targetted.
So... this is just plain a mess.
And... you can also use Lightroom to just remove the field. As best I can tell you cannot just blank it out, you have to first change it to something, then change it to blank. But that will remove the value entirely, which appears to fix it on Smugmug if it is being uploaded new, but not on a replace, because... well, that feature.
What I'd like to know, and maybe someone around here knows -- what is the RIGHT way to fix this. Is it simple enough to just remove the field entirely from an image, and let Date Time Original be the real field?
Note that I am asking a more broad question that Smugmug -- it may be that Smugmug will "fix" this by giving DateTimeOriginal preference. But if one wants to get rid of the bad data -- is deleting it OK? Is removing this field entirely like to cause other problems?
PS. I wrote a little SQL program to find these values also, but updating them is a real pain, as the IPTC field is stored in a column with a bazillion other fields in a set of XML-like text structures (or maybe JSON I didn't look closely).
You are determined! Cool beans.
Got something between a cold and the flu and was all slept out, this at least kept me entertained.
And confused.
Which of course means my conclusions could be affected by cold mediciations.
But I think I've found a way that might fix it which is simple. I want to test some more, but if you have some individual cases and want to experiment, try this:
- From library module, look at the IPTC + EXIF metadata panel
- Select a bunch of photos (raw only -- I have not tried this with non-raw)
- Notice the Date Created field in IPTC is blank (to me this is a bug, it should say "mixed"; if you select only one photo or all photos have the same value it may appear and not be blank).
- Put in a dummy date, say 1/1/2010, tab through field so it updates (this is necessary to make the next line be a change)
- Go back up and blank out the date, tab through the field so it updates (this removes it)
At this point each image, if you look one by one, should have a blank IPTC date created. You could stop at this point and depend on its absence.
- Go back in, select all the same images again
- Under metadata menu, do a save-metadata - this writes out the XMP files
- Under metadata menu, do a read-metadata, acknowledge it is OK if it asks
These last two steps should be a no-change event, but you'll find that the IPTC metadata field for Create Date is now filled in from the original creation time, I think specifically from Exif:DateTimeOriginal + subsecond, but I am not certain where it is coming from. But it seems to be correct.
I haven't gotten quite brave enough to try this on massive numbers of photos, I am still experimenting, but thought I would post this process in case you wanted to try. It's several steps, but not tedious ones, and at least in principle could do 20 images to 20,000.
I had a gallery where every photo had a correct Exif:DateTimeOriginal, but a single IPTC:Create Date. The file names on these, if used for sort, caused the files to be out of order. When viewed, it was not in chronologic order.
I did the above to it -- cleared the IPTC field, write metadata, read metadata, published via LR Plugin (which only updated the metadata).
Then I went in and sorted the gallery by something else, and switched back to date taken (I am told this is needed to prevent it from caching the sort order).
And it is in the right order now.
The metadata changes are time consuming but it is all computer time.
The need to go in and force a re-sort in each gallery by changing it, and changing it back, would be horribly tedious to do by hand for me.
But this is a surprising result because I had understood that these metadata fields would not update on a re-publish. Maybe it is because I'm using the LR Plugin and it is forcing that update, but regardless of why, this provides a path to fix my data and then have it reflected on Smugmug.
I'll try it again and NOT flip the sort back and forth and see if it is indeed caching the sort order.
I have a gallery that I tested. It was out of order due to my bad Date Taken IPTC.
I fixed them, and published. Still out of order.
So I went into organizer, and sorted by file (just to change it), then changed back to Date Taken. In Organizer it showed the thumnails in the right order.
I exited Organizer, refreshed the gallery and The Gallery is out of order.
I've done this now 3 times, and each time it happens -- Organizer CORRECTLY shows the thumbnails sorted, but it won't sort in the actual gallery.
So I looked at the Info bubble and these are NOT updated. So the idea that LR Plugin was correctly updating the date taken was incorrect.
I went back to Lightroom and marked all the images to republish, i.e. to send the actual image data not just metadata.
Still out of order.
Given that at least for me, having the URL's change is not practical (i.e. delete and upload again), it would appear there is no possible way to fix the sort.
This is extremely frustrating. I'm willing to actually upload all photos again to try to fix this, but even that will not fix the problem. The programming shortcuts taken (and treated as Features) block every direction I turn to make it sort correctly.
But still looking.
Go figure. So one doing this with the LR Plugin expecting a fast update is doubly disappointed -- not fast, and it won't fix the problem.
Please fix this Smugmug. Please.
Well... I would not expect "new stuff" to be a problem, if you mean as in really new, since now that you know the metadata issue, you should not be generating bad IPTC information right?
You mean old-new stuff?
I've spent a lot of time talking with Smugmug support and also working on alternatives.
The short answer is they are apparently not going to address this anytime soon, the last email said:
I've pushed back and gotten a pretty consistent answer from three different people.
I think it is possible to fix this yourself, though not in any straightforward way. The three solutions I have found are:
The obvious problem with all of these is a need to go to each gallery and change them. I found my errors had started in mid-2013, so I have a LOT of galleries, though probably not all that many people sitll look at.
The secondary problem is that it will take a long time to fix the actual images on Smugmug. The solution I posted earlier does appear to work:
- Select only master images, raw (I have not tested non-raw)
- Change the IPTC date to something consistent, e.g. 1/1/1 (wait to finish)
- Change it to blank (wait a bit to finish)
- Save metadata to file (this is very slow if there are many)
- Read metadata from file (this is even more slow).
Be VERY VERY sure you save metadata successfully first before reading or you can screw things up, but this sequence will set the IPTC Date Created to the DateTimeOriginal from EXIF, as best I can tell, including subsecond. It will also flag the image for upload to Smugmug, apparently the actual image data not just metadata, so it is (a) quite slow to upload, and (b) fixes your actual original image at Smgumug in case someone downloads it.
It's not pretty, it is tedious, and I am very annoyed at Smugmug's lack of any consistent handling of metadata, or willingness to treat such as a bug. But I did create the problem by coding this field wrongly, so I can only yell so loudly.
Well guess what? It creates a date ANYWAY. It's the exact time a template is made. So there's no escaping this time stamp which is in NO relation to the original image. The site used to sort by Original Date Created and now it doesn't. It's reading the wrong time stamp. It's a bug and now nothing works correctly.
Now a brand new gallery that I want to show in sequence appears like this due to it reading the IPTC time. I can't take the time to create a brand new, in the sequence I want them to appear, File Info template for every single shoot! That's crazy.
OK, there are at least a couple different issues here.
First, if Lightroom is mis-behaving, that is clearly not Smugmug's issue, but Adobe. I think we can hold Smugmug responsible only for what they do with the data we send, and if we send them bad data, it is not their fault.
Now that said -- I think we should also hold them responsible for not giving us a viable mechanism to FIX the data if we send bad data by mistake, and on that they score a miserable failure, because their decisions on when to update which metadata are so inconsistent as to almost be random.
But... to the extent you think "Smumug needs to address" I think you need to open a support ticket. Leftquark or others may show up here, but this is an unofficial support channel. If you want a more official and immediate answer, write the help desk.
But...
I cannot recreate what you are seeing, and I think it is worth (to you) to pursue why it is happening. My problem was well defined -- I was copying a field I did not intend to. For years. Now that I stopped, I am not creating new problems.
I just set up a test, and imported a photo and applied a metadata template. Here is the template:
Notice please that I have stuff in the IPTC image section, and indeed it did create a default for IPTC date, but it did not automatically select the Date Created (check box). I think the date it chose was the prior template that was displayed. And I made sure I did not select it (check box). So while filled in, it will not be applied. In your template you did not show the check box, so not sure what you did.
Here is what happened to the photo on import:
Notice in this actual image that there is an IPTC date, but if you compare to the DateTimeOriginal, they match. That's what is supposed to happen -- a blank field (or more precisely an un-checked field) in the template should produce a missing IPTC date, and then LR will fill it in from the DateTimeOriginal from exif.
If I upload that photo to Smugmug now, it comes out correctly in the Date Taken field.
My Guess is that Smugmug decided to use the IPTC field in preference to Exif because it was more easily modified by the user, and people could fix dates (such as scanned in images). Emphasis on "guess". Right, wrong? Arguable.
But the underlying real problem is that once you get it wrong, you cannot fix it and maintain web links.
Questions for you:
- Did you have that field checked? Can you investigate your workflow, and be sure exactly where you in lightroom you are getting the wrong date?
- Do you link to individual photos from other web sites? If not, your fix is pretty easy if a bit tedious -- fix the photos, delete the photos and re-publish them.
Please understand -- I am not trying to be an apologist for Smugmug. I just think it is best to blame them for the stuff they are responsible for, and how metadata copies/templates work in Lightroom is not their fault, it is Adobe's. Or, in my case, my own fault for doing it wrongly.
I'm sorry, then what program are you using to apply the template?
And what program to upload?
I'm back from a lot of travel and just skimmed through the thread. I need to re-read it in more detail before I can fully grasp everything but I wanted you to know that I'm here and will look further into this. I'm post #28 at this point, so I don't want you to think that SmugMug is ignoring you! As the Product Manager for this area of SmugMug, lets keep the discussion here, as it'll be easier for me to look into things, and you guys already have a ton of great information. Stay tuned!
Former SmugMug Product Team
aaron AT aaronmphotography DOT com
Website: http://www.aaronmphotography.com
My SmugMug CSS Customizations website: http://www.aaronmphotography.com/Customizations
I really don't think there is a lot new here. Elsewhere there's reports that smart galleries do not work which may be a new issue introduced with this.
To me the real fix is "metadata updates all fields all the time correctly when photo is replaced". But we've had that conversation, sounds like it is not on top of the priorities. I still argue it is like a low grade infection -- it may not be killing you today, but it is causing a lot of irritation and annoyance all over the place, some of which likely is not recognized as it being the cause, just people seeing things not working quite right.
Date Taken can be placed in multiple date fields. We look through the following "tree" to determine Date Taken. The first date field we find, is what's used, so if #1 is found, we use that. If not, we drop to #2, etc.
"Composite" is a value that ExifTool outputs based on trying to reconcile overlapping metadata based on the Metadata Working Group's recommendation (http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/MWG.html):
So if we rewrite the above list, we see that we're pulling based on:
What makes me most curious now, is for the photos where the Date Taken is wrong, is the EXIF:DateTimeOriginal different from the IPTC:DateCreated and if so, are we ignoring EXIF and instead pulling IPTC?
Former SmugMug Product Team
aaron AT aaronmphotography DOT com
Website: http://www.aaronmphotography.com
My SmugMug CSS Customizations website: http://www.aaronmphotography.com/Customizations
The part that for me is complicated is what I think happened with the sort order, not the actual date/time fields per se.
It would APPEAR that you put in place a change that affected already uploaded photos. i had a bit over 2 years of photos where a substantial number of the photos had the XMP/IPTC:Date/Time created incorrect.
With apologies I need to ramble a bit, as my concern at the time was not the date/time but the subsecond. I had bursts that were out of order. You put in a place the change that changed how dates worked, but also how it sorted shorts with the same time (specifically to fall back to file name). I tested it at the time, and also found (and you confirmed) that it was necessary to "trigger" a sort because the order was cached. And I did, and it did. And you said you were looking into doing it en mass.
Now for me, for those specific shots, I thought the dates were correct, and it was about a subsecond sort (or secondary sort). But I THINK that around the same time, you also changed where the dates were pulled from, as above.
What I did not realize at the time is I had all these bad IPTC/XMP date/times. As I dug into this, I started seeing galleries completely out of order, which is when I discovered the bad dates.
HOWEVER... that would at least seem to imply that when you put this change in, you went back and recalculated the Date Taken that Smugmug shows for old photos, based on the new technique, and not just for the newly uploaded photos. At least that is what I believed and seems consistent with what the OP here found.
Except... you've been very consistent in saying that Date Taken (among others) is not recalculated except on image replacement.
But I think it was -- as part of this change. But only once?
Is that speculation correct? You did a mass recalculate of Date Taken using different metadata rules, and applied it to old photos, and not just new photos?
Let me find some examples, but I think the now-current image data will be correct. Or I can upload some wrong in this way and see what it does with new photos.