Why is keyword indexing taking so long?

jfriend

How long should keyword indexing take on a Legacy site? I uploaded 400 images a couple hours ago and the keywords are still not indexed and I can't create virtual galleries using those images yet. I guess I'm starting to wonder if virtual galleries are just busted now I have to upload hundreds of duplicates just to finish my event presentation or if the keywords are going to index in the next few hours.

syntaxerr0r

Mine still aren't indexed after a week...

mbonocore

We are catching up more and more by the hour. All should be back to normal soon.

jfriend

mbonocore wrote: »

We are catching up more and more by the hour. All should be back to normal soon.

How soon? I need to know when I can go live with these virtual galleries.

thenickdude

I'm not even sure if they know, the estimates they give are always pretty vague.

jfriend

Lamah wrote: »

I'm not even sure if they know, the estimates they give are always pretty vague.

This is quite annoying. A piece of the site is essentially down for me (the ability to show virtual galleries based on newly uploaded photos) and they can't tell us when it will be working again for those images? That's a pain. It is more predictable for me to just upload duplicate photos rather than wait an indeterminate amount of time for virtual galleries to work. I'm not doing that yet because it's the less efficient way for SM, but I need to get this photo display done and out and I can't just wait for an indeterminate amount of time.

mishenka

jfriend or Michael - please educate me, I ma just trying to understand how smart galleries work (or should work). What happens on the back end when and after I attached keywords to a files? Aren't the keywords immediately saved to a database as a property item to my file? What does Smart Gallery do? Is there a periodic job that runs and updates all the Smart Galleries with the files that match the keyword filter on a Smart Gallery?

thenickdude

Okay so basically what happens when you save your image is that the keywords are immediately saved to it, so that if you click on that image itself you should immediately see the new keywords appear.

However, for efficient searching, what SmugMug needs to do is build what's called an "inverted index". Basically they create a table where each row has a certain keyword, and then a list of all of the ID numbers of the photos that have that keyword in it. Then if you want to search for a keyword, you just jump to the entry in the table for that keyword, and you can immediately read off a big list of photos which match.

Most systems for building these indexes run as an offline process. So basically what happens is that every hour (or day, or something), a program runs which scans through your photos and checks to see if they have been updated since it last ran. If they have, the inverted index is updated.

It sounds like it's this periodic index update process on SmugMug that is still working through the backlog of updated images that it hasn't seen yet, because it hasn't been able to keep up with the rate of new photos being added to SmugMug, and so hasn't added them to its index yet.

jfriend

Lamah wrote: »

Okay so basically what happens when you save your image is that the keywords are immediately saved to it, so that if you click on that image itself you should immediately see the new keywords appear.

However, for efficient searching, what SmugMug needs to do is build what's called an "inverted index". Basically they create a table where each row has a certain keyword, and then a list of all of the ID numbers of the photos that have that keyword in it. Then if you want to search for a keyword, you just jump to the entry in the table for that keyword, and you can immediately read off a big list of photos which match.

Most systems for building these indexes run as an offline process. So basically what happens is that every hour (or day, or something), a program runs which scans through your photos and checks to see if they have been updated since it last ran. If they have, the inverted index is updated.

It sounds like it's this periodic index update process on SmugMug that is still working through the backlog of updated images that it hasn't seen yet, because it hasn't been able to keep up with the rate of new photos being added to SmugMug, and so hasn't added them to its index yet.

That is my understanding too for how this works. The indexing of new uploads is a "side process" that happens sometime after the upload. When things aren't backed up, it happens almost immediately, but when things are really busy, it's hard to tell how soon it will happen. In normal operation, which we are not in now, it seems to generally happen within 15 minutes, but now the delays are long.

jfriend

mbonocore wrote: »

We are catching up more and more by the hour. All should be back to normal soon.

If my images aren't indexed by morning, I'm going to have to upload about 1000 duplicates to make the gallery structure I designed around virtual galleries work. I can't continue to just sit here and wait an indeterminate amount of time before sharing the images from this major event. If you could tell me how long it's actually taken, I could make a more educated decision about whether to wait for virtual galleries to work or not, but with no data, I can't just sit and wait an indeterminate amount of time.

This is super frustrating that a major feature of your site which I was depending on doesn't work when I need to publish some new images. This is the third time I've run into this same hangup right when I wanted to publish photos from a major event.

missedbeat

Since the upgrade to new SmugMug I allow 24 hours for the keywords to come through!

Oja

missedbeat wrote: »

Since the upgrade to new SmugMug I allow 24 hours for the keywords to come through!

You're an optimist! It has taken several days now that my smart galleries haven't updated.

populus

I've been trying to get smart galleries to work for a week now. Each day, a couple more pictures appear in a keyword-based smart gallery, but at this rate it will be 20 years before they are all indexed properly. Now, this morning, another problem has cropped up - when I go to a smart gallery, about 20% of the pictures are just placeholder icons with no photograph. If I refresh, some of these pictures appear. It takes about 4 refreshes before they all appear. I can't expect my audience to do this, so for right now, my Smugmug site is useless. I can't take the approach John is considering because it would be a management nightmare. So for now, I am stuck hoping the problem is fixed soon, but pretty pissed at Smugmug.

jfriend

jfriend wrote: »

If my images aren't indexed by morning, I'm going to have to upload about 1000 duplicates to make the gallery structure I designed around virtual galleries work. I can't continue to just sit here and wait an indeterminate amount of time before sharing the images from this major event. If you could tell me how long it's actually taken, I could make a more educated decision about whether to wait for virtual galleries to work or not, but with no data, I can't just sit and wait an indeterminate amount of time.

This is super frustrating that a major feature of your site which I was depending on doesn't work when I need to publish some new images. This is the third time I've run into this same hangup right when I wanted to publish photos from a major event.

Nothing indexed this morning. Virtual galleries is completed busted. I guess I'll have to rearrange how I was going to present this event and start uploading duplicates. Too bad the virtual gallery feature can't be counted upon.

missedbeat

Oja wrote: »

You're an optimist! It has taken several days now that my smart galleries haven't updated.

It's true, regularly my keywords work at 24 hours after upload!

ceasley

jfriend: Another option would be to collect just the photos that belong in what should have been your virtual gallery, but even collecting is painfully show. I wish there was a "bulk" collection option instead of having to collect each photo, one by one.

As far as smart galleries, I'm finding it's taking about 24-36 hours for my photos to show up in their respective virtual galleries. I'm also disappointed in this slow process, but since I'm not under any deadline, it hasn't been a huge problem; just a minor frustration. I'm just happy to have this feature because I didn't have that ability when I hosted at Zenfolio. Sure hope they get it working faster soon. We all like instant gratification!

jfriend

ceasley wrote: »

jfriend: Another option would be to collect just the photos that belong in what should have been your virtual gallery, but even collecting is painfully show. I wish there was a "bulk" collection option instead of having to collect each photo, one by one.

As far as smart galleries, I'm finding it's taking about 24-36 hours for my photos to show up in their respective virtual galleries. I'm also disappointed in this slow process, but since I'm not under any deadline, it hasn't been a huge problem; just a minor frustration. I'm just happy to have this feature because I didn't have that ability when I hosted at Zenfolio. Sure hope they get it working faster soon. We all like instant gratification!

I'm past 24 hours and nothing is showing up yet. If I knew it would all be functional in 36 hrs, I'd wait, but I really have no idea when it's going to work. I see occasional reports of people saying they've been waiting more than a week. Manually collecting is not an option for me because of the number of photos and the ways in which they are supposed to get displayed. Uploading dups would be much more practical than manually collecting because I could drive the dup uploads by keyword from Lightroom.

chaselton

It's funny because immediately after the site upgrade it was working for me...in fact I was relieved when I checked my tumblr queue and found it had populated.

The past few days though it hasn't been working for some reason. I'm glad I found this thread; I'll upload duplicates now.

pilotdave

Hey smugmug... something you might want to look into...

I'm noticing keywords are suddenly (since new smugmug was unveiled) being generated from filenames in old galleries. I have always had that option turned off on my site. If smugmug is mistakenly generating billions of keywords from filenames, it might explain why indexing has gotten so far behind.

Also I don't want all these new keywords. New smugmug doesn't make it easy to get rid of them. Glad I still have access to legacy smugmug where i can kill them much more quickly. Still a giant pain for me, but at least its not as bad as new smugmug.

EDIT: Never mind the nice things I had to say about legacy smugmug. The necessary function is now broken. See http://www.dgrin.com/showthread.php?p=1897188#post1897188.

Dave

Oja

jfriend wrote: »

I see occasional reports of people saying they've been waiting more than a week. Manually collecting is not an option for me because of the number of photos and the ways in which they are supposed to get displayed. Uploading dups would be much more practical than manually collecting because I could drive the dup uploads by keyword from Lightroom.

I have images that I uploaded now three days ago that are still not showing up in the right "smart" galleries. Crazy enough if I search the keyword, the images come up but creating a completely new smart gallery with the exact same search doesn't show the right images. Very frustrating.

I really don't like to upload duplicate images either and I prefer to keep only one image on smug and use keyword searching smart galleries all over. Now if only the smugmug Lightroom plugin knew how to deal with smart galleries correctly ;-)

mbonocore

pilotdave wrote: »

Hey smugmug... something you might want to look into...

I'm noticing keywords are suddenly (since new smugmug was unveiled) being generated from filenames in old galleries. I have always had that option turned off on my site. If smugmug is mistakenly generating billions of keywords from filenames, it might explain why indexing has gotten so far behind.

Also I don't want all these new keywords. New smugmug doesn't make it easy to get rid of them. Glad I still have access to legacy smugmug where i can kill them much more quickly. Still a giant pain for me, but at least its not as bad as new smugmug.

EDIT: Never mind the nice things I had to say about legacy smugmug. The necessary function is now broken. See http://www.dgrin.com/showthread.php?p=1897188#post1897188.

Dave

Can you send me the link to the gallery you are referring to Dave? Thanks!

jfriend

jfriend wrote: »

I'm past 24 hours and nothing is showing up yet. If I knew it would all be functional in 36 hrs, I'd wait, but I really have no idea when it's going to work. I see occasional reports of people saying they've been waiting more than a week. Manually collecting is not an option for me because of the number of photos and the ways in which they are supposed to get displayed. Uploading dups would be much more practical than manually collecting because I could drive the dup uploads by keyword from Lightroom.

Just to update my status, now I'm at 48hrs and things are half indexed. So, it appears that the queue is somewhere around 48+ hrs long as of a few days ago. I have no idea when the rest will finish. Since maintenance of keyword changes may have a 2-3 day lag before they take effect and I don't know when all the images will be indexed, I can't work with that. So to finish this up, I'm clearing the smart gallery rules and uploading dups this afternoon to just get this project finally done. It's a bit ironic that uploading dups yields a finished result faster and more reliably than using smart galleries.

Keyword index lag is a really big thorn in the usability of smart galleries. They are such a powerful tool (and probably save SM a lot of disk resources), but not when they don't work reliably.

populus

Good news - my smart galleries finally caught up, and all the galleries are displaying properly. I am going to hold off on uploading any more pix or making any metadata changes for a few days. In the meantime, I think photos.kimmerer.com looks pretty damn good - way better than the old Smugmug.

anglotexian

Seven days now since I unveiled, and my legacy recent photos are still AWOL.

mbonocore

anglotexian wrote: »

Seven days now since I unveiled, and my legacy recent photos are still AWOL.

Can you link to your site? Are your recent photos on your homepage?

anglotexian

My site is PW protected. I have a multiple photos block on my homepage - select from: recent photos. The photos appear in admin mode, but not when logged out. The Smug heroes tell me that recent photos are caught up in the keyword indexing lag and that they will appear in time. OK, but just how much more time? Eight days and counting - is this really the cause?

mbonocore

anglotexian wrote: »

My site is PW protected. I have a multiple photos block on my homepage - select from: recent photos. The photos appear in admin mode, but not when logged out. The Smug heroes tell me that recent photos are caught up in the keyword indexing lag and that they will appear in time. OK, but just how much more time? Eight days and counting - is this really the cause?

Hi, I still can't take a look without knowing your SmugMug URL

jsrucci

Different keywords showing up depending on if I'm logged in or not...
I just noticed something very odd.... while checking my /keyword page to see if some of mine were "catching up" yet, I noticed some are showing up when I am *NOT* logged in, but do not show up if I *AM* logged in.

Smug Heroes, any ideas? Here is what I've found when comparing my /keyword page when not logged in vs. logged in:

Showing up when NOT logged in, but not when logged in:

air force station
arches
caribbean
castle
gorge
shore birds
sts132


Showing up WHEN logged in, but not when not logged in:

absaroka range
england
geyser
portugal
west thumb geyser basin
yellowstone lake

www.steverucci.com

P.S. I'm pretty sure that I have a lot of keywords from pictures that were already on my site that are not displaying on the /keyword page now. Some new ones that I just uploaded in the last couple of days are there though. Does anyone know if there is still a backlog of "legacy" keywords getting processed from people who converted to New Smugmug in the last couple of weeks?

Thanks...

uketeecee

Keywords are broken at present. Re-indexing of keywords has apparently been in progress for some weeks now but is still not working properly.
Since unveiling I have told my visitors to ignore my site until further notice.

anglotexian

mbonocore wrote: »

Hi, I still can't take a look without knowing your SmugMug URL

Sorry, I did not provide my URL since it's PW protected. However, if you are a Smugmug employee please be my guest and take a look: http://www.ianangel.com

Thanks,

Ian

mbonocore

anglotexian wrote: »

my site is pw protected. I have a multiple photos block on my homepage - select from: Recent photos. The photos appear in admin mode, but not when logged out. The smug heroes tell me that recent photos are caught up in the keyword indexing lag and that they will appear in time. Ok, but just how much more time? Eight days and counting - is this really the cause?

I sent you an PM to ask for your password to the site. Thanks!