Position limitations and bugs

rutt · May 13, 2005

So I tired to move all my private albums above all my others by adding 1,000,000 to the position of each one. The result of incrementing all their positions by 1,000,000 was that they all ended up with the same postion: 65535. So is this the maximum number of albums I can have, or just the maximum number I can stably position relative to one another? It also shows what I think is a true bug, because they now all have the same position, and Don thinks that shouldn't happen.

Don, why use shorts for this?

onethumb · May 13, 2005

rutt wrote:

So I tired to move all my private albums above all my others by adding 1,000,000 to the position of each one. The result of incrementing all their positions by 1,000,000 was that they all ended up with the same postion: 65535. So is this the maximum number of albums I can have, or just the maximum number I can stably position relative to one another? It also shows what I think is a true bug, because they now all have the same position, and Don thinks that shouldn't happen.

Don, why use shorts for this?

Easy. Database size and speed are essential, and we sort on Position *a lot* as you can imagine.

If someone blows by 65,000 galleries, we have a big problem. Average # per customer? 14.

Don

rutt · May 13, 2005

onethumb wrote:

Easy. Database size and speed are essential, and we sort on Position *a lot* as you can imagine.

If someone blows by 65,000 galleries, we have a big problem. Average # per customer? 14.

Don

Are shorts really faster than ints? I'd be suprised. Modern computers are very word oriented deep down and it's often slower to discard half a word than just use it.

Anyway, it wasn't the number of galleries, it was trying to establish ranges for each (sub)category so they wouldn't have to be reordered. So I thought I'd keep all my proofsheets above 1M. No big deal.

At a minimum, I think you should have an overflow fault when someone tries to set an Album position out of range.

BTW, doesn't it make sense to you that a pro would have a gallery per shoot? It would take a while to get to 65k that way, but once you start to want to be able to keep all the weddings below all the portraits below all the soccer games, below all the photo journalism, &etc, well, you can see that you might easily just want to block out some space for each.

I think this points up a limitation having the space of position numbers be global throughout all albums and sorting (sub)categories by their minimum positions. If you associated a position number with each category and subcategory it would have a lot of advantages:

Speed. This would do a lot more for sorting than using shorts for positions. Sorting categories and subcategories would no longer require visiting all their member albums.
Much easier to reorder categories and subcategories. Just change their positions instead of all member albums.
No more unintuitive side effects from arranging albums. I was completely confused the first time I ordered by albums within a category and found my categories reordered.

onethumb · May 14, 2005

rutt wrote:

Are shorts really faster than ints? I'd be suprised. Modern computers are very word oriented deep down and it's often slower to discard half a word than just use it.

Oh, yeah. 2 bytes vs 4 bytes might not seem like a lot, but when we have millions and billions of galleries, it'll be a big deal. Half the amount of RAM, double the speed reading from the disk, etc. The storage requirements are multipled because the position values are used in indexes and stuff in addition to raw table value storage.

Anyway, it's pretty easy to see how it could be slower if we wanted to support 4 billion album positions rather than 65 thousand.

Don

rutt · May 14, 2005

onethumb wrote:

Oh, yeah. 2 bytes vs 4 bytes might not seem like a lot, but when we have millions and billions of galleries, it'll be a big deal. Half the amount of RAM, double the speed reading from the disk, etc. The storage requirements are multipled because the position values are used in indexes and stuff in addition to raw table value storage.

Anyway, it's pretty easy to see how it could be slower if we wanted to support 4 billion album positions rather than 65 thousand.

Don

Now that I understand a little more about how smugmug works, I'm guessing that I know of a real world smugmug performance issue. I have over 600 non-public "proofsheet" galleries, one per shoot, since 1998. Do you have to calculate a minimum of the position of each of these in order to render my home page? Is this different if they are all private and the view isn't logged into my account? All these 600 albums are in the Other category and organized into subcategories by year. It seems to take a long time for my home page and for the "Other" page to start to render. Is it sorting in this time?

This is the source of the suggestion for keeping positions with the category and subcategory levels. If you want to reduce the size of the change, you could just cache them there. But I think there are very appealing advantages to having real postions for (sub)categories.

How many categories and subcategories do you have? Not many, I'll bet, given what you say about the average number of albums/user. So this change won't have the memory size implication that changing postions to longs would.

onethumb · May 14, 2005

rutt wrote:

Now that I understand a little more about how smugmug works, I'm guessing that I know of a real world smugmug performance issue. I have over 600 non-public "proofsheet" galleries, one per shoot, since 1998. Do you have to calculate a minimum of the position of each of these in order to render my home page? Is this different if they are all private and the view isn't logged into my account? All these 600 albums are in the Other category and organized into subcategories by year. It seems to take a long time for my home page and for the "Other" page to start to render. Is it sorting in this time?

This is the source of the suggestion for keeping positions with the category and subcategory levels. If you want to reduce the size of the change, you could just cache them there. But I think there are very appealing advantages to having real postions for (sub)categories.

How many categories and subcategories do you have? Not many, I'll bet, given what you say about the average number of albums/user. So this change won't have the memory size implication that changing postions to longs would.

The short answer is no. The time required to sort your albums by position is probably measured in microseconds.

Don

rutt · May 15, 2005

Don, please, can you explain your thinking about deriving category and subcategory positions from album positions instead of keeping independent positions for them? I've known about your design for more than a year, but have never appreciated why you don't want to do this. It seems so natural to me.

onethumb · May 15, 2005

rutt wrote:

Don, please, can you explain your thinking about deriving category and subcategory positions from album positions instead of keeping independent positions for them? I've known about your design for more than a year, but have never appreciated why you don't want to do this. It seems so natural to me.

Couple of reasons:

1 - it's rarely asked for. We tend to work on the things that hundreds and thousands of people want, as you can imagine. We hear about this, but not in droves.

2 - There are "base" categories which are global for all smugmug customers. If I want individual users to be able to organize all their categories, they need control over these base categories, too, which means I have to add another layer of data complexity. Not impossible, but when factored with #1, it's a low priority.

Particularly when it's fairly easy to make a tool like you described which mimics true positioning by intelligently manipulating the album positions.

Don

Position limitations and bugs

Comments