Proposing a Change in SF Judging Method

drdane · August 25, 2007

I'm throwing this out to see what others think.

In addition to my own disappointment at not making the finalist list in #2, I was stunned that other images which I thought should be there had been dropped by the judges, particularly traunerk's Gotcha! (which I thought to be a little stronger than mine), and to a lesser extent, Seastack's Land of the Quinault.

Instead, photos of lesser impact (IMO) took their place in the final 10. To me, this observation was confirmed by the member vote, with the weaker shots receiving the least votes by a considerable margin (though "Toe Squishing" received fewer than I thought it deserved).

In looking back at SF#1, it seems to me that the same thing occurred there, though to a lesser extent. There, the strongest shot left out of the finalist list was Nikolai's "Conductor" IMO. Here are the links, if you'd like to review that round:
SF#1 entry thread: http://www.dgrin.com/showthread.php?t=61375
SF#1 finalists: http://www.dgrin.com/showthread.php?t=62523

Personally, I think the member voting is far more reliable at picking the strongest images. Another way of saying this is 143 (or 198) pairs of eyes are better than five, and given what I've seen here, I'd much prefer to trust my fate to the full membership than a few hard-working and sincere judges who are, by their own admission, basically voting for what moves them. Hey, what else can they do - don't we all?

So I'd like to hear from others, and if there is significant agreement, I'd ask that the SF rounds (at least) be judged solely by member votes - maybe two rounds of 48 hours each to narrow the field to 10, then 3.

If you think I need glasses (or a cranial transplant:D), I'll lay it to rest.

So, to recap, I'm asking two questions:
1) Are stronger images being set aside for weaker ones in the judging of SF rounds?
2) Would deciding SF rounds by member votes alone offer a more effective means of selection?

I think this applies to the qualifying rounds as well, but it feels like the SF rounds should be the main focus for these questions at the moment.

Shay, I hear you about the "real world" thing, and see the value of that approach. Yet I also feel that if we see that something is not working in the "real world" or here in this forum, it is up to us to try to change it.

I rest my case, and invite everyone's input! Thanks!

(I tried to post a poll on this, but screwed it up when I tried to add another option)

Blessings,
Dane

Tentacion · August 25, 2007

Have to say for the most part, I disagree.

I would rather have the judges vote, and then have audience participation....

1. Popularity, whether you like it or not, folks tend to support "friends" been in this predicament before, and one of my main reasons for not participating in contests where photos are picked according to peer voting.

2. Too many EYES see too many things, and have too many opinions. With the photos narrowed down, it gives viewers a set choice to view, and make a decision. Perfect example, the "UNOFFICIAL Feedback Thread" where many pick their choice of Top 10. I would rather take my chances with 5 sets of eyes than 198.

3. There ARE just too many reasons, but I think 1, and 2 narrow down my strongest viewpoints.

4. It would not be ethical at this stage in the game to change the RULES, this would be very UNFAIR to the prior QR's and SF's contestants of this LPS.

This I believe would be (if at all) something for consideration for the next LPS contest, if Shay hasn't by then just thrown up his hands, and said, "I'm giving my time, offering prizes, and still you can't please people, there will be no more LPS contests.",

These are my thoughts.
Donna

PS: But I do believe, the judge selection should be well rounded i.e. their photographic POV's., this way everyone has a fair shake...and Portraiture Photograhers are not being looked soley by photographers who just do Landscape Photos.

Povertybarn · August 25, 2007

drdane wrote:

Personally, I think the member voting is far more reliable at picking the strongest images.

I agree. I have thought all along that the entries should be decided by member vote. Instead of taking a sampled vote of the select few it is always better to get the collective choosings of that of the larger good. You will find the entries will coincide with what the populous deems as "worthy". This stance not only alleviates the thought process of "getting the shaft" but gives ones' feelings that he or she was able to compete in a level playing field.

Angelo · August 25, 2007

I think, if I were one of the fantastic 10 finalists, I'd be insulted by your insinuation that a pair of birds in flight should trump any of the terrific final entries.

IMHO

Harryb · August 25, 2007

So, to recap, I'm asking two questions:
1) Are stronger images being set aside for weaker ones in the judging of SF rounds?

No and the question really has no right answer. No matter how a "winner" is choosen some one will say "stronger images were set aside for weaker ones".

If you do it by popular vore someone will say "its just a popularity contest".
If you use a panel of judges someone will say "150 set of eyes is a fairer way than using just 5 set of eyes".

Come up with another way of judging and I'll find a complaint someone made about using that method also.

I'm currently waiting to see how my shots will fare in a local contest down here. I'll bet you next months mortgage payment that I won't agree with the judges' selections in that contest.

The basic fact here is that these contests are like life, it just ain't fair. When someone wants to buy one of my shots they always ask for a shot that I think is one of my lesser efforts. When you try to sell your work or win a contest you just have to deal with it how it is. You can't always get to choose how your work will be viewed by the public or those making the buying decisions. Just take the best picture you can, submit it, and learn what you can from the process.

Harryb · August 25, 2007

Angelo wrote:

I think, if I were one of the fantastic 10 finalists, I'd be insulted by your insinuation that a pair of birds in flight should trump any of the terrific final entries.

IMHO

I would say, IMHO, that a shot of birds in flight would just about trump anything. :ivar

Angelo · August 25, 2007

Harryb wrote:

I would say, IMHO, that a shot of birds in flight would just about trump anything. :ivar

oh Lord, I just knew you'd be coming along to throw a grenade in my direction!

Harryb · August 25, 2007

Angelo wrote:

oh Lord, I just knew you'd be coming along to throw a grenade in my direction!

Watch your step, if you don't want to get known as an anti-birdite. :grim

You don't want to know what a flock of egrets can do to a parked car. :eek1

fashiznitsngrins · August 25, 2007

Here's my 2 cents FWIW
When I first started LPS (at the very beginning) I didn't realise that the judging would go quite the way that it has. I assumed that there would always be at least 2 mods for each round and I was under the impression that Shay had a whole host of professional photographers at his beck and call to judge any number of challenges. We all know what happens when we assume though, don't we?

I too, sit and scratch my head at the decisions made in both semi finals (not necessarily during the individual rounds). The first semi final hit me hard - not so much that MINE hadn't got through, but that at what had. It took me a good long while to get over that, but once I did, I was much better for it. Water off a duck's back now. I enjoy what I am doing a lot more and it allows me to be more creative. I don't sit and contemplate about what I think will get through, but more about what I want to do. In the first SF I came up with what I thought was a pretty good portrait and I never had thought I could take a good portrait. In the second SF I came up with what I thought was a pretty good landscape and I had never thought I could capture a good landscape. There was a large part of me that didn't want to get through in the top ten, because I want to push myself just as hard to see what I can come up with in SF3 (provided that I qualify) that I didn't think I was capable of.

What would be my ultimate goal? I suppose I would love to take photos of what I love and sell them in galleries. How would I be successful? By appealling to a large and varied audience. So, the more times I can qualify with different judges, the more that I feel I am on the right path. So, although I would love to win 1st place in a SF, I just see this whole experience as a great way to help me accomplish my ultimate goal. I have learned to accept criticism through this entire process, which has been one of the biggest challenges of my life!

I know that I have gotten a little off the subject maybe, but what I am trying to say is that sure, I'd welcome changes in the judging, but if it stays the same, I'll continue to roll with the punches.

Here's a question to ask yourself perhaps... did the voting poll for the SF go the way you thought it "should"? I know it didn't for me, so perhaps having more eyes won't necessarily go any different to how you think it should.

Art Scott · August 25, 2007

Angelo wrote:

oh Lord, I just knew you'd be coming along to throw a grenade in my direction!

At leqst it was a LOVE GRENADE (of the birdie kind):D:D

rofl

Dang had a Nuge moment there

rofl

Art Scott · August 25, 2007

there is no good answer to this type of dilemma ..... every salon (contest) ihave ever been involved in had such gripes.....the only way that was found for the Int'l Salon that is held here in ICT every year was to get totally outside judges.....and then the bi**hing was about the cost of outside judges as they cost money and still that won;t satisfy every one as there will always be someone that still yells "FIXED or BOUGHT OFF JUDGES"............so as Mr. Harry so well put it ...just shoot your best and learn what you can from it......and never give up.............

drdane · August 25, 2007

Tentacion wrote:

Have to say for the most part, I disagree.

I would rather have the judges vote, and then have audience participation....

1. Popularity, whether you like it or not, folks tend to support "friends" been in this predicament before, and one of my main reasons for not participating in contests where photos are picked according to peer voting.

2. Too many EYES see too many things, and have too many opinions. With the photos narrowed down, it gives viewers a set choice to view, and make a decision. Perfect example, the "UNOFFICIAL Feedback Thread" where many pick their choice of Top 10. I would rather take my chances with 5 sets of eyes than 198.

3. There ARE just too many reasons, but I think 1, and 2 narrow down my strongest viewpoints.

4. It would not be ethical at this stage in the game to change the RULES, this would be very UNFAIR to the prior QR's and SF's contestants of this LPS.

This I believe would be (if at all) something for consideration for the next LPS contest, if Shay hasn't by then just thrown up his hands, and said, "I'm giving my time, offering prizes, and still you can't please people, there will be no more LPS contests.",

These are my thoughts.
Donna

PS: But I do believe, the judge selection should be well rounded i.e. their photographic POV's., this way everyone has a fair shake...and Portraiture Photograhers are not being looked soley by photographers who just do Landscape Photos.

Excellent points, Donna, thank you!

I wasn't comfortable with the Unofficial Feedbadk threads, either, but found the actual voting to be pretty consistent with my own assessment.

I heartily endorse your PS! As an example from my perspective, it has seemed to me that there has been a slight, but noticable "urban bias" when it comes to judging these things.

Blessings,
Dane

drdane · August 25, 2007

Angelo wrote:

I think, if I were one of the fantastic 10 finalists, I'd be insulted by your insinuation that a pair of birds in flight should trump any of the terrific final entries.

IMHO

Angelo, thanks for your comment. Please disregard my shot and consider #36 - Gotcha! Do you really think that every one of the final 10 is a stronger shot than this? Better yet, let's ask the finalists themselves. This kind of capture is extremely difficult, and probably required tremendous time, patience and exposures - and it's really good (IMO:D)!

drdane · August 25, 2007

Harryb wrote:

Watch your step, if you don't want to get known as an anti-birdite. :grim

You don't want to know what a flock of egrets can do to a parked car. :eek1

Thanks, Harry and Angelo - you guys just made my day !

Hey, Harry - would you have a photo of the "egret's revenge?":D

Thanks for your comments and Good luck in the contest down there!

Blessings,
Dane

indiegirl · August 25, 2007

I have not read this entire thread, forgive me of I'm way off base.

I would like to see a vote count requirement for the folks voting for the audience favorite to discourage vote-and-run practices. I have no idea if this has happened but worry that someone may bring in a ton of people whose only participation is to vote for their friend.

Also, I think the SF judging portion should be by professional judges (sponsors, mods, professional photographers, smugmug royalty)

Jesse

drdane · August 25, 2007

fashiznitsngrins wrote:

When I first started LPS (at the very beginning) I didn't realise that the judging would go quite the way that it has. I assumed that there would always be at least 2 mods for each round and I was under the impression that Shay had a whole host of professional photographers at his beck and call to judge any number of challenges. We all know what happens when we assume though, don't we?

I too, sit and scratch my head at the decisions made in both semi finals (not necessarily during the individual rounds). The first semi final hit me hard - not so much that MINE hadn't got through, but that at what had. It took me a good long while to get over that, but once I did, I was much better for it. Water off a duck's back now. I enjoy what I am doing a lot more and it allows me to be more creative. I don't sit and contemplate about what I think will get through, but more about what I want to do. In the first SF I came up with what I thought was a pretty good portrait and I never had thought I could take a good portrait. In the second SF I came up with what I thought was a pretty good landscape and I had never thought I could capture a good landscape. There was a large part of me that didn't want to get through in the top ten, because I want to push myself just as hard to see what I can come up with in SF3 (provided that I qualify) that I didn't think I was capable of.

What would be my ultimate goal? I suppose I would love to take photos of what I love and sell them in galleries. How would I be successful? By appealling to a large and varied audience. So, the more times I can qualify with different judges, the more that I feel I am on the right path. So, although I would love to win 1st place in a SF, I just see this whole experience as a great way to help me accomplish my ultimate goal. I have learned to accept criticism through this entire process, which has been one of the biggest challenges of my life!

I know that I have gotten a little off the subject maybe, but what I am trying to say is that sure, I'd welcome changes in the judging, but if it stays the same, I'll continue to roll with the punches.

Here's a question to ask yourself perhaps... did the voting poll for the SF go the way you thought it "should"? I know it didn't for me, so perhaps having more eyes won't necessarily go any different to how you think it should.

Thanks for sharing your experience, Karen! I have come to a similar conclusion from all of this. I have been taking this contest too seriously, and now my goal is to have fun, and shoot what moves ME.

I did feel I needed to say something, and see what happened. Perhaps it will improve things a bit in the future, but as several have pointed out, there are many different opinions as to what that would look like!

In answer to your question to me, yes, the voting was pretty close, with one or exceptions, to my own preferences.

Blessings,
Dane

drdane · August 25, 2007

Art Scott wrote:

. . . as Mr. Harry so well put it ...just shoot your best and learn what you can from it......and never give up.............

Thanks Art!

A lot of wisdom is coming back from you all. In response to your admonition above, I will (keep doing my best), and I won't (give up).

Blessings,
Dane

drdane · August 25, 2007

indiegirl wrote:

I have not read this entire thread, forgive me of I'm way off base.

I would like to see a vote count requirement for the folks voting for the audience favorite to discourage vote-and-run practices. I have no idea if this has happened but worry that someone may bring in a ton of people whose only participation is to vote for their friend.

Also, I think the SF judging portion should be by professional judges (sponsors, mods, professional photographers, smugmug royalty)

Jesse

Thanks, Jessie!

The vote and run thing would be a different issue, and I don't think that's happened here, but its possible. I don't know how that could be prevented - I would just rather trust in each entrant's integrity, unless something really blatant showed up.

I think the Pro judge idea might be an improvement, if there's a balance as Donna suggested in post #2.

How is your daughter, BTW?

Blessings,
Dane

Gary Glass · August 25, 2007

Look, it's very simple. If my stuff loses, the contest is rigged. That's life. I find ways to cope.

But seriously, I shoot what I like. I don't care if I agree with the judges or the public opinion poll or the magic 8-ball. Judging is hard. Judging is subjective. But photography is hard. And subjective. There's no perfect contest because there are no perfect judges because there are no perfect criteria because this is art.

imax · August 25, 2007

And Then What?
After reading through the post, that's pretty much how I feel.

So the powers that be have taken their time and money and put together a contest and now because people are unhappy with the results they feel it should be changed.

So let's change it to make these people happy.

Then what?

When that upsets others do we change it again?

I don't know, maybe it's not a perfect system, what I produce today may move someone to say wow and that same picture in someone elses eyes, they say it's crap. Bottom line is that you cannot make everyone happy, no matter what you do. All you can do is the best that you can do and hope in the end the best will come out on top. Just remember this

it's not our contest to run but it is our contest to participate in or not. The choice is ours.

And for those that are going to say well your in the finals, you have no reason to complain. It's not about complaining, it's about getting better at what I love to do. If that was the case I wouldn't be looking to enter photographs in the next challenge and If I wasn't in, I'd certainly still be trying to get in by my photographs, not by trying to change the rules to suit me better.

And on a side note, if those people feel that less qualified entries got into the finals, they should be happy, it's less competition for them.

Have a great night...........

Joe

indiegirl · August 25, 2007

I think, at least in my case anyway, the changes suggested are more for the discussion rather than an actual call to change. I've wondered what other people think about the whole "vote and run" possibility. I do not expect nor do I think change is needed but in discussing all of this perhaps we can learn from it all.

Jesse

Angelo · August 25, 2007

Harryb wrote:

Watch your step, if you don't want to get known as an anti-birdite.

Me? an anti-birdite? Never... (where's my gun?) :toni

Harryb wrote:

You don't want to know what a flock of egrets can do to a parked car.

Precisely!!! I've been to the marshes. Get my gun!!!

ShepsMom · August 25, 2007

I don't participate in this challenge, oh wait, i think i did ones. I don't have enough confidence in my shots to post them for contest. However, if you want to judge fair and square, here is solution:

Entries should not be displayed for public viewing until contest is closed.
Entries should be posted for voting without names, number them as you wish.

Everyone is voiting based on the picture quality/subject matter, etc.

Isn't it how it should be?

I always scan contest entries, and i feel that if i hadn't seen all the pictures posted, i might have guts to submit someting that i think worth a look. Seeing submission creates too much "keeping up with the Jones" attitude, IMHO, correct me if i'm wrong.

Angelo · August 25, 2007

drdane wrote:

Angelo, thanks for your comment. Please disregard my shot and consider #36 - Gotcha! Do you really think that every one of the final 10 is a stronger shot than this? Better yet, let's ask the finalists themselves. This kind of capture is extremely difficult, and probably required tremendous time, patience and exposures - and it's really good (IMO:D)!

I don't like #36 very much either. Now, please understand, in my original response I was not being condescending just to be mean; I was trying to prove a point.

Photography is art and judging is largely subjective. Having 5 or 20 or 140 pairs of eyes would not guarantee any particular image would be selected in a contest.

IMO the beauty of the process here is that various rotating judges are employed and their selections and application of points is private and confidential until tallied. This ensures the greatest possible honesty in the outcome.

While certain points of judging could be considered rote, such as consideration of composition, exposure, theme (when applicable), much relies on the emotion of the judge's reaction to the image as well.

Crying into one's cup because the "oh so fantastic" image was overlooked and wanting to overhaul the system or move the goalpost is just unfair.

Everyone knows the rules when entering and that includes understanding the judging method.

Felicia · August 26, 2007

I like this thread because it has gotten me thinking about judging art. Like Gary said, it is art, and art is subjective. Yes, there are rules to "good" art that people have come up with by studying art that has moved them. However, those rules are broken many times by art which inspires, captures the imagination, revolts, brings to tears of pain, empathy, or enlightenment, etc.

I see the LPS as a teaching tool with benefits. As we are challenged to submit photos and and have them judged based on a theme, we are being taught to listen and observe - listen to the request of what we are expected to show/illuminate in our work and observe creativity in action. It's about honing skills and unleashing our creativity so that we can be better communicators of what we are observing in this big, amazing, beautiful world. I don't care how professional and amazing you are as a photographer, you can always learn how to be a better communicator. No one is perfect.

That said, we all have different personal filters - different world views and life experiences which make up those filters. Some photos don't communicate well through our filters. Some do. Some photos may be technically brilliant. They follow all of the rules. But those photos may not pass muster through our personal filters.

How do we judge? I like the LPS format of rotating judges with a consistent anchor. It allows folks with different filters to have their opinion count. Just look at some of the opinion polls requesting help in narrowing down entry options. Sometimes, the responses are all over the place.

I also prefer the method of narrowing down the SF photos into a top 10 then opening it up to the DGrin public for the top 3. I, too, was surprised at what photos were left out of the top 10. But then it goes back to everyone's own filter. If I had been a judge, I'm pretty sure there would have been amazement at some of my selections, wondering how on earth I could think that photo rated higher than another. Well, it would be my filter.

(sigh) All of this filter talk probably comes from my desire to fork out some $$ for some nice ones...

peterst6906 · August 26, 2007

My opinion on this, is that the current system works fine.

But I also think that it is a discussion worth having, because I'm sure that at the end of this first LPS, Andy, Shay, Erik and others will review the results and process and decide whether or not to continue with it in the future, and if it continues, how to improve on it and what to maintain the same.

I like the current format, because in my own observations I usually see that, even in the QRs, there are 3-4 strong images that most people pick, then 2-3 images that are in the top 10, but placing varies and then a group of 10 or so 'boundary' images where it comes down to individual choice as to which makes it and which doesn't. The advantage of having a small group of judges in the initial stage is that the top 10 is fairly easy to separate out. We haven't seen too many =10 place ties. I think if it went to a group vote from the start for the semi-final, then there would be many occasions where the top 10 would involve several ties for 10th place, giving a top 20 or so. In the semi-final this doesn't work, not only because it would then dilute the votes for the top 3, but also because there are prizes for placing in the top 10. So in this case, a small group of judges initially makes it easy to identify a top 10 only (even if we don't agree with it).

Just my 0.02.

Regards,

Peter

drdane · August 26, 2007

Angelo wrote:

I don't like #36 very much either.

Surprised me on that one, Angelo, for which I thank you. It seems to be time for me to be shedding the belief that the way I personally see things actually means anything in the larger context. What's more important is to recognize, accept, and even appreciate that we see things differently.

Angelo wrote:

Everyone knows the rules when entering and that includes understanding the judging method.

Please correct me if I an wrong, but from the postings from some of the judges recently, it sounds like there isn't really a "method, and that each judge is free to make up their own.

drdane · August 26, 2007

Felicia wrote:

. . . That said, we all have different personal filters - different world views and life experiences which make up those filters. . .

Thanks, Felicia - I've "known" this for years, but now I think I'm beginning to really "get it."

The feedback offered by everyone so far has definitely encouraged me to look at things from different perspectives.

Dane

drdane · August 26, 2007

peterst6906 wrote:

My opinion on this, is that the current system works fine.

But I also think that it is a discussion worth having, because I'm sure that at the end of this first LPS, Andy, Shay, Erik and others will review the results and process and decide whether or not to continue with it in the future, and if it continues, how to improve on it and what to maintain the same.

Peter, thanks for your .02 worth. Yes, I'm sure they will be looking at the whole thing, and will probably even ask us masses for feedback! Better be prepared for 200 different opinions!

Gary Glass · August 26, 2007

I generally don't do contests for several reasons. One of them is that I want to shoot what I want when I want without reference to anything but my own tastes and desires. That's also why I don't really have any desire to be a "pro." I'm a working stiff and I have to make a living. To make a living as a professional photographer would change my relationship to photography. I'd rather do something absurd for which people are willing to pay me (like write computer programs) and work on my arts on my own dime.

I'm going to enter this round of LPS because, basically, certain people dared me. And also because, in creative defiance of everything I just said above, the constraint of the context theme and rules also inspires creativity, but creativity of a different kind. I'm trying to grow as an artist, to go beyond snapping whatever strikes me, and think more about consciously undertaking considered projects. A contest offers a challenge in that area: here's your theme, do something creative with it. That is something professional photographers are faced with every day, and they have their livelihoods on the line to boot (God help them!).

So what I'm getting around to saying in my long-winded way is that if you're concentrating on winning the contest or how it is judged or whatever, then in my opinion, for whatever it's work, you're not getting out of the experience the best value it has to offer, which is the challenge to stretch and grow as an artist, to think creatively about a subject not of your choosing. Doing so gives us a chance to exercise our critical faculties, to learn the kind of distancing from and cold-blooded critical judgment of our own work which is so essential to developing artistic ability.

Pat664422 · August 26, 2007

There are a lot of interesting points raised on the topic so far -

A point was made that with public voting it can come down to a popularity contest. If the the judges are always a small sample of the users on this site, then I think you would still have the same problem, just on a smaller scale. I would think the only way to eliminate this possibility would be to find judges that do not regularly visit this site. But I am not actually suggesting that be done. I am fine with it the way it is.

drdane wrote:

To me, this observation was confirmed by the member vote, with the weaker shots receiving the least votes by a considerable margin.

The fact that some photos in the top ten received significantly less votes than others doesn't necessarily indicate that some photos shouldn't have been in the top ten. The public votes represented the photo that each voter thought was the best photo of those ten. If the public had been able to rank the photos from 1 to 10 and the bottom three were consistently the same, then I think you can start to wonder.

GaryGlass wrote:

This is art

I agree. There is no formula. And the fact the judges are regularly rotated removes any consistency from the judging which I think is probably best for the sake of the contest and those who participate.

ShepsMom wrote:

I always scan contest entries, and i feel that if i hadn't seen all the pictures posted, i might have guts to submit someting that i think worth a look.

Don't talk yourself out of it! Besides, the great thing about the themed round for those of us who are less confident in our abilities that others is the fact that sometimes even the best quality images are removed from the judging because they didn't fit the theme. As long as your picture is clearly related to the theme, you've got a chance. Plus, the feedback you receive afterwards may really help you grow.

Proposing a Change in SF Judging Method

Comments