At the end of the contest there were over 13,000 images in the Flickr pool. Assuming the card/clean image ratio was 1:1, roughly 6,500 clean images were submitted. Of course, things are never that simple. Multiple views, details and other secondary images complicate estimating the original submission number.

With any submission I needed to check ten aspects of the image (1) Is the photographer registered? (2) Was the image taken in February 2009? (3) Where is the card shot? (4) Is the object on loan or (5) post-1923? [Disqualifying in most circumstances] (6) Is the image public on Flickr? (7) Does it have the right license (CC-BY or CC-BY-SA)? (8) Is it tagged? (9) Does it have a valid category tag from a goal list to which the object relates? (10) Are there any multiple views/details as secondary shots not to be scored? Perhaps some of that could have been more automated—but certainly not all of it. If that was good to go then I could manually add the machine tags to be read by our scoreboard. I could only find a few tools to help the process—it’s difficult to do batch actions to other people’s photos on Flickr.

In total, I added at least 30,000 tags since the contest began. This also included tags marking scoring issues and accession number tags. Why the accession number tags? It wasn’t part of the original tagging plan, but it became rapidly apparent it was necessary. Certainly it helps identify the object without routing around for the card, but more important it allowed me to see all the images of a given object. From the Brooklyn Museum: Four Dianas, Four cartonnages, Five scarabs, Five war councils, Six cupids, Six cats!

It makes captioning so much easier. It’s daunting however to contemplate how much of the pool is similar shots of the same objects. Further, goal lists were as also somewhat similar across all the museums, so one might end up with dozens of shots illustrating the same subject. Take Cupid for example.

Why were certain images disqualified? You can probably gather from my ten-part checklist above situations when images did not qualify. When the contest began the most common problem was forgetting of category tags, but after reminder was sent out it was rarely an issue again. Instead, it became clear the biggest problem for contest would actually be insuring the right licensing on images. Most times it seemed to be people forgetting to mark their WLA-images a different copyright than their photostream, but huge source of confusion was the necessity for a Creative Commons license which permitted commercial use and derivative works. The short answer is because the Wikimedia Commons, where the images will be stored and accessed for Wikipedia, only permits free content which can be used “by anyone for any purpose“, including potential commercial uses. There was even some worry that the Museums, in particular, would use the image for a commercial purpose. I’m not sure we made the reasoning for this licensing clear enough to our participants if concerns like this came up.

Also of note, a response I was not expecting, but was very common in disagreement with a rule, read essentially, “I am not participating for prizes, so I don’t think I need to follow this rule.” Our answer was the same rules for everyone, no exceptions. It was a frustrating situation and perhaps also required more clarity from the start.

In total, around 300 images were removed from the pool at the end of the contest due to unaddressed problems, largely incorrect licensing. What was the worst about this is it often disqualified a person’s entire contribution. How much more opportunity should we have given though to fix the problem in the context of this contest?

I don’t think it’s possible to thank the photographers enough for their work. Given my role in this project I’ve looked, literally, at every photograph submitted—often several times over. Having never been to many of participating institutions, it’s a personal reminder about the type of access an endeavor like this can offer. It’s wonderful. I would like to give special thanks though to those who went the distance, either by contributing an absurd amount of photography or by handling a particularly wonky Flickr and/or scoreboard problem. Many thanks to teams: artifacts, BWillsphotos, Department of Trife, Futons of Rock, niborean, team gene, RandomVariables, shooting brooklyn, TheAdversePossessors, The Wookies, UK FGR, Va Va Val, VeronikaB, Wiki-wiki whaaat? Also super props to The Grotto and Assignment Houston One for tying for the biggest team! And extra super props to Opal Art Seekers 4 who photographed at the most museums of any team—from London to Indianapolis!

Machine Tagging, Captioning, Bonus Points (the boring, tedious stuff)

Erin has cleaned the entire pool and scored every entry. In some cases, this meant 3 or more machine tags per clean photo. I’m sure she’ll give you the total tomorrow, but the most basic math would indicate something along the lines of 30,000+ machine tags.  Please keep in mind, that’s 30,000 tags applied by hand to organize and a pool of 13,000 images.  To say that our plans here didn’t scale is putting it a bit mildly. Erin, we all seriously owe you more than one drink.

Institutions are captioning at a pretty solid rate, but this will take some time. We are all trying to do this in spare hours and had hoped captioning would be done by the end of March, but it will take longer. You can query overall progress here (as of now, roughly 1700 of the 6000 clean shots have been captioned) and you can run queries by institution here.  Because of the sheer volume of the 2,690 images shot at this one venue, the MET will be captioning-on-demand as the wiki community decides what it needs from the pool and we are discussing the best way to coordinate that effort.

Cary Bass has approximately 6000 clean shots to go through to assign bonus points. To be fair to all the photographers, he’s being good about stopping when his brain is on overload. We expect the entire process to take him 46 hours, over many sessions. He’s now sorting entries by museum, so hopefully we can announce winners at each institution as he finishes groups rather than waiting for the entire pool. You can chart his progress and see his picks by running this query.

Uploading (more possible snafus)

We are currently facing issues surrounding how the images are going to get uploaded to Wikipedia (not something Erin or I have to do….yipeee).  When we originally set out, institutions and photographers were told they were going to be used to illustrate Wikipedia articles.  The wiki community would like to upload them into Wikimedia Commons which helps them manage assets and makes the images more accessible across the wiki platform,  so they can be cross posted at Wikipedia.  To the wiki peeps, this is six/one-half-dozen/or-the-other and, in reality, this really is splitting hairs, but I wish it was something I had understood better at the start, so we could have more clearly defined it for the participants.  We will be e-mailing participants soon to clarify this issue. At that point they will have the opportunity to leave the project without it affecting their scores or prizes.

Closing Thoughts

So, how did this project go off the rails?  For starters, we jumped in with large project instead of a much smaller one where we could apply the “keep it simple” rule.   More importantly, the entire process was really designed to work with the Wiki community (people we didn’t know yet) to create a project that would engage our existing Flickr community (people we knew very well).  What we found is that the community that we had a lot of experience with was the one that made this a smooth process, but the one we knew less about got us into rougher waters.  When I look back on this project, what rings true in my mind is that all communities are different and when we are designing a project, it’s best to concentrate on perhaps one of those and start small, so you can chart the waters first.  The issues you find might stop you in your tracks or they might help you design something more appropriate.

Two weeks ago someone e-mailed me to express that we were going about this much too slowly, that it was unacceptable that the winners were not known yet.  On the other hand, we are hearing lots of encouragement and understanding and this has made all the difference right now—thank you for being patient, thank you for being awesome. This seesaw so perfectly encapsulates what this project has been like: on one hand, significant issues with some participants…on the other, the best community members in the world working with us to make this a good experience against enormous odds. When I look at the discussion forum, I’m amazed at how much interaction was going on there in such a short amount of time. I love the fact that this project introduced us to new people via Flickr and kept our relationships with old friends. And sometimes we’d get amazing feedback like this and frustration would melt away.

I’d like to especially thank the above people in this little mosaic and give a special shout-out to the top five contributors.  Thank you for your participation in the discussion forums, for working your hearts out, for overall being awesome—your feedback during this process was incredibly valuable.  These are but a fraction of the people who worked so hard and Erin is going to give even more props to the participants she’s been working with in her post tomorrow—yay! My greatest hope is that as institutions finish up captioning, participating photographers can be recognized for their substantial efforts by getting their work onto Wikipedia with proper credit and notification. We are looking forward to that.

In the end, the Brooklyn Museum won’t do this again and, given the enormity of organizing this project, we will assume we’ve got enough Brooklyn-Museum-Wikipedia-karma to be all set for quite some time.

Tomorrow, Erin will round out the series and I hear there are pie charts!

I admit…index cards…say it with me now…FAIL! (now don’t we all feel better?)

“We ask that you shoot each work twice. First time shoot the object with an index card in the frame that displays the object’s accession number, your team name, and category name so we can assign points. Second time shoot the object again, but this time without the card. Submit both shots to this group. Museum staff will use the information on the index card to properly caption the image with the correct object information and credit line and the second, clean shot will then be used for Wikipedia.”

Wow, this was a silly idea. It was something we adopted from Wikis Takes Manhattan and it seemed straightforward enough in that situation where it worked, but ours was a different story.  Institutions needed to caption submitted images, so index cards to keep all the submissions straight and give us some context seemed like a good idea at the time. We knew it would be tiresome, but we didn’t realize it was going to be comedy and tragedy on so many levels.  Some participants came up with creative solutions to save time and trees: Trish used a spiral notebook and just flipped pages; yonghokim used a laptop; some used Photoshop to do something similar; there was the awesome whiteboard approach from Moria; and check out the home-grown whiteboard from The Wookies!  These inventive ways to deal with this just kept us smiling.  Once everyone added shots to the group everything got out of order anyway and we had a total mess to sort through (Flickr drawback!  More pros/cons on that subject down the page). We actually could have saved time on our end if we hadn’t used them—yes, Erin and I wanted to scream. Looking back, I had to ask myself…what was wrong with tagging, exactly?

Wiki LOVES art (end) via luluinnyc | Amy Dreher on Flickr

In the end, Amy’s shot above and this Twitter status from CJ say it all. This was hard on the photographers, it was hard on us and it was overall a total mess. Sorry folks, by the time we realized what was happening, we didn’t feel it would be fair to switch gears, but at least we all suffered through this part together. So many of you tried to help by creating sets to keep things in order, making sure accession numbers were on your photos and overall just being absolute troopers in the face of this nuttiness. Many, many countless thank yous.

You gotta freeze like ‘ya mean it…

If you are ever running a project like this, please make sure to freeze all lists, rules and deadlines once the competition starts and ensure everyone working with you knows the ground rules. Changes are enough to drive participants off the deep end. To see an example of this madness, see #2 here. We just can’t add items to lists or extend deadlines. The answer just has to be no.  This was something I knew, but sometimes I’d let things slide.  That was a mistake and I know better—if you were one of the participants affected by any of these changes, I apologize (sorry, Trish).

There was only one time when changes couldn’t be helped—Victor, I will never forget your face upon hearing that some works on the MoMA list had been removed from the galleries after the first week of the competition. It was kind of a funny moment for all of us. Thank you, everyone, for taking that in stride!

Flickr = Awesome + Problem

Flickr is an amazing tool and we just couldn’t have done this project without it. The ability to go in and machine tag images, so they could be easily sorted and categorized was vital. The ability to easily delete disqualified images from the group, to have multiple administrators and moderators—all awesome. The API provided a 2-day programming project to bring you the scoreboard. Communication via the discussion forum was a breeze. There are Greasemonkey scripts and they saved us major amounts of time. We could come up with a creative solution to close the submissions across 5 timezones (thank you for this idea, zyrcster). Most of all, Flickr allowed each institution to take part as group admins and allowed participants to get to know us and vice versa.

On the flip side, some participants were new to Flickr which meant things like licensing, adding to groups, tagging, bulk uploading, batch processing…all that was just new enough to be difficult.  CJ helped with a howtoFlickr and Ayelie pitched in with a photography guide (thank you both). We also ran into trouble when we saw participants using free Flickr accounts, uploading high resolution images and then maxing out their account storage (free accounts have limited storage). Users with Flickr Pro accounts didn’t have storage issues, but they curate their feeds and were not keen on flooding their audiences with massive amounts of standard art shots.  As mentioned above, we also had serious issues when images were added to the pool landed out of order.

Collaboration hurdles (btw, I was never very good at hurdles in track)

As I had mentioned, one of the objectives was to ensure correct information was migrating to Wikipedia and this meant creating a funnel of sorts. Everyone shoots and uploads to Flickr, museums have the opportunity to caption properly, so that when images are selected for Wikipedia articles information is correct.

Very early, we started running into a few issues. We noticed some participants uploading to the group and missing steps like adding tagging or changing licensing.  We’d politely e-mail or comment to try and explain what adjustments needed to be made, but we started hearing responses along the lines of “No worries, I’m really not in it for the competition, so I’ll just upload directly to Wikimedia Commons.” You can also start to see this come up over and over again in the forum. We started to realize this was an ideals thing, a cultural thing. I am not here to take sides on this issue, but I am here to say these kinds of issues on a project where we are all trying to work together is a bit problematic. Check out the killer example: Toransuke posts a question about what can/cannot be shot, we explain the situation. Then Toransuke mentions in this thread that in response to the previous thread someone FlickrMailed him to say “hey, just upload to Commons directly.” Toransuke works for a museum, understands why we might not be so happy about this, so he asks about it. The image below so perfectly captures what it felt like to be seeing these things and wondering what I got all of us into.

vaquilla by lawmoment via Flickr. All Rights Reserved.

It’s at this point, that I will ask you to e-mail me if you are thinking about a similar project, so I can explain the nuances in greater detail—this was a serious issue and not one we take lightly. I want to emphasize that there were many awesome participants who worked hard to understand and follow our labyrinth of guidelines—both Erin and I will be giving some serious props to these peeps in the next two posts.

Lessons Learned in Round 2:

• Have contact information for all participants. We asked everyone to register and this was essential. It made informing the group of any major changes and clarifications much simpler. This extra step could not have been more worth it.
• Freeze. No exceptions, no matter how tempting or simple the request seems.
• Keep it simple. Tagging would have been fine, index cards were overkill.
• I wouldn’t take on a project like this without a tool like Flickr, but I’m really lost on how to fix the other issues Flickr caused and they were fairly substantial.
• Clarify when you need to. Fix problems fast.
• Be sure to check the forums at all times to answer questions.
• Always provide e-mail addresses so participants can contact you offline if they need to.
• The nuances of collaboration were serious enough to stop any future participation from us.

OK, tomorrow we will focus on what’s been happening after the close of the competition and single out some really awesome peeps.

Hey! We are doing a lot of tech posting this week, so if this is not your thing, we will be back to more varied material in the blog next week. If this kind of thing interests you, let’s go after the jump…

Planning:

When we were approached to consider this project, one of the things that really sold us on the idea was the chance to work with the community at Wikipedia. As most of our readers know, community is something we take to heart at the Brooklyn Museum and the Wikipedia community is one we didn’t know and this seemed like a great opportunity to start that relationship. The gist from the wiki community was along these lines: “Do you know how difficult it is to spend a lot of time researching and writing an article, then find out you have no way to illustrate it?” I’m sure this issue is something we’ve all dealt with at one time or another, so we could relate. The original plan was to go it alone and host a scavenger hunt for photographs. Plan #2 shifted a bit—it seemed a little odd to just do this ourselves. Wikipedia is a global community, so why not do our best to try and involve other institutions in other areas? The original idea in Plan #2 was to engineer it as a museum vs. museum competition. We talked with small focus group to hear reaction on this and quickly learned that they had no interest in this aspect and they would want to shoot at more than one venue. We restructured into Plan #3 which was designed to allow participants to shoot at as many venues as they wanted, to shoot in teams or on their own and to compete against each other rather than organizations. We started with a few partners and a set of rules that we thought would work for most organizations, then opened it up for anyone to join us. In all, 15 institutions came on board.

Improve Quality, Prevent COI:

If you work in an institution, you are probably familiar with the reaction you get when you so much as say “Wikipedia” in this setting <insert big grin here>. Quality can be a point of contention (nice summary here) and a project like this can be a difficult sell internally because you have to get over this (big) hump. As a collaborator we wanted to recognize that the resource doesn’t get better without help and there’s something we can do about that, but also try and accomplish this project in a way that wouldn’t represent a conflict of interest (COI is a sin in the Wiki world, but there’s debate within our own industry).

Working with the Wiki community, we decided the best way to prevent COI would be to ensure the Wikipedia community were the ones that would create the thematic lists that would form the hunt and these would be based on things that Wikipedia needed, rather than our suggestions. On the quality front, we wanted to insure that each institution had the chance to properly caption each photograph before it was used on Wikipedia. 

Erin will be blogging with her own thoughts on the quality issue, so be on the lookout−she has a lot to say.

Early Logistics:

So, now we’ve got 15 institutions and now we need 15 scavenger hunt lists. Wikipedia is a decentralized community with participants almost everywhere, so the idea was to match each participating institution with a Wikipedian from their local area. The Wikipedian would assist by creating the list, work through issues at that particular venue and help plan events. Due to the decentralized nature of the system, this was a little bit easier said than done. Wiki contacts were found and deployed to institutions.  In some cases this worked extremely well (thank you!), but in other cases we were struggling. As we were getting closer and closer to the opening of the competition, some institutions were emailing me with no list, unable to get a hold of their assigned contact and asking what to do. In some instances we were totally prepared, in others scrambling. As we inched closer and closer to the opening of the competition, you should have seen the amount of DM’ing going on at Twitter (do you have? have you heard from? where are you on x, y or z?).

Early Problems:

In addition to the missing lists, we started to see a few other things happen as a result of the decentralization. For starters, an unsanctioned Flickr group popped up.  Why is that such a problem?  Well, for starters, we wanted this to be something you could look at in the end and see how much collaboration there was as a group effort.  There were more practical issues too, we could create one scoring solution that would work for everyone and a single group could be more easily managed than many.

More alarmingly than the group, we started to see various incarnations of rules appearing on Wikipedia. The rules were complicated enough, but to have different versions running around all saying slightly different things in different styles was a bit of a nightmare. Things started to get confusing and we hadn’t even made it to the February 1 start date yet. I spent January 31 making hundreds of edits to various wiki pages trying to get language and rules consistent and did this until the midnight deadline. Even after all the edits, I still didn’t catch everything and there’s a fantastic example of the craziness these problems caused right here.

Lessons Learned in Round 1:

• Focus groups are awesome (thank you Amy, Trish, Stephen for taking time)
• Work with the community to accomplish your goals (quality) under their rules (prevent COI)
• Much like what we know of Wikipedia already, decentralization is a great asset but it can also be one heck of an enormous drawback. To account for this, you need to allocate a lot of time to work through situations (we didn’t allocate enough). Decentralization created other problems for us and we will discuss that more in the next post, so stay tuned.
• Watch consistency errors like a hawk, because there will be a ton of them.

We will continue the discussion tomorrow, but in good humor, I’ll leave you with the image below and let your imagination take the reigns for a bit.

mad man by McGarry via Flickr

What would Erin and I do for love? Freeze, apparently (it was really, really cold and windy getting this pic)! We hear the weather is going to warm up this weekend, so it should be perfect—the Wikipedia Loves Art Brooklyn Museum meetup is scheduled for this Saturday. Come to Target First Saturday (2/7/09 5-11pm) and when you get to the lobby, look for our table with this sign. It should be a great opportunity for us all to say hello. And, hey, if you register in advance and bring your camera we’ll make sure you have a box of Conversation Hearts with your name on it!

My own personal props have to go out to Victor over at MoMA, who wins the gold star for bending over backwards to figure out the best way they could participate and still ensure everything falls into the public domain. Victor, that’s dedication! CJN212, thanks for the legwork over there. I have to say from an organizer standpoint, I couldn’t be more thrilled about how many institutions took the leap work with us on what will hopefully become a massive cross-institution community collaboration!

Not so super-secret meetup for the Wikipedia NYC Chapter. Topic of discussion? You guessed it: Wikipedia Loves Art. Wiki peeps and guests including Denise and Hazel from the MET, Victor from MoMA and moi. Check out that awesome ceiling w/ globe lights at Columbia University.

In the next week, we’ve got a lot of work to do to get scavenger hunt lists published and many of us are making final preparations for meetups. All details, including the lists will be published to the Wikipedia Loves Art Flickr group, so keep an eye on things over there (and congrats to us for creating what might be the longest Flickr group description…ever). There’s even a discussion getting started about the best way to shoot in museums to avoid glare off cases while working with no flash, no tripod restrictions.

Now that we have institutions, we need photographers! Please help us spread the word. Remember, because of the Film Society of Lincoln Center’s offer, there’s a way you can participate from almost anywhere in the United States even if your local museum is not on the participant list. Good luck everyone, we are looking forward to seeing your shots!

Wow, there was such a great response to my first post about prepping for Wikipedia Loves Art! Since that announcement, we’ve been joined by Art Gallery of New South Wales, the Carnegie Museum of Art, the Film Society of Lincoln Center and the Houston Museum of Natural Science. The overwhelming cross-institution support for the project has been fantastic and, as you can see from the Flickr Group, organizations are thinking outside the box about how they can participate, which is awesome. If you could see my inbox, you’d know there will be more museums joining in the coming days—exciting!

A quick note about an upcoming deadline. If you represent an institution and would like to participate in Wikipedia Loves Art, please contact me. The deadline for institution entry is this Friday, January 23rd, which will give us just enough time to get everything in order for the big month!

This is just a quick note to any of the peeps at cultural institutions who may read our blog. We are helping organize Wikipedia Loves Art, an event being held in February at museums across the globe. To get an idea of how all this is going to work, check out all the details and rules in the Flickr group. So far, in addition to the Brooklyn Museum, we have the MET from across the river…the V&A from across the pond…IMA right in the middle and LACMA over on the West Coast all participating. We will all be doing a lot of blogging about this as the date gets closer, but if you represent a cultural institution (large or small) and you have a decently open photography policy and would like to take part—please email me so I can get you hooked up! Many thanks—this is going to be really, really fun.