Andy Gapin

A couple of months ago, I posted about trying to figure how to organize all my photos, roughly 30k, I think. Even though I just got my first DSLR six months ago and got seriously into photography, I’ve always been a bit of a picture taker. But I never go back and look at pictures and even though my old organization technique wasn’t awful—usually a folder with the event name and some sort of date—I always had trouble finding the right picture for something when I didn’t have a specific one in mind. So I decided that one of my summer projects would be to figure this about and go through all of my pictures and tag them.

I spent some time trying to figure out the best way to do this and now that I’m about 8,500 pictures in, I think I’ve got it down pretty solid. Or at least it works and it’s already been useful for finding stuff (like pictures for the wedding site my fiancée and I are working on).

If this is the kind of thing that bores you, you should probably quit reading, but if you’re interested in photography, organization, and workflow, read on! Oh, and I’ll note that I am strictly a hobbyist when it comes to photography, so all of my photos are personal photos, nothing is professional or anything like that.

Software
What makes everything below possible is a bit of software that I was already really in love with. I’ve been using Adobe Lightroom and it’s simply amazing. Even though it is just a single application, it provides both strong post-processing and strong library management features that are both good enough to almost entirely eliminate the need for a second piece of software.

For years, I was a big supporter of Google’s Picasa which is great for photo management, but the processing and editing side of things is practically useless. I also found that it was very slow and laggy on my Mac. The feature I liked the most was the face detection, but, as good as it is, I found that this left a lot to be desired. If someone in the photo wasn’t facing the camera, Picasa doesn’t pick it up as a person. So if you want to be able to tag everyone in every photo, you still end up going through every photo manually.

With Lightroom, I just tag everyone as keyword in each photo and it’s entirely manual since Lightroom doesn’t have face detection. This would probably be an annoyance, but since I am going through every single photo anyway and keywording other stuff, I’m already in there, I can do it myself.

Other than that, Lightroom does everything that Picasa does…better and snappier. I have plugins for Facebook and Picasaweb so those are covered and with more options.

Workflow
My workflow is simple in theory, but has a few steps. Whether photos are coming in fresh from my camera or I’m working on organizing older photos, every group of photos gets three passes. One for processing, one for keywording, and one for rating.

When I import the photos, I first import them to a local temporary folder in my Mac. Eventually, everything ends up on my file server, but I’ve found that when processing the RAWs things are much faster when they files are local rather than on a network drive. RAW files are converted to DNG during the import so this step can take a little while, but it’s easier to do it here than anywhere else. At this point, it seems that DNG has caught on enough that it’s a safe file format to use for archival and I really like the idea that all data is stored in a single file rather than needing to keep an XMP sidecar file with each photo. JPGs stay as JPGs which is fine because Lightroom can store all of its edits and info in the metadata in the file, making sure that the original photo data itself is untouched.

Each photo gets processed in my first pass through the bunch. Terrible shots that are either completely out of focus, so badly exposed that they can’t be saved, or pretty much useless for any other reason are marked as rejected in this step and I don’t bother doing any processing on them. Photos from my DSLR obviously take quite a bit longer to process, but even photos from my point and shoot and phone still get run through this step incase they need some tweaking.

Once the photos are processed and ready to go, I zip through them and keyword the hell out of them. I’m trying to be very thorough with this step and erring on the side of doing too much. I also make sure that all of the location fields are set for each photo and the timestamps are correct…or at least close.

The timestamps have been a little of a pain in the ass. I was surprised to find that only a handful of really old photos didn’t have timestamps that (I believe) are correct. I was afraid that that was going to be a problem. But there have been other problems. One issue is that some events, like my Disney trip this year, have photos from multiple cameras and the time isn’t exactly synced up on them. This leaves them sequenced oddly unless I fix them. That Disney trip has photos from my DSLR, point-and-shoot, my fiancée’s iPhone, and my iPhone. I had to adjust a lot of times.

The second issue is that my point-and-shoot was somehow off by 12 hours (am/pm wasn’t set correctly) for at least six months. I’m not sure how that happened or exactly when it started, but it’s something I’ve needed to keep an eye out for as I go back through my older photos and organize them. The third issue, and the one that seems to be the most annoying, is that a lot of times the timestamps from my iPhone photos don’t make it over to Lightroom properly. I’ve noticed this is especially so when I use third party camera apps. Lightroom imports them and names the files correctly, most of the time, but shows the wrong capture time. I’m not sure what causes this, but the capture time ends up getting set to the import time. It’s very annoying.

In the final pass, I assign each picture a rating of 0-5. Since I’ve already been through all the pictures at least twice at this point, I’ve got a general idea of how good each shot is in comparison to the rest of the group. A rating of 1 or higher means that it would at least be good enough to export and/or show other people. A 2 rating is generally a decent shot that would definitely make a slideshow or Facebook or whatever if it’s a larger group of photos where it wouldn’t make sense to show everything that’s a 1 or above. Rating a photo with a 3 means that it’s one of the best of the group. 4 and 5 are reserved only for my best shots that are really good in terms of content and quality. These are my favorites and if I pulled up all the pictures of a particular keyword, these would be the best.

Also during this step, I group shots if necessary. If I snap a bunch of the exact same thing from the same angle, I’ll usually group them together with the highest rated one on top.

After that, I make sure all the metadata is saved to the file and move the files over to my server for safe-keeping.

File organization
This was one of the toughest things to make a decision on. Like everything else, once you start down a path, you’re kind of stuck, but this would undoubtedly be the toughest to make changes to.

Each photo is named with the format of YYYYMMDD_HHMMSS_O. I’ve never read of anyone naming their files like this, but I like it because even if this is all I have to go on, I can figure out very closely where a file should go. It also automatically keeps the files sorted chronologically when viewing them outside of Lightroom. The ‘O’ at the end stands for original. That’s my indication that a file is the most original version of a photo that I have and I should never make any destructive changes to it. Any other version of the file will get an ‘E’ for export or some other distinction.

My folder structure is a bit more standard. There are top-level folders for year and then folders within that for month. Within each month, I have a folder for each “event” or group of shots. Events that span days are broken into a folder for each day and if different groups of photos are taken on the same day, I break them up into their own folders. Once you’re in Lightroom, the folders don’t really matter, but at least I’ll always know where to find a particular photo I’m looking for outside of Lightroom as well.

The one issue I found with this structure was that I have tons of photos of a specific subject that are thrown in a single folder. For example, I have hundreds of pictures of my cat that I’ve randomly taken over the last couple of years. They’re all just thrown into a single folder because they’re pretty random. At first, I had a “XXXX” top-level folder with some subjects in there, but I decided that was going to end up being a bad idea, so everything gets to fit into the above structure. This means that if I take just one photo of my fiancée while she’s sitting on the couch, it gets it’s own folder (ex. “Danielle hanging out 2010-07-09″). I’ve ended up with a bunch of folders with just one photo in them, but at least it keeps everything in the same file structure.

Each event folder has at least one folder inside of it. All of the most original versions of each photo are in a folder called “original” and, again, originals do not get any destructive changes made to them. However, these originals are where all the metadata and edits are stored–the edits are saved as metadata by Lightroom, the image data isn’t touched. Exported versions would go in “processed,” but I haven’t been exporting processed versions of everything so many folders don’t have this. There may also be a “video” folder if video was taken and a “PSD” folder if I’ve done some work in Photoshop…which is extremely rare.

So an example of all that would be something like “2010/06/Father’s Day 2010-06-20/original/20100620_143523_O.dng.” You’ll notice that the event folder has the name before the day. I kind of screwed up and was way too far in when I realized I didn’t like it so I’m kind of stuck with it at this point. My only reasoning for this was that in the left sidebar in Lightroom, you can only see about 12 characters of the folder name at that level without expanding the width and taking up more workspace. I thought it would be more useful to be able to see a few letters than always being stuck seeing pretty much just the year and month. It wasn’t until too late that it hit me that I could just hide that pane altogether when I’m not actively looking for a folder and keep it fully expanded when I am. Fail.

Keywords
I’m trying to be very thorough with my keywording. I’ve got tons of them organized hierarchically in Lightroom, but they could use a little more organization at this point as some groups of keywords expanded way past what I thought they would. I’ll take care of this soon.

Every photo is tagged with the full names of every person in the photo (that I know, of course) and various other content is tagged as well. Anything that I think I could ever want to use to pull up a photo is tagged. Places and locations beyond the simple location fields, landmarks, things that are being done, things people are wearing, references to anything, ingredients for food pictures, objects. You get the idea.

One of the more odd things that I tag is the name of the t-shirt I’m wearing in pictures that include me. It sounds weird, but I’ve got a TON of t-shirts and every once in a while, I actually find a reason to pull up pictures with a certain t-shirt or brand/site/store. For example, Threadless has done contests involving pictures of people wearing their shirts. I can very simply pull up these photos by the “threadless” keyword. If I want a specific shirt, the name of the shirt is a keyword under the “threadless” keyword.

That’s it!
That’s really about it. I’m working on going through my entire collection and I’m making progress, but it’s very time consuming. Getting all my old pictures organized was supposed to be one of my summer projects, but this will definitely extend a bit past that.

If you’re into photography and have a method to all the madness, I’d love to hear about it!

Normally, I don’t do any Summer projects, but I think it’s time to change that. I always have a ton of things I want to get done, but the bigger ones never make it. Tasks that I can’t do in a couple of hours tend to just sit on my to-do list without ever being started. Half of it is because I just don’t have the time for most of these things, but the other half is laziness and lack of motivation for tackling these things. However, this Summer I want to do a few of the things on my list. Hopefully, it’s not too much, but I’ve got four projects that I want to do.

First, I want to get on the task of organizing all of my photos. I was thinking about this a lot last week and wrote an entire entry about it here, but I think this is a doable task. I’ve mostly worked out how I’m going to tackle the project and I’ve actually started already, which is good because this will probably be the project that takes the longest to complete. I’m just a hobbyist when it comes photography, but I’ve been getting into it a lot since getting my DSLR camera. Even before that though, I took tons and tons of pictures, I have tens of thousands of them on my computer. But for all of these pictures that I take, I rarely ever go back and look at them later on. Not only will this project allow me to get all of these photos better organized so that I can find anything I’m looking for later–I’ve got a pretty comprehensive keywording system that I’m planning on using and blogging about soon–but it will also be fun to actually go through and look at all of these pictures that I’ve taken over the years.

My next project is to actually clear my to-do list for the software I develop for the radio station. Okay, not the overall to-do list, but the one that’s been growing recently with about two dozen items on it. I started development on it again for the first time in a while this year and I have a pretty awesome new feature to push out, but there’s a lot more I want and need to do with the software. It’s hard to find the motivation to come home from developing all day at work only to do the same thing at home, but I think over the course of the summer, I can push out everything I want. My general process in the past was to do a ton of enhancements and then push them out in big groups. This was great at first, but it ended up causing some things to sit forever before going out. My new approach is going to be to push out each enhancement and bug fix as I finish it. This will get the new stuff to the users much quicker and also make it easier to check things off my list to make it feel like I’m actually getting things done.

I’ve got one more nerdy indoor project that I want to do, this one is not super important to me, but I think it would be a great learning experience. I want to develop an iPhone app. I downloaded the SDK a few weeks ago when I couldn’t sleep one night, but I haven’t had a chance to actually play around with it. I don’t know Objective-C yet, but I don’t expect it to be too hard to learn given the fact that I do software development for a living. I’m still trying to think of a good idea for an app though. It doesn’t to be the next big thing or even make me any money via the App Store, but I don’t want to make another to-do or fart app or anything else that will just get lost in a sea of similar apps.

And my last project is going to be to take enough pictures to be able to walk away with at least one “useable” picture everyday. I think this will be the most fun project out of the group. I know some people do photo 365 projects and make it a point to shoot everyday, but I’ve totally botched that idea for the year already, so I’m going to make it a Summer thing. I’m not going to try to limit myself to any particular theme or anything, but given that it will be Summer, I think I want to try some action photography. Maybe I’ll take some pictures of my soccer buddies one week or I’ll just go hang out at the park one day and take pictures of what’s going on. Who knows!

So that it. It’s a lot of nerdy stuff that will keep me indoors, but given that I’ll be playing soccer once a week, running a few times a week still, hopefully mountain biking here and there, and my fiancée and I finally got a table and chairs for the backyard so we can eat dinner out there, I’ll still be spending plenty of time outside. And besides, once the temperature starts to approach 80° I’m pretty miserable being outside anyway.

Recently, I added a to-do for myself to figure out a better way to organize my photos. It’s going to be a bitch of a task, I’ve got tens of thousands of them spanning over the last decade or so. For the most part, I’ve got everything grouped into folders by event or something else like person or subject for more open-ended collections of photos that don’t really belong to a specific event. So for at least 90% of my photo collection, I can work by folder and not individual files.

But this still presents a big problem for me, one that is probably somewhat unique because of the way I have my computers set up at home. I have a file server where I store everything, but I want to access the photos from at least three different computers (my MacBook Pro which is my main machine, my Linux/Windows dual-boot desktop, and my Windows 7 netbook). I also have photos that come in from my point-and-shoot, my iPhone, my fiancée’s iPhone, my fiancée’s point-and-shoot, and my dSLR (I save RAW files and JPEGs of the processed photos). That’s a lot of devices to deal with and since I’m just a hobbyist when it comes to photography, everything is personal, I don’t feel the need or want to separate anything. I want everything to fall under the same organization.

My current workflow has been to import photos from my dSLR using Lightroom (recently, I’ve been converting the RAWs straight to DNG and will eventually convert all my existing RAWs as well). These photos go in a sub-folder called “Raw” under a folder for the event or subject. Once I do all my processing, I export JPEGs into the event folder. Photos from our point-and-shoots and iPhones just get dumped straight to the event folder. From here, I tag faces and add captions to the JPEGs in Picasa and then post select photos to Facebook or a blog of some sort. I’ve found that Picasa’s face-tagging feature is pretty great. It does a great job of detecting faces and guessing which ones belong to the same person. The user interface for it is a little clunky, but it gets the job done. It’s incredibly convenient to be able to just select a person and pull up all the pictures with them in it…or even better, all the pictures with that person and myself.

That’s been kind of it, really. It’s not terrible, but it’s a pain and it kind of ties me to one machine. I end up having to do all my Picasa related stuff on my MacBook since the face tags aren’t stored in the files, they’re stored in a local Picasa database and a separate file in the folder containing the photos. At least with Lightroom, all my edits can be stored in the DNG (or XMP file for RAWs) so they can easily be accessed from any other machine or application that supports it.

This whole process is very clunky and relies a lot on data that is stored on only one machine and is proprietary to a certain piece of software (Picasa). I can’t do anything at all with the face tags in any other application and if I even want to be able to use them in Picasa on another computer, it’s a big pain in the ass. I honestly love the feature set of Picasa, but I’d really like to ditch it altogether. It’s very slow and laggy and has a lot of odd quirks that have gotten to be too annoying to put up with. It’s great for free software, but I’m ready to move on.

The other frustration is that none of this data is tied to the original files–there is no way that I will let Picasa touch my DNG/RAW files–so if I decide to export from the original again (either because I reprocessed it or want to export as a smaller file or different file type), the face data doesn’t make it over to the new file. Essentially, the issue is that for photos from my DSLR, Picasa only gets to touch the end result, not the source. Photos from anywhere else aren’t as much of a problem because the JPEGs are the only versions, but I would like to be able to store as much data with the most original version of the file as possible.

So back to the whole wanting to reorganize my photo collection thing, I really want to clean everything up and get a good system going, I just don’t know how to do it. This is something that I’d probably continue to put off if not for realizing last night that I made a couple horrible assumptions a few months ago that are going to cause me to have to reprocess almost every shot I’ve taken with my DSLR. I totally assumed that when I imported my RAWs into Lightroom, it would automatically set the white balance to whatever setting was on the camera when it applied the presets. I mean, it has this data and there’s no reason not to default it to that. At any rate, it didn’t do this and I didn’t notice at all that it wasn’t set. You might wonder why I didn’t notice this immediately by looking at the pictures. Well, I just assumed that I wasn’t taking good pictures or using the lighting very well since I’m a complete newbie. So since I didn’t know or think to look to see if my white balance was set correctly, I adjusted as much as I could with the other settings. After I realized this last night, I went through a handful of pictures and set the correct white balance and they immediately looked better, but I had to kind of revert back to the original and remove any other changes I made first and then go from there.

That’s entirely my fault for being a complete idiot and not noticing that as well has just assuming that I was sucking. Not all of my photos need to be reprocessed, it’s mostly just the indoor ones, I think. But it’s still a lot. I’d really like to take care of this soon though and re-export them so I can put the whole thing behind me and not have to worry about it later on when I want to grab one of these pictures. Since I figure there are about 4,500 photos that I’ll have to sort through during this process, I might as well just reorganize my whole collection at the same time and improve my workflow. I’ll have to re-export my JPEGs anyway. Luckily, I think that Picasa stores the face tags based on the file name so as long as I export with the same name, Picasa shouldn’t even know the difference. I guess this is one plus to the fact that Picasa doesn’t store this in the file. And since Picasa also writes these tags to a text file in the folder where the pictures are, I should be able to move the whole folder around and then have Picasa re-read the tags based on that file. I’ll probably have to rebuild its database entirely though. But after all of this, I’m still stuck with the face tags being tied to the JPEGs and Picasa.

I’m thinking that the best solution will be the say screw the face tags and screw Picasa. I’d do everything in Lightroom and just add names as keywords to the metadata on the original files, along with keywords for the event/location/subject/whatever. Then just preserve this when I export to JPEG. That should at least provide keywords that will be visible in any photo application and on any computer. I think. I won’t have face tags per se, but I really don’t care much if there are names attached to the specific faces in the photo as long as there are names attached to the photo itself. I know who the people in my pictures are, I just need to be able to search easily. And as far as exporting to Picasaweb or Facebook, I’ve seen plugins for Lightroom to handle this so I could pretty safely rely on that as well.

Since this method would be storing everything with the original file (stored on my server), it will also allow me to make changes and edits to any photo from my netbook, MacBook, or desktop without worry. If I’m away for a few days and dump my photos to my netbook and process them, I can just drop the folder onto my server when I get home and then add it to my Lightroom library on my MacBook. Nice and easy.

Though after that, I’m still left figuring out a folder hierarchy for where the photos are actually stored. What I have now isn’t terrible, but it’s not great either. There’s organization and I know where everything is, but it could be better. I know a lot of photographers store based mostly on date (e.g. /YYYY/MM/YYYYMMDD-filename), but since I also like to keep those open-ended collections based solely on a subject (like my cat) that don’t need to be stored by date, I have no idea what would be the best way to do this. I’d also like a clean way to deal with the fact that I’ll have multiple versions of the same photo (original, exported JPEG, differently processed versions, etc) that I would like to be tied to the same tags/keywords/etc. without looking like duplicates. I like to keep JPEGs of everything because it’s much more convenient for when I have to grab a quick photo for something. I guess I could just have a personal policy of only having the originals imported and visible to Lightroom, but then I’d probably want to dump all photos from all sources into the “raw” folder and then export everything, even photos that were originally JPEGS as well, into the main event folder when I’m done my post-processing. Maybe not a bad solution.

But anyway, I’m really curious as to whether or not other photo happy people have any of these same issues and what solutions they’ve come up with. Or even separate from these issues, how do other people effectively manage a huge collection of photos. If you’ve got anything, let me know!