Data Hoarders Anonymous: Hello, my name is Broacher and

I’m still digging up slides and early digital photos I took more than 10 or 20 years ago that I submit to INaturalist. Some I can’t pin down to a precise location but many I can.

2 Likes

You shouldn’t need to convert the RAW images to DNG. Lightroom handles RAW just fine.

I don’t much like Lightroom, but I do use it periodically and have no issues with it reading and filing Sony, Nikon, Canon, & GoPro (the camera systems I use at home and/or for work) RAW images.

Dropping the DNG step will save you time and a lot of storage space.

You’re right, there is no real need for me to covert my files to DNG, but I choose to do so for the versatility of DNG and its readability on a wide range of platforms, hopefully for a very long time to come (the same can’t be said for many other proprietary raw formats). I also use various cameras with different raw formats and I prefer to have all my “digital negatives” in the same format. The DNG files are in any case slightly smaller than the CR3 raw files from my Canon camera, so I actually save some space (not that that’s of great importance).
Batch processing a whole day’s shooting takes less time than enjoying a well-deserved coffee/tea/beer on my return :-).

So say we all!

1 Like

It is interesting to see everyone’s workflows and approach to storing photos. Thank you.

I have multiple cameras, including 3 Canon EOS digital. I shoot in RAW plus low-res jpg. My workflow is to download and sort into folders for date and location. The folder names are the date and a location keyword or possibly a keyword for the most interesting thing I saw that day. I do an immediate backup to a portable hard drive. Then I look through the photos in Adobe Bridge: assign keywords, delete anything really bad; select one photo for my website; select what’s going on to iNat; etc. I do basic edits in Camera Raw then open into Photoshop where I do further editing, including cropping and file-sizing. Photos with minimal editing are saved with the original file number plus codes for cropped (c), edited (e) and resized (r). If I do any extensive work on a photo I’ll save a full resolution TIF version too. Then when done, I move the photos into subject folders for future reference/use. These are also on portable hard drives - I don’t store anything on the computer’s C-drive.

This works well - now. I had a period when I was too busy to be consistent and missed a lot of indexing and sorting so am going back retrospectively through everything digital (from 2005) although I’m getting more selective in what I keep as I go back in time. A lot of the photos “dumped” into subject folders will need more sorting and identification but at least they are now where I can find them.

I don’t use Lightroom - I didn’t really “click” with it and couldn’t get my head around importing images “into” Lightroom, and was totally put off when I realised that it couldn’t cope with moving images to different folders when not “in” Lr.

I don’t use cloud services because I’m mean with data costs (I’ve taken over 200gb of photos so far this year so far) and also don’t have a high level of trust in them. There is of course a risk of loss of the hard drives I do use, but I accept that.

Here’s one. My husband, The Doctor (Ph. D, not MD), got a call a couple of years back from a company for which he was the startup database administrator, back before he started grad school. (He was the third employee of this business, after the founder and founder’s business partner. Now it’s international.) The caller was one of the IT staff; the current DBA was wondering if The Doctor would be willing to consult on some changes that they were making.

Now, he started grad school while he was still working for them, but that was only up to the first Master’s. Since then, there’s been a cross-country move, a second Master’s, another cross-country move back, the Ph. D., a post-doc, a university position (which kind of imploded due to administrative politics), and now a private sector gig that he really likes, even if he misses some aspects of academia. Short version, it’s been a minute since he worked for these guys, and he hadn’t looked at the database (that he built and that they were still using) in all that time.

But, they were really good to him while he was working for them, and he was happy to help out. He pulled up the files, studied them for a couple of hours; and, because he’s a stickler for documenting and annotating his code, he was able to answer all of their questions, make suggestions on the tweaks that they wanted to do, and write some of those tweaks himself. He even added a few more annotations to the code files, and sent them along for testing. Bear in mind that they were still using the database code that he wrote after all this time because it still ran like a champ, was easily updated, and all of those notes in the code files let everyone coming after him see why he did something in a specific way. When he builds a database, he builds for the ages. :grin:

So there’s one instance where data hoarding was quite useful. The check that they set him for consulting also paid for a very nice “us day”.

2 Likes

Apart from the insane energy use of cloud servers.

2 Likes

Think of Lightroom assigning your photos in different “virtual” folders based on tags or keywords in your metadata. E.g. the day is a folder. Each taxon/name is a folder. Each camera or lens, say macrolens can be a folder. Each location can be a folder. The possibilities and the time saving is massive.

I only use Lightroom for editing, and delete things from it after (so can easily see what is left to do), so moving folders, etc. I do outside of it. You can try different versions of Lr, as it and Lr Classic are fairly different with how they handle files, new one uses cloud and classic uses copies of files.

This is correct. I just want to point out that the image you see (on LR) is essentially a DNG. If you edit it in Photoshop you send it directly from within LR as a DNG. The two are so well integrated. Each image is saved as a preview (also a DNG) which make it accessible as well as save a huge amount of space on a old MacBook SSD (with the original on harddrives).

Perhaps pointing out with the latest CR3 RAW you can also select compress Raw with will save about 40% space. The data lost is minimal (as DPReview tested it comprehensively: the only additional noise visible is when they are 4 stops underexposes). However, if you convert that back to DNG the image file will be much more (about 30%) more. This is because DNG is also a lossy format.

Happens to me constantly. I don’t use something for 5 years, I give it away, then whammo, I need it. Particularly with clothing. When I moved to NC I got rid of some old heavy winter clothing. A few years later, I had to travel to North Dakota. In the winter. Cold. Really cold. Same with some kitchen things… and other things. And sewing things… I have even found I needed something I hadn’t touched in 20 years. The weird thing was that I both remembered that I had it and I was able to find it (it was carbon paper–I don’t think most people even know what that is nowadays. )
And as you might imagine, I save most of my RAW files except those that are hopelessly out of focus. So I guess I am a data hoarder too…

2 Likes

Ever listen to the Freakonomics podcast? They often mention the ‘loss aversion’ bias in their shows, which describe a very universal, deep-rooted behavioural bias that drives a surprisingly huge part of the investment world.

Simply put, we have a greater tendency to hang onto things that we have already invested time or money in, then to get rid of them and replace them with something that’s proven to be better.

I think I have the copy of this one great podcast about this, um… somewhere on one of my old hard drives. If I could find the right box.

2 Likes

I know what carbon paper is, but that’s because I use it to transfer embroidery patterns to fabric. :slightly_smiling_face:

3 Likes

Do you use it in school or not? Here it’s both a toy thing for first classes where you can copy things from books and later for cloth classes too.

I never used it for sewing classes when I was in school. (We only had a couple of sewing classes offered to us from first classes to high school, and only one was required. I don’t know if that is common across the United States, or just in that specific school district.)

The embroidery that I do comes from my history research, but it’s mostly for my own enjoyment. Some of the patterns that I have collected, though, are between 90 and 130 years old. I’m slowly scanning them to digital formats so that I can make them available again.

2 Likes

No, i am not in school. I was using it to trace part of a map. :)

What a great thread! Yes, long live the data hoarders!

My workflow is very similar to @lynkos, just with darktable instead of Lightroom:

  • download from camera or phone into darktable
  • prune unneeded photos
  • geotag everything with gpx track from GPS unit (if I had it on me, otherwise a Cyclemeter gpx track from my iPhone), and tag photos with the source of the GPS info
  • tag with taxon names and places and iNat observation fields (iNat’s Uploader conveniently grabs all of these automatically)
  • export JPG to upload taxon photos to iNat and other photos (habitats, etc.) to Flickr
  • back-up originals to local NAS and AWS Deep Glacier (cloud)

Admittedly, most of my photos from recent years are not tagged, and so not yet on iNat, because I also write down all the tag info (species place etc) in consistently formatted geotagged and timestamped notes on my phone, when I take the photos. I know I can use these data to automatically tag all my photos, but I haven’t had the time yet to script that. Soon!

3 Likes

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.