Data Hoarders Anonymous: Hello, my name is Broacher and

Nature Talk
1

I think I may have a data hoarding problem.

That is, in regards to iNat stuff, I shoot a lot and I am starting to have some trouble keeping my picture collection storage size under control.

Not that I am currently super-worried about this, but I have noticed that in the nearly 3 months I’ve been on iNat I have probably filled close to 130 Gb of photos on my drive. Even knowing that we’re blessed to be living in the age of such cheap storage options (I remember my first computer – 8k user storage, 1978), I have a tendency to hoard things a bit, and I feel I could do better. Scaling this into the not-too-distant future, I can see how things could get tight pretty quickly. In any case, I was hoping for some advice from the rest of you more-disciplined lot regarding data storage management.

My current system is converting my RAW files into DNGs (which are pretty huge) and then importing them into LightRoom which organizes them into the right folders based on capture dates. From Lightroom, I quickly cull the herd, so maybe a fifth goes straight out (too out-of-focus, incomplete specimen, mistaken commoner, etc.) and then I start dragging things around to group them, possibly keyword and file rename too (usually in batches).

Then a round of cropping on the DNGs and some ACR (from within LightRoom) on those to roughly tune the images as much as I can. But at this point, being DNGs, they’re still huge (over 100Mb each), because they’re really just cropped master copies of the RAW exploded into DNG space. So…

Then I open these as copies into Photoshop as 16 bit TIFs where I really do most of the final tweaking, and then, I lower the colour space to 8bit and save as TIFs, with compression bringing most images down below 10Mb. Then I delete the DNGs and any other RAW files.

After exporting the results to JPGs into my ‘Submission’ folder from LR, I drag these into iNat and start working out the ID stuff more thoroughly. Inevitably, I have stuff that I don’t think I want to keep as observations but I want to hang onto them, nonetheless.

If this sounds overwhelmingly technical, it probably is. I am just very fluid with image editing (should be, spent 30+ years learning it) so I actually don’t find it slow or go through any brain wedgies doing it all.

I also have some cameras (like the phone) which I have to toss into the fray, depending on the shooting experience, but that’s not a big deal, time wise, and it’s pretty streamlined by now.

For my main system, I use iDrive, an online service that regularly backs up my drives to the cloud, so I’m not overly concerned about data protection at this point.

What about other options? I don’t know this already, if you have an Amazon Prime account, it allows unlimited storage space to picture files (you pay for video though) and this INCLUDES RAW and DNG files. I have my phone set to automatically upload pics (via Wifi) using the Prime Photos app. It’s usually faster for me to download my phone pics from the Amazon folders than going through what I need to do to get the full images across by USB, etc.

So if you have tons of pic files, and good upload speed, this is not a bad deal. It’s not as flexible or maybe accessible as more sophisticated cloud storage, but it certainly cheap mass storage when you’re generating massive picture collections.

I would always advise keeping local copies first though, of course.

So sorry for yet again, another super-long post, but I thought I’d like to hear how others are handling the storage issue, and what procedures, services, workflows they use to manage things.

There is (of course) an environmental angle to all this – the more online storage we use, rely on – the more energy is consumed. So that’s something to think about.

What say ye?

6 Likes
2

I just buy hard drives and store everything there, both RAWs and edited JPG from Lightroom (e.g. 400Gb of RAWs from this summer), it can be dangerous, but I’d refuse having everything in the cloud for reasons of safety, security and environmental ones too. I use folders for season/year, and when everything is uploaded on iNat I move it to the disk to keep computer clear. I would never delete RAWs just for the sake of future editing, “eye” is changing, I don’t like what I did years ago and know more now, so I wish to have that ability to redo things. Nowadays there’re drives for many Tb, so if you shoot a lot, that may be a way to store everything with less headache.

7 Likes
3

I back up the photos on my camera onto my laptop and then onto an external storage drive that still has a lot of space, organized by the date I shot them. The photos on my iPhone get dumped automatically into my Flickr account where I can access them for downloading to iNat. Not the best organized process but I can usually find what I’m looking for. My problem is I’m not aggressive enough in deleting duplicate or sub-par photos.

2 Likes
4

Unless you do edit it in Photoshop, there is no real need for PNG and/or especially the large 16 or 8 bit TIFF. Why? Please also keep in mind iNat is reducing your images to a maximum size of 2024 by 2024.

2 Likes
5

We’ve all been there. In fact, most of us are still there!

7 Likes
6

Yes, and this is why I do a crop on everything as a first step. I have noticed that (particularly with macro work) with more active species, it’s better to shoot wider and ask cropping questions later.

Hence, the RAW files are usually much larger in pixels than the finals.

Not sure if you caught that it was DNGs, not PNGs. And 16 bits is great to have for edits as you have more tonal and colour range correction power. Not too significant, perhaps, compared to 8 bit, but if your best find is blown out by highlights or sunk into shadowland, you’ll be grateful for the difference.

7

My old laptop died (problem with the motherboard) but I took out the hard drive (1 Tb) and bought an inexpensive housing so I can use it as an external hard drive to back up my images. Before then I used only a mix of memory cards, memory sticks, and DVDs. It wasn’t a big issue since my old photos were pretty small. I bought a used dSLR in 2016 so I definitely needed to expand my storage. I leave some of my duplicates at my dad’s house so I have backup in case there is a fire at my place. I try to be pretty ruthless in deleting photos when I’m taking multiples of a subject. But also I probably don’t have nearly as much data as some others because of my limits due to my health problems. I also don’t take video very often because I can’t keep my arms/hands steady. I have a trailcam since 2020 but it only takes videos up to 1 min long.

3 Likes
8

Yes, and in particular with new species that you may not be too familiar with. Was that ID key hidden in the shadow/highlights? Is that the right colour balance?

I normally would keep all RAWs too, except my workflow requires the RAW files to all go through a denoiser app (PureRaw) and even though the results are huge DNGs, they’re much more ready for tuning/tweaking than the RAW itself, as I tend to shoot at 5000-10000 ISO for all my close macro stuff.

2 Likes
9

Yes it is massively overkill if you can directly export your final 2024 by 2024 jpg image from the RAW file with Lightroom. Including editing. And perhaps more importantly no going back if you already cropped.

I saw looking one of your images was reduced from 4000 to 2048 px by iNat.

10

And I am super aggressive in deleting duplicate or sub-par photos.

1 Like
11

will inat autocrop images that are too large in 2024?

12

No, the reduction is removing pixels until the longest side is 2048 (not 2024 my fault). If it is a landscape photo (not a square) the image suffers from more severe reduction. E.g. from 4000 to 2048 in the above example.

2 Likes
13

I installed a Network-Attached Storage (NAS) at home for photo management. It’s essentially a file server that one can install at home and your machine treats it as a ‘normal’ drive over a wired or wifi home network. Mine has a 8TB hard disk, which is mirrored by another 8TB hard disk (so that if one HD fails, the other has a real-time mirror backup image, with both hard disks living in the same NAS chassis), but I could have also chosen to just have two 8TB hard disks masquerading as a single 16TB drive (the NAS s/w takes care of this “under the hood”). You can of course choose different capacity HDs. Or if you wanted, you could purchase two NAS units, one at home and another off-site, and have automated off-site backups (which is better: but then, you’d need to make sure your internet service provider doesn’t ding you for excessive traffic if you’re managing a lot of photos).

My NAS (from Synology) makes configuring these different options very easy without things getting too technical. And you can even set things up (very easily) to access RAW (not sure about DNGs) files outside your home on a mobile device if you wish to show your picts over beer at a party or whatever. If you decide to purchase a NAS, read the reviews carefully, and make sure the manufacturer has devoted resources to making the unit not only easy to install, but easy to maintain (through s/w patches and the like), via a user-friendly configuration interface.

I found that using a physical internet cable to connect to the NAS much faster than through wifi because of the size of RAW (and DNG) files. So I do most of my heavy post-processing of RAW files on my laptop, and once I’m done with ~95% of the tasks, transfer it to the NAS, where the files appear just as another mounted drive to your software, and can continue post-processing as required using the NAS.

My Synology unit also has the option of configuring photos on my Android phone to be automatically backed up to its storage. So I’ve been able to combine the photos from my mirrorless camera with my cell phone camera and manage them by updating their metadata all through my photo management s/w (I use ACDSee), without worrying too much about different images living elsewhere in cyberspace.

3 Likes
14

but still auto
in other forums f.e. you have to manually crop the image to the correct max. size

15

I’m pretty ruthless about culling my images. Unless they’re really special - either visually or for personal reasons - I get rid of them pretty quickly. Also a good mindless excercise if you’re ever on an airplane. :-)

6 Likes
16

What if you save your image archive in high-quality 10 or 12 bit HEIC? The files are pretty small.

17

They have pretty inexpensive terabyte+ drives, I have several. By the time you’ve filled up 1-5TB of photos, label the dates/years(because its a lot more space than it feels like) on the drive, and get another one

2 Likes
18

This topic made me smile. I, too, have too many photos. In the last few years I’ve taken hundreds of thousands of photos, and I tended to keep nearly all of them. I filled up the hard drive on my desk computer early this year.

So I’ve started on an effort to remove photos. I’m going through them and asking myself, “Will I ever use this outside of iNaturalist? Or do I just love it?” If not, I delete it. This is, of course, easier for me at this point in my life than it is for many of you young whippersnappers. So far I’ve removed over 3,000 photos (from my “permanent” files, not counting the new ones I don’t keep), so I have a ways to go. But my computer runs better than it did a month ago.

7 Likes
19

You should not reduce the file size on you computer - always retain full sized files.

1 Like
20

Hello, my name is Lynkos and… yes, I certainly DO have a data hoarding problem. Or rather, I hoard data, but I don’t think of it so much as a problem as a resource for use by me and by anyone else who might find it useful.
I do use a fair bit of storage space, but the cost is now relatively accessible, so I don’t give it much thought. Will there be a time when I decide to cancel some photos? Probably not, because as well as being useful for “taxonomic” purposes, each image, even the poorest, contains a memory and information that may (or may not) be valuable at a future date and for a future purpose I am currently unable to imagine.
In case it’s of interest to someone, below is my workflow. As I wrote it, I realised it probably sounds absurd to many, but like Broacher, I’m experienced and agile in data/image processing and so for me it’s just a natural extension of the original act of finding the organism and taking the photo. I actually find it fun :-).

  • Download the raw images from the camera to my PC and import into Lightroom
  • Perform a first selection of the images worth processing
  • Change the default name to a coded name that indicates roughly the area where they were taken, date, camera and a progressive number
  • Convert the raw images to DNG
  • Georeference the images against the GPS track I always record and also tag with relevant geographical data (placename, municipality, etc.)
  • Process the images in Lightroom, including any crop necessary
  • Export a georeferenced low-resolution JPG for iNaturalist
  • Identify the subject
  • Add the name of the species to the filename and also as an IPTC metatag, together with the family
  • Export a low-resolution JPG archive copy that can be viewed on multi-platforms (particularly necessary when the original image has been heavily processed/cropped) and stack this with the original DNG
  • Archive the DNG/JPG stack in folders with a taxonomic structure stored on an external hard drive (with two hard copies and one cloud copy)

Long live the data hoarders!!!

1 Like