Human-in-the-loop AI for wildlife camera trap annotation

In the summer of 2021, after three years of volunteering with Code Nation as an instructor and a brief stint with Code for SF, I was ready for something new!

Wading through the web on a mission to find a new nonprofit, I stumbled onto a LinkedIn post from Felidae Conservation Fund. As their name suggested, they were a wildlife research & conservation organization that studied wild cats, specifically mountain lions.

For someone who grew up on David Attenborough, the word “conservation” alone was enough to spark some excitement. But when I learned they needed a cloud-based AI solution to improve their data pipelines, I was sold!

In this post, I briefly chronicle my journey designing and building this AI-powered, human-in-the-loop system to manage Felidae’s camera trap images & annotations.

If you’re interested in the role of technology in wildlife conservation, building computer vision applications, or working with nonprofits, you might find this post useful. And if you want a high-level overview first, check out this slide deck.

1. smile cougar, you’re on camera!

Before we jump in, let’s quickly cover what camera traps are and how they help wildlife research. Here’s a tl;dr from this wwf post.

• Camera traps are cameras with sensors that detect activity in front of them and automatically trigger an image or a video capture.
• They let wildlife researchers gather data about our fellow creatures in a way that doesn’t disturb their natural behavior.

Now, you’re probably thinking, “but why can’t we just walk through the forest, interview these critters like we do with our census, and high five them at the end?” To that, I’d obviously say, “Great idea! I love to high five cougars!” but I presume the ecologists would be rolling their eyes.

On a serious note, “walking around the forest and gathering data” is actually a valid technique, one I volunteered for over a decade ago. But it’s an expensive, labor-intensive process. It requires folks to tip-toe around the forest with the risk of, in my case, getting trampled into a pancake by unsuspecting, startled elephants!

Different ways of collecting data come with different tradeoffs. For Felidae, camera traps are an excellent source since cats are typically shy and nocturnal. So, to support their research, Felidae operates well over a hundred camera traps, primarily in the San Francisco Bay Area, that generate hundreds of thousands of images every year!

2. the data supply chain

While camera traps are great, the raw images they generate have some ways to go before turning into the sweet, sweet science we all love! Typically, researchers need to know

• where the image was taken - this is collected when a camera trap is set up by noting its latitude & longitude.
• when the image was taken - camera traps generate a timestamp for each image indicating when it was taken.
• what animal is in the image - once images are collected, annotators look at each image and make a note of the animals in it.

As you probably guessed, manually annotating these images is by far the most time-consuming step in this pipeline.

But factor in camera misfires, errors, occlusions, blowing leaves, moonwalking leprechauns (ok, I may have made this one up), etc., and you’re left with a lot of images that don’t even have any animals in them!

And if that isn’t enough, just to mess with researchers, these camera traps occasionally reset their internal clocks to some time in 1970, messing up all the timestamps. They just can’t catch a break!

To manage this gnarly pipeline, researchers and organizations set up different workflows depending on their size and access to resources.

To get a better sense of this, let’s take a closer look at Felidae’s data pipeline, the problems they run into, and the suite of tools they use to address them. Broadly, their workflow can be split into three tasks.

1. Set up camera traps - Camera traps are laid out at predetermined locations. To track them, Felidae uses Google Sheets to link each camera trap’s ID with its latitude & longitude.
2. Collect images - At regular intervals, memory cards from these camera traps are collected and sent to the office, either over dropbox or by physical mail. Felidae staff then save these images onto a physical computer, along with information about the source camera trap.
3. Annotate images - Felidae uses a Microsoft Access based tool called Camera Base to handle annotations. Since it runs on a single physical computer, only one annotator can use it at a time. So, annotators book slots on a shared Google calendar and then remotely log in using TeamViewer to annotate images.

Once images are annotated and cleaned, the data is exported out of Camera Base and distilled using R/Excel to answer their research questions. This entire process is powered by a dedicated, hardworking group of hundreds of volunteers and a small set of interns & full-time staff.

3. up, up and away to the cloud

When I first learned about Felidae’s workflow, this was my reaction… 😳😧💀 and my guess is, you’re probably not far off now that you’ve learned about it too! I remember thinking, “Microsoft Access? TeamViewer? Shared Calendar? Did I just time travel?”

But they knew that already!

They were painfully aware of the inefficiencies in their workflow, especially the one-annotator-at-a-time bottleneck and the considerable backlog of un-annotated images it left behind. Not to mention the risk of a hard disk failure and losing all that data!

They also knew that moving to the cloud was the answer. They had taken a shot at it but were stuck without a clear path forward.

3.1. the holy grail

Let’s take a step back and look at the workflow again to find areas to improve.

The first, and the most obvious one, is moving away from Camera Base as the annotation tool. To be fair, it probably wasn’t built with simultaneous users in mind. A web application that replaces it would remove the bottleneck by letting multiple annotators work simultaneously and independently.

The second, less obvious one, is using Machine Learning, specifically Computer Vision, to automatically identify animals in the image. This technology has advanced significantly in recent years and, with the potential to massively improve annotator throughput, it is widely considered a “game-changer” by wildlife researchers!

So, from Felidae’s perspective, progress in these directions would improve their workflow significantly and free up time to do more sciency things.

But, as Morpheus famously says, “there’s a difference between knowing the path and walking the path”, a difference that’s even more dicey for nonprofits.

3.2. the nonprofit conundrum

It should be no surprise that nonprofits are resource-strapped, especially the smaller, more local ones. They have a hard enough time funding their core work, so when it comes to investing in new technology, they tend to play it safe. Even when they lean ambitious, they often lack the in-house expertise to know where to start. As a result, they look to volunteers to come in and lend a helping hand!

While this might look like “free work” at first, it comes with costs that aren’t immediately apparent. Reality, unfortunately, doesn’t hand out free lunch!

So, what are these costs? They’re not that different from what a for-profit organization might face, like attrition, onboarding, decision missteps, etc. The magnitude, however, is significantly amplified. Why? Because volunteers tend to have a day job, and their availability & commitment levels are highly variable!

The pool of available volunteers is also often shallow, with a high degree of variance in experience, expertise & expectations. So when volunteers flake or ghost, projects tend to stall, change direction, or in the worst case, fail completely.

3.3. highway to integration hell

Felidae’s tech endeavors started in 2020 and had since seen a string of volunteers work on different parts of their broader vision. The outcome was multiple threads of work at varying levels of maturity, spanning a range of tech that closely mirrored the backgrounds of the people who built them. Here’s a quick look at what they were.

• A database schema connecting camera traps, volunteers, images, annotations, etc., drafted as raw SQL queries. It was well thought out but hadn’t been in production yet and was sitting, dataless, on a $60/month Google CloudSQL instance.
• A partially built Node app that interfaced with this database. Most of the code, however, implemented basic CRUD (Create, Read, Update, Delete) operations on a handful of tables and wasn’t deployed anywhere.
• A partially built React UI that hadn’t been integrated with the back-end yet.
• A computer vision model, trained in-house, to separate images with and without animals in them. It was distributed as raw PyTorch code that staff ran on the office computer with a series of commands. This was the only piece actively in use, and it provided a lot of value!

By summer 2021, when I first spoke to them, most of the folks who had worked on these threads weren’t around. The rest had been trying, for quite some time, to finish these threads and stitch them into a cohesive product.

What they didn’t know at the time was that they were marching straight down the highway to what is fondly known as “Integration Hell”.

4. sometimes, a step back is a step forward

While it’s reasonable to jump straight into a project on its existing trajectory, it’s important to remember that as someone new, you bring a fresh perspective, relatively free from prevailing thought bubbles. This empowers you to find better solutions, but it also makes you prone to repeating past mistakes! Needless to say, there was homework to be done, and a lot of it!

4.1. homework

Starting with no domain knowledge, I spent the first few weeks talking to Felidae’s staff and past volunteers. They were very gracious in helping me get a strong sense of their workflows, their pain points, and what utopia looked like. They also chronicled, in detail, their journey to modernize so far, the decision points along the way, and the rationale behind them.

Since conversations like these tend to follow a narrative, I needed to separate what their problems were from the soup of how they tried to solve them, or thought they should be solved. The best way to do that was to produce a structured document. It would serve not just as a written version of their oral history but also as a useful anchor for future conversations. Simply put, a review paper!

To complement this, I also had a fair bit of ground to cover on camera trap data pipelines in general, which led me to this fantastic blog post from Dan Morris! It soon became apparent that Felidae wasn’t alone in navigating what seemed like a fragmented landscape of software tools. What piqued my interest, however, was the community’s significant effort in recent years to address this very problem!

Most notably, since 2018, using machine learning to automatically detect animals in camera trap images had been formalized as a competition at a workshop in CVPR, a popular computer vision conference. Microsoft’s AI for Earth had also open-sourced one such model, called MegaDetector, for everyone to use! And another collaborative effort, Wildlife Insights, aimed to standardize camera trap image workflows by providing a cloud-based AI solution, something Felidae was attempting to build!

4.2. bum first, buy next, build last

The urge to build something from scratch isn’t easy to resist, especially under the seductive forces of flexibility & customization. After all, the desire to create is high, and so is the discomfort from the constraints that external solutions impose!

But every decision is a trade-off and, with flexibility, come upfront and maintenance costs. As I mentioned earlier, these decisions aren’t unlike the ones for-profit organizations have to make. But against the backdrop of rapidly advancing technology, higher churn and a lack of in-house expertise, nonprofits make them under far greater uncertainty, which makes those costs easier to underestimate.

In other words, nonprofits face the same build vs. buy predicament that for-profit organizations face, albeit with slightly different constraints. On the one hand, they have a shallower pool of capital and labor to build or buy. On the other, thanks to the open source community, they have a deeper pool of free options to bum from!

Given this context, a good rule of thumb is

1. Bum first - Embrace a strong bias in favor of free solutions, even if it means relaxing quality thresholds.
2. Buy next - Look at options from large, mature organizations that offer low-cost, stable products, often with discounts for nonprofits.
3. Build last - Only when adapting workflows to existing solutions is far too inefficient, or when other nonprofits could reuse what you build.

Felidae, in their journey, had leaned heavily toward building far too many things. With over 18 months of time and effort invested, they had started to succumb to the infamous sunk cost trap.

Moving forward clearly meant culling most threads of work and taking a step back. But I also had to be cognizant that proposals of radical change, especially from a new volunteer, would likely, and rightfully, induce fear and doubt. So, as I let time build trust, I worked on untangling what they had while nudging them towards external solutions, even if it meant adding an extra step or two to their workflows.

5. less is more

At a very high level, Felidae needed a single system that integrated the following functional components:

1. Inventory management to catalog physical inventory like camera traps, boxes, memory cards, etc. and track their deployment in the field.
2. Camera Trap management to track the physical locations these cameras monitored during the course of their study period.
3. Image management to store, browse and share the hundreds of thousands of images they collected in a simple, structured manner.
4. Annotation management to tag these images with the animals they contain.
5. User management for the volunteers and staff that collected, uploaded and annotated these images.

As mentioned earlier, Google Sheets kept track of inventory, camera traps & users. Dropbox and the Windows file system managed images. And finally, Camera Base handled annotations. None of these, however, were linked to each other, and stitching them together took a lot of manual labor!

At the time, there was no silver bullet that did it all (Wildlife Insights came close). There were, however, fairly mature sub-components backed by large, active working groups. So, the most reasonable option was to simply adopt those in place of the in-house threads, and pare things down to a thin gluing layer, the web application.

My guiding principle here, molded by numerous hackathons, was speed. So, as I got started, my singular priority was to deploy a barebones but fully functioning end-to-end product as soon as possible.

This approach fit the context well. It was imperative to bring end-users into the fold early in development, and as an internal app, there was a higher tolerance for bugs and glitches. But, perhaps more importantly, it offered something much needed at the time, a tangible sense of progress after 18+ months that would boost both morale and recently placed trust!

5.1. django unchained

The central gluing component, the web application, existed as thousands of lines of partially functioning code spread across SQL, Node & React. But to my surprise, most, if not all of it, simply implemented CRUD operations on base tables.

There were probably third-party packages in those frameworks offering the same functionality, but neither I nor anyone else knew those ecosystems well enough to continue those threads. I did, however, have a fair bit of experience with another popular framework, Django!

Now, adding a new technology dependency is not something to take lightly, especially when that dependency trickles down to a single point of failure. In this case, me! But I was sure of my commitment and, given the situation, the risk seemed more than justified considering the numerous benefits that Django offered:

1. A built-in GUI for basic CRUD operations, the Django Admin interface, and an array of skins to choose from (I use Jazzmin).
2. A built-in user management system beefed up by the amazing django-allauth package.
3. ORMs over raw SQL, which I find simpler & cleaner (a personal bias).
4. A single unifying language, Python, spanning the web app, the database and the AI/ML model, lowering barriers for volunteers to contribute across different areas of the system.

Armed with this reasoning atop an undying love for Django, I dusted off my web development skills and got down to it. Luckily, just as I’d hoped, I was able to not just replace the thousands of lines of code that existed but also add significantly more functionality with only about a hundred lines of code!

And just like that, over a few coffee-fueled evenings, I stood up a web application that knocked out three of the five functional components - inventory, camera trap & user management. All thanks to the amazing power of Django & the ecosystem around it!

5.2. more dropbox, please!

The next, slightly trickier component was image management. The tricky bit here wasn’t so much handling image uploads as it was the sheer volume of them! Straight outta memory cards, images would come in bursts of thousands, and in some cases, tens of thousands. That meant the web app would need to pause/resume uploads, deduplicate images, verify image integrity and so on.

On top of that, it would need user interfaces to browse, share and manipulate these images. Unfortunately, Felidae had grossly underestimated the effort this would take. They had lofty plans of not just building it but also layering in an in-browser ML model to filter out empty images before uploading. A pipe dream for a small nonprofit!

In line with the theme, my goal was to build less, not more. Luckily, this was a problem solved by pretty much any company that offered cloud storage. And, as icing on the cake, Felidae was already using Dropbox to transport their images. So more Dropbox actually meant less work, since it offset their need to download images onto the office computer!

More importantly, Dropbox came with two main advantages - (1) a simple, intuitive Python SDK for easy integration with Django, and (2) an easy-to-use feature called “File Requests” that let even non-Dropbox users upload. That made it an obvious choice over alternatives like Google Drive.

From the user’s perspective, once a memory card was collected from a camera trap, they simply had to log into the website and submit an online form (previously done on paper). They’d then receive a Dropbox link to upload images. That’s it!

Behind the scenes, Django would automatically look through the uploaded images and add a thin layer of necessary metadata to its database. At this stage, Dropbox again proved extremely useful by exposing a suite of functionality through its API. Specifically,

1. It made it easy to identify file types (images, video, documents, etc.), and for media in particular, it also exposed Exif attributes like timestamps and location (latitude/longitude), critical for camera traps!
2. It automatically generated content hashes to uniquely identify images.
3. Lastly and most importantly, it offered a simple thumbnail API. This meant Django could pull only smaller, compressed versions of the larger source images (~5MB) for additional processing & rendering.

Less was definitely more!

5.3. megadetector & annotorious, the dynamic duo

Lastly, and most critically, it was time to replace Camera Base with something more efficient. Broadly, this entailed two components - (1) a user interface for people to add annotations, and (2) an AI/ML model that could learn from and replicate those annotators.

Here, Felidae did have a win, a trained model that could detect the presence of animals in an image with reasonable accuracy. But they had completely missed MegaDetector, a similar model backed by a large community and trained on orders of magnitude more data.

As an added bonus, the in-house model attached a “blank”/“not blank” label to the entire image (an image classifier). MegaDetector, on the other hand, detected people, animals and vehicles along with their positions in the image (an object detector). Needless to say, the axe had to come down on the in-house model. And with a new version of MegaDetector recently announced, that decision has clearly paid off many times over!

AI/ML models are tricky little creatures that make everything overtly stochastic, and MegaDetector is no different. Without it, every image would need manual inspection. With it, each image would have a probability of needing it!

Unfortunately, interpreting these numbers is no easy task, and uncaught error patterns at this stage could significantly skew downstream analyses. To be fair, human annotators are also stochastic, with their own tenure-dependent error rates, even if they’re afforded a little more trust.

So, a robust system would be one of collaboration, where annotators wouldn’t just cover the AI model’s blind spots, but also each other’s, all while minimizing manual labor over time.

The first step meant a user interface that would render the bounding boxes MegaDetector predicted, while also letting annotators edit them or add their own. This is a common but important task in Computer Vision applications, with entire companies built around it. While examining my options, I stumbled onto a free but pretty awesome library called Annotorious that seemed tailor-made!

With a simple, intuitive interface, Annotorious made it trivial for the Django website to serve as an annotation platform. It didn’t just replace Camera Base, it also enabled collecting more nuanced, object-level data that MegaDetector and other AI models could learn from!

5.4. faas, paas & iaas

Since my priority was to build with Felidae staff in the loop, it was important to have the application live and accessible, even in barebones form. So, decisions about the underlying software infrastructure had to be made from the get-go.

A decade ago, my instinct would have been to rent a bare-metal instance and stand everything up myself. But now, with a buffet of mature cloud services, there were a lot more options to choose from! With Felidae having already started on Google Cloud Platform (GCP), the task boiled down to picking the right set of services, leaning strongly toward speed and ease of maintenance.

So, let’s take a step back and look at the different components of the system, only this time as pieces of software, and where each one landed. Broadly, we have…

1. the web application, Django, to serve as the primary interface for users to interact with the system. App Engine was almost a no-brainer here, with its generous free tier, built-in autoscaling and great documentation to get started.
2. a relational database to house structured data about inventory, camera traps, users, images, and annotations. Cloud SQL made setting up Postgres easy, and its proxy made local development a breeze!
3. media assets, mostly compressed versions of camera trap images to render on the website. django-storages made storing to and serving from Cloud Storage trivial.
4. AI/ML models like MegaDetector to detect and identify objects in the image. More on this one in a bit.

Along the way, Cloud Logging added a ton of visibility and made debugging a delight! All of this meant the Django app went live as soon as it was built, and the staff could browse, grok and give their valuable feedback early on in the development process!

The remaining piece, MegaDetector, was a tad more tricky. For starters, it was only triggered when large batches of images were uploaded, which happened a few times a month. So it made no sense to bundle it with the Django app. It also had a large memory footprint (~6GB), well beyond App Engine’s limits.

With that in mind, I initially considered Cloud Run, but eventually landed on Cloud Functions, primarily due to its simplicity! With a simple http trigger, Django could invoke these cloud functions whenever images were uploaded. And, decoupled from the app, model development and deployment became more modular and independent.

And with that in place, the site was functionally complete, and I totally did not scream “It’s alive! It’s ALIVE!!!” maniacally!

6. the view

By the start of 2022, after sporadic evening & weekend hack sessions in the preceding months, the site was ready! After a few months of staff-led user testing and ironing out UX kinks, it had started to collect annotations by Spring ‘22.

Zooming out, here is a quick look at what that meant for Felidae’s workflow.

The site lives at WildePod.org and is currently gated to vetted volunteers, with plans to open it up in the coming year! For now, here are some screengrabs to show what it looks like.

7. the trail ahead

The journey so far is really just a foundation for the interesting and impactful journey that lies ahead. With basic tooling in place, here are some broad areas of work that I think can create a lot of value, not just for Felidae but for wildlife research & conservation beyond!

• Data goldmine - Over the years, Felidae’s camera traps have collected close to a million images, almost 5TB worth! Better yet, they also have a passionate group of volunteers who, using Camera Base, have painstakingly annotated a large subset of those images with both the species and the number of animals in them!

  So, the first step is migrating all those images into the new system. Stitched with MegaDetector, they could generate hundreds of thousands of species-level annotations, an invaluable resource for ML models!

• Species detection - Currently, AI/ML in the app is limited to detecting animals in general. Accurately identifying the species is a much harder task due to data sparsity, and is very much on the bleeding edge of research.

  For now, this means a two-step annotation process, a MegaDetector-assisted animal detection phase followed by a purely manual species annotation phase. In the second, users have to pick from a long list of species, which makes for a slow and poor user experience.

  But with Felidae’s vast trove of in-house data, there’s a big opportunity to not just improve species detection for Felidae but also contribute to the larger research effort. And, in classic human-in-the-loop fashion, trained models can greatly improve the user experience by simply sorting that list of species. In return, those user annotations help the models get better over time, in a near-continuous, collaborative loop!

• Insights and analytics - Once images are collected and annotated, the last stage of this pipeline is often massaging data to answer research questions. At the moment, this means downloading annotations and loading them into special-purpose software like Excel or R.

  While these tasks are generally harder to abstract, there’s an opportunity to integrate high-level insights directly into the website, freeing researchers to focus on deeper, more nuanced questions.

Underscoring this trail ahead is the opportunity to open-source data, models, and software to the wider community, to better understand and help our fellow creatures in the wild!

If you’re interested in helping, or if you have any questions, comments or feedback, feel free to leave me a message on LinkedIn!

citation

If you found this post useful, please consider citing it as follows:

@misc{
    kashyap2022hitlcameratraps,
    title={Building a simple AI-powered, human-in-the-loop system to manage wildlife camera trap images & annotations},
    url={https://abhaykashyap.com/blog/ai-powered-camera-trap-image-annotation-system/},
    author={Kashyap, Abhay}
}