Birdly v2

Motivation

Birdly is a bird species classifier and was one of my first machine learning projects which I built in late 2022 during the first semester for my undergrad. It was the project that ultimately got me into machine learning. Now, after almost 4 years and having learned a lot more about ML since, I decided to revisit this project and improve on it.

The first version had many issues like using an unecessarily large model, an unecessarily large dataset, and test data leakage to name a few. So, I decided to rebuild the project from scratch, starting with a new dataset and a new model, and then building a full inference service with a frontend.

In this version, I covered the following:

• dataset construction from iNaturalist Open Data
• ConvNeXt-Tiny training and evaluation
• CPU and GPU inference benchmarking
• FP16 and INT8 experiments
• ONNX Runtime, TensorRT, and Torch JIT compilation with torch.compile
• a FastAPI inference service
• concurrency and reliability testing
• deployment to Google Cloud Run
• a Next.js frontend deployed on Vercel

The first version of Birdly was built to classify all of the roughly 1250 bird species of India, but for this version I changed the training target.

The final model covers 236 bird species of New York State, United States and reached 86.59% top-1 accuracy, 96.75% top-5 accuracy, 0.8655 macro F1, and 0.8653 MCC on a 4,712-image test set.

Architecture

Dataset

This time I decided to use the iNaturalist Open Data dataset instead of scraping manually.

These main source files are large enough that I did not want the data preparation process to depend on loading full tables into memory. So, I used Polars streaming for the main filtering and join jobs.

I also used DVC to track and version control the processed data artifacts.

Taxonomy and observation filtering

Birdly currently targets species found in New York state.

The geographic filter uses a bounding box covering the state:

• latitude: 40.45 to 45.05
• longitude: -79.80 to -71.80

I kept only research-grade observations and restricted to active species in the bird lineage.

I also filtered based on the iNaturalist anomaly score. Observations with an anomaly score below 1.0 were excluded.

The allowed image licenses were restricted to CC0 and CC-BY.

Sampling

The main sampling conditions I used were:

• maximum 1,000 images per species
• minimum 300 total observations per species
• minimum 50 observations from the target region

If a species did not have enough New York observations for training, global observations were added to supplement the training split. But, validation and test images were intentionally kept regional.

I did this to keep the evaluation set closer to the intended use case but still allowing the training set to use more available data if required.

I also sampled only one image per observation to avoid duplicates. However, if training set did not have sufficient images after global supplementation, I allowed further supplementation with multiple images per observation using observation-level round-robin sampling. Like before, I did not allow any supplementation for validation or test splits.

The final processed manifest contained about 234,000 images across 236 species.

The test set contained 4,712 images.

Model training

The final model uses a ConvNeXt-Tiny backbone with a 224 × 224 input.

Training experiments were tracked with MLflow. Each run recorded its parameters, metrics, and model artifacts.

The final FP32 checkpoint produced the following results on the test set:

Top-1 accuracy, macro F1, macro precision, and macro recall were all similar. So, that’s a good sign that the model is performing equally well across the classes.

I also measured NLL and expected calibration error because prediction confidence is later used by the API.

The final FP32 model had an ECE of approximately 6.1 percentage points. So, the model’s confidence is reasonably well calibrated. While there is certainly room for improvement, this is good enough for my intended use case.

Inference baseline

Now that the model was trained, it was time to benchmark inference.

The main measurements I used were:

• model-only latency
• end-to-end latency
• throughput
• RAM or VRAM usage
• model load time
• cold inference time

Model-only timing measures just the forward pass.

End-to-end timing also includes image decoding and preprocessing.

CPU baseline

The first CPU backend was PyTorch eager FP32.

At batch size 1:

The four-thread result of approximately 95 ms end to end was clearly the best and so I chose it as the main CPU reference point.

GPU baseline

I ran the GPU benchmarks on an RTX 3050 Laptop GPU.

Model-only FP32 results:

End-to-end FP32 results:

End-to-end throughput peaked at batch size 16 with a value of 102.75 images/s.

torch.compile

With the baseline established, I experimented with torch.compile to see if it could improve CPU or GPU performance. So, I tested torch.compile on both CPU and GPU.

Compilation introduced a substantial one-time startup cost:

These measurements include compilation and the first inference. Once compiled, the GPU results improved as batch size increased, but batch-1 inference (which is what I expect for the serving workload) was actually slower than eager execution.

GPU results

At batch 16, compilation increased model throughput from 233.64 to 260.73 images/s, which is an improvement of about 11.6%. Batch 32 also showed a similar improvement. At batch 1, however, p50 latency increased from 8.75 ms to 10.75 ms.

The improvement became smaller when preprocessing and the rest of the inference path were included:

The compiled run reached 107.18 images/s end-to-end at batch 16, versus 260.73 images/s for model execution alone. So, accelerating the model did not directly cause the same improvement for the complete inference path.

CPU results

The CPU result was less useful for the serving workload. Since the API would process individual requests rather than large offline batches, batch-1 latency was the main comparison that I cared about:

Compiled model was slower with one and two threads. And at four threads, the difference was less than 1 ms. The compiled four-thread run measured 93.92 ms p50 and 10.58 images/s at batch 1.

CPU batch-1 latency did not improve enough to justify the startup cost.

torch.compile was therefore useful only for sustained GPU batch inference, where it gave roughly 10–12% higher model throughput at larger batches. But, it was not a good fit for the CPU serving path I was targeting since batch-1 performance was effectively unchanged and compilation was expensive. So, I kept eager PyTorch as the reference implementation and moved on to testing inference-specific runtimes.

ONNX Runtime on CPU

Now, I exported the model to ONNX and benchmarked ONNX Runtime on CPU.

At batch size 1:

Compared with the original PyTorch FP32 baseline, ONNX Runtime reduced batch-1 end-to-end p50 latency from 94.59 ms to 63.87 ms at four CPU threads, while increasing throughput from 10.56 to 15.60 images/s.

ONNX Runtime eventually became the CPU backend used by the API.

INT8 quantization

For my next trick, I created a statically quantized INT8 ONNX model.

The quality loss from this was thankfully very low:

Top-1 accuracy fell by approximately 1.17 percentage points. Whereas top-5 accuracy fell by just 0.62 percentage points. Since the Birdly US service returns up to five predictions, this was a small enough loss to be acceptable.

But was there a speedup in inference?

INT8 was slightly faster with one thread but slower with two and four threads.

Reduced precision alone was not enough to reduce inference latency in my case. The quantized model was therefore not used for the main CPU deployment.

GPU inference

I tested four main GPU inference paths:

• PyTorch FP32
• PyTorch FP16
• ONNX Runtime FP32 CUDA
• TensorRT FP16

PyTorch FP16

FP16 largely preserves model quality.

Top-1 accuracy was 86.629%, compared with 86.587% for FP32.

The difference is too small to be treated as a meaningful accuracy change.

But batch size 1 inference was significantly slower than FP32:

• FP16 model p50: 13.65 ms

Even though larger batches were much better.

At batch size 32:

• model p50: 69.67 ms
• throughput: 459.37 images/s
• VRAM: 356.9 MiB

So, FP16 was more useful for large batch sizes than for single-image inference jobs.

ONNX Runtime CUDA

ONNX Runtime CUDA did not improve over the other PyTorch GPU paths.

At batch size 32:

• model p50: 215.59 ms
• model throughput: 148.41 images/s
• end-to-end throughput: 80.43 images/s

So, I did not use this backend further.

TensorRT FP16

TensorRT gave the best GPU measurements.

The TensorRT engine was about 56.1 MiB.

The selected runtimes were therefore:

API

I implemented the inference API with FastAPI.

It exposes:

• POST /predict
• GET /health
• GET /metrics

I load the model once at application startup.

A prediction request passes through:

1. upload validation
2. image decoding
3. preprocessing
4. ONNX Runtime inference
5. probability calculation
6. confidence filtering
7. response generation

The service also includes the following:

• 10 MiB upload limit
• maximum image dimensions of 4096 × 4096
• JPEG, PNG, and WebP support
• animated-image rejection
• 30-second request/inference timeout
• bounded inference concurrency
• CORS restrictions
• IP-based rate limiting
• Prometheus-compatible metrics
• non-root container execution

The production rate limit is 15 predictions per minute per IP.

Local load testing

I used load tests to measure how inference concurrency affected the API.

The tests used:

• ONNX Runtime FP32
• two ONNX CPU threads
• 30-second test windows
• virtual-user counts of 1, 2, 4, 8, and 16

Inference concurrency 1

Throughput stayed close to 7–8 requests per second even as the number of clients increased.

The extra requests primarily waited for the inference slot.

Inference concurrency 8

The highest measured throughput was about 16.5 requests per second.

The improvement over concurrency 4 was much smaller than the improvement from concurrency 1 to 4.

I used these results to avoid setting inference concurrency higher than the CPU could use effectively.

Reliability tests

The local reliability suite covered 18 cases.

These included:

• valid image
• corrupted file
• unsupported content type
• unsupported image format
• empty upload
• oversized upload
• malformed multipart request
• concurrent valid requests
• mixed valid and invalid requests
• burst traffic
• rate limiting
• queue timeout
• missing model
• metrics updates
• process restart and recovery

All 18 local tests passed.

I later ran the applicable tests against the Cloud Run deployment as well.

The remote tests passed as well.

Cloud Run deployment

The deployment image only contains the FastAPI service code, ONNX Runtime CPU, FP32 ONNX model, class mapping, preprocessing metadata, and the required Python dependencies.

It does not include PyTorch, CUDA, TensorRT, training code, or training data.

The model bundle was about 107.45 MiB and idle container memory was approximately 260.8 MiB.

The API runs on Google Cloud Run.

The service can scale to zero because I expect Birdly to receive relatively little traffic.

A warm /health request from my laptop took 202 ms on average.

After the service had scaled to zero, the same request took 6.48 seconds on average.

The expected scale-to-zero cold-start overhead was therefore about 6.3 seconds.

I kept minimum instances at zero because avoiding idle compute cost was more important for me than removing the first-request delay.

Cloud Run tuning

The Cloud Run service has 2 vCPUs, 2 GiB RAM, and maximum one instance.

I tested various configurations by varying ONNX Runtime thread count, Birdly inference concurrency, and Cloud Run request concurrency

The selected backend settings were:

• ONNX CPU threads: 2
• inference concurrency: 4
• Cloud Run request concurrency: 16

I ran a final load test with these settings.

The instance sustained approximately 12 requests per second in the load tests.

The main results are:

Frontend

The frontend is a Next.js application deployed to Vercel.

The user flow is:

1. upload an image
2. select a square crop
3. submit the crop
4. view the returned predictions

Since cropping happens in the browser, only the selected crop is sent to the backend.

The result list supports between zero and five predictions.

The deployed request path is:

Raspberry Pi deployment

I also checked whether the INT8 ONNX model could run on a Raspberry Pi Zero W v1.1.

The board only has a single-core ARMv6 CPU with 512 MB RAM

But, the immediate problem was runtime support.

There is no normal ONNX Runtime Python package for that ARMv6 environment.

Cross-compiling ONNX Runtime or moving the model to another runtime would have required additional work for an old hardware target, so I stopped the experiment there.

The Pi Zero W was not useful for running the current classifier, but it can still be used as a camera node.

I then tried deploying the classifier on a Raspberry Pi 5 with 1 GB RAM. Unlike the Zero W, the Pi 5 supports ONNX Runtime on ARM64, so I was able to run the same FP32 model through a FastAPI service. I set it up as a systemd service with one inference worker and a limit of two active prediction requests.

I also ran load and reliability tests on the Pi 5. With one virtual user, it handled 3.43 requests per second with a p50 latency of 284.53 ms. With two virtual users, throughput increased to 3.65 requests per second, with a p50 of 538.70 ms. At four virtual users, most excess requests returned HTTP 503 instead of increasing successful throughput. The reliability tests passed with the two-request limit.

Observations

Runtime choice changed inference performance substantially without changing the trained model.

On CPU, moving from PyTorch eager to ONNX Runtime reduced batch-1 end-to-end latency from about 95 ms to 64 ms at four threads.

On GPU, TensorRT reduced model-only batch-1 latency to 3.62 ms and increased batch-32 throughput to more than 730 images per second.

Reduced precision did not behave the same way on every target.

FP16 was useful for GPU throughput.

INT8 maintained reasonable model quality but did not improve CPU latency on the tested Ryzen system.

The service benchmarks also showed that single-request latency alone was not enough to choose serving settings. Inference concurrency improved throughput until CPU contention reduced the benefit.

Model-only and end-to-end measurements were also materially different. Once TensorRT reduced the forward pass to a few milliseconds, preprocessing became a much larger part of total latency.

Cloud deployment also added a tradeoff. Scale-to-zero reduced idle cost but added about 6.3 seconds to the first request after the service had been inactive.