Raunak Dey

As an AI R&D intern at Elanco Animal Health, I am building a production computer-vision system to classify animal behavior from video for pharmaceutical efficacy studies — across many individuals, multiple cameras and scenes, and strongly imbalanced behavior classes.

The core challenge is that raw video is high-dimensional, expensive, and noisy. The behavior labels come from human-annotated clinical protocols that are inconsistent in time format, duplicated, and misaligned with video timestamps. Before any model can run, the data must be made tractable and trustworthy.

Data engineering: I built an automated pipeline that cleans and time-aligns annotations — deduplicating records, normalizing date and time formats, merging behavior intervals, and matching labels to video frames using Gemini as a multimodal timestamp parser. Exploratory analysis then reduced the label space to most informative behaviors grounded in class frequency and clinical relevance.

Scalable vision pipeline: With ~1 TB of video, compute choices matter. A pilot sampling study determined the minimum frame rate and resolution that preserve behavioral signal — dramatically cutting storage and GPU cost. Segmentation then crops behavior-relevant regions per frame, reducing redundant background tokens before they reach the model.

Probabilistic modeling: Zero-shot baselines with multimodal foundation models (Gemini) produced weak results on this specialized domain. The approach I am developing layers probabilistic structure on top of fine-tuned representations: location-conditioned priors (behaviors are not equally likely at all locations in the scene) and hidden Markov model / Viterbi decoding to enforce temporal consistency across frames. This is the same inverse-problem instinct as the phage and drone work — injecting structure the way a Bayesian would, rather than letting the model hallucinate transitions freely.

LLM and foundation-model time-series forecasting: Large language models trained on token sequences can, in principle, forecast numerical time series by treating numbers as tokens and next-token prediction as next-step forecasting. I investigated how well zero-shot and few-shot LLM forecasting works on chaotic and stochastic dynamical systems — probing fundamental limits of predictability. A novel aggregated tokenization scheme improved forecasting accuracy significantly over standard baselines; LoRA adapters fine-tuned across GPT-4, LLaMA-2, and Mistral further improved task-specific performance while halving training time on HPC clusters. I also benchmarked zero/few-shot forecasting with time-series foundation models (TimesFM, Prophet) on these systems.

Network inference from population dynamics: Given observed abundance trajectories of interacting species, can we recover the underlying interaction network? I developed a multitask ML framework that jointly estimates interaction structure, mechanistic model parameters, and forecasting targets from population-dynamics data — without requiring direct observation of interactions. The framework couples differential-equation constraints with learned representations to regularize the inversion. Presented at NetSci-2024 and APS Global Summit 2025.

GPU-accelerated constrained optimization for microbiome networks: Predicting the dynamics of thousands of coupled microbial interaction networks requires solvers that scale. I designed and deployed a GPU-accelerated adaptive optimizer (TensorFlow/Adam) for large-scale constrained network time-series forecasting, achieving 5× better prediction across 10,000 interaction networks versus a CVXPY baseline.

Generative models for time series: VAE, GAN, and diffusion architectures can approximate underlying probability distributions from samples. For physics-derived dynamical systems with sparse or undersampled observations, I explored generative approaches for predictive extrapolation and uncertainty-aware synthesis of time series.

Before applying probabilistic methods to biological and health problems, I spent several years studying stochastic processes at a fundamental level — building the theoretical and experimental foundations I now use in ML contexts.

Arcsine laws in nonequilibrium systems: In equilibrium statistical mechanics, the fraction of time a Brownian particle spends on one side of its starting point follows a striking distribution: the arcsine law. We experimentally verified that this result extends to nonequilibrium systems driven by active forces — and showed that the distribution of thermodynamic currents exhibits a non-monotonic skewness as a function of observation timescale that classical theory does not predict. Published in Physical Review E (2022, first author) and Physical Review Research (2022).

Random number extraction from stochastic trajectories: True random numbers are generated by physical processes, not algorithms. Using experimentally recorded Brownian trajectories from an optical trap, I developed ML-based extraction algorithms that pass all NIST statistical randomness tests — and showed that the entropy of the extracted bits improves asymptotically with sampling rate, connecting experimental physics to cryptographic-grade validation. Published in Frontiers in Physics (2021, first author) and presented at SPIE Photonics (2021, oral).

Early in my research career I designed algorithms for getting the most out of noisy physical measurement systems — a signal-processing and optimal-control problem that turned out to be excellent training for the Bayesian inference work that followed.

Optimal sensing in microrheology: In optical-tweezers experiments, a microscopic probe embedded in a complex fluid reveals the fluid's viscoelastic properties through its motion. The challenge is that useful signal only exists in a narrow linear-response regime, and noise limits broadband measurement. I designed optimal and feedback-control algorithms — including a multi-sinusoid modulation scheme — that maximize signal-to-noise across a 2 kHz bandwidth, improving measurement speed 20× over single-frequency methods. This work was applied to study the viscoelastic properties of mutated Lamin A (linked to cardiomyopathy) and resulted in an Indian patent (No. 539208, 2024). Published in Physical Review Fluids (2021) and Soft Matter (2021).

Signal processing for radio astronomy: At NCRA (Giant Metrewave Radio Telescope, India), I built a frame-stacking pipeline to filter RFI-corrupted frames from pulsar observations, combining multi-telescope data and frame-to-frame comparison algorithms to recover clean signal from noisy integrations (99% corrupt-frame detection).

Microscopy and imaging systems: At the Li Lab (University of British Columbia), I built a custom dual-lens microscope with 3D motorized control via gaming joysticks and real-time contour-based particle tracking in MATLAB/LabVIEW. At the Silva Lab (Georgia Tech), I worked on time-synchronized dual single-photon detection for quantum spectroscopy with entangled photon pairs.