Real-Time Segmentation with Limited Training Frames and a 50ms latency requirement

A machine learning write-up on building a real-time segmentation model for a low-contrast grayscale video stream, with almost no labeled data and a hard latency requirement. Over one work term, I went from ideation to shipping a real-world product. My first solution entailed a pixel-thresholding baseline, through a hand-tuned pipeline designed to catch false positives, to a learned U-Net detector, and then finally working on making its evaluation trustworthy and the inference extremely fast.

The ML Task

Watch a live, low-contrast grayscale video stream and detect the moment an injected dark fluid spreads into a user-defined region of the image. Three properties made it challenging:

• Almost no labeled data. No data for this task existed, so I had to create data from physical simulations, and annotate afterwards.
• Extreme class imbalance. The signal is a small patch of darker pixels in an otherwise unchanged frame.
• A tight latency and precision budget. Detections have to arrive faster than a person could notice them, without producing enough false alarms that the output gets ignored.

Can you detect a specific kind of local darkening inside a region of interest, fast enough to be useful, without drowning the user in false alarms?

The false positives were the most important consideration for the next 4 months.

Development Arc: Three Physical Data Sources

I couldn't develop against real patient-data, so the work progressed through three physical flow simulations of increasing fidelity. From an ML point of view, each one is a different data distribution, and each exposed a different failure mode.

Simulator 1: the baseline

The first prototype ran on a 3D-printed flow simulator built at SickKids by previous co-ops (thank you!). Detection here was deliberately naive: straight pixel thresholding, comparing the value of an empty channel against the value of dark fluid entering it.

It worked as a proof of concept. It established that the signal was there and that real-time detection was worth pursuing. It also established a baseline that everything afterward had to beat.

Simulator 2 (Simone): a pulsatile model, and a useful negative result

The second simulator, developed by Katie, Connor, Lucas, and Mattie (Simone Team at University of Waterloo), added a motorized pump to reproduce pulsatile flow rather than steady-state flow. It was closer to reality, and made the temporal behaviour of the signal more realistic. I imaged Simone under X-ray, and was surprised. The X-ray was filled with noise.

This was the most unexpected result of the project. The motors and metallic hardware, alongside our fluid reservoir, that made the model physically realistic actually worsened our imaging result. Under X-ray they appear as high-contrast structures that swamp the frame, and detection became extremely inconsistent. The fidelity improvement that made the phantom better physically made it unusable as a computer vision target.

Two things came out of this. First, a hard requirement that the detector explicitly reject metal-like pixels rather than hope they wouldn't appear. Second, the decision to migrate to a third simulator which would have no metal.

Simulator 3: the training platform

The final development platform was a commercial flow simulator, radiolucent apart from the flow channels themselves, with branching geometry realistic enough to mimic human biology such that results would transfer. All feature development and model training from this point forward ran on this simulator.

The False-Positive Problem

Once the detector was sensitive enough to catch real fluid, it caught everything else too. Moving hardware, shifts in the scene, compression artifacts, global brightness drift; all of it registered as darker pixel values.

A false alarm is not a minor annoyance. An alerting system that cries wolf gets ignored, and an ignored system is worse than no system, because it consumes attention while providing nothing.

I built a five-stage cascade where each stage is aimed at a specific, observed failure mode.

1. Baseline differencing. The baseline is a per-pixel median of the first 20 frames, collapsed into one reference frame. Median rather than mean, because it suppresses flicker, compression noise, and one-frame shadows instead of averaging them in. A pixel becomes a candidate only if it clears two conditions: darkened relative to baseline, and below an absolute intensity cutoff. Requiring both matters: with only the relative test, anything merely less bright than before qualifies.
2. Artifact rejection. Qualifying pixels form a binary mask, from which two categories are removed: structural edges (slight misalignment there produces apparent darkening that registers as fluid, but in actuality is not a real change) and metal-like pixels, identified by taking the extreme ends of the baseline distribution. Both are regions where baseline subtraction is unreliable, this stage is implemented from the learnings of the second simulator.
3. Region-level gates. Four whole-region checks, any of which failing causes an immediate rejection.

   Gate	Rejects
   ROI restriction	Anything outside the user's chosen region
   Detection surface	Regions whose baseline brightness indicates black margin, not imaged area
   Border contact	Too much of the ROI border lit: a scene change or baseline mismatch, not a local pool
   Coverage cap	Too much area darkening at once: global change, not local flow

4. Blob analysis. The surviving mask is split into connected components and each is scored on size (area, bounding-box width and height), shape (fill ratio and aspect ratio, which separates a compact pool from a thin sliver or a solid rectangle), how many bounding-box edges it touches, overlap with the detection surface, and mean darkening. A component must pass every gate. Instrumenting this stage surfaced a real bug where valid large blobs were being silently discarded.
5. Temporal confirmation. A detection requires two consecutive hot frames. Importantly, a single cold frame does not reset the streak (this was causing a lot of flickering earlier), it costs one frame of credit instead. Strict resetting meant one dropped frame killed an otherwise-real detection run, which happened often enough to matter.

Why it was replaced

The false-positive pipeline worked, and building it was the most valuable part of the project, because each stage encodes something specific I learned about how this signal fails. But it was hand-tuned, and every threshold in it was a parameter that would need re-tuning for a new imaging setup or a new simulator.

In the final system, the pipeline is gone entirely, replaced by a learned segmentation model. The model adopted the same skepticisms of the classical detection pipeline without the brittleness. Keeping the pipeline as a first-pass filter was tempting, but it reintroduced exactly the tuning burden the learned approach removed.

I'm including it here because the failure taxonomy led to a more successful approach later.

The Learned Detector

Architecture: U-Net with a ResNet-34 encoder, ImageNet-pretrained. Binary segmentation, one output class: fluid or no fluid.

U-Net because the task needs precise spatial localization, not classification. Knowing that fluid is present somewhere in the frame is useless; what matters is where. The encoder-decoder structure with skip connections concatenates high-resolution features from the encoder directly into the decoder, preserving the fine spatial detail that repeated downsampling destroys, and the skip paths keep gradients from vanishing in early layers.

ResNet-34 as the encoder for three reasons: ImageNet pretraining supplies useful low-level features (edges, tubular structures) for free, which matters enormously when the training set is 19 frames; its residual connections again help gradient flow; and it's deep enough to be detailed without the compute cost of ResNet-50 or ResNet-101, which matters when there's a hard latency budget.

Loss: Binary cross-entropy + Dice. BCE alone fails on this task because of severe class imbalance. With roughly 200,000 background pixels against 5,000 fluid pixels in a typical frame, a model that predicts "no fluid" everywhere scores about 97.5% precision while being completely useless. Dice directly optimizes overlap, so total failure to detect fluid scores 0% regardless of how much background it gets right. BCE contributes a per-pixel probabilistic signal that heavily penalizes confident false positives. Together each covers the other's weakness and gives us more realistic metrics.

Optimizer: AdamW, for adaptive learning rates and faster convergence. Checkpoint selection: best validation Dice.

Training on 19 Frames Without Overfitting

The training set was 19 hand-annotated frames drawn from two separate videos, labeled in CVAT and exported as COCO 1.0.

Nineteen frames is small enough that the real risk isn't underfitting, it's producing an impressive number that means nothing due to learned patterns and memorization. Most of the methodology here exists to prevent that.

• Held-out videos, not held-out frames. Train, validation, and test came from separate video sequences. Splitting randomly across frames would be deceptive: adjacent frames in a video are near-duplicates, so a random split effectively puts the test data in the training set.
• 19-fold cross-validation. Eighteen frames train, one held out, repeated nineteen times, this is the protocol behind the metrics reported below, rather than a single fixed 70/15/15 split, precisely because a single split over 19 frames is too small a sample to trust on its own.
• Neighbour exclusion. For each fold, the held-out frame's temporal neighbour was set aside as the validation frame rather than left in training, closing the same leakage path within a fold.
• Untouched test set. The final held-out frames weren't examined until the end.
• Hard negative sampling. Training ran on 256px crops sampled to include both positives and hard negatives, so the model saw the confusing cases rather than mostly-empty background. (The 256px figure is the training crop size; inference below runs at the full 448×448 resolution, the two numbers describe different stages, not a contradiction.)

Transfer Learning for Automatic Region Proposal

Asking a user to mark regions by hand costs time, so I also built automatic proposal of the regions worth monitoring. The first approach used a YOLO model, which performed poorly at Dice 0.41.

Replacing it with a U-Net/ResNet-34 encoder, ImageNet-pretrained and then fine-tuned on retinal vessel data, roughly doubled that to Dice 0.82. Retinal vessel segmentation is a well-resourced public task with the same underlying structure, thin branching tubular networks in low-contrast grayscale, which made it an unusually good transfer source for a domain with almost no labeled data (Thanks Connor for this insight).

Latency Engineering

A detection that arrives after a person has already noticed the event is worthless. The initial pipeline ran at roughly 350ms per detection. The target was ~50ms, faster than a human blink.

Four changes closed most of the gap:

1. Moved U-Net inference to GPU. The largest improvement since U-Net inference is spearheaded by matrix multiplication which GPUs perform in parallel.
2. Restricted inference to the regions of interest rather than full frames. The user has already declared what matters; segmenting the entire frame spends most of the budget on irrelevant regions.
3. Vectorized preprocessing in NumPy, removing per-pixel Python loops from the hot path and using optimized NumPy operations.
4. Dropped a redundant RTSP relay that was adding a full network hop for little reason. Not a clever optimization, just a pipeline stage that shouldn't have existed which was hurting the latency. Measuring the whole path end-to-end found this latency issue.

These changes brought detection to running 1-2 frames behind the live feed, a final measured residual delay of roughly 45ms, which was successful in meeting the target.

Results

Evaluated with a U-Net/ResNet-34 at 448×448 input, threshold 0.45, scored ROI-restricted against a hand-drawn region.

The result I'd point to is the second column. The model was trained entirely on simulated frames and had never seen real-world imaging, yet Dice fell only from 0.85 to 0.80, and recall held essentially flat at 0.88. For a monitoring application, recall is crucial, missing a real event is the failure mode with most consequences.

Precision drops more sharply, from 0.82 to 0.73, which is the honest limitation. Real imaging contains structures the simulation doesn't produce, and the model flags some of them. That sim-to-real gap is the direct target of the next phase of work.

What I'd Do Next

• Annotate the known confusers. A specific class of thin, high-contrast structures accounts for much of the precision gap; labeling it explicitly as a negative class should close a good part of it.
• Broaden the training distribution. Performance is strongest on one imaging mode and weaker on another; the model needs to hold up on both.
• Generalize to other imaging setups. Nothing in the architecture ties it to a single acquisition setup, so the same approach should extend to new ones.

A Note on Scope

This write-up covers the machine learning side only: model design, training methodology, evaluation, and latency work. Application details, demonstrations, and domain specifics are intentionally left out, and the results are reported at a level intended to describe the approach rather than reproduce the system.

Work completed at PCIGITI, The Hospital for Sick Children, with Connor Lee, Thomas Looi, Dale Podolsky, and Prakash Muthusami.