# TrackMate Accuracy

The problem with tracking algorithms is that they always give an answer.

This answer can be completely irrelevant, even non-physical, and there is no built-in flags that would indicate something wrong. The best way to avoid basing your downstream analysis on faulty tracking results is to know in what situation the tracker works the best, and what are its limitations. This is the aim of this page for the trackers and detectors shipped with TrackMate.

## The ISBI 2012 single particle challenge.

In 2011-2012, an ISBI Grand Challenge was organized for the Single-Particle Tracking algorithms. Though TrackMate does not offer a completely new algorithm, product of an original Research work, we took the chance and participated in the challenge. The results and the methodology to compute the accuracy of a tracking algorithms were published[1] thereafter.

Unsurprisingly, we did not score amongst the best. At the time, TrackMate was in version 1.1, and ship a stripped down version of the better performing Jaqaman et al. LAP framework[2]. See the LAP trackers section for algorithm details. Plus, TrackMate was was young at the time, and some bugs did not help.

## TrackMate v2.7.x series accuracy against the ISBI dataset.

From v2.7.x, TrackMate ships a new tracker that can deal specifically with linear motion. We though it was the right time to re-run the accuracy assessment with the ISBI challenge data. The people behind Icy offered the website to host the challenge data, and it is still available today[3] for download.

The figures below shows the comparison of accuracy for the 3 classes of tracking algorithms available in TrackMate:

• The LAP framework derived from Jaqaman et al.[4].
• The linear motion tracker based on Kalman filter.
• The plain Nearest neighbor tracker for reference.

### Scenarios.

It’s best to directly read the paper[5] to know what is behind these measures, but here is a brief survey of how they are done. The ISBI dataset covers four scenarios:

Scenario name

Particle shape

Motion type

MICROTUBULE

Slightly elongated shape to mimic MT tip staining.

Roughly constant velocity motion.

RECEPTOR

Spherical.

Tethered motion: switch between Brownian and directed motion with random orientation for the later.

VESICLE

VIRUS

Switch between Brownian and directed motion with fixed orientation for the later.

For each scenario, images covers several particle density:

• low: 60-100 / frame
• mid: 400-500 / frame
• high: 700-1000 / frame

to check how a tracking algorithm behaves when particles get very dense.

Also, particles SNR spans several values: 1, 2, 4, 7 (plus 3 for the RECEPTOR scenario). As said on the challenge page: “SNR=4 is a critical level at which methods may start to break down”.

### Example images from the challenge dataset.

Below are shown typical images taken from the challenge.

#### Varying particle density.

snr 7 density low-1.png\|Low density snr 7 density mid-1.png\|Medium density snr 7 density high-1.png\|High density Contrast stretched to the 0-150 8-bit range. #### Varying particle SNR. snr 7 density low-1.png\|SNR = 7 snr 4 density low-1.png\|SNR = 4 snr 3 density low-1.png\|SNR = 3 snr 2 density low-1.png\|SNR = 2 snr 1 density low-1.png\|SNR = 1 Contrast stretched to the 0-50 8-bit range. #### The MICROTUBULE scenario particle shape. snr 4 density mid-1.png\|MICROTUBULE, SNR = 4, density = mid ### Accuracy measurements. For each scenario and condition, the method returns numerous values that characterizes the accuracy of a tracking algorithm. They are detailed on [this technical paper](http://bioimageanalysis.org/track/PerformanceMeasures.pdf). We plot below only three of them: - The Jaccard similarity between tracks, that quantifies how well the tracks returned by the algorithm match the ground truth. This value assesses the accuracy of the [spot tracker](TrackMate_algorithms#Spot_trackers) you pick in TrackMate. It ranges from 0 (terrible) to 1 (found tracks = ground truth). - The Jaccard similarity between detections, that quantifies how well the particle detected by the detection algorithm match the ground truth. It depends strongly on the [spot detector](TrackMate_algorithms#Spot_detectors) you pick in TrackMate, and ranges from 0 to 1 like the above quantity. - The RMSE of detection positions that quantifies how precise is the location of the detected particles. The smaller the better. I fully relied on [Icy](/software/icy) to compute these values. TrackMate ships an action that exports tracking results to the XML format imposed by the ISBI challenge, and that can be found [here](https://github.com/fiji/TrackMate/blob/master/src/main/java/fiji/plugin/trackmate/action/ISBIChallengeExporter.java). I generated these files for all the conditions of a scenario, and used the [Icy ISBI challenge scoring plugin](http://icy.bioimageanalysis.org/plugin/ISBI_Tracking_Challenge_Batch_Scoring) to yield metrics. I then used [MATLAB](MATLAB) to plot them. ### Parameter used. Unless otherwise specified below, I always used the LoG detector as a spot detector, with an estimated particle diameter of 2, and used sub-pixel accuracy. For SNR below 4, this detector was completely confused and the detection results are dominated by noise. I did not make anything special to improve its sensitivity below this limit. When the histogram of detection quality returned by the detector as not bimodal, I pick a quality threshold that yielded approximately the expected number of particles in the sequence. The three spot trackers were configured as indicated in the table below. It's not very sensible to keep always the same parameter across different scenario, but with what you can tune in TrackMate, there is little room for fine-tuning.

Spot tracker Parameter Value
Linear motion tracker Initial search radius 10
Finally, for SNR>4, I filtered out tracks that had less than 4 detections.