Matching and Classification Rules
This section describes the rules behind EdgeFirst Validator's matching algorithm which is used to compute the deployment metrics for object detection. The matching algorithm is responsible for matching the prediction bounding boxes to the ground truth based on the best overlap (IoU + label). The prediction matches are then classified into true positives, false positives, and false negatives.
Rule 1: Requirements of True Positives
A detection is a true positive if it meets all requirements described below.
- The detection label matches the ground truth label
- The detection and ground truth IoU >= IoU threshold
- The detection confidence score >= score threshold (NMS)
Rule 2: Requirements of False Positives
False positives are categorized into localization and classification.
Classification false positives meets all requirements described below.
- The detection label does not match the ground truth label
- The detection and ground truth IoU >= IoU threshold
- The detection confidence score >= score threshold (NMS)
Localization false positives meets any requirements described below.
- The detection and ground truth IoU < IoU threshold
- The detection does not overlap any ground truth (IoU = 0)
Rule 3: Requirements of False Negatives
- Any ground truth where the IoU < IoU threshold
- Any ground truth that does not overlap a detection
Rule 4: Unique Matches
There exists only one detection (true positive or classification false positive) that is matched per ground truth.
Rule 4.1: True Positive Count
The number of true positives should not exceed the number of ground truths nor the number of predictions.
- Each ground truth may have at most, one true positive
- Each prediction may have at most, one true positive
Rule 4.2: Classification False Positive Count
The number of classification false positives should not exceed the number of ground truths nor the number of predictions.
- If a ground truth does not have a true positive, it may have at most, one classification false positive
- If a prediction does not have a true positive, it may have at most, one classification false positive
Rule 4.3: Localization False Positives Count
For cases without any ground truth to detection overlaps (matches), the number of localization false positives should be the total number of predictions.
Rule 4.4: False Negatives
For cases with no model detections, the number of false negatives is equal to the number of ground truth annotations.
Rule 5: Prioritization Cases
If a detection meets the requirements of a true positive, it will be matched to the ground truth. Otherwise, if a detection meets the requirements of a classification false positive, it will be matched to the ground truth. Otherwise, it is a localization false positive. Consider the following cases.
Rule 5.1: Matching Labels Are Prioritized Over IoU
Considering two detections and one ground truth, the detection with a lower IoU, but with the same label as the ground truth should be matched over a detection with a higher IoU, but with a mismatching label as long as both detections have IoU >= IoU threshold.
The following images provides visualization to this rule. The IoU threshold is set to 0.10 and the score threshold is set to 0.50 in these cases. Here the green bounding box prediction is the true positive, the red bounding box is the localization false positive (LOC).
Otherwise, if these detections, are lower than the IoU threshold, they will be regarded as localization false positives (LOC) and one false negative from the ground truth as shown below. Here both predictions have IoUs lower than the IoU threshold which is set to 0.90.
Rule 5.2: Prioritization of Highest IoU Matches
For cases where multiple detections have labels that matches the ground truth, the highest IoU is matched.
The following image shows visualization of this rule where the IoU threshold is set to 0.10 and the score threshold is set to 0.50. Here the green bounding box prediction is the true positive since it has the highest IoU and its label matches the ground truth. The rest of the predictions are then treated as localization false positives (LOC).
These detections have the same label as the ground truth, but the IoU varies, each detection overlaps the ground truth. The greatest overlap is taken as the match. Since this match meets the criteria of a true positive, it is classified as such.
Rule 5.3: Matching Labels, Highest IoU Matches
For cases that concerns both scenarios as the rules described above. A match is based on a matching label and the highest IoU with the ground truth if multiple detections are present with varied labels and varied IoUs.
The following image shows a visualization of this rule following the same IoU threshold set to 0.10 and the score threshold set to 0.50. Here the green bounding box prediction is the true positive as it has the highest IoU of all the matching labels. The rest of the bounding boxes are then treated as localization false positives (LOC).
In this case, matching labels are prioritized and the highest IoU is taken as the match.
Rule 5.4: Mismatching Labels, Highest IoU Matches
For cases of having detections with matching labels as the ground truth, but do not meet the IoU threshold requirement, the detections with mismatching labels are matched if it meets the IoU threshold requirement.
The following image with visualization shows an IoU threshold set to 0.50. In this case, there is a prediction with a matching label as the ground truth, but it does not meet the IoU requirement, so it is treated as a localization false positive (LOC). The one mismatching label with the highest IoU with the ground truth would then be matched since it meets the IoU requirement, but it will be treated as a classification false positive (CLF).
Despite a detection with a matching label of "ace" as the ground truth being present, it is still regarded as a localization false positive (LOC) since it does not meet the IoU requirement. The detection with the highest IoU is matched instead which is with the label "king".
Otherwise, if all detections do not meet the IoU threshold requirement, they will be treated as localization false positives (LOC) and one false negative for the ground truth as shown in the image results below where the IoU threshold was to 1.
Rule 5.5: Matches Meets IoU Threshold Requirement
Any detections that do not overlap any ground truth or any detections with IoU < IoU threshold will be regarded as localization false positives (LOC).
Conclusion
This article has shown the rules established in EdgeFirst Validator that governs the behaviour of the matching and classification algorithms to process the raw detections into their classifications of true positives, false positives, and false negatives. Rules 1-3 shows the requirements of these classifications. Rule 4 describes the limits to the count of these classifications. Rule 5 shows the prioritization of matches to ensure classification requirements are met by prioritizing finding of true positives first, then classification false positives, and then localization false positives. For a closer look into how predictions are classified please see Object Detection Classifications.