Rimeng Society
AI AI:Keras PyTorch MXNet TensorFlow PaddlePaddle deep learning real combat (irregular update)
CNN: RCNN,SPPNet,Fast RCNN,Faster RCNN,YOLO V1 V2 V3,SSD,FCN,SegNet,U-Net,DeepLab V1 V2 V3,Mask RCNN
Target estimation model Kalman filter
Object detection based on yoloV3
Cross riding: traffic flow statistics based on virtual coil method
Traffic flow statistics in video
4.4. Target estimation model Kalman filter
Learning objectives
- Understand the state variables and observation inputs in Kalman filter
- Understand the status update vector of the target box
- Understand the estimation of the prediction bounding box
Here we mainly complete the implementation of Kalman filter tracking.
- Initialization: state variables and observation inputs of Kalman filter
- Update status variable
- The bounding box of the target is predicted according to the state variable
-
initialization:
The setting of state quantity x is a seven dimensional vector:
Respectively represent the X and Y coordinates of the target center position, the area s and the aspect ratio of the current target frame. The last three are the change rates of transverse, longitudinal and area, in which the speed part is initialized to 0, and others are input according to the observation.
The initialization Kalman filter parameters, 7 state variables and 4 observation inputs, the determination of motion form and transformation matrix are based on the uniform motion model, and the state transition matrix F is determined according to the kinematics formula:
The measurement matrix H is a 4 * 7 matrix, which corresponds the observed value to the state variable:
And the corresponding covariance parameters are set according to the empirical value.
def __init__(self, bbox):
# Define isokinetic model
# Kalman filter is used internally, with 7 state variables and 4 observation inputs
self.kf = KalmanFilter(dim_x=7, dim_z=4)
# F is the state transformation model, which is a 7 * 7 square matrix
self.kf.F = np.array(
[[1, 0, 0, 0, 1, 0, 0], [0, 1, 0, 0, 0, 1, 0], [0, 0, 1, 0, 0, 0, 1], [0, 0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0], [0, 0, 0, 0, 0, 1, 0], [0, 0, 0, 0, 0, 0, 1]])
# H is the measurement matrix, which is a 4 * 7 matrix
self.kf.H = np.array(
[[1, 0, 0, 0, 0, 0, 0], [0, 1, 0, 0, 0, 0, 0], [0, 0, 1, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0, 0]])
# R is the covariance of the measurement noise, that is, the covariance of the difference between the real value and the measured value
self.kf.R[2:, 2:] *= 10.
# P is the covariance of a priori estimates
self.kf.P[4:, 4:] *= 1000. # give high uncertainty to the unobservable initial velocities
self.kf.P *= 10.
# Q is the covariance of the process excitation noise
self.kf.Q[-1, -1] *= 0.01
self.kf.Q[4:, 4:] *= 0.01
# state estimation
self.kf.x[:4] = convert_bbox_to_z(bbox)
# Update of parameters
self.time_since_update = 0
self.id = KalmanBoxTracker.count
KalmanBoxTracker.count += 1
self.history = []
self.hits = 0
self.hit_streak = 0
self.age = 0-
Update status variable
Update the state variable with the observed target box
def update(self, bbox):
# Reset
self.time_since_update = 0
# Empty history
self.history = []
# hits count plus 1
self.hits += 1
self.hit_streak += 1
# Modify the internal state according to the observation results x
self.kf.update(convert_bbox_to_z(bbox))-
Predict the target box
Advance the state variable and return the predicted bounding box result
def predict(self):
# Propulsion state variable
if (self.kf.x[6] + self.kf.x[2]) <= 0:
self.kf.x[6] *= 0.0
# Make predictions
self.kf.predict()
# Times of Kalman filter
self.age += 1
# If it has not been updated during the process, it will hit_ Set streak to 0
if self.time_since_update > 0:
self.hit_streak = 0
self.time_since_update += 1
# Append forecast results to history
self.history.append(convert_x_to_bbox(self.kf.x))
return self.history[-1]The entire code is as follows:
class KalmanBoxTracker(object):
count = 0
def __init__(self, bbox):
"""
Initialize bounding box and tracker
:param bbox:
"""
# Define isokinetic model
# Kalman filter is used internally, with 7 state variables and 4 observation inputs
self.kf = KalmanFilter(dim_x=7, dim_z=4)
self.kf.F = np.array(
[[1, 0, 0, 0, 1, 0, 0], [0, 1, 0, 0, 0, 1, 0], [0, 0, 1, 0, 0, 0, 1], [0, 0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0], [0, 0, 0, 0, 0, 1, 0], [0, 0, 0, 0, 0, 0, 1]])
self.kf.H = np.array(
[[1, 0, 0, 0, 0, 0, 0], [0, 1, 0, 0, 0, 0, 0], [0, 0, 1, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0, 0]])
self.kf.R[2:, 2:] *= 10.
self.kf.P[4:, 4:] *= 1000. # give high uncertainty to the unobservable initial velocities
self.kf.P *= 10.
self.kf.Q[-1, -1] *= 0.01
self.kf.Q[4:, 4:] *= 0.01
self.kf.x[:4] = convert_bbox_to_z(bbox)
self.time_since_update = 0
self.id = KalmanBoxTracker.count
KalmanBoxTracker.count += 1
self.history = []
self.hits = 0
self.hit_streak = 0
self.age = 0
def update(self, bbox):
"""
Update the state vector with the observed target box. filterpy.kalman.KalmanFilter.update Internal state estimates are modified based on observations self.kf.x.
Reset self.time_since_update，empty self.history.
:param bbox:Target box
:return:
"""
self.time_since_update = 0
self.history = []
self.hits += 1
self.hit_streak += 1
self.kf.update(convert_bbox_to_z(bbox))
def predict(self):
"""
Advance the state vector and return the predicted bounding box estimation.
Append forecast results to self.history. because get_state Direct access self.kf.x，therefore self.history Not used
:return:
"""
if (self.kf.x[6] + self.kf.x[2]) <= 0:
self.kf.x[6] *= 0.0
self.kf.predict()
self.age += 1
if self.time_since_update > 0:
self.hit_streak = 0
self.time_since_update += 1
self.history.append(convert_x_to_bbox(self.kf.x))
return self.history[-1]
def get_state(self):
"""
Returns the current bounding box estimate
:return:
"""
return convert_x_to_bbox(self.kf.x)summary
-
Understand initialization, Kalman filter state variables and observation inputs
-
Update status variable (update)
-
Predict the bounding box of the target according to the state variable
from filterpy.kalman import KalmanFilter
import numpy as np
def convert_bbox_to_z(bbox):
"""
take[x1,y1,x2,y2]The detection frame of the form is transformed into the state representation of the filter[x,y,s,r].
among x,y Is the center coordinate point of the frame, s Is the area scale, r Is the aspect ratio w/h
:param bbox: [x1,y1,x2,y2] They are the upper left corner coordinates and the lower right corner coordinates, i.e [Top left corner x Coordinates, top left y Coordinates, lower right x Coordinates, lower right y coordinate]
:return: [ x, y, s, r ] 4 Row 1 column, where x,y yes box Coordinates of the center position, s Is the area, r Is the aspect ratio w/h
"""
w = bbox[2] - bbox[0] #x coordinate of lower right corner - x coordinate of upper left corner = width of detection box
h = bbox[3] - bbox[1] #y coordinate of lower right corner - y coordinate of upper left corner = height of detection box
x = bbox[0] + w / 2. #X coordinate of the upper left corner + width / 2 = x coordinate of the center of the detection frame
y = bbox[1] + h / 2. #Y coordinate of the upper left corner + height / 2 = y coordinate of the center of the detection frame
s = w * h #Width * height of detection frame = area of detection frame
r = w / float(h) #Width w / height h of detection frame = width height ratio
#Because the input format of Kalman filter requires 4 rows and 1 column, the shape of [x, y, s, r] should be converted into 4 rows and 1 column and then input to Kalman filter
return np.array([x, y, s, r]).reshape((4, 1))
"""
Change the candidate box from the form of central area[x,y,s,r] Convert to coordinate form[x1,y1,x2,y2]
"""
def convert_x_to_bbox(x, score=None):
"""
take[cx，cy，s，r]The target box of indicates to[x_min，y_min，x_max，y_max]Form of
:param x:[ x, y, s, r ],among x,y yes box Coordinates of the center position, s Is the area, r Is the aspect ratio w/h
:param score: Confidence
:return:[x1,y1,x2,y2],Upper left and lower right coordinates
"""
"""
x[2]: s Is the area, the original formula s The source of is s = w * h，That is, the width of the detection frame * high = Detection frame area.
x[3]: r Is the aspect ratio w/h，Original formula r The source of is r = w / float(h)，That is, the width of the detection frame w / high h = Aspect ratio.
x[2] * x[3]: s*r Namely(w * h) * (w / float(h)) = w^2
sqrt(x[2] * x[3]): sqrt(w^2) = w
"""
w = np.sqrt(x[2] * x[3]) #sqrt(w^2) = w
h = x[2] / w #w * h / w = h
if score is None:
return np.array([x[0] - w / 2., #X coordinate of the center position of the detection frame - width / 2 = x coordinate of the upper left corner
x[1] - h / 2., #Y coordinate of the center of the detection frame - height / 2 = y coordinate of the upper left corner
x[0] + w / 2., #X coordinate of the center of the detection frame + width / 2 = x coordinate of the lower right corner
x[1] + h / 2.] #Y coordinate of the center of the detection frame + height / 2 = y coordinate of the lower right corner
).reshape((1, 4))
else:
return np.array([x[0] - w / 2.,
x[1] - h / 2.,
x[0] + w / 2.,
x[1] + h / 2.,
score]).reshape((1, 5))
"""
Implementation of Kalman filter for tracking
Target estimation model:
1.The target frame state of the current frame is predicted according to the target frame result of the previous frame, and the boundary frame is predicted(Target box)The model is defined as a constant velocity motion/Uniform motion model.
2.Each target frame has a corresponding Kalman filter(KalmanBoxTracker Instance object)，
KalmanBoxTracker The instance attribute in the class is specifically responsible for recording various statistical parameters in its corresponding target box,
The class attribute is used to record the number of Kalman filters created. Adding a target box adds a Kalman filter(KalmanBoxTracker Instance object).
3.yoloV3,Kalman filter prediction/Update process steps
1.Step 1:
yoloV3 Target detection stage:
--> 1.If a target is detected, a detection target chain is created/Track the target chain, otherwise, if the target cannot be detected, the target detection cycle will be repeated.
--> 2.Detection target chain/If the tracking target chain is not empty, it enters the Kalman filter predict In the prediction stage, if it is empty, the target detection cycle will be repeated.
2.Step 2:
Kalman filter predict Prediction stage:
If the prediction is made several times without an update operation, it means that the similarity matching between the prediction target and the detection target after each prediction is unsuccessful,
Therefore, there will be "prediction and then similarity matching failure" for many consecutive times, resulting in not entering the update stage.
If the similarity matching is successful after one prediction, then it will enter the update stage.
--> 1.Target location prediction
1.kf.predict(): Target location prediction
2.Total number of target box predictions: age+=1.
3.if time_since_update > 0:
hit_streak = 0
time_since_update += 1
1.Number of consecutive predictions per execution predict One time time_since_update+=1.
2.In continuous prediction(Continuous execution predict)Once executed update If so, time_since_update Will be reset to 0.
3.In continuous prediction(Continuous execution predict)In the process, as long as the number of continuous predictions time_since_update If greater than 0,
Will put hit_streak(Number of consecutive updates)Reset to 0, indicating that there is no update state update vector in the process of continuous prediction x(state variable x)Operation of,
That is, the continuous prediction process has not been executed once update.
4.In continuous update(Continuous execution update)The process, once started, is executed continuously predict Two or more times,
When executed for the first time in a row predict When, because time_since_update Still 0 does not hit_streak Reset to 0,
Then it will be carried out time_since_update+=1；
When executed for the second time in a row predict When, because time_since_update If it is already 1, it will hit_streak Reset to 0,
Then proceed time_since_update+=1.
--> 2.If the similarity between the predicted target and the detected target is successfully matched, enter update In the update phase, otherwise, if the matching fails, the tracking target will be deleted.
3.Step 3:
Kalman filter update Update phase:
If the similarity match between the prediction target and the detection target is successful, then it will enter the update stage.
kf.update([x,y,s,r]): Used by yoloV3 The obtained "and matched with the prediction box" detection box to update the prediction box.
--> 1.The target location information is updated to the detection target chain/Tracking target chain
1.Total target box updates: hits+=1.
2.history = []
time_since_update = 0
hit_streak += 1
1.history The list is used to save multiple results of continuous prediction of a single target box in the prediction stage. Once executed update Will be emptied history List.
2.Number of consecutive updates per execution update One time hit_streak+=1.
3.In continuous prediction(Continuous execution predict)Once executed update If so, time_since_update Will be reset to 0.
4.In continuous prediction(Continuous execution predict)In the process, as long as the number of continuous predictions time_since_update If greater than 0,
Will put hit_streak(Number of consecutive updates)Reset to 0, indicating that there is no update state update vector in the process of continuous prediction x(state variable x)Operation of,
That is, the continuous prediction process has not been executed once update.
5.In continuous update(Continuous execution update)The process, once started, is executed continuously predict Two or more times,
When executed for the first time in a row predict When, because time_since_update Still 0 does not hit_streak Reset to 0,
Then it will be carried out time_since_update+=1；
When executed for the second time in a row predict When, because time_since_update If it is already 1, it will hit_streak Reset to 0,
Then proceed time_since_update+=1.
--> 2.Target position correction.
1.kf.update([x,y,s,r]):
Use observed target box bbox Update status variable x(State update vector x).
Used by yoloV3 The obtained "and matched with the prediction frame" detection frame is used to update the prediction frame obtained by Kalman filter.
1.Initialization, prediction, update
1.__init__(bbox):
Initialize the state update vector of Kalman filter x(state variable x),Observation input[x,y,s,r](adopt[x1,y1,x2,y2]Transformed from),State transition matrix F,
Measurement matrix H(Observation matrix H),Covariance matrix of measurement noise R,Covariance matrix of a priori estimation P,Covariance matrix of process excitation noise Q.
2.update(bbox): Update the state vector according to the observation input x(state variable x)Update
3.predict(): Update vector based on status x(state variable x)Update the results to predict the bounding box of the target
2.State variable, state transition matrix F,Measurement matrix H(Observation matrix H),Covariance matrix of measurement noise R,Covariance matrix of a priori estimation P,Covariance matrix of process excitation noise Q
1.State update vector x(state variable x)
State update vector x(state variable x)The setting of is a 7-dimensional vector: x=[u,v,s,r,u^,v^,s^]T.
u,v Each represents the position of the center point of the target box x,y Coordinates, s Represents the area of the target box, r Represents the aspect ratio of the target box/Aspect ratio.
u^,v^,s^Transverse direction u(x direction),portrait v(y direction),the measure of area s The rate of change of motion.
u,v,s,r Initialization: initialize according to the observation results of the first frame.
u^,v^,s^Initialization: it is initialized to 0 at the beginning of the first frame, and will change according to the prediction results in the next frame.
2.State transition matrix F
It defines a 7*7 Square matrix of(Its diagonal values are all 1). .
The determination of motion form and transformation matrix is based on uniform motion model and state transition matrix F According to the kinematic formula, the tracked target is assumed to be a uniform moving target.
Pass 7*7 State transition matrix F Multiply by 7*1 Status update vector x(state variable x)You can get an updated 7*1 Status update vector x，
Of which, the updated u,v,s This is the result of the current frame.
3.Measurement matrix H(Observation matrix H)
Measurement matrix H(Observation matrix H)，It defines a 4*7 Matrix of.
Pass 4*7 Measurement matrix H(Observation matrix H) Multiply by 7*1 Status update vector x(state variable x) You can get a 4*1 of[u,v,s,r]Estimate of.
4.Covariance matrix of measurement noise R,Covariance matrix of a priori estimation P,Covariance matrix of process excitation noise Q
1.Covariance matrix of measurement noise R: diag([1,1,10,10]T)
2.Covariance matrix of a priori estimation P: diag([10,10,10,10,1e4,1e4,1e4]T). 1e4: 1x10 The fourth power of.
3.Covariance matrix of process excitation noise Q: diag([1,1,1,1,0.01,0.01,1e-4]T). 1e-4: 1x10 of-4 Power.
4.1e Meaning of numbers
1e4: 1x10 The fourth power of
1e-4: 1x10 of-4 Power
5.diag Represents a diagonal matrix, written as diag(a1，a2,...,an)The diagonal matrix of is actually represented by the values on the main diagonal in turn a1，a2,...,an，
A matrix whose elements outside the main diagonal are all 0.
Diagonal matrix(diagonal matrix)Is a matrix whose elements outside the main diagonal are all 0. It is often written as diag(a1，a2,...,an) .
Diagonal matrix can be considered as the simplest kind of matrix. It is worth mentioning that the elements on the diagonal can be 0 or other values, and the diagonal matrix with equal elements on the diagonal is called quantity matrix;
A diagonal matrix whose elements are all 1 on the diagonal is called an identity matrix. The operations of diagonal matrix include sum, difference, number multiplication and the product of diagonal matrix of the same order, and the result is still diagonal matrix.
"""
"""
1.Tracker chain(list):
In fact, it is multiple Kalman filters KalmanBoxTracker A list of instance objects of a custom class.
Each target frame has a corresponding Kalman filter(KalmanBoxTracker Instance object)，
KalmanBoxTracker The instance attribute in the class is specifically responsible for recording various statistical parameters in its corresponding target box,
The class attribute is used to record the number of Kalman filters created. Adding a target box adds a Kalman filter(KalmanBoxTracker Instance object).
Put each Kalman filter(KalmanBoxTracker Instance object)Are stored in the tracker chain(list)Yes.
2.unmatched_detections(list):
1.A new target appears in the detection box, but the prediction box appears(Tracking box)The target still does not exist in,
Then you need to create the prediction box corresponding to the new target/Tracking box(KalmanBoxTracker Class)，
Then put the corresponding KalmanBoxTracker Class into the tracker chain(list)Yes.
2.At the same time, if the matching degree of the pairwise combination of "tracking frame and detection frame" IOU Value less than iou Threshold,
The target detection box should also be placed in the unmatched_detections Yes.
3.unmatched_trackers(list):
1.When the tracking target fails or the target leaves the screen, that is, the target disappears from the detection box, the tracking box corresponding to the target shall be closed(Prediction frame)Remove from tracker chain.
unmatched_trackers What is saved in the list is the target that leaves the screen when tracking fails, but the prediction box corresponding to the target/Tracking box(KalmanBoxTracker Class)
It still exists in the tracker chain(list)Therefore, it is necessary to put the prediction box corresponding to the target/Tracking box(KalmanBoxTracker Class)Slave tracker chain(list)Delete from.
2.At the same time, if the matching degree of the pairwise combination of "tracking frame and detection frame" IOU Value less than iou Threshold,
The tracking target box should also be placed in the unmatched_trackers Yes.
"""
#Target estimation model Kalman filter
class KalmanBoxTracker(object):
"""
Each target frame has a corresponding Kalman filter(KalmanBoxTracker Instance object)，
KalmanBoxTracker The instance attribute in the class is specifically responsible for recording various statistical parameters in its corresponding target box,
The class attribute is used to record the number of Kalman filters created. Adding a target box adds a Kalman filter(KalmanBoxTracker Instance object).
"""
count = 0 #Class attribute is responsible for recording the number of Kalman filters created. Adding a target box adds a Kalman filter (Kalman boxtracker instance object)
"""
__init__(bbox)
Use target box bbox Initialize the state of the Kalman filter. Passed in during initialization bbox，That is, initialization is performed according to the results of the observed detection frame.
Each target frame has a corresponding Kalman filter(KalmanBoxTracker Instance object)，
KalmanBoxTracker The instance attribute in the class is specifically responsible for recording various statistical parameters in its corresponding target box,
The class attribute is used to record the number of Kalman filters created. Adding a target box adds a Kalman filter(KalmanBoxTracker Instance object).
1.kf = KalmanFilter(dim_x=7, dim_z=4)
A Kalman filter is defined to estimate the state of the target.
dim_x=7 The definition is a 7-dimensional state update vector x(state variable x): x=[u,v,s,r,u^,v^,s^]T.
dim_z=4 The definition is a 4-dimensional observation input, that is, the form of central area[x,y,s,r]，Namely[Center position of detection frame x coordinate,y coordinate,the measure of area,Aspect ratio].
2.kf.F = np.array(7*7 Square matrix of)
State transition matrix F，It defines a 7*7 Square matrix of its(The values on the diagonal are all 1).
Pass 7*7 State transition matrix F Multiply by 7*1 Status update vector x(state variable x)You can get an updated 7*1 Status update vector x，
Of which, the updated u,v,s This is the result of the current frame.
The current observation results are estimated through the state transition matrix to obtain the prediction results, and then the current prediction results are used as the basis for the next estimation and prediction.
3.kf.H = np.array(4*7 Matrix)
Measurement matrix H(Observation matrix H)，It defines a 4*7 Matrix of.
Pass 4*7 Measurement matrix H(Observation matrix H) Multiply by 7*1 Status update vector x(state variable x) You can get a 4*1 of[u,v,s,r]Estimate of.
4.The corresponding covariance parameters are set according to the empirical value.
1.R Is the covariance matrix of the measurement noise, that is, the covariance of the difference between the true real value and the measured value.
R=diag([1,1,10,10]T)
kf.R[2:, 2:] *= 10.
2.P Is the covariance matrix of a priori estimation
diag([10,10,10,10,1e4,1e4,1e4]T). 1e4: 1x10 The fourth power of.
kf.P[4:, 4:] *= 1000. # A large value is set, which brings great uncertainty to the unobservable initial velocity
kf.P *= 10.
3.Q Is the covariance matrix of process excitation noise
diag([1,1,1,1,0.01,0.01,1e-4]T). 1e-4: 1x10 of-4 Power.
kf.Q[-1, -1] *= 0.01
kf.Q[4:, 4:] *= 0.01
5.kf.x[:4] = convert_bbox_to_z(bbox)
convert_bbox_to_z Be responsible for[x1,y1,x2,y2]Form detection box bbox Change to the form of central area[x,y,s,r].
State update vector x(state variable x)The setting is a seven dimensional vector: x=[u,v,s,r,u^,v^,s^]T.
x[:4]That is to say u,v,s,r Initialize to first frame bbox Observed results[x,y,s,r].
6.Update of statistical parameters in a single Kalman filter corresponding to a single target frame
Each target frame has a corresponding Kalman filter(KalmanBoxTracker Instance object)，
KalmanBoxTracker The instance attribute in the class is specifically responsible for recording various statistical parameters in its corresponding target box,
The class attribute is used to record the number of Kalman filters created. Adding a target box adds a Kalman filter(KalmanBoxTracker Instance object).
1.Number of Kalman filters
There are as many Kalman filters as there are target boxes. Each target box will have a Kalman filter, that is, each target box will have a Kalman filter KalmanBoxTracker Instance object.
count = 0: Class attribute is responsible for recording the number of Kalman filters created. Adding a target box adds a Kalman filter(KalmanBoxTracker Instance object.
id = KalmanBoxTracker.count: The number of Kalman filters, that is, the number of target frames.
KalmanBoxTracker.count += 1: Every time you add a target box, you add one KalmanBoxTracker Instance object(Kalman filter )，Then class properties count+=1.
2.Count the statistics times of each parameter in the Kalman filter corresponding to a target frame
1.age = 0:
The total number of times the target box is predicted. Every execution predict Once, then age+=1.
2.hits = 0:
The total number of times the target box was updated. Every execution update Once, then hits+=1.
3.time_since_update = 0
1.Number of consecutive predictions per execution predict One time time_since_update+=1.
2.In continuous prediction(Continuous execution predict)Once executed update If so, time_since_update Will be reset to 0.
3.In continuous prediction(Continuous execution predict)In the process, as long as the number of continuous predictions time_since_update If greater than 0,
Will put hit_streak(Number of consecutive updates)Reset to 0, indicating that there is no update state update vector in the process of continuous prediction x(state variable x)Operation of,
That is, the continuous prediction process has not been executed once update.
4.hit_streak = 0
1.Number of consecutive updates per execution update One time hit_streak+=1.
2.In continuous update(Continuous execution update)The process, once started, is executed continuously predict Two or more times,
When executed for the first time in a row predict When, because time_since_update Still 0 does not hit_streak Reset to 0,
Then it will be carried out time_since_update+=1；
When executed for the second time in a row predict When, because time_since_update If it is already 1, it will hit_streak Reset to 0,
Then proceed time_since_update+=1.
7.history = []:
Save multiple results of continuous prediction of a single target box to history List, once executed update Will be emptied history List.
The predicted candidate box is in the form of central area[x,y,s,r]Convert to coordinate form[x1,y1,x2,y2] of bbox Save to history In the list.
"""
def __init__(self, bbox):
# Define isokinetic model
# Kalman filter is used internally, with 7 state variables and 4 observation inputs
self.kf = KalmanFilter(dim_x=7, dim_z=4)
# F is the state transformation model, which is a 7 * 7 square matrix
self.kf.F = np.array([[1, 0, 0, 0, 1, 0, 0],
[0, 1, 0, 0, 0, 1, 0],
[0, 0, 1, 0, 0, 0, 1],
[0, 0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0],
[0, 0, 0, 0, 0, 1, 0],
[0, 0, 0, 0, 0, 0, 1]])
# H is the measurement matrix, which is a 4 * 7 matrix
self.kf.H = np.array([[1, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0],
[0, 0, 1, 0, 0, 0, 0],
[0, 0, 0, 1, 0, 0, 0]])
# R is the covariance of the measurement noise, that is, the covariance of the difference between the real value and the measured value
self.kf.R[2:, 2:] *= 10.
# P is the covariance of a priori estimates
self.kf.P[4:, 4:] *= 1000. # It brings great uncertainty to the unobservable initial velocity
self.kf.P *= 10.
# Q is the covariance of the process excitation noise
self.kf.Q[-1, -1] *= 0.01
self.kf.Q[4:, 4:] *= 0.01
# state estimation
self.kf.x[:4] = convert_bbox_to_z(bbox)
# Update of parameters
self.time_since_update = 0
self.id = KalmanBoxTracker.count
KalmanBoxTracker.count += 1
self.history = []
self.hits = 0
self.hit_streak = 0
self.age = 0
"""
update(bbox): Use observed target box bbox Update status update vector x(state variable x)
1.time_since_update = 0
1.Number of consecutive predictions per execution predict One time time_since_update+=1.
2.In continuous prediction(Continuous execution predict)Once executed update If so, time_since_update Will be reset to 0.
2.In continuous prediction(Continuous execution predict)In the process, as long as the number of continuous predictions time_since_update If greater than 0,
Will put hit_streak(Number of consecutive updates)Reset to 0, indicating that there is no update state update vector in the process of continuous prediction x(state variable x)Operation of,
That is, the continuous prediction process has not been executed once update.
2.history = []
empty history List.
history The list saves multiple results of continuous prediction of a single target box([x,y,s,r]Converted[x1,y1,x2,y2])，Once executed update Will be emptied history List.
3.hits += 1:
The total number of times the target box was updated. Every execution update Once, then hits+=1.
4.hit_streak += 1
1.Number of consecutive updates per execution update One time hit_streak+=1.
2.In continuous update(Continuous execution update)The process, once started, is executed continuously predict Two or more times,
When executed for the first time in a row predict When, because time_since_update Still 0 does not hit_streak Reset to 0,
Then it will be carried out time_since_update+=1；
When executed for the second time in a row predict When, because time_since_update If it is already 1, it will hit_streak Reset to 0,
Then proceed time_since_update+=1.
5.kf.update(convert_bbox_to_z(bbox))
convert_bbox_to_z Be responsible for[x1,y1,x2,y2]The detection frame of the form is transformed into the state representation of the filter[x,y,s,r]，Then the passed in is kf.update([x,y,s,r]).
Then modify the internal state according to the observation results x(State update vector x).
Used by yoloV3 The obtained "and matched with the prediction frame" detection frame is used to update the prediction frame obtained by Kalman filter.
"""
#Update the status variable and use the observed target box bbox to update the status variable
def update(self, bbox):
"""
Update the state vector with the observed target box. filterpy.kalman.KalmanFilter.update Internal state estimates are modified based on observations self.kf.x.
Reset self.time_since_update，empty self.history.
:param bbox:Target box
:return:
"""
# Reset
self.time_since_update = 0
# Empty history
self.history = []
# hits count plus 1
self.hits += 1
self.hit_streak += 1
# Modify the internal state x according to the observation results.
#convert_bbox_to_z is responsible for transforming the detection frame in the form of [x1,y1,x2,y2] into the state representation of the filter [x,y,s,r], then the input from update is (x,y,s,r)
self.kf.update(convert_bbox_to_z(bbox))
"""
predict: Predict the target box and return the predicted bounding box results
1.if(kf.x[6] + kf.x[2]) <= 0:
self.kf.x[6] *= 0.0
State update vector x(state variable x)by[u,v,s,r,u^,v^,s^]T，that x[6]by s^，x[2]by s.
If x[6]+x[2]<= 0，that x[6] *= 0.0，Immediately s^Set to 0.0.
2.kf.predict()
Predict the target box.
3.age += 1
The total number of times the target box is predicted. Every execution predict Once, then age+=1.
4.if time_since_update > 0:
hit_streak = 0
time_since_update += 1
1.Number of consecutive predictions per execution predict One time time_since_update+=1.
2.In continuous prediction(Continuous execution predict)Once executed update If so, time_since_update Will be reset to 0.
3.In continuous prediction(Continuous execution predict)In the process, as long as the number of continuous predictions time_since_update If greater than 0,
Will put hit_streak(Number of consecutive updates)Reset to 0, indicating that there is no update state update vector in the process of continuous prediction x(state variable x)Operation of,
That is, the continuous prediction process has not been executed once update.
4.In continuous update(Continuous execution update)The process, once started, is executed continuously predict Two or more times,
When executed for the first time in a row predict When, because time_since_update Still 0 does not hit_streak Reset to 0,
Then it will be carried out time_since_update+=1；
When executed for the second time in a row predict When, because time_since_update If it is already 1, it will hit_streak Reset to 0,
Then proceed time_since_update+=1.
5.history.append(convert_x_to_bbox(kf.x))
convert_x_to_bbox(kf.x): The result predicted by the target box is in the form of central area[x,y,s,r] Convert to coordinate form[x1,y1,x2,y2] of bbox.
history The list saves multiple results of continuous prediction of a single target box([x,y,s,r]Converted[x1,y1,x2,y2])，Once executed update Will be emptied history List.
6.predict Return value: history[-1]
Put the target box to the current prediction result([x,y,s,r]Converted[x1,y1,x2,y2])Return output.
"""
#Predict the target box, advance the state variables, and return the predicted bounding box results
def predict(self):
"""
Advance the state vector and return the predicted bounding box estimation.
Append forecast results to self.history. because get_state Direct access self.kf.x，therefore self.history Not used
:return:
"""
# Propulsion state variable
if (self.kf.x[6] + self.kf.x[2]) <= 0:
self.kf.x[6] *= 0.0
# Make predictions
self.kf.predict()
# Times of Kalman filter
self.age += 1
# If it has not been updated during the process, it will hit_ Set streak to 0
if self.time_since_update > 0:
self.hit_streak = 0
self.time_since_update += 1
# Append forecast results to history
self.history.append(convert_x_to_bbox(self.kf.x))
return self.history[-1]
"""
get_state():
Get the predicted result of the current target box([x,y,s,r]Converted[x1,y1,x2,y2]).
return convert_x_to_bbox(kf.x): Change the candidate box from the form of central area[x,y,s,r] Convert to coordinate form[x1,y1,x2,y2] of bbox And return output.
Direct access kf.x And return, so history Not used.
"""
def get_state(self):
"""
Returns the current bounding box estimate.
because get_state Direct access self.kf.x，therefore self.history Not used.
:return:
"""
#Convert the candidate box from the form of central area [x,y,s,r] to bbox of coordinate form [x1,y1,x2,y2]
return convert_x_to_bbox(self.kf.x)