Object detection and tracking have been crucial concerns in the 2D^[1,2] and 3D^[3,4] vision community. Multiple object tracking (MOT) aims to analyze videos to identify and track objects. Because of the flexibility of unmanned aerial vehicles (UAVs) equipped with cameras, MOT in UAV videos has become a new research hotspot and trend in recent years.

Online MOT algorithms can be reduced to the two-step and one-shot approaches. The two-step approach^[1,2] mainly follows the tracking-by-detection (TBD) paradigm^[5]. It formulates the MOT task as two steps of object detection and data association. The one-shot approach^[6,7] joints detection and embedding learning, and does not separate the re-identification (Re-ID) model as usual in TBD. As of now, almost all UAV-based MOT algorithms adopt the TBD paradigm. Different from the common observation perspectives in MOT, such as fixed security cameras and moving cars, MOT in UAV videos must pay more attention to the following challenges. (1) Scale changes and small objects: the flight altitude of the UAV changes with time, and the object scale varies greatly at different altitudes. Because of the wide field of view and high altitude, the objects are usually small. (2) Real-time performance: MOT in UAV videos needs to locate fast moving ground objects, so algorithms should be fast enough to be used in industrial applications.

Previous efforts have been made to solve these problems. IPGAT^[8] predicts complex motions using a conditional generative adversarial networks model. However, it neglects to utilize multi-scale appearance information of an object. M-CMSN-M^[9] unifies single object tracking and MOT for multi-task learning using a Siamese network. Nevertheless, the speed of M-CMSN-M^[9] is extremely slow due to the numerous objects in UAV videos.

In this Letter, we are committed to coping with both aforementioned problems simultaneously. To address the problem of scale changes, we design a feature-aligned attention network (FAANet), which is mainly composed of two modules: the channel and spatial attention (CSA) module and feature-aligned aggregation (FAA) module. The CSA module adaptively enhances multi-scale features, and the FAA module successively generates alignment bias of two different resolution features. FAANet integrates multi-scale features to improve the robustness of object scale changes. To address the problem of real-time, we adopt the joint-detection-embedding (JDE) paradigm^[6] and adopt the structural re-parameterization technique^[10] to increase the network inference speed.

The major contributions of this paper are summarized as follows. (1) We propose an FAANet to enhance and aggregate multi-scale features so as to cope with drastic scale changes in UAV videos. (2) We introduce the JDE paradigm to MOT in UAV videos and use the structural re-parameterization technique to increase the network inference speed. (3) Extensive experiments are conducted on UAV detection and tracking (UAVDT)^[9] benchmarks to verify effectiveness of the proposed method.

In this section, we first present the architecture of the proposed FAANet and then explain the details of the CSA module, FAA module, and online inference.

As shown in Fig. 1, the framework of proposed FAANet contains four components: feature extractor backbone, feature fusion neck, detection and Re-ID prediction heads, and online tracking association. We adopt RepVGG^[10] as our backbone to extract multi-scale features with minor modifications to their channels and layers. Let the shape of the input image be $3\times H_{\text{input}}\times H_{\text{input}}$; then the shape of multi-scale output features is, respectively, $C_i\times H_i\times W_i$, where $C_i=64\times 2^{i-1}$, $H_i=H_{\text{input}}{/2}^{i+1}$, $W_i=W_{\text{input}}{/2}^{i+1}$, and $i=1,2,3,4$. We adopt the same detection and Re-ID heads as FairMOT^[7], including heat map, box size, center offset, and Re-ID embeddings.

In general, the input and output dimensions of the channel attention (CA) or spatial attention (SA) are the same^[11,12]. In order to improve real-time performance and reduce the risk of overfitting, we propose a method combining feature dimension reduction, channel attention, and spatial attention, as shown in Fig. 2. We perform channel attention first, and then spatial attention. We refer to the effective channel attention mechanism^[11] and modify it by 2D $1\times 1$ convolution and one-dimensional convolution with variant kernel and stride.

Specifically, let the one output of the backbone be $F\in R^{C\times H\times W}$, where $C$, $H$, and $W$ are channel dimension, height, and width. Accordingly, the weights of channels can be computed as $$w_{\mathrm{channel}}=\sigma (\mathrm{Conv}1\mathrm{d}(g(F))),$$(1)where $g(F)=\frac{1}{WH}{\sum }_{i,j=1}^{W,H}{F}_{i,j}$ is the channel-wise global average pooling (GAP), and $\sigma$ is the sigmoid nonlinear activation function. We set the $\mathrm{Con}\mathrm{v}1\mathrm{d}$ with a kernel as 3, 7, 11, 15 and stride as 1, 2, 4, 8, respectively, in four multi-scale features. In this way, all channel dimensions of multi-scale features can be reduced to 64. Subsequently, the output of channel attention can be computed as $$F_{\text{channel}}=w_{\text{channel}}\odot \text{Conv}2\mathrm{d}(F)+\text{Conv}2\mathrm{d}(F),$$(2)where $\odot$ denotes element-wise product. The output dimension of $\mathrm{Con}\mathrm{v}2\mathrm{d}(1\times 1)$ is 64. Specially, for $i=1,C=64$, we replace $\mathrm{Con}\mathrm{v}2\mathrm{d}(1\times 1)$ with an identity function.

In contrast to channel attention, which focuses on channel dimension, spatial attention mainly focuses on the height and width dimension. Inspired by polarized self-attention^[12], we also adopt the polarized method to perform spatial attention. Specifically, let the output of channel attention be $F_{\text{channel}}\in R^{64\times H\times W}$, so the spatial weights can be computed by the following formulas: $$F_1=\text{soft}\max (f_1(g(\text{Conv}2\mathrm{d}(F_{\text{channel}})))),$$(3)$$F_2=f_2(\text{Conv}2\mathrm{d}(F_{\text{channel}})),$$(4)$$w_{\text{spatial}}=\sigma (f_3(F_1\otimes F_2)),$$(5)where $f_1$, $f_2$, and $f_3$ denote different reshape operation, and $\text{softmax}(\text{·})$ denotes the softmax function. After calculating the spatial weights, the final output of the CSA module can be computed as $$F_{\text{spatial}}=w_{\text{spatial}}\odot F_{\text{channel}},$$(6)$$F_{\text{output}}=F_{\text{channel}}+F_{\text{spatial}}.$$(7)

Because the channel attention adopts one-dimensional convolution instead of the full connection as usual, the number of parameters is greatly reduced, and the inference speed is improved. The number of channels is reduced to 64 dimensions through the proposed CSA module, which also reduces the number of parameters for real-time performance and provides a channel consistent input to the FAA module.

Traditional multi-scale feature aggregation usually adopts the method of bi-linear interpolation up-sampling and element-size summation. However, because of the feature misalignment, feature degradation will occur, which leads to the degradation of the ability to locate objects at different scales. After our meticulous research, we refer to the feature-aligned mechanism in AlignSeg^[13] and apply it to the UAV-based MOT domain.

The FAA module is shown in Fig. 3. Specifically, let the corresponding two outputs of the CSA module be $F_{\text{high}}\in R^{64\times H\times W}$ and $F_{\mathrm{low}}\in R^{64\times (H/2)\times (W/2)}$, where $F_{\text{high}}$ and $F_{\mathrm{low}}$, respectively, denote the feature map with high and low resolution. The network will learn to generate the offsets of two feature maps for alignment. Let the corresponding two offsets be ${\mathrm{\Delta }}_{\text{high}}\in R^{64\times H\times W}$ and ${\mathrm{\Delta }}_{\mathrm{low}}\in R^{64\times (H/2)\times (W/2)}$; then they can be computed by the following formulas: $$F_{\text{concat}}=\text{Concat}(\text{upsample}(F_{\mathrm{low}}),F_{\text{high}}),$$(8)$${\mathrm{\Delta }}_{\mathrm{low}}=\text{Conv}2{\mathrm{d}}_{3\times 3}(\mathrm{ReLU}(\mathrm{BN}(\text{Conv}2{\mathrm{d}}_{1\times 1}(F_{\text{concat}})))),$$(9)$${\mathrm{\Delta }}_{\text{high}}=\text{Conv}2{\mathrm{d}}_{3\times 3}(\mathrm{ReLU}(\mathrm{BN}(\text{Conv}2{\mathrm{d}}_{1\times 1}(F_{\text{concat}})))),$$(10)where $\text{Concat}(F,F)$ denotes the concatenation operation of two feature maps along the channel dimension, $\mathrm{upsample}(F)$ denotes the bilateral interpolation function, $\text{Conv}2{\mathrm{d}}_{3\times 3}$ and $\text{Conv}2{\mathrm{d}}_{1\times 1}$ denote the convolution layer with $3\times 3$ and $1\times 1$ kernels, and ReLU and BN, respectively, denote the rectified linear unit activation function and batch normalization layer. Let the output of the FAA module be $F_{\text{output}}$; then $F_{\text{output}}$ can be computed as $$F_{\text{output}}=f(\text{upsample}(F_{\mathrm{low}}),{\mathrm{\Delta }}_{\mathrm{low}})+f(F_{\mathrm{high}},{\mathrm{\Delta }}_{\mathrm{high}}),$$(11)where $f(F,\mathrm{\Delta })$ is defined in AlignSeg^[13]. It can be described like this: suppose $A_{h,w}$ is the output of the alignment function $f(F,\mathrm{\Delta })$ in the spatial coordinates for position $(h,w)$; then $A_{h,w}$ can be computed as $$\begin{gathered} A_{h,w}=\sum _{h^{\prime }=1}^H\sum _{w^{\prime }=1}^W{F}_{h^{\prime },w^{\prime }}·\max (0,1-|h+{\mathrm{\Delta }}_{1hw}-h^{\prime }|) \\ ·\max (0,1-|w+{\mathrm{\Delta }}_{2hw}-w^{\prime }|), \end{gathered}$$(12)where ${\mathrm{\Delta }}_{1hw}$ and ${\mathrm{\Delta }}_{2hw}$ indicate the learned 2D transformation offsets for position $(h,w)$.

Herein, the two important components of online inference are data association and structural re-parameterization, which we will illustrate further.

We follow the standard online tracking algorithm to associate boxes. As shown in Fig. 4, we first initialize a few tracklets based on the estimated boxes in the first frame and use a Kalman filter to predict the locations of the tracklets in the next frame. We perform two Hungarian matchings between detections and tracklets sequentially. The first matching considers the appearance information (Re-ID embedding) measured by cosine distance and the motion information measured by Mahalanobis distance. The second matching only considers intersection over union (IOU) distance, which is simple but useful. Finally, we initialize new tracklets that meet confidence thresholds and mark the lost tracklets.

Structural re-parameterization is used in RepVGG^[10] to decouple a multi-branch topology with a plain architecture. Herein, we also utilize structural re-parameterization to increase the backbone network inference speed. As shown in Fig. 5, during the training phase, multi-branch topology is adopted to avoid the problem like gradient vanishing. After training, we perform the transformation with simple algebra in RepVGG^[10] and save the plain backbone architecture for online inference.

We evaluate our method on the dataset UAVDT^[9]. The dataset offers 50 sequences recorded from a UAV (60% for training and 40% for testing). We compare our method with various classic and recent algorithms on the testing sequences. We use multiple metrics, including MOT accuracy (MOTA), identification F1 score (IDF1), MOT precision (MOTP), mostly tracked targets (MT), mostly lost targets (ML), false positive (FP), false negative (FN), identification switches (IDS), and fragmented (FM) to evaluate different aspects of the tracking performance. MOTA is computed based on FP, FN, and IDS. Considering the amount of FP and FN is larger than that of IDS, MOTA focuses more on the detection performance. IDF1 focuses more on the association performance.

We choose RepVGG^[10] as our backbone. Its weights are initialized by the ImageNet-pretrained model. Specifically, we choose a slightly modified RepVGG-B0 as our default backbone, which has [2,4,6,16] blocks and [64,128,256,512] output channels. Our model is trained and tested on a NVIDIA GEFORCE RTX 1080Ti graphics processing unit. We adopt standard data augmentation techniques including scaling, rotation and color jittering. The input image is resized to $1024\times 544$, and the feature map resolution is $256\times 136$. We choose Adam as our optimizer. We set the starting learning rate as 8 × 10⁻⁵ and batch size as 10. The learning rate will decrease by a factor 10 per 15 epochs. The training step takes about 18 h with 35 epochs in total.

We evaluate our FAANet together with various classic and recent algorithms including CEM^[14], CMOT^[15], SORT^[1], DeepSORT^[2], GOG^[16], IOUT^[17], MDP^[18], SMOT^[19], DeepAlign^[20], SBMA^[21], IPGAT^[8], M-CMSN-M^[9], and Quadruplet^[22]. All the results of the comparison algorithms are obtained from recent published papers and the UAVDT benchmark^[9].

Since most of the current UAV-based MOT algorithms follow the TBD paradigm, many algorithms do not publish their own speed or only publish the speed of the association phase. That leads to difficulty and ambiguity in speed comparison of algorithms. As much as we can, we collect the currently publicly available algorithm speed and performance, which are shown in Fig. 6. Note that the three algorithms in comparison only calculate the time consumption in the association phase. It can be observed that the speed of our FAANet is 60 times higher than that of the current state-of-the-art M-CMSN-M^[9], while MOTA and IDF1 are slightly ahead. Detailed comparison between algorithms is shown in Table 1. We list the seven kinds of the most excellent trackers in recent years. We have the best performance on the 7 out of 10 metrics.

The comparison based on scene attributes is shown in Fig. 7. Our algorithm performs better than other algorithms in the high-alt, bird-view, and fog scenes. It demonstrates the effectiveness of the proposed module.

To validate the effectiveness of CA, SA, and FAA modules, we introduce a baseline RepVGG-B0 with a re-parameterization technique. The baseline reduces the feature dimension by $1\times 1$ convolution and performs multi-scale fusion by bi-linear interpolation up-sampling and element-size summation. Except for the above, all the other settings are consistent, such as training hyperparameters and association details in tracking. As shown in Table 2, the contributions of CA and SA are similar, but their combination can lead to better performance. Compared with CSA, FAA improves more IDF1, which may be due to the feature-aligned multi-scale aggregation boosting robustness to small objects and scale changes.

As shown in Table 3, we illustrate the speed improvement of the re-parameterization technique. It decreases the number of model parameters from 15.9 × 10⁶ to 14.4 × 10⁶ and the amount of floating-point operations (FLOPs) from 62.3 × 10⁹ to 58.3 × 10⁹. Generally, it increases frames per second (FPS) from 30.32 to 38.24. This is a 26% speed improvement without degeneration of any accuracy performance.

Figure 8 visualizes several typical scenes tracking comparison results between DeepSORT^[2] and FAANet on the UAVDT^[9] test dataset. From the results of M0403, M1301, and M1004, we can see that FAANet can better track small objects at the end of the road, which is caused by the wide field of vision. From the results of M0701, we can see that FAANet performs better in scenes of scale changes, which is caused by flight altitude.

The vertical numbers denote the number of objects tracked in the three frames. On average, FAANet can track 29% more objects than the classical DeepSORT^[2] method in the four typical scene examples. This is mainly attributed to the multi-scale feature enhancement and fusion of the CSA and FAA modules.

In this Letter, we propose an FAANet for MOT in UAV videos. Experimental results demonstrate that our methods can better cope with the problem of scale changes and small object with real-time speed. We hope that our method is attractive for application to industry due to its high accuracy and fast speed.