Moving object classification across a subset of scenes from an aerial drone video.

This article explains the use of Fast-Fourier Transform and Short-Time Fourier Transform for object tracking. 

While standard optical object detection relies heavily on spatial image pixel convolutions, FFT and STFT are critical to detect moving objects. FFT (Fast Fourier Transform) converts a discrete signal from the time domain to the frequency domain where the frequency shift aka Doppler effect or time delay aka frequency beat reveals an objects distance and velocity. STFT (Short-Time Fourier Transform): Applies the FFT to localized, overlapping time windows. This captures how frequency changes over time, producing a spectrogram. It is essential for detecting moving objects, classifying micro-Doppler signatures (e.g., distinguishing a pedestrian from a cyclist). The squared magnitude of the STFT yields the Spectrogram, matrix data that object detection models (like 2D CNNs or Transformers) ingest to localize and classify targets.

To extract frequency-domain features from video pixel tracking, we must perform a 3D-to-1D reduction. A 2D CNN or Transformer cannot directly process a raw spatial image with an FFT/STFT along the temporal axis without creating a spatial-temporal bottleneck. Instead, we extract the 1D spatial trajectory vectors (the X and Y coordinates over time) or the temporal pixel intensity shifts of a tracking bounding box, and apply the STFT to those 1D trajectories. This translates the object's physical acceleration, micro-movements, and brief erratic motion into a 2D Time-Frequency Spectrogram that a standard image-based neural network can classify.

While deep learning frameworks like Swin Transformer 3D or TimeSformer handle video natively, engineering frequency features helps classify fast-moving vs. slow-moving targets (e.g., separating a speeding vehicle from a pedestrian) when data is sparse. 

Below is a complete, working pipeline that takes a sequence of aerial drone frames, tracks an object's spatial coordinates across the scenes, computes the STFT on its movement dynamics, and formats the output into a tensor ready for a 2D CNN or Vision Transformer (ViT):

#! /usr/bin/python

import numpy as np

import cv2

import matplotlib.pyplot as plt

from scipy.signal import stft

# ==========================================

# 1. SIMULATE AERIAL DRONE SCENE DATA

# ==========================================

def generate_mock_drone_video(num_frames=120, height=512, width=512):

    """

    Simulates a continuous aerial video snippet from 100m.

    An object (e.g., a cyclist) moves across the frame with micro-vibrations.

    """

    frames = []

    # Base drone scene background (textured noise)

    background = np.random.randint(100, 130, (height, width), dtype=np.uint8)

    

    # Simulate a target moving diagonally across the frame over time

    for t in range(num_frames):

        frame = background.copy()

        

        # Base trajectory + micro-oscillations (pedaling frequency/road bumps)

        center_x = int(50 + 3.2 * t + 2 * np.sin(0.8 * t))

        center_y = int(80 + 2.5 * t + 1.5 * np.cos(0.8 * t))

        

        # Draw the target if it is within bounds

        if 0 < center_x < width and 0 < center_y < height:

            # Simulated target boundary box footprint

            cv2.circle(frame, (center_x, center_y), radius=6, color=255, thickness=-1)

            

        frames.append(frame)

    return np.array(frames)

# ==========================================

# 2. EXTRACT PIXEL TRACKING TRAJECTORIES

# ==========================================

def track_object_centroid(video_frames):

    """

    Simulates an upstream Object Tracker (e.g., ByteTrack / Kalman Filter).

    Returns a 1D array of spatial positions over time.

    """

    trajectory_x = []

    trajectory_y = []

    

    # Simple centroid extraction loop via thresholding for demo purposes

    for frame in video_frames:

        _, thresh = cv2.threshold(frame, 240, 255, cv2.THRESH_BINARY)

        moments = cv2.moments(thresh)

        

        if moments["m00"] != 0:

            cx = moments["m10"] / moments["m00"]

            cy = moments["m01"] / moments["m00"]

        else:

            # Handle brief occlusion/disappearance by holding last known position

            cx = trajectory_x[-1] if trajectory_x else 0

            cy = trajectory_y[-1] if trajectory_y else 0

            

        trajectory_x.append(cx)

        trajectory_y.append(cy)

        

    return np.array(trajectory_x), np.array(trajectory_y)

# ==========================================

# 3. COMPUTE STFT TENSOR FOR DEEP LEARNING

# ==========================================

def generate_stft_features(trajectory_x, trajectory_y, fps=30):

    """

    Converts 1D motion tracking coordinates into a 2D Time-Frequency Spectrogram map.

    """

    # Convert absolute coordinates to velocity vectors (pixel displacement delta)

    vel_x = np.diff(trajectory_x, prepend=trajectory_x[0])

    vel_y = np.diff(trajectory_y, prepend=trajectory_y[0])

    

    # Compute Magnitude of the velocity vector

    velocity_magnitude = np.sqrt(vel_x**2 + vel_y**2)

    

    # Apply STFT to the velocity sequence

    # Short segment length (nperseg) is vital because targets appear briefly

    nperseg = min(32, len(velocity_magnitude)) 

    frequencies, times, Zxx = stft(velocity_magnitude, fs=fps, nperseg=nperseg, noverlap=nperseg-4)

    

    # Extract Power Spectral Density (Magnitude Squared)

    spectrogram = np.abs(Zxx)**2

    

    # Normalize to 0-255 range for standard 2D Image CNN/Transformer consumption

    log_spectrogram = 10 * np.log10(spectrogram + 1e-10)

    norm_spectrogram = cv2.normalize(log_spectrogram, None, 0, 255, cv2.NORM_MINMAX)

    

    return norm_spectrogram.astype(np.uint8), frequencies, times

# ==========================================

# 4. EXECUTION PIPELINE

# ==========================================

# Step A: Load video sequence (Simulated 30 FPS drone clip)

video_data = generate_mock_drone_video(num_frames=150, height=512, width=512)

# Step B: Get object tracking data across frames

x_coords, y_coords = track_object_centroid(video_data)

# Step C: Generate the 2D STFT target signature

stft_tensor, freqs, timeline = generate_stft_features(x_coords, y_coords, fps=30)

# Step D: Resize to square dimensions for standard networks (e.g., 224x224 for ViT/ResNet)

network_input = cv2.resize(stft_tensor, (224, 224), interpolation=cv2.INTER_CUBIC)

print(f"Processed Tracking Coordinates Shape: {x_coords.shape}")

print(f"Generated STFT Tensor Spectrogram Shape: {stft_tensor.shape}")

print(f"Final 2D CNN/Transformer Input Shape: {network_input.shape} (Ready for network integration)")

# ==========================================

# VISUALIZATION

# ==========================================

plt.figure(figsize=(10, 4))

plt.subplot(1, 2, 1)

plt.plot(x_coords, y_coords, '-o', color='teal', markersize=3)

plt.title("Spatial Pixel Path (Aerial View)")

plt.xlabel("X Coordinate")

plt.ylabel("Y Coordinate")

plt.grid(True)

plt.subplot(1, 2, 2)

plt.imshow(network_input, aspect='auto', cmap='magma', origin='lower')

plt.title("Resized Motion STFT Signature (224x224)")

plt.xlabel("Temporal Windows")

plt.ylabel("Frequency Components")

plt.colorbar(label='Normalized Energy')

plt.tight_layout()

plt.show()

Sample output:

Processed Tracking Coordinates Shape: (150,)

Generated STFT Tensor Spectrogram Shape: (17, 39)

Final 2D CNN/Transformer Input Shape: (224, 224) (Ready for network integration)

 

Conclusion:

The STFT translates continuous time-domain raw sensor signals into structured 2D spatial-frequency representations (spectrograms), enabling standard computer vision models and CFAR filters to accurately segment, classify, and track objects based on their range and Doppler velocity signatures.