comda2k19 dataset, the sequence length as 800 costs too much memory

[ztb-35]

Dear authors:
In the comma2k19datset, the length of image sequences is set as 800. It costs too much memory, and the supercombo model inputs are only the last 2 frames of the image sequences.
Why don't we set the image sequences length shorter to save memory?

[ElectronicElephant]

Yes you are correct. Unfortunately, while it's the best solution at least theoretically, properly reading the frames from the videos and batchfying them on-the-fly requires much development. Therefore, we decide to extract all the frames of the videos in the batch first before the training batch begins. That's why it consumes much memory. If you are interested in implementing the "on-the-fly" mechanism, discussions and PRs are always welcomed.

[ztb-35]

I remove the imges sequnce in the dataset, and only return the last two images, in this way I save a lot memory

[aoeno]

I have a question about the GRU part:

The GRU module is designed to process image sequences and rely on the hidden state being passed step-by-step across the entire sequence to capture long-term temporal dependencies. If the DataLoader now only returns two consecutive frames each time, does that mean:

• We are effectively only training on two-frame short sequences?
• The hidden state can no longer be propagated through the full historical sequence (e.g., the previous 798 frames)?
• Have we completely lost the ability to model long-term temporal information?

In other words, after this “only return the last two frames” modification, is the model still learning meaningful long-range dependencies via GRU, or has it silently degraded to a model that only sees adjacent frames (similar to optical flow or simple two-frame difference models)?

Additionally, if the implementation does NOT simply return two frames and actually preserves the propagation of the GRU hidden state across the long sequence (so long-term dependencies are still modeled), could you please explain in detail how this is achieved? For example:

• How are the images loaded and fed into the model?
• How is the hidden state passed across hundreds of time steps while keeping memory usage low?
• Is the sequence split into chunks? Is the hidden state manually cached and carried over between batches/clips? Or is some kind of pre-computed hidden state loaded?

Looking forward to your explanation, thank you!

[ztb-35]

Thanks for your questions! I am also curious about how the training process is, and why they load a long sequence in the dataloader. We know that the model (e.g., the Supercombo model in OpenPilot) only takes two consecutive frames as input.

I only load the two frames as input to the Supercombo and check the results during inference (no training). And I found the results(predicted ego vehicle trajectory) are not reliable. For example, the video shows the ego vehicle's current velocity is zero(stopped), the predicted trajectory shows the predicted ego vehicle velocity is high(20 meters in 1 second).

I think it's because the hidden state is always initialized as zeros during my inference. I checked the output from the OpenPilot, and the OpenPilot results are either not reliable. The predicted velocity is also very high with a stopped ego vehicle in the video. But it converges to zero after about 25 frames.

I built a recurrent pipeline in my inference stage. The input sequence contains the last 26 frames, and each time the model loads two consecutive frames from frame_1 to frame_26 and updates the hidden state. After 25 iterations, I can get the updated hidden state, and combining it with the current two consecutive frames the desire, and the traffic as the input to the Supercombo, the predicted trajectory is much more reliable, and the ego vehicle velocity is close to zero.

I think the hidden state captures long-term temporal dependencies and is updated during inference, but the training procedure is not known.

[aoeno]

Thanks for your reply！

The current implementation loads an entire sequence at a time. Since seq_input_img takes in the full sequence of images (e.g., 400 frames) and temporal information is propagated via a GRU module during training, the data cannot be loaded frame-by-frame; it must be loaded as a sequence. GPU memory limitations prevent the loading of such long temporal sequences, forcing the batch size to be very low, which leads to unstable training. Essentially, the core bottleneck during training is BPTT (Backpropagation Through Time).When the sequence length is long (e.g., 400 frames), PyTorch must store the entire computational graph for all 400 steps in GPU memory because gradients need to be backpropagated from step 400 all the way to step 1. Each step contains feature maps extracted by EfficientNet, which consumes a massive amount of GPU memory. Consequently, the Batch Size is restricted to 1 or 2. Batch Normalization statistics are inaccurate at such small batch sizes (<8), resulting in extreme training instability or even a failure to converge.

I am currently attempting a new training approach where each sample retains a history of 20 frames. Features are extracted from these 20 frames via the backbone, and temporal feature information is fused using Self-Attention or TCN (Temporal Convolutional Networks) to replace the GRU. When running on the real vehicle, inference is performed on a single-frame basis (inputting only the current frame). After feature extraction, historical frame features are stored in a Memory Bank, where they are fused with the current frame's features before proceeding to decoding.

[ElectronicElephant]

Thanks for your reply！

The current implementation loads an entire sequence at a time. Since seq_input_img takes in the full sequence of images (e.g., 400 frames) and temporal information is propagated via a GRU module during training, the data cannot be loaded frame-by-frame; it must be loaded as a sequence. GPU memory limitations prevent the loading of such long temporal sequences, forcing the batch size to be very low, which leads to unstable training. Essentially, the core bottleneck during training is BPTT (Backpropagation Through Time).When the sequence length is long (e.g., 400 frames), PyTorch must store the entire computational graph for all 400 steps in GPU memory because gradients need to be backpropagated from step 400 all the way to step 1. Each step contains feature maps extracted by EfficientNet, which consumes a massive amount of GPU memory. Consequently, the Batch Size is restricted to 1 or 2. Batch Normalization statistics are inaccurate at such small batch sizes (<8), resulting in extreme training instability or even a failure to converge.

I am currently attempting a new training approach where each sample retains a history of 20 frames. Features are extracted from these 20 frames via the backbone, and temporal feature information is fused using Self-Attention or TCN (Temporal Convolutional Networks) to replace the GRU. When running on the real vehicle, inference is performed on a single-frame basis (inputting only the current frame). After feature extraction, historical frame features are stored in a Memory Bank, where they are fused with the current frame's features before proceeding to decoding.

Hi, thanks for your interest, but I am sure that GPU memory is related with model.module.optimize_per_n_step, which is 40 by default. This means PyTorch stores the computational graph within 40 steps, not 400.

The 400 length is more related with CPU RAM consumption.