3D Reconstruction from a Single Video

Module

COMP5530 Group Project

Year

2025/26 · Year 4

Team

5 students, supervised by Sharib Ali

My role

Team lead, pipeline integration

Getting 3D geometry out of a single video is hard. Depth models give detailed depth for each frame, but it flickers from one frame to the next, so the frames don't line up when you stack them. Classical methods line frames up well but only recover sparse points. We combined both: two pretrained models do the perception, and a classical optimisation step makes their outputs agree with each other.

The input is one RGB video. The output is a textured mesh, a coloured point cloud and the camera's path through the scene, all in metres.

1. 1. Depth. Depth Anything V3 predicts depth in metres for every frame, plus a confidence map and a rough camera pose. Long videos are processed in overlapping 128-frame windows, each aligned to the first.
2. 2. Tracking. CoTracker3 follows thousands of points through the whole video, including through occlusion. As the camera moves into new areas, lost points are dropped and new ones are added.
3. 3. Geometry. A rigidity filter (iteratively reweighted least squares) removes tracks that don't fit a rigid scene, such as reflections or things that move. Then bundle adjustment refines every camera pose and a depth correction for every frame, so the tracked points project back to where they were seen.
4. 4. Fusion. The corrected depth maps are merged into a single volume (TSDF fusion in Open3D), and a mesh and point cloud are extracted from it.

Both models run with frozen weights, so nothing is trained per scene. Each stage saves its output, so a later stage can be re-run without repeating the expensive model passes. On a single GPU, a 2,000-frame video takes about 16 minutes from start to finish.

• • I led the pipeline integration: joining the depth and tracking outputs, writing the bundle adjustment, and adding the rigidity filter to make the optimisation robust.
• • I was the team lead. Most of the group hadn't covered this area before, so early on I ran knowledge-sharing sessions and helped break down the papers we were building on.
• • I split the work by what each person was strongest at: tracking, depth, integration, and two people on evaluation.

• • Replica (indoor, synthetic): our camera error of 1.56 cm was in line with NICE-SLAM's 1.95 cm, even though NICE-SLAM uses measured depth for every frame and we used colour video alone.
• • ScanNet (indoor, real): short, textured sequences came close to the Replica results. Long sequences that revisit the same room drifted, because the pipeline has no loop closure. Over the six standard sequences our mean camera error was 31.1 cm against NICE-SLAM's 9.63 cm.
• • Tanks and Temples (outdoor): the base pipeline struggled with large outdoor depth ranges. We added spatial depth corrections and matched the official evaluation protocol, which raised the share of surface within 5 cm from 0.35 to 0.52 on average, and from 0.29 to 0.60 on the Truck scene.

Our original plan ended with an AR/VR demo. When we tried it, the fused meshes were too detailed to render at a smooth frame rate, and our supervisor pointed out that the project needed far more rigorous benchmarking. We dropped the VR work and spent that time on evaluation across three datasets instead. We also tightened how we worked once integration got harder: pull requests with branch protection on GitHub, Jira for the Kanban board, Discord for day-to-day coordination and Overleaf for the report.