Skip to content

zju3dv/InfiniDepth

BranchesTags

Repository files navigation

Telescope InfiniDepth: Arbitrary-Resolution and Fine-Grained Depth Estimation with Neural Implicit Fields

Hao Yu* • Haotong Lin* • Jiawei Wang* • Jiaxin Li • Yida Wang • Xueyang Zhang • Yue Wang
Xiaowei Zhou • Ruizhen Hu • Sida Peng

InfiniDepth Demo

📢 News

[2026-04] 🎉 Training and evaluation code of InfiniDepth (RGB Only & Depth Sensor Augmentation) is available now!

[2026-03] 🎉 Inference code of InfiniDepth (RGB Only & Depth Sensor Augmentation) is available now!

[2026-02] 🎉 InfiniDepth has been accepted to CVPR 2026! Code coming soon!

✨ What can InfiniDepth do?

InfiniDepth supports three practical capabilities for single-image 3D perception and reconstruction:

Capability Input Output
Monocular & Arbitrary-Resolution Depth Estimation RGB Image Arbitrary-Resolution Depth Map
Monocular View Synthesis RGB Image 3D Gaussian Splatting (3DGS)
Depth Sensor Augmentation (Monocular Metric Depth Estimation) RGB Image + Depth Sensor Metric Depth + 3D Gaussian Splatting (3DGS)

⚙️ Installation

Please see INSTALL.md for manual installation.

🤗 Hugging Face Space Demo

If you want to test InfiniDepth before running local CLI inference, start with the hosted demo:

This repo also includes a Gradio Space entrypoint at app.py:

  • Input: RGB image (required), depth map (optional)
  • Task Switch: Depth / 3DGS
  • Model Switch: InfiniDepth / InfiniDepth_DepthSensor

Local run

python app.py

Notes

  • In this demo, InfiniDepth_DepthSensor requires a depth map input; RGB-only inference should use InfiniDepth.
  • Supported depth formats in the demo upload: .png, .npy, .npz, .h5, .hdf5, .exr.

🚀 Inference

Quick Command Index

If you want ... Recommended command
Relative Depth from Single RGB Image bash example_scripts/infer_depth/courtyard_infinidepth.sh
3D Gaussian from Single RGB Image bash example_scripts/infer_gs/courtyard_infinidepth_gs.sh
Metric Depth from RGB + Depth Sensor bash example_scripts/infer_depth/eth3d_infinidepth_depthsensor.sh
3D Gaussian from RGB + Depth Sensor bash example_scripts/infer_gs/eth3d_infinidepth_depthsensor_gs.sh
Multi-View / Video Depth + Global Point Cloud bash example_scripts/infer_depth/waymo_multi_view_infinidepth.sh
1. Relative Depth from Single RGB Image (inference_depth.py)

Use this when you want a relative depth map from a single RGB image and, optionally, a point cloud export.

Required input

  • RGB image

Required checkpoints

  • checkpoints/depth/infinidepth.ckpt
  • checkpoints/moge-2-vitl-normal/model.pt recover metric scale for point cloud export

Optional checkpoint

  • checkpoints/sky/skyseg.onnx additional sky filtering

Recommended command

python inference_depth.py \
  --input_image_path=example_data/image/courtyard.jpg \
  --model_type=InfiniDepth \
  --depth_model_path=checkpoints/depth/infinidepth.ckpt \
  --output_resolution_mode=upsample \
  --upsample_ratio=2

Replace example_data/image/courtyard.jpg with your own image path.

For the example above, outputs are written to

  • example_data/pred_depth/ for the colorized depth map
  • example_data/pred_pcd/ for the exported point cloud when --save_pcd=True

Example scripts

bash example_scripts/infer_depth/courtyard_infinidepth.sh
bash example_scripts/infer_depth/camera_infinidepth.sh
bash example_scripts/infer_depth/eth3d_infinidepth.sh
bash example_scripts/infer_depth/waymo_infinidepth.sh

Most useful options

Argument What it controls
--output_resolution_mode Choose upsample, original, or specific.
--upsample_ratio Used when output_resolution_mode=upsample.
--output_size Explicit output size (H,W) when output_resolution_mode=specific.
--save_pcd Export a point cloud alongside the depth map.
--fx_org --fy_org --cx_org --cy_org Camera intrinsics in the original image resolution.
2. 3D Gaussian + Novel-View Video from Single RGB Image (inference_gs.py)

Use this when you want a 3D Gaussian export from a single RGB image and an optional novel-view video.

Required input

  • RGB image

Required checkpoints

  • checkpoints/depth/infinidepth.ckpt
  • checkpoints/gs/infinidepth_gs.ckpt
  • checkpoints/moge-2-vitl-normal/model.pt recover metric scale for 3D Gaussian export

Optional checkpoint

  • checkpoints/sky/skyseg.onnx additional sky filtering

Recommended command

python inference_gs.py \
  --input_image_path=example_data/image/courtyard.jpg \
  --model_type=InfiniDepth \
  --depth_model_path=checkpoints/depth/infinidepth.ckpt \
  --gs_model_path=checkpoints/gs/infinidepth_gs.ckpt

Replace example_data/image/courtyard.jpg with your own image path.

For the example above, outputs are written to

  • example_data/pred_gs/InfiniDepth_courtyard_gaussians.ply
  • example_data/pred_gs/InfiniDepth_courtyard_novel_orbit.mp4

If --render_size is omitted, the novel-view video is rendered at the original input image resolution.

Example scripts

bash example_scripts/infer_gs/courtyard_infinidepth_gs.sh
bash example_scripts/infer_gs/camera_infinidepth_gs.sh
bash example_scripts/infer_gs/fruit_infinidepth_gs.sh
bash example_scripts/infer_gs/eth3d_infinidepth_gs.sh

Most useful options

Argument What it controls
--render_novel_video Turn novel-view rendering on or off.
--render_size Output video resolution (H,W).
--novel_trajectory Camera motion type: orbit or swing.
--sample_point_num Number of sampled points used for gaussian construction.
--enable_skyseg_model Enable sky masking before gaussian sampling.
--sample_sky_mask_dilate_px Dilate the sky mask before filtering.

The exported .ply files can be visualized in 3D viewers such as SuperSplat.

3. Depth Sensor Augmentation (Metric Depth and 3D Gaussian from RGB + Depth Sensor)

Use this mode when you have an RGB image plus metric depth from a depth sensor.

Required inputs

  • RGB image
  • Sparse depth in .png, .npy, .npz, .h5, .hdf5, or .exr

Required checkpoints

  • checkpoints/depth/infinidepth_depthsensor.ckpt
  • checkpoints/moge-2-vitl-normal/model.pt
  • checkpoints/gs/infinidepth_depthsensor_gs.ckpt

Required flags

  • --model_type=InfiniDepth_DepthSensor
  • --input_depth_path=...

Metric Depth Inference Command

python inference_depth.py \
  --input_image_path=example_data/image/eth3d_office.png \
  --input_depth_path=example_data/depth/eth3d_office.npz \
  --model_type=InfiniDepth_DepthSensor \
  --depth_model_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
  --fx_org=866.39 \
  --fy_org=866.04 \
  --cx_org=791.5 \
  --cy_org=523.81 \
  --output_resolution_mode=upsample \
  --upsample_ratio=1

3D Gaussian Inference Command

python inference_gs.py \
  --input_image_path=example_data/image/eth3d_office.png \
  --input_depth_path=example_data/depth/eth3d_office.npz \
  --model_type=InfiniDepth_DepthSensor \
  --depth_model_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
  --gs_model_path=checkpoints/gs/infinidepth_depthsensor_gs.ckpt \
  --fx_org=866.39 \
  --fy_org=866.04 \
  --cx_org=791.5 \
  --cy_org=523.81

Example scripts

bash example_scripts/infer_depth/eth3d_infinidepth_depthsensor.sh
bash example_scripts/infer_depth/waymo_infinidepth_depthsensor.sh
bash example_scripts/infer_gs/eth3d_infinidepth_depthsensor_gs.sh
bash example_scripts/infer_gs/waymo_infinidepth_depthsensor_gs.sh

Most useful options

Argument What it controls
--fx_org --fy_org --cx_org --cy_org Strongly recommended when you know the sensor intrinsics.
--output_resolution_mode Output behavior for inference_depth.py.
--render_size Video resolution for inference_gs.py.
--output_ply_dir Custom output directory for gaussian export.
4. Multi-View / Video Depth + Global Point Cloud (inference_multi_view_depth.py)

Use this when you want sequence-level depth inference from an RGB image folder or video, plus per-frame aligned point clouds and one merged global point cloud. By default the script runs DA3 once on the whole sequence, then aligns each InfiniDepth depth map to the corresponding DA3 depth map before export. When you already know the camera intrinsics and extrinsics, you can instead provide them directly and skip DA3 entirely.

Required inputs

  • RGB image directory, single RGB image, or video
  • Sparse depth directory / single file / depth video when --model_type=InfiniDepth_DepthSensor

Required checkpoints / dependencies

  • checkpoints/depth/infinidepth.ckpt for RGB-only inference
  • checkpoints/depth/infinidepth_depthsensor.ckpt for RGB + depth sensor inference
  • checkpoints/moge-2-vitl-normal/model.pt recover metric scale for RGB-only frame inference
  • depth-anything-3 installed in the current environment when using the default DA3-based sequence mode; default DA3 model is depth-anything/DA3-LARGE-1.1

Optional checkpoint

  • checkpoints/sky/skyseg.onnx additional sky filtering

RGB-Only Multi-View / Video Command

python inference_multi_view_depth.py \
  --input_path=example_data/multi-view/waymo/image \
  --model_type=InfiniDepth \
  --depth_model_path=checkpoints/depth/infinidepth.ckpt \

RGB + Depth Sensor Multi-View / Video Command

python inference_multi_view_depth.py \
  --input_path=example_data/multi-view/waymo/image \
  --input_depth_path=example_data/multi-view/waymo/depth \
  --model_type=InfiniDepth_DepthSensor \
  --depth_model_path=checkpoints/depth/infinidepth_depthsensor.ckpt \

For video input, replace --input_path with a video file. When --model_type=InfiniDepth_DepthSensor, --input_depth_path can also be a depth video and must contain the same number of frames as the RGB input.

Explicit Camera-Parameter Multi-View Command

python inference_multi_view_depth.py \
  --input_path=example_data/multi-view/waymo/image \
  --camera_intrinsics_dir=/path/to/intrinsics \
  --camera_extrinsics_dir=/path/to/extrinsics \
  --model_type=InfiniDepth \
  --depth_model_path=checkpoints/depth/infinidepth.ckpt \

The explicit camera mode expects Waymo-style text files under intrinsics/ and extrinsics/. Files are sorted lexicographically and matched one-to-one against the sorted RGB image list, so the number of camera files must exactly match the number of images. In this mode the script skips DA3 loading, DA3 cache export, DA3 RANSAC conditioning, and DA3 post scale alignment. This mode currently supports image inputs only, not video.

For the RGB-only example above, outputs are written to

  • example_data/multi-view/waymo/pred_sequence/image/frames/depth/ for aligned raw depth maps
  • example_data/multi-view/waymo/pred_sequence/image/frames/depth_vis/ for colorized depth maps
  • example_data/multi-view/waymo/pred_sequence/image/frames/pcd/ for per-frame aligned point clouds
  • example_data/multi-view/waymo/pred_sequence/image/frames/meta/ for per-frame camera and alignment metadata
  • example_data/multi-view/waymo/pred_sequence/image/da3/sequence_pose.npz for cached DA3 predictions
  • example_data/multi-view/waymo/pred_sequence/image/merged/sequence_merged.ply for the merged global point cloud

Example scripts

bash example_scripts/infer_depth/waymo_multi_view_infinidepth.sh
bash example_scripts/infer_depth/waymo_multi_view_infinidepth_depthsensor.sh
bash example_scripts/infer_depth/waymo_multi_view_infinidepth_explicit_camera.sh

Most useful options

Argument What it controls
--input_path RGB image directory, single image, or video path.
--input_depth_path Depth directory, single depth file, or depth video; required for InfiniDepth_DepthSensor.
--camera_intrinsics_dir --camera_extrinsics_dir Enable explicit camera mode from sorted Waymo-style txt directories. Image inputs only; file counts must match the RGB frame count.
--input_mode Force images or video instead of auto detection.
--align_to_da3_depth Align each InfiniDepth depth map to the corresponding DA3 depth map before export. Ignored in explicit camera mode.
--save_frame_pcd Save one aligned point cloud per frame.
--save_merged_pcd Save the merged global point cloud across the whole sequence.
--da3_scale_align_conf_threshold Minimum DA3 confidence used during per-frame scale estimation.
--output_root Override the default pred_sequence/<sequence_name>/ output directory.
5. Common Argument Conventions
Argument Used in Description
--input_image_path depth + gs Path to the input RGB image.
--input_path multi-view Path to an RGB image directory, single image, or video.
--input_depth_path depth + gs + multi-view Optional metric depth prompt; required for InfiniDepth_DepthSensor. In multi-view mode this can be a depth directory, single depth file, or depth video.
--camera_intrinsics_dir --camera_extrinsics_dir multi-view Optional sequence camera parameter directories. When both are set, multi-view inference skips DA3 and uses the provided sorted txt files directly.
--model_type depth + gs + multi-view InfiniDepth for RGB-only, InfiniDepth_DepthSensor for RGB + sparse depth.
--depth_model_path depth + gs Path to the depth checkpoint.
--gs_model_path gs only Path to the gaussian predictor checkpoint.
--moge2_pretrained depth + gs MoGe-2 checkpoint used when --input_depth_path is missing.
--fx_org --fy_org --cx_org --cy_org depth + gs Camera intrinsics in original image resolution. Missing values fall back to MoGe-2 estimates or image-size defaults.
--input_size depth + gs Network input size (H,W) used during inference.
--enable_skyseg_model depth + gs + multi-view Enable sky masking before depth or gaussian sampling.
--sky_model_ckpt_path depth + gs Path to the sky segmentation ONNX checkpoint.

Depth output modes

  • --output_resolution_mode=upsample: output size = input_size * upsample_ratio
  • --output_resolution_mode=original: output size = original input image size
  • --output_resolution_mode=specific: output size = output_size

Default output directories

Script Default directory
inference_depth.py depth images pred_depth/ next to your input data folder
inference_depth.py point clouds pred_pcd/ next to your input data folder
inference_gs.py gaussians and videos pred_gs/ next to your input data folder
inference_multi_view_depth.py sequence outputs pred_sequence/<sequence_name>/ next to your input data folder

🏋️ Training and Validation

The repo also provides main.py training and validation entrypoints for InfiniDepth and InfiniDepth_DepthSensor.

First, prepare the training/validation data and the pretrained weight as described in DATA.md.

Before running any command, export the environment variables required by main.py:

export workspace=/path/to/your/experiments
export commonspace=/path/to/your/common_space
  • workspace stores experiment outputs under outputs/<task>/<exp_name>/
  • commonspace stores datasets and pretrained weights shared across experiments

Quick Command Index

If you want ... Recommended command
Fine-tune from an existing checkpoint Add ckpt_path=... to the training command
Train from scratch Omit ckpt_path and use a fresh exp_name
Validate on the mixed real-data benchmark Run main.py with --entry val
1. Fine-Tuning from an Existing Checkpoint (main.py, default train_net entry)

Use this when you want to initialize training from an existing InfiniDepth checkpoint. The training config referenced below uses Hypersim as the training set and runs validation on a mixed real-data benchmark at the end of each epoch.

Required environment

  • workspace=/path/to/your/experiments
  • commonspace=/path/to/your/common_space

RGB-Only Fine-Tuning Command

python3 main.py \
  --cfg_file training/exp_configs/exps/infinidepth.yaml \
  --include training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml \
  ckpt_path=checkpoints/depth/infinidepth.ckpt \
  exp_name=finetune_infinidepth_on_hypersim \
  model.compute_abs_metric=True \
  model.save_orig_pred=True \
  model.save_metrics=True \
  pl_trainer.devices=8

RGB + Depth Sensor Fine-Tuning Command

python3 main.py \
  --cfg_file training/exp_configs/exps/infinidepth_depthsensor.yaml \
  --include training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml \
  ckpt_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
  exp_name=finetune_infinidepth_depthsensor_on_hypersim \
  model.compute_abs_metric=True \
  model.save_orig_pred=True \
  model.save_metrics=True \
  pl_trainer.devices=8

Equivalent launch scripts

bash launch_scripts/train/infinidepth.sh
bash launch_scripts/train/infinidepth_depthsensor.sh

Outputs

  • ${workspace}/outputs/${task}/${exp_name}/checkpoints/ for saved checkpoints
  • ${workspace}/outputs/${task}/${exp_name}/tb/ for TensorBoard logs

Notes

  • ckpt_path must point to an existing checkpoint.
  • The training data config is training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml.
  • If the same exp_name already has checkpoints in ${workspace}/outputs/${task}/${exp_name}/checkpoints/, training will resume from the latest saved checkpoint in that directory.
2. Train from Scratch (main.py, no ckpt_path)

Use this when you want to start training without loading an InfiniDepth .ckpt. In this mode, do not pass ckpt_path. The model will still load the DINOv3 backbone from ${commonspace}/pretrained_models/dinov3/. You need to download the DINOv3 weights yourself and place them there before running.

RGB-Only Training-from-Scratch Command

python3 main.py \
  --cfg_file training/exp_configs/exps/infinidepth.yaml \
  --include training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml \
  exp_name=train_infinidepth_from_scratch \
  model.compute_abs_metric=True \
  model.save_orig_pred=True \
  model.save_metrics=True \
  pl_trainer.devices=2

RGB + Depth Sensor Training-from-Scratch Command

python3 main.py \
  --cfg_file training/exp_configs/exps/infinidepth_depthsensor.yaml \
  --include training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml \
  exp_name=train_infinidepth_depthsensor_from_scratch \
  model.compute_abs_metric=True \
  model.save_orig_pred=True \
  model.save_metrics=True \
  pl_trainer.devices=2

Notes

  • Use a new exp_name for a clean scratch run.
  • If you intentionally want to reuse an old exp_name, set resume_training=False to prevent automatic resume. Be careful: when resume_training=False, the code will delete the old output directory before training.
3. Validation from a Checkpoint (main.py with --entry val)

Use this when you want to run validation metrics on the mixed real-data benchmark defined in training/exp_configs/components/data/test/infinidepth_mix_data.yaml.

RGB-Only Validation Command

python3 main.py \
  --cfg_file training/exp_configs/exps/infinidepth.yaml \
  --include training/exp_configs/components/data/test/infinidepth_mix_data.yaml \
  --entry val \
  ckpt_path=checkpoints/depth/infinidepth.ckpt \
  exp_name=eval_infinidepth \
  model.compute_abs_metric=True \
  model.save_orig_pred=True \
  model.save_metrics=True \
  pl_trainer.devices=2

RGB + Depth Sensor Validation Command

python3 main.py \
  --cfg_file training/exp_configs/exps/infinidepth_depthsensor.yaml \
  --include training/exp_configs/components/data/test/infinidepth_mix_data.yaml \
  --entry val \
  ckpt_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
  exp_name=eval_infinidepth_depthsensor \
  model.compute_abs_metric=True \
  model.save_orig_pred=True \
  model.save_metrics=True \
  pl_trainer.devices=2

Reference launch scripts

bash launch_scripts/eval/infinidepth.sh
bash launch_scripts/eval/infinidepth_depthsensor.sh

Validation datasets

  • Kitti val split
  • ETH3D val split
  • NYU val split
  • ScanNet val split
  • DIODE indoor val split
  • Synth4K (CyberPunk, DeadIsland, SpiderMan2, SpiderManMM, WatchDogLegion)

These datasets are read from ${commonspace}/datasets/ using the meta files referenced in training/exp_configs/components/data/test/infinidepth_mix_data.yaml.

Outputs

  • ${workspace}/outputs/${task}/${exp_name}/val_metrics/ for validation logs
  • ${workspace}/outputs/${task}/${exp_name}/default/metrics/metrics.json for aggregated validation metrics
  • ${workspace}/outputs/${task}/${exp_name}/default/metrics/all_scenes.csv when model.save_metrics=True
4. Common Overrides
Argument What it controls
ckpt_path Initialization or evaluation checkpoint path. Omit it for training from scratch.
exp_name Experiment name used to build ${workspace}/outputs/${task}/${exp_name}.
pl_trainer.devices Number of GPUs used by PyTorch Lightning.
model.compute_abs_metric Enable absolute-metric evaluation during training or validation.
model.save_orig_pred Save original prediction outputs alongside logs and metrics.
model.save_metrics Save metric files for later inspection.
--entry val Switch main.py from the default training entry to the validation entry.
--include Merge an extra data config, such as training/exp_configs/components/data/test/infinidepth_mix_data.yaml.

🙏 Acknowledgments

We thank Yuanhong Yu, Gangwei Xu, Haoyu Guo and Chongjie Ye for their insightful discussions and valuable suggestions, and Zhen Xu for his dedicated efforts in curating the synthetic data.

📖 Citation

If you find InfiniDepth useful in your research, please consider citing:

@article{yu2026infinidepth,
    title={InfiniDepth: Arbitrary-Resolution and Fine-Grained Depth Estimation with Neural Implicit Fields},
    author={Hao Yu, Haotong Lin, Jiawei Wang, Jiaxin Li, Yida Wang, Xueyang Zhang, Yue Wang, Xiaowei Zhou, Ruizhen Hu and Sida Peng},
    booktitle={arXiv preprint},
    year={2026}
}
Thanks

Thank you for your interest in InfiniDepth!

⭐ Star this repo if you find it interesting!

About

[CVPR 2026] InfiniDepth: Arbitrary-Resolution and Fine-Grained Depth Estimation with Neural Implicit Fields

zju3dv.github.io/InfiniDepth/

Resources

Readme

License

Apache-2.0 license
Activity
Custom properties

Stars

922 stars

Watchers

65 watching

Forks

35 forks
Report repository

Releases 1

InfiniDepth code and models - Official Release 🎉 Latest
Mar 24, 2026

Packages

 
 
 

Contributors

Languages