about dynamic videos

[WenjiaWang0312]

Dear author, I noticed that you claim "Furthermore, if certain pixels in the input frames are unwanted (e.g., sea, glass), you can mask them by setting the corresponding pixel values to 0 or 1, and then pass the masked images to the model."
But I found when I masked out the walking person in this video, the model takes the black part as a black walking person rather than non-sense parts. So how could we achieve the masking similar to in VGGsfm or colmap? Like ignoring the masked pixels during feature matching.

[jytime]

wow thanks for such an interesting observation! I did not meet this (model predicting black pixels as human) before and will be glad to have a try myself and then try to answer your question. In theory the model should ignore these points... Would you mind sharing your images somewhere?

[WenjiaWang0312]

https://drive.google.com/file/d/161R3VWqqCXbgp92bMi1ODRtHfyD83JS6/view?usp=drive_link

Hi, I have shared it here. Please check

[jytime]

Hi @WenjiaWang0312 ,

It seems a bit counterintuitive, but very detailed segmentation masks still make the model think a person is there. In contrast, using simple bounding box masks appears to work better. I manually cropped the walking person using a rectangle, and the results look correct to me—please let me know if anything seems off on your end. Hope this workaround helps!

(Note: I didn’t filter out the masked white points, so the point cloud might look odd, but the camera poses appear reasonable.)

[WenjiaWang0312]

Thank you for your help!

[DocterStrawberry]

Hi both, I have an opposite question: how to also reconstruct the human in the scene instead of masking them out? I tried to input a video or an image, with a human in the scene, but there is no human in the reconstruction result but only the background there