WebGPURenderer: Persistent video texture approach

[sunag]

Closes #31330, #31397

Description

Replaces texture_external with a persistent texture using copyExternalImageToTexture() updating by requestVideoFrameCallback of VideoTexture, reusing all of the core's texture management, including sharing between materials, and adds support for vertex stage usage too.

@Mugen87 I think this is the best way, in addition to the benefits, it reduces the total code.

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	338.28 78.93	338.28 78.93	+0 B +0 B
WebGPU	558.52 154.58	557.63 154.34	-893 B -248 B
WebGPU Nodes	557.45 154.37	556.55 154.12	-893 B -249 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	469.62 113.66	469.62 113.66	+0 B +0 B
WebGPU	634.18 171.62	633.28 171.4	-892 B -220 B
WebGPU Nodes	589.31 160.95	588.42 160.71	-892 B -239 B

[Mugen87]

Yes, that sounds like a good solution 👍 .

Are parts of the PR missing though? I don't see any changes to VideoTexture or usage of copyExternalImageToTexture()?

[sunag]

Great, right? The current texture system was reused, and most things worked auto-magically.

[Mugen87]

Ah I see, the code goes now in the WebGPUTextureUtils._copyImageToTexture() branch. Neat!

[Mugen87]

It seems this PR broke VideoFrameTexture in webgpu_video_frame when using WebGPU. The WebGL backend works fine.

[ycw]

This introduced copy overheads that would be hamstringed by LPDDR bandwidth in mobiles.

[Mugen87]

This introduced copy overheads that would be hamstringed by LPDDR bandwidth in mobiles.

@greggman We were hoping to simplify our code base by not using texture_external and importExternalTexture() for video textures in WebGPURenderer. Instead the idea is to copy the video frame via copyExternalImageToTexture() and then treat it as a normal texture.

Does this approach via copyExternalImageToTexture() indeed introduce a performance issue?

[greggman]

Does this approach via copyExternalImageToTexture() indeed introduce a performance issue?

I'm not really sure how to define "performance issue" 😛

copying is slower than not copying. importExternalTexture is designed to allow no copy. Conceptually you're directly using some external texture. That said, it's up to the implementation whether or not it lets you directly use some external texture or makes a temporary copy.

In Chrome, if you enable webgpu-developer-features, then GPUExternalTexture has a isZeroCopy boolean you can use to see if it's using the external texture directly or making a temporary copy. This is affected by platform/OS/gpu/video format.

copyExternalImageToTexture on the other hand, definitely always makes a copy. This is the same as texImage2D(,,,, someElement) which also always makes a copy. Whether either is fast (GPU based copy) or slow (CPU based copy) up to the implementation. That said, making a copy was your only option in WebGL and people have been using video there so it's not bad/wrong to use this path. And, as you discovered, it's easier and more flexible since you don't have to change any shaders.

[Mugen87]

Then I would say we keep the status quo and see how it performs in production. Maintaining a separate code path for video textures in the WebGPU backend was a bit of a pain and the new approach is so much more elegant.