标题： X2Video：适应扩散模型的多模态可控制神经视频渲染 显示英文标题

标题： X2Video: Adapting Diffusion Models for Multimodal Controllable Neural Video Rendering

摘要： 我们提出X2Video，这是第一个基于内在通道（包括反照率、法线、粗糙度、金属度和辐照度）生成逼真视频的扩散模型，同时支持通过参考图像和文本提示对全局和局部区域进行直观的多模态控制。 内在引导允许对颜色、材质、几何和光照进行精确操作，而参考图像和文本提示在缺乏内在信息的情况下提供了直观的调整方式。 为了实现这些功能，我们通过采用一种新颖且高效的混合自注意力机制，将基于内在引导的图像生成模型XRGB扩展到视频生成，这确保了视频帧之间的时序一致性，并提高了与参考图像的一致性。 我们进一步开发了一种掩码交叉注意力机制，以分离全局和局部文本提示，并将其有效地应用于相应的局部和全局区域。 为了生成长视频，我们的新方法递归采样结合了关键帧预测和帧插值，以保持长距离时序一致性的同时防止误差累积。 为了支持X2Video的训练，我们整理了一个名为InteriorVideo的视频数据集，包含来自295个室内场景的1,154个房间，配有可靠的地面真实内在通道序列和平滑的相机轨迹。 定性和定量评估表明，X2Video可以在内在条件的引导下生成长时序一致且逼真的视频。 此外，X2Video能够有效适应多模态控制，包括参考图像、全局和局部文本提示，并通过参数调节能同时支持对颜色、材质、几何和光照的编辑。 项目页面：https://luckyhzt.github.io/x2video 显示英文摘要

摘要： We present X2Video, the first diffusion model for rendering photorealistic videos guided by intrinsic channels including albedo, normal, roughness, metallicity, and irradiance, while supporting intuitive multi-modal controls with reference images and text prompts for both global and local regions. The intrinsic guidance allows accurate manipulation of color, material, geometry, and lighting, while reference images and text prompts provide intuitive adjustments in the absence of intrinsic information. To enable these functionalities, we extend the intrinsic-guided image generation model XRGB to video generation by employing a novel and efficient Hybrid Self-Attention, which ensures temporal consistency across video frames and also enhances fidelity to reference images. We further develop a Masked Cross-Attention to disentangle global and local text prompts, applying them effectively onto respective local and global regions. For generating long videos, our novel Recursive Sampling method incorporates progressive frame sampling, combining keyframe prediction and frame interpolation to maintain long-range temporal consistency while preventing error accumulation. To support the training of X2Video, we assembled a video dataset named InteriorVideo, featuring 1,154 rooms from 295 interior scenes, complete with reliable ground-truth intrinsic channel sequences and smooth camera trajectories. Both qualitative and quantitative evaluations demonstrate that X2Video can produce long, temporally consistent, and photorealistic videos guided by intrinsic conditions. Additionally, X2Video effectively accommodates multi-modal controls with reference images, global and local text prompts, and simultaneously supports editing on color, material, geometry, and lighting through parametric tuning. Project page: https://luckyhzt.github.io/x2video