Artemis: Articulated Neural Pets with Appearance and Motion Synthesis

We, humans, are entering into a virtual era and indeed want to bring animals to the virtual world as companions. Yet computer-generated furry animals remain limited by tedious offline rendering and lack practical interactive motion control. ARTEMIS introduces a neural modeling and rendering pipeline for articulated neural pets with appearance and motion synthesis. The system combines an efficient octree-based animal representation, volumetric rendering, a neural shading network, multi-view animal motion capture, and neural character control, enabling real-time animation, photorealistic rendering, and immersive interaction with virtual animals in VR.

Given multi-view RGBA images of traditionally modeled animals rendered in canonical space, ARTEMIS first extracts a sparse voxel grid and allocates a corresponding feature look-up table together with an octree for efficient indexing. The character is then posed to training poses using the rig of the traditional animal asset, followed by efficient volumetric rendering to generate view-dependent appearance feature maps and coarse opacity maps. A convolutional neural shading network decodes these maps into high-quality appearance and opacity images, while adversarial training further enhances high-frequency details.

We show synthesized RGBA results of different neural volumetric animals in representative motions.

ARTEMIS also supports VR applications with multiple levels of interaction. Users can view neural pets from both third-person and first-person perspectives, issue low-level commands such as jump, move, and sit, provide high-level destination guidance, and interact with virtual animals that can follow, accompany, and explore their environment autonomously.

The authors thank Junyu Zhou and Ya Gao from DGene Digital Technology Co., Ltd. for processing the CGI animal models and motion capture data. We also thank Zhenxiao Yu and Heyang Li from ShanghaiTech University for producing supplementary video and figures.