human capture
67740839
Human capture system for virtual reality applications
The growing competition between the manufacturing companies results in constant optimization of their processes. This applies not only to the production process, but also to the process of product development. Digitization plays a very important role in this development. Through the process of digitization, things can be quickly designed, changed, and visualized. One of the great innovations that helps visualize the process of three dimensions is virtual reality (VR) With virtual reality, you are able to test and improve simulated processes interactively with user help [PD0915] In this paper, an experimental hardware and software system is to be presented, called "Human Capture System" This system allows for integration of various data sources like sensors or simple data streams.
Helge Drumm
3847788
MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Prior
Recovering 3D full body human pose is a challenging problem with many applications. It has been successfully addressed by motion capture systems with body worn markers and multiple cameras. In this paper, we address the more challenging case of not only using a single camera but also not leveraging markers: going directly from 2D appearance to 3D geometry. Deep learning approaches have shown remarkable abilities to discriminatively learn 2D appearance features. The missing piece is how to integrate 2D, 3D, and temporal information to recover 3D geometry and account for the uncertainties arising from the discriminative model. We introduce a novel approach that treats 2D joint locations as latent variables whose uncertainty distributions are given by a deep fully convolutional neural network. The unknown 3D poses are modeled by a sparse representation and the 3D parameter estimates are realized via an Expectation Maximization algorithm, where it is shown that the 2D joint location uncertainties can be conveniently marginalized out during inference. Extensive evaluation on benchmark datasets shows that the proposed approach achieves greater accuracy over state of the art baselines. Notably, the proposed approach does not require synchronized 2D 3D data for training and is applicable to "in the wild" images, which is demonstrated with the MPII dataset.
IEEE Transactions on Pattern Analysis and Machine Intelligence
Xiaowei Zhou Menglong Zhu Georgios Pavlakos Spyridon Leonardos Konstantinos G Derpanis Kostas Daniilidis
52931186
LiveCap: Real Time Human Performance Capture From Monocular Video
We present the first real time human performance capture approach that reconstructs dense, space time coherent deforming geometry of entire humans in general everyday clothing from just a single RGB video. We propose a novel two stage analysis by synthesis optimization whose formulation and implementation are designed for high performance. In the first stage, a skinned template model is jointly fitted to background subtracted input video, 2D and 3D skeleton joint positions found using a deep neural network, and a set of sparse facial landmark detections. In the second stage, dense non rigid 3D deformations of skin and even loose apparel are captured based on a novel real time capable algorithm for non rigid tracking using dense photometric and silhouette constraints. Our novel energy formulation leverages automatically identified material regions on the template to model the differing non rigid deformation behavior of skin and apparel. The two resulting non linear optimization problems per frame are solved with specially tailored data parallel Gauss Newton solvers. To achieve real time performance of over 25Hz, we design a pipelined parallel architecture using the CPU and two commodity GPUs. Our method is the first real time monocular approach for full body performance capture. Our method yields comparable accuracy with off line performance capture techniques while being orders of magnitude faster.
ACM Trans. Graph.
Marc Habermann Weipeng Xu Michael Zollhofer Gerard Pons-Moll Christian Theobalt
163164866
Resting brain dynamics at different timescales capture distinct aspects of human behavior
Linking human behavior to resting state brain function is a central question in systems neuroscience. In particular, the functional timescales at which different types of behavioral factors are encoded remain largely unexplored. The behavioral counterparts of static functional connectivity (FC) at the resolution of several minutes, have been studied but behavioral correlates of dynamic measures of FC at the resolution of a few seconds remain unclear. Here, using resting state fMRI and 58 phenotypic measures from the Human Connectome Project, we find that dynamic FC captures task based phenotypes (e.g. processing speed or fluid intelligence scores) whereas self reported measures (e.g. loneliness or life satisfaction) are equally well explained by static and dynamic FC. Furthermore, behaviorally relevant dynamic FC emerges from the interconnections across all resting state networks, rather than within or between pairs of networks. Our findings shed new light on the timescales of cognitive processes involved in distinct facets of behavior.An individual's pattern of resting state brain connectivity, as measured with fMRI, has been shown to predict cognitive and behavioral traits. Here, the authors show that different traits are predicted by different time scales of resting state activity (dynamic vs. static)
Nature Communications
Raphael Liegeois Jingwei Li Ru Kong Csaba Orban Dimitri Van de Ville Tian Ge Mert Rory Sabuncu B T Thomas Yeo
80628373
SimulCap Single View Human Performance Capture With Cloth Simulation
This paper proposes a new method for live free viewpoint human performance capture with dynamic details (e.g. cloth wrinkles) using a single RGBD camera. Our main contributions are: (i) a multi layer representation of garments and body, and (ii) a physics based performance capture procedure. We first digitize the performer using multi layer surface representation, which includes the undressed body surface and separate clothing meshes. For performance capture, we perform skeleton tracking, cloth simulation, and iterative depth fitting sequentially for the incoming frame. By incorporating cloth simulation into the performance capture pipeline, we can simulate plausible cloth dynamics and cloth body interactions even in the occluded regions, which was not possible in previous capture methods. Moreover, by formulating depth fitting as a physical process, our system produces cloth tracking results consistent with the depth observation while still maintaining physical constraints. Results and evaluations show the effectiveness of our method. Our method also enables new types of applications such as cloth retargeting, free viewpoint video rendering and animations.
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Tao Yu Zerong Zheng Yuan Zhong Jianhui Zhao Qionghai Dai Gerard Pons-Moll Yebin Liu
76665326
Recurrence required to capture the dynamic computations of the human ventral visual stream
The visual system is an intricate network of brain regions that enables us to recognize the world around us. Despite its abundant lateral and feedback connections, human object processing is commonly viewed and studied as a feedforward process. Here, we measure and model the rapid representational dynamics across multiple stages of the human ventral stream using time resolved brain imaging and deep learning. We observe substantial representational transformations during the first 300 ms of processing within and across ventral stream regions. Categorical divisions emerge in sequence, cascading forward and in reverse across regions, and Granger causality analysis suggests bidirectional information flow between regions. Finally, recurrent deep neural network models clearly outperform feedforward models in terms of their ability to jointly capture the multi region cortical dynamics. These results establish that recurrent models are required to understand information processing in the human ventral stream.
ArXiv
Tim C Kietzmann Courtney J Spoerer Lynn K A Sorensen Radoslaw Martin Cichy Olaf Hauk Nikolaus Kriegeskorte
203902672
Recurrence is required to capture the representational dynamics of the human visual system
Significance Understanding the computational principles that underlie human vision is a key challenge for neuroscience and could help improve machine vision. Feedforward neural network models process their input through a deep cascade of computations. These models can recognize objects in images and explain aspects of human rapid recognition. However, the human brain contains recurrent connections within and between stages of the cascade, which are missing from the models that dominate both engineering and neuroscience. Here, we measure and model the dynamics of human brain activity during visual perception. We compare feedforward and recurrent neural network models and find that only recurrent models can account for the dynamic transformations of representations among multiple regions of visual cortex. The human visual system is an intricate network of brain regions that enables us to recognize the world around us. Despite its abundant lateral and feedback connections, object processing is commonly viewed and studied as a feedforward process. Here, we measure and model the rapid representational dynamics across multiple stages of the human ventral stream using time resolved brain imaging and deep learning. We observe substantial representational transformations during the first 300 ms of processing within and across ventral stream regions. Categorical divisions emerge in sequence, cascading forward and in reverse across regions, and Granger causality analysis suggests bidirectional information flow between regions. Finally, recurrent deep neural network models clearly outperform parameter matched feedforward models in terms of their ability to capture the multiregion cortical dynamics. Targeted virtual cooling experiments on the recurrent deep network models further substantiate the importance of their lateral and top down connections. These results establish that recurrent models are required to understand information processing in the human ventral stream.
Proceedings of the National Academy of Sciences
Tim C Kietzmann Courtney J Spoerer Lynn K A Sorensen Radoslaw Martin Cichy Olaf Hauk Nikolaus Kriegeskorte
207560964
MonoPerfCap: Human Performance Capture from Monocular Video
We present the first marker less approach for temporally coherent 3D performance capture of a human with general clothing from monocular video. Our approach reconstructs articulated human skeleton motion as well as medium scale non rigid surface deformations in general scenes. Human performance capture is a challenging problem due to the large range of articulation, potentially fast motion, and considerable non rigid deformations, even from multi view data. Reconstruction from monocular video alone is drastically more challenging, since strong occlusions and the inherent depth ambiguity lead to a highly ill posed reconstruction problem. We tackle these challenges by a novel approach that employs sparse 2D and 3D human pose detections from a convolutional neural network using a batch based pose estimation strategy. Joint recovery of per batch motion allows to resolve the ambiguities of the monocular reconstruction problem based on a low dimensional trajectory subspace. In addition, we propose refinement of the surface geometry based on fully automatically extracted silhouettes to enable medium scale non rigid alignment. We demonstrate state of the art performance capture results that enable exciting applications such as video editing and free viewpoint video, previously infeasible from monocular video. Our qualitative and quantitative evaluation demonstrates that our approach significantly outperforms previous monocular methods in terms of accuracy, robustness and scene complexity that can be handled.
ACM Trans. Graph.
Weipeng Xu Avishek Chatterjee Michael Zollhofer Helge Rhodin Dushyant Mehta Hans-Peter Seidel Christian Theobalt
77387993
Using Wearable Sensors to Capture Posture of the Human Lumbar Spine in Competitive Swimming
Motion capture based on wearable inertial sensors is a promising technique for swimmers' training. To apply motion capture techniques properly, a swimming motion evaluation method based on inertial motion capture technology is proposed. Our proposed method uses a multisensor data fusion algorithm for swimmers' attitude estimation, and the swimming posture is reconstructed in combination with a human biomechanical model. Furthermore, a comparative experiment between our proposed motion capture system and the NDI motion tracking system shows that our system performs reliably and accurately, and the estimation errors are well controlled. In addition, the accuracy of the orientation estimation algorithm ranges from <inline formula><tex math notation="LaTeX"\text{1.65}\circ /tex math>/inline formula> to <inline formula><tex math notation="LaTeX"\text{3.66}\circ /tex math>/inline formula> The system can capture swimmers' lumbar spine in four competitive swimming styles. A kinematic analysis of lumbar spine movement indicates that the patterns of swimmers' lumbar spine movement can be used to evaluate training performance and provide quantitative data for swimmers.
IEEE Transactions on Human Machine Systems
Zhelong Wang Jiaxin Wang Hongyu Zhao Sen Qiu Jie Li Fengshan Gao Xin Shi
4891972
DoubleFusion: Real Time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor
We propose DoubleFusion, a new real time system that combines volumetric dynamic reconstruction with data driven template fitting to simultaneously reconstruct detailed geometry, non rigid motion and the inner human body shape from a single depth camera. One of the key contributions of this method is a double layer representation consisting of a complete parametric body shape inside, and a gradually fused outer surface layer. A pre defined node graph on the body surface parameterizes the non rigid deformations near the body, and a free form dynamically changing graph parameterizes the outer surface layer far from the body, which allows more general reconstruction. We further propose a joint motion tracking method based on the double layer representation to enable robust and fast motion tracking performance. Moreover, the inner body shape is optimized online and forced to fit inside the outer surface layer. Overall, our method enables increasingly denoised, detailed and complete surface reconstructions, fast motion tracking performance and plausible inner body shape reconstruction in real time. In particular, experiments show improved fast motion tracking and loop closure performance on more challenging scenarios.
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Tao Yu Zerong Zheng Kaiwen Guo Jianhui Zhao Qionghai Dai Hao Li Gerard Pons-Moll Yebin Liu