Depth consistency and vertical disparities in stereoscopic panoramas
Gurrieri, Luis E.
Dubois, Eric
CONTEXT: In recent years, the problem of acquiring omnidirectional stereoscopic imagery of dynamic scenes has gained commercial interest and, consequently, new techniques have been proposed to address this problem. The goal of many of these novel panoramic methods is to provide practical solutions for acquiring real-time omnidirectional stereoscopic imagery suitable to stimulate binocular human stereopsis in any gazing direction. In particular, methods based on the acquisition of partially overlapped stereoscopic snapshots of the scene are the most attractive for real-time omnistereoscopic capture. However, there is a need to rigorously model these acquisition techniques in order to provide useful design constraints for the corresponding omnidirectional stereoscopic systems. OBJECTIVE: Our main goal in this work is to propose an omnidirectional camera model, which is sufficiently flexible to describe a variety of omnistereoscopic camera configurations. We have developed a projective camera model suitable to describe a range of omnistereoscopic camera configurations and usable to determine constraints relevant to the design of omnistereoscopic acquisition systems. In addition, we applied our camera model to estimate the system constraints for the rendering approach based on mosaicking partially overlapped stereoscopic snapshots of the scene. METHOD: First, we grouped the possible stereoscopic panoramic methods, suitable to produce horizontal stereo for human viewing in every azimuthal direction, into four camera configurations. Then, we propose an omnistereoscopic camera model based on projective geometry which is suitable for describing each of the four camera configurations. Finally, we applied this model to obtain expressions for the horizontal and vertical disparity errors encountered when creating a stereoscopic panorama by mosaicking partial stereoscopic snapshots of the scene. RESULTS: We simulated the parameters of interest using the proposed geometric model combined with a ray tracing approach for each camera model. From these simulations, we extracted conclusions that can be used in the design of omnistereoscopic cameras for the acquisition of dynamic scenes. One important parameter used to contrast different camera configurations is the minimum distance to the scene to provide a continuous perception of depth in any gazing direction after mosaicking partial stereoscopic views. The other important contribution is to characterize the vertical disparities that cause ghosting at the stitching boundaries between mosaics. In the simulation, we studied the effect of the field-of-view of the lenses, and the pixel size and dimension of the sensor in the design of the system. NOVELTY: The main contribution of this work is to provide a tractable method for analyzing multiple camera configurations intended for omnistereoscopic imaging. In addition, we estimated and compared the system constraints to attain a continuous depth perception in all azimuth directions. Also important for the rendering process, we characterized mathematically the vertical disparities that would affect the mosaicking process in each omnistereoscopic configuration. This work complements and extends our previous work in stereoscopic panoramas acquisition by proposing a mathematical framework to contrast different omnistereoscopic image acquisition strategies.
 I'm going to present now a kind of follow up of the conclusion of my first presentation for those that were here. I was talking about the review of the technology to produce a telescopic panoramas for human viewing and with a particular twist of including dynamic scenes in the problem. In that case, I conclude that, well, a multi-camera approach using a reduced number of stereoscopic samples partially overlapped could be mosaic into a full omni-stereoscopic image. And the mosaicing approach actually is an easy approach to do a frame by frame rendering and then enable omni-stereoscopic videos. So there are remaining two problems in that case and that was the follow up that we started working after doing this literature review of the technologies. One is how to mosaic these partially overlapped stereoscopic images, maintaining a consistent depth in between mosaics. So basically you don't want to see artifact. If you see artifact, there is a sign that there is something going on there. And as you see in the example I showed that stereoscopic panorama, there are not noticeable artifacts, at least no very noticeable artifacts. There are artifacts still that are difficult to erase, but that was done with the prototype. So I decided to go into modeling this, how to model how much depth, how to maintain the depth illusion in between mosaics. And also another problem that is a regular problem in panoramic vision that is at least in stereoscopic panoramas, how to deal with the vertical disparities, at least how to explain whether these vertical disparities came from and where you are expected to find those vertical disparities. So we propose a model, a very simple model to start with. And usually those models are the models that work when they are simple and represent something meaningful or give some meaningful results. And that model that I'm going to explain now is based on basically two cameras and stereoscopic rig. And with that you can explain the majority of the technologies I explained in my first presentation based on horizontal stereo. And we have some experimental results based on simulations to test this model and to extract some conclusions. And that basically is the roadmap of this presentation. Basically the model consists of these two pinhole cameras separated by a baseline. In the center we have the reference viewpoint of the panorama and these two cameras could represent a rotated pair of cameras around that center. Oh, with certain displacement it could represent the real case of a multi-camera system in which you cannot align or you cannot position these cameras altogether given the dimension of the cameras in the nodal points. And, well, we have the reference for each one of the cameras and a point in the space and in the scene could be projected into these two image planes. And using projected geometry and simple equations, basically we can infer a result, basically the horizontal disparity. We can calculate the vertical disparity for different configurations. And the different configurations are these. This simple model could be expressed in four forms. And we found that with those four forms, even though it's difficult to believe, but it could represent most of the technologies we were, I was talking in the first presentation, but not those technologies based on vertical parallax. But I don't care about those technologies because basically we were interested in the horizontal parallax problem. So the first case is two cameras that could be rotated or could be a multi-camera approach. Actually this is difficult to implement in the multi-camera configuration. The second one is basically shifting these and start rotating the cameras around the center. And one of the papers from 1998, the things that was basically could be modeled with this configuration. The other configuration is this place in this radial distance RC from the center. And that basically represented in a multi-camera approach based on this configuration, on this stereoscopic rig, when you put multiple stereoscopic rig around that virtual rotation center. The case in which the nodal point of these cameras cannot be in the center all the time. In this case, for example, you cannot represent this in multi-camera pressure because all the cameras cannot be around coexisting in a space, co-located in a space with this, the same nodal point. And the last case is basically this one rotated when shifted BW2, B divided by 2 to the other direction. So basically you have this stereoscopic rig, one is centered in one of the cameras and the other is centered in the midpoint in between the nodal points of two cameras. These four configurations, for example, let's take this last configuration, configuration for that I was showing before. Well, this represents a single rotating stereoscopic rig at that distance with respect to the center, a multiple cameras approach which each one of the cameras have certain spatial location with respect to the other stereoscopic rig, which is this multi-camera approach, or this multi-camera approach with our right, multiple n number of cameras rotated around the center or fixed around the center, all represented with the same model. To give an example, let's say that we have this model and we have two sequential acquisitions separated by an azimuth angle of theta2e. So we have one pair of the same, one stereoscopic pair and the other stereoscopic pair. Now let's say that in that configuration we decided to take six stereoscopic samples. And this is starting by 0 to 5. We have six stereoscopic samples of the same, partially overlapped, constitute these two sequence of images and this is the dataset we acquired. Could be sequential, could be simultaneously. Now what areas in the, when we start most hiking this, we can differentiate. We have an overlapping region here when the point in the scene is visible by the two stereoscopic rigs or by the, by two samples of the scene. And these areas that are visible by only one stereoscopic rig at a time. When we mosaic, we can define a fixed position or two mosaic that depends on the number of the telescopic samples. In this case, it's this position in between the scene. This is the image. This is the reference center of the image and this is a position that is symmetrically located with respect to the center. Now is that the optimal? No, it's not the optimal. It's a position reference that you can use as a, you have six stereoscopic samples and you have this width of the image and then you can basically by eye say, well, this is the point when I had to stitch all the images in order to create the 360 degree view of the image at least in a scene mode. But it's not the optimal point. We can see that. Those point that we saw before, well, this P2 and P3 are the point that are visible only by, in this case, of P2 by this stereoscopic rig and this P3 by this. P4 is the one that is in the overlapping region of the two images, which I call the homestereoscopic field of view of the whole arrangement. Now coming back to this model, any point of the scene could be represented in the reference frames of each one of the cameras by this equation. With R is the rotation. In this case, the two cameras are restricted to be in the XZ plane. And the theta, so I represent the Asimut sample angles between the two. Y is always parallel to the Y, the global reference frame. So basically the two cameras are restricted to one single plane and they have the parallel axis and basically they are rotated in this direction. So you can be represented all the different positions by this single equation. And by this translation, they represent the translation in between left and right. From this relative coordinates of the scene in each one of the cameras, you can, using projective equations, find the coordinates in each one of the image, the X and Y coordinates for the left and right for each one of the theta, so I. Following up that logic, you can talk about the horizontal disparity error, which is what is the difference in horizontal disparity that is going to be in the overlapping region of the images or two images or two telescopic pairs that we need to mosaic. When that difference is larger than the threshold of human resolution of horizontal disparity, then we're going to be noticed. So if it's below what we can perceive, therefore it's not going to be visible and we can teach it safely. So we can infer what is the horizontal disparity expression from those previous equations for theta sub i, for any theta sub i and n theta sub i plus 1. So for two consecutive samples, in any, the symmetry of the problem, we just need to study two consecutive samples. There are certain simplifications that could be done here, for example, the set or the depth with respect to one telescopic pair or the left or the right, it's going to be the same. So we can reduce this to this singular distance with respect to one single camera and it's going to be different than the one that is seen by the other camera. And then talking about the error, the absolute difference between these two magnitudes. When manipulating the equations, you can reach something very simple, which is the horizontal disparity error, it depends on the difference of the reciprocal of the depth as seen by each one of the stereoscopic pair. Now translate it to each one of the four variations of the same model and then you can describe all you want, at least in terms of horizontal disparity. Now what is the traditional horizontal disparity for in humans? And that is a nice topic of research because basically for some, in some literatures, I found that, for example, this equation came out by comparing the same basic parallel optical axis, the stereoscopic pair model using pinhole cameras with the resolution of the human eye based on psychological styles, based on several tests in certain group of human with certain selection of images, whether those images have certain texture or not and that number change. But the evaluation, when you put the two equations together, you can reach a horizontal threshold. This is the error, the threshold in the horizontal disparity error that human perceive based on this pinhole stereoscopic pair and the human model, which is baseline, the baseline between the human eye and this is the distance between human eyes. So basically if you, the baseline of this virtual stereoscopic rig is the same than the distance between the interocular distance between humans, you can simplify this. This parameter is the angular resolution perceived by human and that threshold is about 20 arc of second, which is quite slow, quite low and this number here of threshold gives something that is 25 times smaller than the width of a pixel. So basically it's no use, it's no use in this form. So this is going to be lower than the width of a pixel for a regular camera, you cannot use it. So what we always move to use the width of a pixel as a threshold, which is a practical approach given that the width of the pixel is basically what we need when we mosaic stereoscopic images. If the error is larger than the width of a pixel, it's going to be noticeable. If it's low, it's not going to be noticeable. And it happens also that it maintains the continuity in the horizontal disparity. So if you consider something larger, 25 times larger, that 20 arc of second is close to one arc of a minute of resolution. So we said probably that is the correct one, the correct answer, even the technology we had today. And basically it looked good, so you must be correct. So we started simulating based on our model using that threshold of one pixel. For this particular camera, using real numbers, the sensor width of this kind of 400D, an angular field of view based on a fisheye commercial lens that we could use in our experiments. And in the baseline of more or less half the interlocular distance in the human population, 35 millimeters, and a radial distance for this particular configuration, 3 and 4, equal to the baseline, 35 millimeters. So we tried to compare the different cases and we came out with this plot. The minimum distance between the scene and the camera to have a continuous horizontal disparity. So basically what we are plotting here is that the difference between the point we choose by XB in the image, moving 5 percent towards the center of the image or minus 5 percent towards the edge of the image. And that correction give us this plot that, for example, if we are using that point choosing deterministically, we found that, for example, the configuration 3 and the configuration 3, for example, the scene had to be 2 meters from the scene when you use 6 samples. When you use 8 samples, basically that overlapping region moves closer to the center of stereoscopic pairs and that changes a little bit closer to the camera. So that is important. No, it's not important because basically, for practical reasons, if I build a camera and I want to determine one model is better than the other or one configuration of multicameras is better than the other based on this, I will say, well, I really don't care if the scene is 2 meters from the camera or 1 meter from the camera. And listen to micro-omnistereoscopic images where things are very close to the camera and the thing is small. But in normal cases, if I put my camera here and I want to create an omnistereoscopic image that you guys over there and me here and I am at 3 meters, well, I'm not going to have any problem if I choose that threshold to mosaic the images. But it had to have some uses. Well, the use is basically the other thing that we're interested in using this result to say we can define an optimal cut for the mosaics that is based in an energy fraction that consider this horizontal disparity resolution in humans. So when you introduce that into the algorithm of optimal cut, the same scene could be cut in a different way when you mosaic for elements that are closer to the camera and elements that are far away from the camera. You can find the point in between two mosaics where you can basically blend them and then make invisible any artifact that will happen otherwise if I use a fixed stitching point. And basically you can put this for each one of the models or all the configuration you can imagine. And that's how, well, that is an interesting result. There is a vertical disparities. Vertical disparities in panoramas always appear. But based in this model, the vertical disparities, based in an stereoscopic vertical disparity in the stereoscopic area of each one of the camera should be zero because based on this equation, the projection on the image plane, this value and this value of the distance of the scene is equal to this. For each one of the stereoscopic pair, we say that the set is equal to each other. So this value simplifies. And basically if they are aligned, happen that in reality this never happened because the optical axes are not really parallel and they are misaligned. So you have to do some correction, some stereoscopic registration. But if you do that, basically you wouldn't have any problem in these areas. But you will have problems in the overlapping areas. And that problem happened not in between left and right, but in between left and in the left sequence of images and in the right sequence of images. And that happened when you do this comparison. You see this expression that gives this number is a multiplicative factor over this Y. Why is the elevation with respect to the center? When we are talking about the equatorial area here, it should be zero. So it's going to be aligned. But the more you look up or you look down, you will have a multiplicative factor that affects that. And the larger is going to be this part. The more you need to warp the images in order to make it aligned. But we are talking again with respect left to left, right to right. And to see this graphically, you see when the scene is far away from the camera, that effect doesn't matter. Because basically that disparity is going to be not noticeable. But when you're taking a panorama here and imagine what happened in the roof or in the floor that are closer to the camera, meaning closer distance from the camera, this is the distance to the camera, the multiplicative factor is going to be larger. And that's what is playing, what is always look misaligned and always have to do some kind of warping in that direction. I'm talking always when I don't have extra information about the depth in order to match pixels. All right. And here you see what happened at different distances. So basically all these analysis was to put together a model that could represent all the different omnistereoscopic technologies with horizontal parallax. And it's flexible and simple enough to represent this. We came out with a result that could be used for the device at optimal cut. And we explained some issues that happened with vertical disparities. Well that is my presentation and I hope you enjoyed it. Thank you.
 I'm going to present now a kind of follow up of the conclusion of my first presentation  for those that were here.  I was talking about the review of the technology to produce a telescopic panoramas for human  viewing and with a particular twist of including dynamic scenes in the problem.  In that case, I conclude that, well, a multi-camera approach using a reduced number of stereoscopic  samples partially overlapped could be mosaic into a full omni-stereoscopic image.  And the mosaicing approach actually is an easy approach to do a frame by frame rendering  and then enable omni-stereoscopic videos.  So there are remaining two problems in that case and that was the follow up that we started  working after doing this literature review of the technologies.  One is how to mosaic these partially overlapped stereoscopic images, maintaining a consistent  depth in between mosaics.  So basically you don't want to see artifact.  If you see artifact, there is a sign that there is something going on there.  And as you see in the example I showed that stereoscopic panorama, there are not noticeable  artifacts, at least no very noticeable artifacts.  There are artifacts still that are difficult to erase, but that was done with the prototype.  So I decided to go into modeling this, how to model how much depth, how to maintain the  depth illusion in between mosaics.  And also another problem that is a regular problem in panoramic vision that is at least  in stereoscopic panoramas, how to deal with the vertical disparities, at least how to  explain whether these vertical disparities came from and where you are expected to find  those vertical disparities.  So we propose a model, a very simple model to start with.  And usually those models are the models that work when they are simple and represent something  meaningful or give some meaningful results.  And that model that I'm going to explain now is based on basically two cameras and stereoscopic  rig.  And with that you can explain the majority of the technologies I explained in my first  presentation based on horizontal stereo.  And we have some experimental results based on simulations to test this model and to extract  some conclusions.  And that basically is the roadmap of this presentation.  Basically the model consists of these two pinhole cameras separated by a baseline.  In the center we have the reference viewpoint of the panorama and these two cameras could  represent a rotated pair of cameras around that center.  Oh, with certain displacement it could represent the real case of a multi-camera system in  which you cannot align or you cannot position these cameras altogether given the dimension  of the cameras in the nodal points.  And, well, we have the reference for each one of the cameras and a point in the space  and in the scene could be projected into these two image planes.  And using projected geometry and simple equations, basically we can infer a result, basically  the horizontal disparity.  We can calculate the vertical disparity for different configurations.  And the different configurations are these.  This simple model could be expressed in four forms.  And we found that with those four forms, even though it's difficult to believe, but it  could represent most of the technologies we were, I was talking in the first presentation,  but not those technologies based on vertical parallax.  But I don't care about those technologies because basically we were interested in the  horizontal parallax problem.  So the first case is two cameras that could be rotated or could be a multi-camera approach.  Actually this is difficult to implement in the multi-camera configuration.  The second one is basically shifting these and start rotating the cameras around the  center.  And one of the papers from 1998, the things that was basically could be modeled with this  configuration.  The other configuration is this place in this radial distance RC from the center.  And that basically represented in a multi-camera approach based on this configuration, on this  stereoscopic rig, when you put multiple stereoscopic rig around that virtual rotation center.  The case in which the nodal point of these cameras cannot be in the center all the time.  In this case, for example, you cannot represent this in multi-camera pressure because all  the cameras cannot be around coexisting in a space, co-located in a space with this,  the same nodal point.  And the last case is basically this one rotated when shifted BW2, B divided by 2 to the other  direction.  So basically you have this stereoscopic rig, one is centered in one of the cameras and  the other is centered in the midpoint in between the nodal points of two cameras.  These four configurations, for example, let's take this last configuration, configuration  for that I was showing before.  Well, this represents a single rotating stereoscopic rig at that distance with respect to the center,  a multiple cameras approach which each one of the cameras have certain spatial location  with respect to the other stereoscopic rig, which is this multi-camera approach, or this  multi-camera approach with our right, multiple n number of cameras rotated around the center  or fixed around the center, all represented with the same model.  To give an example, let's say that we have this model and we have two sequential acquisitions  separated by an azimuth angle of theta2e.  So we have one pair of the same, one stereoscopic pair and the other stereoscopic pair.  Now let's say that in that configuration we decided to take six stereoscopic samples.  And this is starting by 0 to 5.  We have six stereoscopic samples of the same, partially overlapped, constitute these two  sequence of images and this is the dataset we acquired.  Could be sequential, could be simultaneously.  Now what areas in the, when we start most hiking this, we can differentiate.  We have an overlapping region here when the point in the scene is visible by the two stereoscopic  rigs or by the, by two samples of the scene.  And these areas that are visible by only one stereoscopic rig at a time.  When we mosaic, we can define a fixed position or two mosaic that depends on the number of  the telescopic samples.  In this case, it's this position in between the scene.  This is the image.  This is the reference center of the image and this is a position that is symmetrically  located with respect to the center.  Now is that the optimal?  No, it's not the optimal.  It's a position reference that you can use as a, you have six stereoscopic samples and  you have this width of the image and then you can basically by eye say, well, this is  the point when I had to stitch all the images in order to create the 360 degree view of the  image at least in a scene mode.  But it's not the optimal point.  We can see that.  Those point that we saw before, well, this P2 and P3 are the point that are visible only  by, in this case, of P2 by this stereoscopic rig and this P3 by this.  P4 is the one that is in the overlapping region of the two images, which I call the homestereoscopic  field of view of the whole arrangement.  Now coming back to this model, any point of the scene could be represented in the reference  frames of each one of the cameras by this equation.  With R is the rotation.  In this case, the two cameras are restricted to be in the XZ plane.  And the theta, so I represent the Asimut sample angles between the two.  Y is always parallel to the Y, the global reference frame.  So basically the two cameras are restricted to one single plane and they have the parallel  axis and basically they are rotated in this direction.  So you can be represented all the different positions by this single equation.  And by this translation, they represent the translation in between left and right.  From this relative coordinates of the scene in each one of the cameras, you can, using  projective equations, find the coordinates in each one of the image, the X and Y coordinates  for the left and right for each one of the theta, so I.  Following up that logic, you can talk about the horizontal disparity error, which is what  is the difference in horizontal disparity that is going to be in the overlapping region  of the images or two images or two telescopic pairs that we need to mosaic.  When that difference is larger than the threshold of human resolution of horizontal disparity,  then we're going to be noticed.  So if it's below what we can perceive, therefore it's not going to be visible and we can teach  it safely.  So we can infer what is the horizontal disparity expression from those previous equations for  theta sub i, for any theta sub i and n theta sub i plus 1.  So for two consecutive samples, in any, the symmetry of the problem, we just need to study  two consecutive samples.  There are certain simplifications that could be done here, for example, the set or the  depth with respect to one telescopic pair or the left or the right, it's going to be the  same.  So we can reduce this to this singular distance with respect to one single camera and it's  going to be different than the one that is seen by the other camera.  And then talking about the error, the absolute difference between these two magnitudes.  When manipulating the equations, you can reach something very simple, which is the horizontal  disparity error, it depends on the difference of the reciprocal of the depth as seen by  each one of the stereoscopic pair.  Now translate it to each one of the four variations of the same model and then you can describe  all you want, at least in terms of horizontal disparity.  Now what is the traditional horizontal disparity for in humans?  And that is a nice topic of research because basically for some, in some literatures, I  found that, for example, this equation came out by comparing the same basic parallel optical  axis, the stereoscopic pair model using pinhole cameras with the resolution of the human eye  based on psychological styles, based on several tests in certain group of human with certain  selection of images, whether those images have certain texture or not and that number  change.  But the evaluation, when you put the two equations together, you can reach a horizontal threshold.  This is the error, the threshold in the horizontal disparity error that human perceive based  on this pinhole stereoscopic pair and the human model, which is baseline, the baseline  between the human eye and this is the distance between human eyes.  So basically if you, the baseline of this virtual stereoscopic rig is the same than  the distance between the interocular distance between humans, you can simplify this.  This parameter is the angular resolution perceived by human and that threshold is about 20 arc  of second, which is quite slow, quite low and this number here of threshold gives something  that is 25 times smaller than the width of a pixel.  So basically it's no use, it's no use in this form.  So this is going to be lower than the width of a pixel for a regular camera, you cannot  use it.  So what we always move to use the width of a pixel as a threshold, which is a practical  approach given that the width of the pixel is basically what we need when we mosaic  stereoscopic images.  If the error is larger than the width of a pixel, it's going to be noticeable.  If it's low, it's not going to be noticeable.  And it happens also that it maintains the continuity in the horizontal disparity.  So if you consider something larger, 25 times larger, that 20 arc of second is close to  one arc of a minute of resolution.  So we said probably that is the correct one, the correct answer, even the technology we  had today.  And basically it looked good, so you must be correct.  So we started simulating based on our model using that threshold of one pixel.  For this particular camera, using real numbers, the sensor width of this kind of 400D, an  angular field of view based on a fisheye commercial lens that we could use in our experiments.  And in the baseline of more or less half the interlocular distance in the human population,  35 millimeters, and a radial distance for this particular configuration, 3 and 4, equal  to the baseline, 35 millimeters.  So we tried to compare the different cases and we came out with this plot.  The minimum distance between the scene and the camera to have a continuous horizontal  disparity.  So basically what we are plotting here is that the difference between the point we choose  by XB in the image, moving 5 percent towards the center of the image or minus 5 percent  towards the edge of the image.  And that correction give us this plot that, for example, if we are using that point choosing  deterministically, we found that, for example, the configuration 3 and the configuration 3,  for example, the scene had to be 2 meters from the scene when you use 6 samples.  When you use 8 samples, basically that overlapping region moves closer to the center of stereoscopic  pairs and that changes a little bit closer to the camera.  So that is important.  No, it's not important because basically, for practical reasons, if I build a camera  and I want to determine one model is better than the other or one configuration of multicameras  is better than the other based on this, I will say, well, I really don't care if the  scene is 2 meters from the camera or 1 meter from the camera.  And listen to micro-omnistereoscopic images where things are very close to the camera  and the thing is small.  But in normal cases, if I put my camera here and I want to create an omnistereoscopic image  that you guys over there and me here and I am at 3 meters, well, I'm not going to have  any problem if I choose that threshold to mosaic the images.  But it had to have some uses.  Well, the use is basically the other thing that we're interested in using this result  to say we can define an optimal cut for the mosaics that is based in an energy fraction  that consider this horizontal disparity resolution in humans.  So when you introduce that into the algorithm of optimal cut, the same scene could be cut  in a different way when you mosaic for elements that are closer to the camera and elements  that are far away from the camera.  You can find the point in between two mosaics where you can basically blend them and then  make invisible any artifact that will happen otherwise if I use a fixed stitching point.  And basically you can put this for each one of the models or all the configuration you  can imagine.  And that's how, well, that is an interesting result.  There is a vertical disparities.  Vertical disparities in panoramas always appear.  But based in this model, the vertical disparities, based in an stereoscopic vertical disparity  in the stereoscopic area of each one of the camera should be zero because based on this  equation, the projection on the image plane, this value and this value of the distance  of the scene is equal to this.  For each one of the stereoscopic pair, we say that the set is equal to each other.  So this value simplifies.  And basically if they are aligned, happen that in reality this never happened because  the optical axes are not really parallel and they are misaligned.  So you have to do some correction, some stereoscopic registration.  But if you do that, basically you wouldn't have any problem in these areas.  But you will have problems in the overlapping areas.  And that problem happened not in between left and right, but in between left and in the  left sequence of images and in the right sequence of images.  And that happened when you do this comparison.  You see this expression that gives this number is a multiplicative factor over this Y.  Why is the elevation with respect to the center?  When we are talking about the equatorial area here, it should be zero.  So it's going to be aligned.  But the more you look up or you look down, you will have a multiplicative factor that  affects that.  And the larger is going to be this part.  The more you need to warp the images in order to make it aligned.  But we are talking again with respect left to left, right to right.  And to see this graphically, you see when the scene is far away from the camera, that  effect doesn't matter.  Because basically that disparity is going to be not noticeable.  But when you're taking a panorama here and imagine what happened in the roof or in the  floor that are closer to the camera, meaning closer distance from the camera, this is the  distance to the camera, the multiplicative factor is going to be larger.  And that's what is playing, what is always look misaligned and always have to do some  kind of warping in that direction.  I'm talking always when I don't have extra information about the depth in order to match  pixels.  All right.  And here you see what happened at different distances.  So basically all these analysis was to put together a model that could represent all  the different omnistereoscopic technologies with horizontal parallax.  And it's flexible and simple enough to represent this.  We came out with a result that could be used for the device at optimal cut.  And we explained some issues that happened with vertical disparities.  Well that is my presentation and I hope you enjoyed it.  Thank you.
SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_0_131.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_132_263.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_264_311.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_312_359.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_360_659.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_660_767.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_768_911.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_912_983.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_984_1211.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_1212_1259.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_1260_1355.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_1356_1475.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_1476_5987.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_5988_9359.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_9360_9803.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_9804_12071.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_12072_12167.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_12168_12395.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_12396_16679.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_16680_29603.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_29604_30227.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_30228_30287.png SD_A_2014__Depth_consistency_and_vertical_disparities_in_stereoscopic_panoramas__9011-35__SKo0iCBUTi_30288_30532.png
I'm going to present now a kind of follow up of the conclusion of my first presentation for those that were here. I was talking about the review of the technology to produce a telescopic panoramas for human viewing and with a particular twist of including dynamic scenes in the problem. In that case, I conclude that, well, a multi-camera approach using a reduced number of stereoscopic samples partially overlapped could be mosaic into a full omni-stereoscopic image. And the mosaicing approach actually is an easy approach to do a frame by frame rendering and then enable omni-stereoscopic videos. So there are remaining two problems in that case and that was the follow up that we started working after doing this literature review of the technologies. One is how to mosaic these partially overlapped stereoscopic images, maintaining a consistent depth in between mosaics. So basically you don't want to see artifact. If you see artifact, there is a sign that there is something going on there. And as you see in the example I showed that stereoscopic panorama, there are not noticeable artifacts, at least no very noticeable artifacts. There are artifacts still that are difficult to erase, but that was done with the prototype. So I decided to go into modeling this, how to model how much depth, how to maintain the depth illusion in between mosaics. And also another problem that is a regular problem in panoramic vision that is at least in stereoscopic panoramas, how to deal with the vertical disparities, at least how to explain whether these vertical disparities came from and where you are expected to find those vertical disparities. So we propose a model, a very simple model to start with. And usually those models are the models that work when they are simple and represent something meaningful or give some meaningful results. And that model that I'm going to explain now is based on basically two cameras and stereoscopic rig. And with that you can explain the majority of the technologies I explained in my first presentation based on horizontal stereo. And we have some experimental results based on simulations to test this model and to extract some conclusions. And that basically is the roadmap of this presentation. Basically the model consists of these two pinhole cameras separated by a baseline. In the center we have the reference viewpoint of the panorama and these two cameras could represent a rotated pair of cameras around that center. Oh, with certain displacement it could represent the real case of a multi-camera system in which you cannot align or you cannot position these cameras altogether given the dimension of the cameras in the nodal points. And, well, we have the reference for each one of the cameras and a point in the space and in the scene could be projected into these two image planes. And using projected geometry and simple equations, basically we can infer a result, basically the horizontal disparity. We can calculate the vertical disparity for different configurations. And the different configurations are these. This simple model could be expressed in four forms. And we found that with those four forms, even though it's difficult to believe, but it could represent most of the technologies we were, I was talking in the first presentation, but not those technologies based on vertical parallax. But I don't care about those technologies because basically we were interested in the horizontal parallax problem. So the first case is two cameras that could be rotated or could be a multi-camera approach. Actually this is difficult to implement in the multi-camera configuration. The second one is basically shifting these and start rotating the cameras around the center. And one of the papers from 1998, the things that was basically could be modeled with this configuration. The other configuration is this place in this radial distance RC from the center. And that basically represented in a multi-camera approach based on this configuration, on this stereoscopic rig, when you put multiple stereoscopic rig around that virtual rotation center. The case in which the nodal point of these cameras cannot be in the center all the time. In this case, for example, you cannot represent this in multi-camera pressure because all the cameras cannot be around coexisting in a space, co-located in a space with this, the same nodal point. And the last case is basically this one rotated when shifted BW2, B divided by 2 to the other direction. So basically you have this stereoscopic rig, one is centered in one of the cameras and the other is centered in the midpoint in between the nodal points of two cameras. These four configurations, for example, let's take this last configuration, configuration for that I was showing before. Well, this represents a single rotating stereoscopic rig at that distance with respect to the center, a multiple cameras approach which each one of the cameras have certain spatial location with respect to the other stereoscopic rig, which is this multi-camera approach, or this multi-camera approach with our right, multiple n number of cameras rotated around the center or fixed around the center, all represented with the same model. To give an example, let's say that we have this model and we have two sequential acquisitions separated by an azimuth angle of theta2e. So we have one pair of the same, one stereoscopic pair and the other stereoscopic pair. Now let's say that in that configuration we decided to take six stereoscopic samples. And this is starting by 0 to 5. We have six stereoscopic samples of the same, partially overlapped, constitute these two sequence of images and this is the dataset we acquired. Could be sequential, could be simultaneously. Now what areas in the, when we start most hiking this, we can differentiate. We have an overlapping region here when the point in the scene is visible by the two stereoscopic rigs or by the, by two samples of the scene. And these areas that are visible by only one stereoscopic rig at a time. When we mosaic, we can define a fixed position or two mosaic that depends on the number of the telescopic samples. In this case, it's this position in between the scene. This is the image. This is the reference center of the image and this is a position that is symmetrically located with respect to the center. Now is that the optimal? No, it's not the optimal. It's a position reference that you can use as a, you have six stereoscopic samples and you have this width of the image and then you can basically by eye say, well, this is the point when I had to stitch all the images in order to create the 360 degree view of the image at least in a scene mode. But it's not the optimal point. We can see that. Those point that we saw before, well, this P2 and P3 are the point that are visible only by, in this case, of P2 by this stereoscopic rig and this P3 by this. P4 is the one that is in the overlapping region of the two images, which I call the homestereoscopic field of view of the whole arrangement. Now coming back to this model, any point of the scene could be represented in the reference frames of each one of the cameras by this equation. With R is the rotation. In this case, the two cameras are restricted to be in the XZ plane. And the theta, so I represent the Asimut sample angles between the two. Y is always parallel to the Y, the global reference frame. So basically the two cameras are restricted to one single plane and they have the parallel axis and basically they are rotated in this direction. So you can be represented all the different positions by this single equation. And by this translation, they represent the translation in between left and right. From this relative coordinates of the scene in each one of the cameras, you can, using projective equations, find the coordinates in each one of the image, the X and Y coordinates for the left and right for each one of the theta, so I. Following up that logic, you can talk about the horizontal disparity error, which is what is the difference in horizontal disparity that is going to be in the overlapping region of the images or two images or two telescopic pairs that we need to mosaic. When that difference is larger than the threshold of human resolution of horizontal disparity, then we're going to be noticed. So if it's below what we can perceive, therefore it's not going to be visible and we can teach it safely. So we can infer what is the horizontal disparity expression from those previous equations for theta sub i, for any theta sub i and n theta sub i plus 1. So for two consecutive samples, in any, the symmetry of the problem, we just need to study two consecutive samples. There are certain simplifications that could be done here, for example, the set or the depth with respect to one telescopic pair or the left or the right, it's going to be the same. So we can reduce this to this singular distance with respect to one single camera and it's going to be different than the one that is seen by the other camera. And then talking about the error, the absolute difference between these two magnitudes. When manipulating the equations, you can reach something very simple, which is the horizontal disparity error, it depends on the difference of the reciprocal of the depth as seen by each one of the stereoscopic pair. Now translate it to each one of the four variations of the same model and then you can describe all you want, at least in terms of horizontal disparity. Now what is the traditional horizontal disparity for in humans? And that is a nice topic of research because basically for some, in some literatures, I found that, for example, this equation came out by comparing the same basic parallel optical axis, the stereoscopic pair model using pinhole cameras with the resolution of the human eye based on psychological styles, based on several tests in certain group of human with certain selection of images, whether those images have certain texture or not and that number change. But the evaluation, when you put the two equations together, you can reach a horizontal threshold. This is the error, the threshold in the horizontal disparity error that human perceive based on this pinhole stereoscopic pair and the human model, which is baseline, the baseline between the human eye and this is the distance between human eyes. So basically if you, the baseline of this virtual stereoscopic rig is the same than the distance between the interocular distance between humans, you can simplify this. This parameter is the angular resolution perceived by human and that threshold is about 20 arc of second, which is quite slow, quite low and this number here of threshold gives something that is 25 times smaller than the width of a pixel. So basically it's no use, it's no use in this form. So this is going to be lower than the width of a pixel for a regular camera, you cannot use it. So what we always move to use the width of a pixel as a threshold, which is a practical approach given that the width of the pixel is basically what we need when we mosaic stereoscopic images. If the error is larger than the width of a pixel, it's going to be noticeable. If it's low, it's not going to be noticeable. And it happens also that it maintains the continuity in the horizontal disparity. So if you consider something larger, 25 times larger, that 20 arc of second is close to one arc of a minute of resolution. So we said probably that is the correct one, the correct answer, even the technology we had today. And basically it looked good, so you must be correct. So we started simulating based on our model using that threshold of one pixel. For this particular camera, using real numbers, the sensor width of this kind of 400D, an angular field of view based on a fisheye commercial lens that we could use in our experiments. And in the baseline of more or less half the interlocular distance in the human population, 35 millimeters, and a radial distance for this particular configuration, 3 and 4, equal to the baseline, 35 millimeters. So we tried to compare the different cases and we came out with this plot. The minimum distance between the scene and the camera to have a continuous horizontal disparity. So basically what we are plotting here is that the difference between the point we choose by XB in the image, moving 5 percent towards the center of the image or minus 5 percent towards the edge of the image. And that correction give us this plot that, for example, if we are using that point choosing deterministically, we found that, for example, the configuration 3 and the configuration 3, for example, the scene had to be 2 meters from the scene when you use 6 samples. When you use 8 samples, basically that overlapping region moves closer to the center of stereoscopic pairs and that changes a little bit closer to the camera. So that is important. No, it's not important because basically, for practical reasons, if I build a camera and I want to determine one model is better than the other or one configuration of multicameras is better than the other based on this, I will say, well, I really don't care if the scene is 2 meters from the camera or 1 meter from the camera. And listen to micro-omnistereoscopic images where things are very close to the camera and the thing is small. But in normal cases, if I put my camera here and I want to create an omnistereoscopic image that you guys over there and me here and I am at 3 meters, well, I'm not going to have any problem if I choose that threshold to mosaic the images. But it had to have some uses. Well, the use is basically the other thing that we're interested in using this result to say we can define an optimal cut for the mosaics that is based in an energy fraction that consider this horizontal disparity resolution in humans. So when you introduce that into the algorithm of optimal cut, the same scene could be cut in a different way when you mosaic for elements that are closer to the camera and elements that are far away from the camera. You can find the point in between two mosaics where you can basically blend them and then make invisible any artifact that will happen otherwise if I use a fixed stitching point. And basically you can put this for each one of the models or all the configuration you can imagine. And that's how, well, that is an interesting result. There is a vertical disparities. Vertical disparities in panoramas always appear. But based in this model, the vertical disparities, based in an stereoscopic vertical disparity in the stereoscopic area of each one of the camera should be zero because based on this equation, the projection on the image plane, this value and this value of the distance of the scene is equal to this. For each one of the stereoscopic pair, we say that the set is equal to each other. So this value simplifies. And basically if they are aligned, happen that in reality this never happened because the optical axes are not really parallel and they are misaligned. So you have to do some correction, some stereoscopic registration. But if you do that, basically you wouldn't have any problem in these areas. But you will have problems in the overlapping areas. And that problem happened not in between left and right, but in between left and in the left sequence of images and in the right sequence of images. And that happened when you do this comparison. You see this expression that gives this number is a multiplicative factor over this Y. Why is the elevation with respect to the center? When we are talking about the equatorial area here, it should be zero. So it's going to be aligned. But the more you look up or you look down, you will have a multiplicative factor that affects that. And the larger is going to be this part. The more you need to warp the images in order to make it aligned. But we are talking again with respect left to left, right to right. And to see this graphically, you see when the scene is far away from the camera, that effect doesn't matter. Because basically that disparity is going to be not noticeable. But when you're taking a panorama here and imagine what happened in the roof or in the floor that are closer to the camera, meaning closer distance from the camera, this is the distance to the camera, the multiplicative factor is going to be larger. And that's what is playing, what is always look misaligned and always have to do some kind of warping in that direction. I'm talking always when I don't have extra information about the depth in order to match pixels. All right. And here you see what happened at different distances. So basically all these analysis was to put together a model that could represent all the different omnistereoscopic technologies with horizontal parallax. And it's flexible and simple enough to represent this. We came out with a result that could be used for the device at optimal cut. And we explained some issues that happened with vertical disparities. Well that is my presentation and I hope you enjoyed it. Thank you.
the technology to produce a telescopic panoramas for human viewing and with a particular twist of including dynamic scenes in the problem. In that samples partially overlapped could be mosaic into frame by frame rendering and then enable literature review of the technologies. One is how to mosaic these partially overlapped stereoscopic panorama, there are not noticeable artifacts, at between mosaics. And also another problem that is least in stereoscopic panoramas, how to deal with disparities. So we propose a model, a very simple horizontal stereo. And we have some experimental results based on simulations to test this model and to extract some conclusions. And that a multi-camera system in which you cannot align given the dimension of the cameras in the nodal image planes. And using projected geometry and result, basically the horizontal disparity. We different configurations. And the different modeled with this configuration. The other represented in a multi-camera approach based on the nodal point of these cameras cannot be in the divided by 2 to the other direction. So basically multi-camera approach with our right, multiple n you can be represented all the different between left and right. From this relative coordinates of the scene in each one of the cameras, you can, using projective equations, samples, in any, the symmetry of the problem, we each one of the four variations of the same model and then you can describe all you want, at least human model, which is baseline, the baseline This parameter is the angular resolution of a pixel as a threshold, which is a practical approach given that the width of the pixel is basically what we need when we mosaic camera, using real numbers, the sensor width of this plot. The minimum distance between the scene and the camera to have a continuous horizontal So that is important. No, it's not important build a camera and I want to determine one model I want to create an omnistereoscopic image that basically you can put this for each one of the in this model, the vertical disparities, based in an stereoscopic vertical disparity in the camera, this is the distance to the camera, the depth in order to match pixels. All right. And different omnistereoscopic technologies with horizontal parallax. And it's flexible and simple result that could be used for the device at