Feature based no-reference continuous video quality prediction model for coded stereo video
Sazzad, Z. M. Parvez
Bensalma, Rafik
Larabi, Mohamed Chaker
In this paper, we propose a continuous no-reference video quality evaluation model for MPEG-2 MP@ML coded stereoscopic video based on spatial, temporal, and disparity features with the incorporation of human visual system characteristics. We believe edge distortion is a major concern to perceive spatial distortion throughout any image frame which is strongly dependent on smooth and non-smooth areas of the frame. We also claim that perceived depth of any image/ video is mainly dependent on central objects/ structures of the image/ video contents. Thus, visibility of depth is firmly dependent on the objects’ distance such as near, far, and very far. Subsequently, temporal perception is mostly based on jerkiness of video and it is dependent on motion as well as scene content of the video. Therefore, segmented local features such as smooth and non-smooth area based edge distortion, and the objects’ distance based depth measures are evaluated in this method. Subsequently, video jerkiness is estimated based on segmented temporal information. Different weighting factors are then applied for the different edge distortion and depth features to measure the overall features of a temporal segment. All features are calculated separately for each temporal segment in this method. Subjective stereo video database, which considered both symmetric and asymmetric coded videos, is used to verify the performance of the model. The result indicates that our proposed model has sufficient prediction performance.
 So, good morning everybody. So, please don't look at me like a UFO because, you know, the topic is completely different from the other presentations. So, this work has been performed in a major part by Parviz Sazad from University of Dhaka in Bangladesh during his stay in our lab. And I will present our collaboration on quality assessment for stereoscopic video. So, just before starting, you know, you know that quality is a very big issue nowadays, and especially stereoscopic quality or 3D quality, because this quality is not, we cannot handle it, we cannot measure it directly from the left and the right images. So, if you have, for example, such a scene, you know that you have the left and the right view, but our brain reconstructs the depth, and the quality is measured starting from this information. So, if you use 2D metrics on the left and the right views, you will only have 2D quality, but the third dimension that the human will perceive will not capture it, because we need something to handle how we can reconstruct this in the brain and use what we call, you know, the disparity, use the depth information in order to be able to give a score to the metric, to the image or video. And you have many applications nowadays for 3D, and the best way to measure quality, to measure 3D quality is to do subjective measurement. But, you know, it is tedious time consuming, you cannot do it on all applications. So we need some models that can handle the quality and that can mimic the human visual system in order to measure this quality. It is not a very easy task because even for 2D, the problem is not solved yet. Imagine when you add a dimension and it's subjective dimension, it's a very big issue. So in quality measurement, you have three families, depending on the reference. If you have a reference, so then you can construct a full reference method. If you have some cues about the reference, you can develop a reduced reference method, so we can assume that you embed this reduced reference within the image or within the sequence and then send it and use this reference to, to this reduced reference to calculate the quality of the receiver stage. But the best, best way to, to measure the quality is to have a no reference method, especially for video, because if you have to send only one image, this is, will not offer high cost. But if you have video, if you have to measure the quality of a movie theater in 3D or a 3D movie received through IP over, over your internet box, so this will, will cost a lot. So you cannot, you cannot have the reference. And even if you have reduced reference, it can be, can be of a high cost. So this, the third family is the best one in order to monitor the, the quality. So what is the objective of this work? So this work is intended to, to develop a metric for video, for monitoring the quality of a video. It has to be of a low cost metric, because we need to do it in real time, if you want to have a feedback about the quality. We need to have some cues about what is important in, in human perception, like the sensitivity to, to edge or to structural information. We have to, to take care of the special information, but also about the temporal information, because when you look, when you watch a movie, you don't watch only images. You have successive images and the motion and the, the, the, the, the speed of the, of the sequence is important and you have to take it into account. And when dealing with 3D, you have a third part, which is the disparity or the depth. You have to take it into account, because if you have an object which is close to your eyes or if you have an object which is far, you will not perceive them in the same way. And the impairment you will have on both objects will not have the same annoyance. So as, as I said, so you have the special part, the depth part and the temporal part also that has to be taken into account. And this, these are parts of our objectives. So this is the, the method, the proposed method based on what I said before. So there is many things related to disparity measurement, to special, special information and to temporal information. But as you will see in the next slide, the method is rather simple and the idea is really to have easy, easy to compute features in order to make our regression afterwards using subjective measurement and then having a model that can predict quality in real time. So for the special feature, we used an edge distortion measure. It's relatively simple. It's based on the zero crossing for each 8 by 8 block. So why 8 by 8? This is because of the construction of coding algorithms. Here coders are used for video are using DCT and DCT is calculated on 8 by 8 blocks. So then afterward, we, when we have the zero crossing on, for each block, we will have average zero crossing for edge and non-edge areas of the image because we know that the quality in regions with high activity is not the same than the quality in flat area or regions with low activity or no texture. So what is done in horizontal way here is this is done also in vertical and this is just an example to show how our zero crossing is working. It's really simple and the idea is to find for each pixel what is its activity and what is the activity of the edge. So then it is not done only on one frame, but we have chosen to do this on five successive frames. Why? In order to be sure that we have enough cues about the special quality and because there is the stability in the frames, so it means that we reduce the computation and also we can have more stable features in order to compute the quality. So for the disparities feature, we have chosen to decompose the disparity in three parts. So apart for very low disparity, apart for average disparity and the final part for the highest disparity. So starting from that for each disparity we extract the information and from this we construct our feature. And at the end our disparity feature is a combination of the high, the medium and the low disparity weighted here as you see and these weights are obtained afterwards thanks to the regression and to the training that we will do on subjective database. So this is an example of what I call low, medium and high disparity. For example here you can see that we need to make a difference between this object which is in the front and this one which is in the back. While here we have more information about the object which is in the front and so on and so forth. So this information may allow us to say for example for this frame. We know that since this object is taking a big place in terms of special disposition, so it means that in terms of 3D this object will attract the gaze of the observer and the quality will fall in this part. So this is the simplistic temporal feature which is done also within 4 successive frames. It's very simple, it's just a difference within a temporal interval and as I said before the main idea here wasn't to try to develop very complex features. Here the idea behind what we propose here is really to use simple features, very easy to compute features but try to combine them in the best way. So this is how we combine them. So you have here the disparity feature, you have the special feature and the temporal here and thanks to that database that I will describe in the next slide we have identified through an optimization process all the weightings that we have here and that will allow us afterward to be able to mimic the observers in terms of monitoring 3D quality. So the subjective test has been done on a 3D monitor using SSCQE method, so with a single stimulus and the material that has been used here is an MPEG-coded material. The main important information that I need to give you here is that the compression has been done in a symmetric and asymmetric way. What does it mean? It means that even you compress your left and right views at the same bit rate. So this is what is originally done but we know that in our brain we have what we call the binocular compensation. So if you have one view with lower quality than the other the brain will compensate and we will have the best view will compensate the lower view and we will have a good one. So what we call also the binocular rivalry, there is a good experiment that you can do. If you take a paper then you make a cylinder, you look, you put it on your left eye and then you put your hand on the right eye you will see the hole on your hand. Why? Because you have two different information, you have a binocular rivalry and the brain has to decide which region is important and in this case the region with highest details with highest contrast will win. So in this case we wanted to know if we were able to predict the quality when one view is of lower quality than the other. So these are the sequences that have been combined in order to do the test. So these sequences come from NHK and these are the quantitative results. Here we are not talking about the accuracy of the prediction or something else because what we want here is to monitor the quality. So we cannot measure, give a measure saying okay it's better than the other. What we need here since we are in a continuous evaluation, we need only to know whether our model can predict the quality and the information that we need here is the outlayer ratio. What is the ratio of the samples when our method is out of the confidence interval that we have obtained with the subjective evaluation. You can see here that we need also the test with and without disparity because what we wanted to show here is whether the disparity has an important impact on our perceived quality or not. You can see for example here that the outlayer ratio for the system without quality is higher than the one with disparity. Lower is the outlayer ratio better at the results. It means that you have less values outside the confidence interval. Let me show you some curves. So what you have here is, so you have the quality score and here you have the time. Here you have the coding information. This means that both views have been encoded at 5 megabits per second. The black curve represents the confidence interval, the blue curve is the MOS, the subjective score, and the red curve is the predicted score. So we can see that of course it is difficult to mimic completely what has been done by the observers, but the idea is try to keep our model within the confidence interval. If you can achieve this, it means that the model is able to predict the video quality, the stereo video quality. Of course I will not show you only the case when it works because otherwise it's not fair. So you can see here for example we are in a quite complicated evaluation because you have one view coded at 8 megabits per second and you have the other view encoded at 2 megabits per second. It means that you have a big difference of quality between both views. And you can see here that the model can predict the quality, but you have some parts that are not really well predicted. Let me show you why. This is related to the nature of the scenes. Here we have two sequences with very high motion. And here we have a flat or smooth sequence. What we can say here, this prediction comes from the fact of having a very different quality between both views. So the model tries to combine the quality, but on one view you have a lot of impairment, on the other view it's quite correct in terms of quality. So this is why we see that there is a gap in quality. But you can also see it on the subjective itself. So the subjective evaluation says the same thing, but not with the same intensity. This means that we need in the model, we need to take into account the motion intensity in order to maybe try to... So we have two parameters here. The motion intensity and also the difference between the bit rate of the right and the left views. So these parameters have to be taken into account in order to improve the performance of the metric. These are other results. You can see here, for example, when you have similar encoding parameters, there is a gap, of course, but the gap is still within the confidence interval, which means that the model is able to predict the quality and it's not falling very low because the difference between both views is not very high. Same thing between three and three, even if we can say also that the model is not perfect. So we are trying to do something that is quite difficult, try to predict 3D video quality in real time using very simple features. So we can show the same thing here between three and two with a gap here, which is not very, very important. But imagine if you have a video provider, if you have something that can predict for you the quality, even if you have some wrong... Not really wrong, but not in the confidence interval, it can be useful to know whether the perceived quality of the users is high or not. So same thing with other results. And let's go to the conclusion. So here we try to propose a model to predict the quality of 3D sequences by using very simple features and by trying to combine these features using a subjective evaluation and try to optimize the model using this subjective evaluation. So the results are very encouraging in this case and we need to go first in this direction in order to improve the model with some cues about the motion, about the... So maybe add something related to the binocular rivalry if the both sequences are of very big different quality. Maybe this information has to be included in order to improve the prediction of our model. And thank you for your attention. Thank you, Shekhar. One quick question before the break. Thank you, Shekhar. That was a really interesting talk and I think you should be encouraged by the performance of your model. It's actually quite good, I think, given such a complex task. Early in the talk, you alluded to trying to determine where people might gaze, the important areas of visual attention. One thing that struck me about the failure case you had with the lady in the basket of flowers is that there's going to be a fairly strong top-down salience effect with that person's face. Yes. And it may not have been predicted because I'm sure you don't have top-down features in your model yet. Yes. Thank you for this question, Jeff. It's really the good one. One of the things that we wanted to integrate into the model is a kind of 3D saliency information, especially things that are related to high-level features. We know, for example, that text faces are attracting more of the gaze of the observers. And also, if you add to this the disparity, if you have a face in the first plane and or in the second or third plane, the quality will not measure in the same way. So this is the direction we want to adopt. Okay. Thank you, Shaker. Let's thank all the speakers again for this session. So this concludes my presentation.
 So, good morning everybody.  So, please don't look at me like a UFO because, you know, the topic is completely different  from the other presentations.  So, this work has been performed in a major part by Parviz Sazad from University of Dhaka  in Bangladesh during his stay in our lab.  And I will present our collaboration on quality assessment for stereoscopic video.  So, just before starting, you know, you know that quality is a very big issue nowadays,  and especially stereoscopic quality or 3D quality, because this quality is not, we cannot  handle it, we cannot measure it directly from the left and the right images.  So, if you have, for example, such a scene, you know that you have the left and the right  view, but our brain reconstructs the depth, and the quality is measured starting from  this information.  So, if you use 2D metrics on the left and the right views, you will only have 2D quality,  but the third dimension that the human will perceive will not capture it, because we need  something to handle how we can reconstruct this in the brain and use what we call, you  know, the disparity, use the depth information in order to be able to give a score to the  metric, to the image or video.  And you have many applications nowadays for 3D, and the best way to measure quality, to  measure 3D quality is to do subjective measurement.  But, you know, it is tedious time consuming, you cannot do it on all applications.  So we need some models that can handle the quality and that can mimic the human visual  system in order to measure this quality.  It is not a very easy task because even for 2D, the problem is not solved yet.  Imagine when you add a dimension and it's subjective dimension, it's a very big issue.  So in quality measurement, you have three families, depending on the reference.  If you have a reference, so then you can construct a full reference method.  If you have some cues about the reference, you can develop a reduced reference method,  so we can assume that you embed this reduced reference within the image or within the sequence  and then send it and use this reference to, to this reduced reference to calculate the  quality of the receiver stage.  But the best, best way to, to measure the quality is to have a no reference method, especially  for video, because if you have to send only one image, this is, will not offer high cost.  But if you have video, if you have to measure the quality of a movie theater in 3D or a  3D movie received through IP over, over your internet box, so this will, will cost a lot.  So you cannot, you cannot have the reference.  And even if you have reduced reference, it can be, can be of a high cost.  So this, the third family is the best one in order to monitor the, the quality.  So what is the objective of this work?  So this work is intended to, to develop a metric for video, for monitoring the quality  of a video.  It has to be of a low cost metric, because we need to do it in real time, if you want  to have a feedback about the quality.  We need to have some cues about what is important in, in human perception, like the sensitivity  to, to edge or to structural information.  We have to, to take care of the special information, but also about the temporal information, because  when you look, when you watch a movie, you don't watch only images.  You have successive images and the motion and the, the, the, the, the speed of the,  of the sequence is important and you have to take it into account.  And when dealing with 3D, you have a third part, which is the disparity or the depth.  You have to take it into account, because if you have an object which is close to your  eyes or if you have an object which is far, you will not perceive them in the same way.  And the impairment you will have on both objects will not have the same annoyance.  So as, as I said, so you have the special part, the depth part and the temporal part  also that has to be taken into account.  And this, these are parts of our objectives.  So this is the, the method, the proposed method based on what I said before.  So there is many things related to disparity measurement, to special, special information  and to temporal information.  But as you will see in the next slide, the method is rather simple and the idea is really  to have easy, easy to compute features in order to make our regression afterwards using  subjective measurement and then having a model that can predict quality in real time.  So for the special feature, we used an edge distortion measure.  It's relatively simple.  It's based on the zero crossing for each 8 by 8 block.  So why 8 by 8?  This is because of the construction of coding algorithms.  Here coders are used for video are using DCT and DCT is calculated on 8 by 8 blocks.  So then afterward, we, when we have the zero crossing on, for each block, we will have average  zero crossing for edge and non-edge areas of the image because we know that the quality  in regions with high activity is not the same than the quality in flat area or regions with  low activity or no texture.  So what is done in horizontal way here is this is done also in vertical and this is  just an example to show how our zero crossing is working.  It's really simple and the idea is to find for each pixel what is its activity and what  is the activity of the edge.  So then it is not done only on one frame, but we have chosen to do this on five successive  frames.  Why?  In order to be sure that we have enough cues about the special quality and because there  is the stability in the frames, so it means that we reduce the computation and also we  can have more stable features in order to compute the quality.  So for the disparities feature, we have chosen to decompose the disparity in three parts.  So apart for very low disparity, apart for average disparity and the final part for the  highest disparity.  So starting from that for each disparity we extract the information and from this we construct  our feature.  And at the end our disparity feature is a combination of the high, the medium and the  low disparity weighted here as you see and these weights are obtained afterwards thanks  to the regression and to the training that we will do on subjective database.  So this is an example of what I call low, medium and high disparity.  For example here you can see that we need to make a difference between this object which  is in the front and this one which is in the back.  While here we have more information about the object which is in the front and so on  and so forth.  So this information may allow us to say for example for this frame.  We know that since this object is taking a big place in terms of special disposition,  so it means that in terms of 3D this object will attract the gaze of the observer and  the quality will fall in this part.  So this is the simplistic temporal feature which is done also within 4 successive frames.  It's very simple, it's just a difference within a temporal interval and as I said before  the main idea here wasn't to try to develop very complex features.  Here the idea behind what we propose here is really to use simple features, very easy  to compute features but try to combine them in the best way.  So this is how we combine them.  So you have here the disparity feature, you have the special feature and the temporal here  and thanks to that database that I will describe in the next slide we have identified through  an optimization process all the weightings that we have here and that will allow us afterward  to be able to mimic the observers in terms of monitoring 3D quality.  So the subjective test has been done on a 3D monitor using SSCQE method, so with a single  stimulus and the material that has been used here is an MPEG-coded material.  The main important information that I need to give you here is that the compression has  been done in a symmetric and asymmetric way.  What does it mean?  It means that even you compress your left and right views at the same bit rate.  So this is what is originally done but we know that in our brain we have what we call  the binocular compensation.  So if you have one view with lower quality than the other the brain will compensate and  we will have the best view will compensate the lower view and we will have a good one.  So what we call also the binocular rivalry, there is a good experiment that you can do.  If you take a paper then you make a cylinder, you look, you put it on your left eye and  then you put your hand on the right eye you will see the hole on your hand.  Why?  Because you have two different information, you have a binocular rivalry and the brain  has to decide which region is important and in this case the region with highest details  with highest contrast will win.  So in this case we wanted to know if we were able to predict the quality when one view is  of lower quality than the other.  So these are the sequences that have been combined in order to do the test.  So these sequences come from NHK and these are the quantitative results.  Here we are not talking about the accuracy of the prediction or something else because  what we want here is to monitor the quality.  So we cannot measure, give a measure saying okay it's better than the other.  What we need here since we are in a continuous evaluation, we need only to know whether our  model can predict the quality and the information that we need here is the outlayer ratio.  What is the ratio of the samples when our method is out of the confidence interval that  we have obtained with the subjective evaluation.  You can see here that we need also the test with and without disparity because what we  wanted to show here is whether the disparity has an important impact on our perceived quality  or not.  You can see for example here that the outlayer ratio for the system without quality is higher  than the one with disparity.  Lower is the outlayer ratio better at the results.  It means that you have less values outside the confidence interval.  Let me show you some curves.  So what you have here is, so you have the quality score and here you have the time.  Here you have the coding information.  This means that both views have been encoded at 5 megabits per second.  The black curve represents the confidence interval, the blue curve is the MOS, the subjective  score, and the red curve is the predicted score.  So we can see that of course it is difficult to mimic completely what has been done by  the observers, but the idea is try to keep our model within the confidence interval.  If you can achieve this, it means that the model is able to predict the video quality,  the stereo video quality.  Of course I will not show you only the case when it works because otherwise it's not  fair.  So you can see here for example we are in a quite complicated evaluation because you  have one view coded at 8 megabits per second and you have the other view encoded at 2 megabits  per second.  It means that you have a big difference of quality between both views.  And you can see here that the model can predict the quality, but you have some parts that  are not really well predicted.  Let me show you why.  This is related to the nature of the scenes.  Here we have two sequences with very high motion.  And here we have a flat or smooth sequence.  What we can say here, this prediction comes from the fact of having a very different quality  between both views.  So the model tries to combine the quality, but on one view you have a lot of impairment,  on the other view it's quite correct in terms of quality.  So this is why we see that there is a gap in quality.  But you can also see it on the subjective itself.  So the subjective evaluation says the same thing, but not with the same intensity.  This means that we need in the model, we need to take into account the motion intensity  in order to maybe try to...  So we have two parameters here.  The motion intensity and also the difference between the bit rate of the right and the  left views.  So these parameters have to be taken into account in order to improve the performance  of the metric.  These are other results.  You can see here, for example, when you have similar encoding parameters, there is a gap,  of course, but the gap is still within the confidence interval, which means that the model  is able to predict the quality and it's not falling very low because the difference between  both views is not very high.  Same thing between three and three, even if we can say also that the model is not perfect.  So we are trying to do something that is quite difficult, try to predict 3D video quality  in real time using very simple features.  So we can show the same thing here between three and two with a gap here, which is not  very, very important.  But imagine if you have a video provider, if you have something that can predict for  you the quality, even if you have some wrong...  Not really wrong, but not in the confidence interval, it can be useful to know whether  the perceived quality of the users is high or not.  So same thing with other results.  And let's go to the conclusion.  So here we try to propose a model to predict the quality of 3D sequences by using very  simple features and by trying to combine these features using a subjective evaluation  and try to optimize the model using this subjective evaluation.  So the results are very encouraging in this case and we need to go first in this direction  in order to improve the model with some cues about the motion, about the...  So maybe add something related to the binocular rivalry if the both sequences are of very  big different quality.  Maybe this information has to be included in order to improve the prediction of our  model.  And thank you for your attention.  Thank you, Shekhar.  One quick question before the break.  Thank you, Shekhar.  That was a really interesting talk and I think you should be encouraged by the performance  of your model.  It's actually quite good, I think, given such a complex task.  Early in the talk, you alluded to trying to determine where people might gaze, the important  areas of visual attention.  One thing that struck me about the failure case you had with the lady in the basket of  flowers is that there's going to be a fairly strong top-down salience effect with that  person's face.  Yes.  And it may not have been predicted because I'm sure you don't have top-down features  in your model yet.  Yes.  Thank you for this question, Jeff.  It's really the good one.  One of the things that we wanted to integrate into the model is a kind of 3D saliency information,  especially things that are related to high-level features.  We know, for example, that text faces are attracting more of the gaze of the observers.  And also, if you add to this the disparity, if you have a face in the first plane and  or in the second or third plane, the quality will not measure in the same way.  So this is the direction we want to adopt.  Okay.  Thank you, Shaker.  Let's thank all the speakers again for this session.  So this concludes my presentation.
Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_0_83.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_84_203.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_204_347.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_348_899.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_900_2711.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_2712_3683.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_3684_11831.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_11832_11903.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_11904_12095.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_12096_13499.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_13500_14771.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_14772_19007.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_19008_19175.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_19176_19475.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_19476_19823.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_19824_22031.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_22032_24551.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_24552_26387.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_26388_28379.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_28380_29723.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_29724_29831.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_29832_29879.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_29880_30035.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_30036_31187.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_31188_31847.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_31848_31895.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_31896_32171.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_32172_32279.png Feature_based_no-reference_continuous_video_quality_prediction_model_for_coded_stereo_video_32280_32327.png
So, good morning everybody. So, please don't look at me like a UFO because, you know, the topic is completely different from the other presentations. So, this work has been performed in a major part by Parviz Sazad from University of Dhaka in Bangladesh during his stay in our lab. And I will present our collaboration on quality assessment for stereoscopic video. So, just before starting, you know, you know that quality is a very big issue nowadays, and especially stereoscopic quality or 3D quality, because this quality is not, we cannot handle it, we cannot measure it directly from the left and the right images. So, if you have, for example, such a scene, you know that you have the left and the right view, but our brain reconstructs the depth, and the quality is measured starting from this information. So, if you use 2D metrics on the left and the right views, you will only have 2D quality, but the third dimension that the human will perceive will not capture it, because we need something to handle how we can reconstruct this in the brain and use what we call, you know, the disparity, use the depth information in order to be able to give a score to the metric, to the image or video. And you have many applications nowadays for 3D, and the best way to measure quality, to measure 3D quality is to do subjective measurement. But, you know, it is tedious time consuming, you cannot do it on all applications. So we need some models that can handle the quality and that can mimic the human visual system in order to measure this quality. It is not a very easy task because even for 2D, the problem is not solved yet. Imagine when you add a dimension and it's subjective dimension, it's a very big issue. So in quality measurement, you have three families, depending on the reference. If you have a reference, so then you can construct a full reference method. If you have some cues about the reference, you can develop a reduced reference method, so we can assume that you embed this reduced reference within the image or within the sequence and then send it and use this reference to, to this reduced reference to calculate the quality of the receiver stage. But the best, best way to, to measure the quality is to have a no reference method, especially for video, because if you have to send only one image, this is, will not offer high cost. But if you have video, if you have to measure the quality of a movie theater in 3D or a 3D movie received through IP over, over your internet box, so this will, will cost a lot. So you cannot, you cannot have the reference. And even if you have reduced reference, it can be, can be of a high cost. So this, the third family is the best one in order to monitor the, the quality. So what is the objective of this work? So this work is intended to, to develop a metric for video, for monitoring the quality of a video. It has to be of a low cost metric, because we need to do it in real time, if you want to have a feedback about the quality. We need to have some cues about what is important in, in human perception, like the sensitivity to, to edge or to structural information. We have to, to take care of the special information, but also about the temporal information, because when you look, when you watch a movie, you don't watch only images. You have successive images and the motion and the, the, the, the, the speed of the, of the sequence is important and you have to take it into account. And when dealing with 3D, you have a third part, which is the disparity or the depth. You have to take it into account, because if you have an object which is close to your eyes or if you have an object which is far, you will not perceive them in the same way. And the impairment you will have on both objects will not have the same annoyance. So as, as I said, so you have the special part, the depth part and the temporal part also that has to be taken into account. And this, these are parts of our objectives. So this is the, the method, the proposed method based on what I said before. So there is many things related to disparity measurement, to special, special information and to temporal information. But as you will see in the next slide, the method is rather simple and the idea is really to have easy, easy to compute features in order to make our regression afterwards using subjective measurement and then having a model that can predict quality in real time. So for the special feature, we used an edge distortion measure. It's relatively simple. It's based on the zero crossing for each 8 by 8 block. So why 8 by 8? This is because of the construction of coding algorithms. Here coders are used for video are using DCT and DCT is calculated on 8 by 8 blocks. So then afterward, we, when we have the zero crossing on, for each block, we will have average zero crossing for edge and non-edge areas of the image because we know that the quality in regions with high activity is not the same than the quality in flat area or regions with low activity or no texture. So what is done in horizontal way here is this is done also in vertical and this is just an example to show how our zero crossing is working. It's really simple and the idea is to find for each pixel what is its activity and what is the activity of the edge. So then it is not done only on one frame, but we have chosen to do this on five successive frames. Why? In order to be sure that we have enough cues about the special quality and because there is the stability in the frames, so it means that we reduce the computation and also we can have more stable features in order to compute the quality. So for the disparities feature, we have chosen to decompose the disparity in three parts. So apart for very low disparity, apart for average disparity and the final part for the highest disparity. So starting from that for each disparity we extract the information and from this we construct our feature. And at the end our disparity feature is a combination of the high, the medium and the low disparity weighted here as you see and these weights are obtained afterwards thanks to the regression and to the training that we will do on subjective database. So this is an example of what I call low, medium and high disparity. For example here you can see that we need to make a difference between this object which is in the front and this one which is in the back. While here we have more information about the object which is in the front and so on and so forth. So this information may allow us to say for example for this frame. We know that since this object is taking a big place in terms of special disposition, so it means that in terms of 3D this object will attract the gaze of the observer and the quality will fall in this part. So this is the simplistic temporal feature which is done also within 4 successive frames. It's very simple, it's just a difference within a temporal interval and as I said before the main idea here wasn't to try to develop very complex features. Here the idea behind what we propose here is really to use simple features, very easy to compute features but try to combine them in the best way. So this is how we combine them. So you have here the disparity feature, you have the special feature and the temporal here and thanks to that database that I will describe in the next slide we have identified through an optimization process all the weightings that we have here and that will allow us afterward to be able to mimic the observers in terms of monitoring 3D quality. So the subjective test has been done on a 3D monitor using SSCQE method, so with a single stimulus and the material that has been used here is an MPEG-coded material. The main important information that I need to give you here is that the compression has been done in a symmetric and asymmetric way. What does it mean? It means that even you compress your left and right views at the same bit rate. So this is what is originally done but we know that in our brain we have what we call the binocular compensation. So if you have one view with lower quality than the other the brain will compensate and we will have the best view will compensate the lower view and we will have a good one. So what we call also the binocular rivalry, there is a good experiment that you can do. If you take a paper then you make a cylinder, you look, you put it on your left eye and then you put your hand on the right eye you will see the hole on your hand. Why? Because you have two different information, you have a binocular rivalry and the brain has to decide which region is important and in this case the region with highest details with highest contrast will win. So in this case we wanted to know if we were able to predict the quality when one view is of lower quality than the other. So these are the sequences that have been combined in order to do the test. So these sequences come from NHK and these are the quantitative results. Here we are not talking about the accuracy of the prediction or something else because what we want here is to monitor the quality. So we cannot measure, give a measure saying okay it's better than the other. What we need here since we are in a continuous evaluation, we need only to know whether our model can predict the quality and the information that we need here is the outlayer ratio. What is the ratio of the samples when our method is out of the confidence interval that we have obtained with the subjective evaluation. You can see here that we need also the test with and without disparity because what we wanted to show here is whether the disparity has an important impact on our perceived quality or not. You can see for example here that the outlayer ratio for the system without quality is higher than the one with disparity. Lower is the outlayer ratio better at the results. It means that you have less values outside the confidence interval. Let me show you some curves. So what you have here is, so you have the quality score and here you have the time. Here you have the coding information. This means that both views have been encoded at 5 megabits per second. The black curve represents the confidence interval, the blue curve is the MOS, the subjective score, and the red curve is the predicted score. So we can see that of course it is difficult to mimic completely what has been done by the observers, but the idea is try to keep our model within the confidence interval. If you can achieve this, it means that the model is able to predict the video quality, the stereo video quality. Of course I will not show you only the case when it works because otherwise it's not fair. So you can see here for example we are in a quite complicated evaluation because you have one view coded at 8 megabits per second and you have the other view encoded at 2 megabits per second. It means that you have a big difference of quality between both views. And you can see here that the model can predict the quality, but you have some parts that are not really well predicted. Let me show you why. This is related to the nature of the scenes. Here we have two sequences with very high motion. And here we have a flat or smooth sequence. What we can say here, this prediction comes from the fact of having a very different quality between both views. So the model tries to combine the quality, but on one view you have a lot of impairment, on the other view it's quite correct in terms of quality. So this is why we see that there is a gap in quality. But you can also see it on the subjective itself. So the subjective evaluation says the same thing, but not with the same intensity. This means that we need in the model, we need to take into account the motion intensity in order to maybe try to... So we have two parameters here. The motion intensity and also the difference between the bit rate of the right and the left views. So these parameters have to be taken into account in order to improve the performance of the metric. These are other results. You can see here, for example, when you have similar encoding parameters, there is a gap, of course, but the gap is still within the confidence interval, which means that the model is able to predict the quality and it's not falling very low because the difference between both views is not very high. Same thing between three and three, even if we can say also that the model is not perfect. So we are trying to do something that is quite difficult, try to predict 3D video quality in real time using very simple features. So we can show the same thing here between three and two with a gap here, which is not very, very important. But imagine if you have a video provider, if you have something that can predict for you the quality, even if you have some wrong... Not really wrong, but not in the confidence interval, it can be useful to know whether the perceived quality of the users is high or not. So same thing with other results. And let's go to the conclusion. So here we try to propose a model to predict the quality of 3D sequences by using very simple features and by trying to combine these features using a subjective evaluation and try to optimize the model using this subjective evaluation. So the results are very encouraging in this case and we need to go first in this direction in order to improve the model with some cues about the motion, about the... So maybe add something related to the binocular rivalry if the both sequences are of very big different quality. Maybe this information has to be included in order to improve the prediction of our model. And thank you for your attention. Thank you, Shekhar. One quick question before the break. Thank you, Shekhar. That was a really interesting talk and I think you should be encouraged by the performance of your model. It's actually quite good, I think, given such a complex task. Early in the talk, you alluded to trying to determine where people might gaze, the important areas of visual attention. One thing that struck me about the failure case you had with the lady in the basket of flowers is that there's going to be a fairly strong top-down salience effect with that person's face. Yes. And it may not have been predicted because I'm sure you don't have top-down features in your model yet. Yes. Thank you for this question, Jeff. It's really the good one. One of the things that we wanted to integrate into the model is a kind of 3D saliency information, especially things that are related to high-level features. We know, for example, that text faces are attracting more of the gaze of the observers. And also, if you add to this the disparity, if you have a face in the first plane and or in the second or third plane, the quality will not measure in the same way. So this is the direction we want to adopt. Okay. Thank you, Shaker. Let's thank all the speakers again for this session. So this concludes my presentation.
in a major part by Parviz Sazad from University quality assessment for stereoscopic video. So, such a scene, you know that you have the left and human will perceive will not capture it, because the human visual system in order to measure this measure the quality of a movie theater in 3D or a a third part, which is the disparity or the impairment you will have on both objects will not have the special part, the depth part and the proposed method based on what I said before. So to temporal information. But as you will see in time. So for the special feature, we used an edge based on the zero crossing for each 8 by 8 block. calculated on 8 by 8 blocks. So then afterward, edge and non-edge areas of the image because we activity of the edge. So then it is not done only say for example for this frame. We know that features. Here the idea behind what we propose the compression has been done in a symmetric and Because you have two different information, you a continuous evaluation, we need only to know means that the model is able to predict the video quality, the stereo video quality. Of course I Here we have two sequences with very high motion. there is a gap in quality. But you can also see the performance of the metric. These are other whether the perceived quality of the users is propose a model to predict the quality of 3D order to improve the prediction of our model. And