text:
Synthesis of Tongue Motion and Acoustics From Text using a Multimodal Articulatory Database

# Synthesis of Tongue Motion and Acoustics From Text using a Multimodal Articulatory Database

Ingmar Steiner, Sébastien Le Maguer, and Alexander Hewer All authors are with the Cluster of Excellence “Multimodal Computing and Interaction,” Saarland University, Saarbrücken 66123, Germany (e-mail: steiner@coli.uni-saarland.de; slemaguer@coli.uni-saarland.de; hewer@coli.uni-saarland.de). This paper has supplementary downloadable material available at http://ieeexplore.ieee.org. This includes videos of all synthesized utterances in the test set, produced using the tongue model parameters (cf., Section III-G). This material is in size. Digital Object Identifier 10.1109/TASLP.2017.2756818
###### Abstract

We present an end-to-end text-to-speech (TTS) synthesis system that generates audio and synchronized tongue motion directly from text. This is achieved by adapting a 3D model of the tongue surface to an articulatory dataset and training a statistical parametric speech synthesis system directly on the tongue model parameters. We evaluate the model at every step by comparing the spatial coordinates of predicted articulatory movements against the reference data. The results indicate a global mean Euclidean distance of less than 2.8 mm, and our approach can be adapted to add an articulatory modality to conventional TTS applications without the need for extra data.

Text-to-speech, multimodal synthesis, tongue modeling, articulatory animation, electromagnetic articulography
\DeclareAcronym

2Dlong=two-dimensional \DeclareAcronym3Dlong=three-dimensional \DeclareAcronymBAPlong=aperiodicity per band \DeclareAcronymCAPTlong=computer-assisted pronunciation training \DeclareAcronymDFKIlong=German Research Center for Artificial Intelligence \DeclareAcronymDoFshort-plural=, long=degree of freedom, long-plural-form=degrees of freedom \DeclareAcronymEGGlong=electroglottography \DeclareAcronymEMAlong=electromagnetic articulography \DeclareAcronymEPGlong=electropalatography \DeclareAcronymESTlong=Edinburgh Speech Tools \DeclareAcronymF0short=F\textsubscript0, long=fundamental frequency \DeclareAcronymGMMlong=Gaussian mixture model \DeclareAcronymHMMlong=hidden Markov model \DeclareAcronymHTSlong=\acHMM based synthesis \DeclareAcronymMCDlong=mel cepstral distortion \DeclareAcronymMGClong=mel-generalized cepstral coefficients \DeclareAcronymMLSAlong=mel log spectrum approximation \DeclareAcronymMRIlong=magnetic resonance imaging \DeclareAcronymPCAlong=principal component analysis \DeclareAcronymRMSElong=root mean square error \DeclareAcronymTTSlong=text-to-speech \DeclareAcronymUTIlong=ultrasound tongue imaging \DeclareAcronymVUVlong=voiced-unvoiced \DeclareAcronymXRMBlong=X-ray microbeam \DeclareAcronymHOSVDlong=higher order singular value decomposition \addbibresourcereferences.bib \pdfstringdefDisableCommands\pdfstringdefDisableCommands

## I Introduction

The sound of human speech is the direct result of production mechanisms in the human vocal tract. Air flows from the lungs through the glottis, whose vocal folds can be set to vibrate, the sound of which is then filtered by the shape of the tongue, lips, and other articulators, generating what we perceive as audible signals such as spoken language. Researchers in phonetics and linguistics have studied these speech production mechanisms for many years, but while the acoustic signal and facial movements can be observed and measured directly, doing the same for partially or fully hidden articulators such as the tongue and glottis is not as straightforward.

Consequently, sensing and imaging techniques have been applied to the challenge of observing speech production mechanisms in vivo, which has greatly improved our understanding of these processes. The corresponding modalities include, fluoroscopy [Munhall1995JASA], \acUTI [Stone2005], \acXRMB [Westbury1994], \acEMA [Schoenle1987, Hoole2010], and real-time \acMRI [Niebergall2012, Narayanan2014], among others. Some of these involve health hazards (due to ionizing radiation), and all are more or less invasive, but they produce biosignals which, in combination with simultaneous acoustic recordings, represent multimodal articulatory speech data. The benefits are tempered by the challenges of processing the imaging and/or point-tracking data, which in the field of speech processing has created new opportunities for collaboration with areas such as medical imaging and computer vision.

The biosignals that can be obtained using such modalities to record spoken language, provide opportunities to greatly enhance models of speech by integrating measurements of the underlying processes directly with the acoustic signal. This leads to more elegant and powerful approaches to speech analysis and synthesis [Ling2009, Ling2010SpeCom, Richmond2015]. However, it must be borne in mind that all of the biosignals produced by the modalities mentioned above represent a sampling of the articulators that is sparse in the temporal domain, the spatial domain, or both.

Depending on the manner in which the data is used for analysis or applications, the resolution may need to be increased, but the missing samples cannot be restored without prior knowledge, typically provided by a statistical model trained on other data.

In this study, we present an approach to multimodal \acTTS synthesis that generates the fully animated, \ac3D surface of the tongue, synchronized with synthetic audio, using data from a single-speaker, articulatory corpus that includes \acEMA motion capture of three tongue fleshpoints [Richmond2011]. The audio and articulatory motion are synthesized using the \acHTS framework [Zen2009], while the surface restoration is performed by means of a multilinear statistical tongue model [Hewer2018CSL] trained on a multi-speaker, volumetric \acMRI dataset [Richmond2012]. The potential application domains of this approach include audiovisual speech synthesis and \acCAPT, among others.

### I-a Background

Deriving models suitable for producing speech related tongue motion is an active field of research. Such models can, for example, help to analyze and understand articulatory data that is very sparse in the spatial domain. Ideally, such tongue models should offer a good compromise between accuracy of the generated shape and the available \acpDoF for manipulating it. This means that biomechanical models such as those presented by \citetLloyd2012, \citetXu2015, \citetWrench2015, or \citetYu2017 might be too complex for this purpose. While such models aim to simulate the underlying mechanics of the human tongue as closely as possible, and can be used to visualize existing articulatory data, they can be challenging to control efficiently.

Geometric tongue models are less complex than their biomechanical counterparts. Here, we distinguish between generic and statistical tongue models. Generic tongue models are 3D models of the tongue that may be deformed and animated by using standard methods in computer graphics.

Statistical tongue models, on the other hand, are constructed by analyzing the \acpDoF of the tongue shape in recorded articulatory data, such as \acMRI recordings of speech related vocal tract shapes. Roughly speaking, such an analysis can be carried out in two ways. The first variant investigates shape variations related to the tongue pose that are specific for speech production. Examples of such approaches are the works by \citetEngwall2000, Badin2002, and \citetBadin2006, who examined those variations in \ac3D \acMRI scans from a single speaker, respectively. These methods only estimate the \acDoF that are tongue pose related, while shape variations that may describe anatomical differences are missing.

Another class of methods aims at investigating those anatomy and tongue pose related shape variations separately. This paradigm offers several advantages: First, the results give access to tongue models that may be adapted to new speakers. Second, this type of analysis may also provide insight into how anatomical differences affect human articulation. For \ac2D \acMRI, such work was conducted, e.g., by \citetHoole2000 and \citetAnanthakrishnan2010. \citetZheng2003 investigated those variations in a sparse point cloud extracted from 3D \acMRI. Most recently, we performed such an analysis on mesh representations of the tongue that were extracted from 3D \acMRI scans \citepHewer2018CSL.

Such geometrical models have been successfully used in previous work to generate animations from provided articulatory data: \citetKatz2014 presented a real-time visual feedback system that deforms a generic tongue model using \acEMA data. However, due to the generic nature of the model, their approach did not take anatomical differences into account. A statistical model was used in the approach by \citetBadin2008, who used volumetric imaging data of one speaker to derive the tongue model, and \acEMA data of the same speaker to animate it. \citetEngwall2003 followed a similar approach. Our own previous work utilized a multilinear statistical model to visualize \acEMA data, which allowed it to be adapted to different speakers \citepJames2016.

Independently, there is a growing body of work on application-oriented research to combine articulatory data, and features derived from it, with speech technology applications, such as to recover articulatory movements from the acoustic signal (“articulatory inversion mapping”, cf. \citepKing2007, Mitra2011 for examples), provide articulatory control for reactive \acTTS synthesis (e.g., \citepAstrinaki2013, Ling2013), or predict sparse articulatory movements from a symbolic representation (e.g., \citepLing2010SpeCom, Cai2015).

Early studies on animating full 3D tongue surface models using \acEMA data for multimodal speech synthesis, such as those of \citetEngwall2002ICSLP or \citetFagel2004SpeCom, used concatenative \acTTS systems. Other approaches (e.g., \citepBenYoussef2011PhD) for \acHMM based \acTTS with intra-oral animation also rely on acoustic-articulatory inversion mapping. However, to our knowledge, no previous study has presented an end-to-end system to directly synthesize acoustics and the motion of a full \ac3D model of the tongue surface from text using statistical parametric speech synthesis, particularly with a tongue model that can be easily adapted to the anatomy of different speakers.

## Ii Method

### Ii-a Multilinear Shape Space Model

In our approach, we utilize a multilinear model to describe different tongue shapes. This is achieved by using this model to create a function

 f:Rm×Rn→M (1)

that maps the parameters and to a polygon mesh . Such a mesh consists of a vertex set that contains positional data and a face set that uses these vertices to form the collection of surface patches of the represented shape. We note that these meshes have the same face set and only differ in the positional data of their vertices. The used parameters in the function describe two distinct sets of features: On the one hand, the speaker parameter determines the anatomical features of the generated tongue. The pose parameter , on the other hand, represents the shape properties that are related to articulation.

To compute the multilinear model, we use a database that consists of \acMRI scans of speakers showing their vocal tract configuration for different phonemes. By means of image processing and template matching methods, we extract tongue meshes from the \acMRI data, such that in the end, for each speaker, one mesh is available for each considered phoneme. This processing is described in detail by \citetHewer2016PSA, Hewer2018CSL. We then proceed to derive the \acDoF of the anatomy and speech related variations. To this end, we center the obtained meshes and turn them into feature vectors by serializing the positional data of their vertices. Afterwards, we construct a tensor of third order consisting of these feature vectors, such that the first mode of the tensor corresponds to the speakers, the second one to the considered phonemes, and the third one to the positional data.

In a final step, we apply \acHOSVD [Tucker1966] to obtain the following tensor decomposition:

 A=C×1U1×2U2 (2)

In this decomposition, the tensor is of third order and represents our multilinear model. The operation is the -th mode multiplication of the tensor with the matrix . The two matrices and contain the parameters for reconstructing the original feature vectors: Each row of is a speaker parameter and each row of a pose parameter. Basically, each speaker parameter represents a point in the -dimensional speaker subspace and each pose parameter a point in the -dimensional pose subspace that are linked together by the tensor . We remark that, compared to a \acPCA model, such a multilinear model offers the advantage that it aims at capturing anatomical and articulation related shape variations separately.

The tensor can be used to create new positional data for provided parameters and :

 v(→s,→p)=μ+C×1→s×2→p (3)

where is a feature vector consisting of the positional data that corresponds to the mean mesh of the tongue shape collection. This generated information can be utilized to construct a new tongue shape: We reconstruct the vertex set by using the created positional data and combine it with the original face set to obtain our mesh. More details on how the model was derived and evaluated can be found in [Hewer2018CSL].

In our framework, we use this model to register data of an \acEMA corpus in order to obtain the corresponding parameters, which is done as follows: In a first step, we manually align the \acEMA data to the model space by using a provided reference coil. As we want to register the \acEMA data, we have to decide which coil corresponds to which vertex of the model mesh. This process is done in a semi-supervised way: The parameters are first set to random values and the associated mesh is generated. Next, for each considered coil the nearest vertex on the mesh is found. We then refine these correspondences iteratively by fitting the model to the coils and updating the nearest vertices. In the end, we keep the correspondences that resulted in the smallest average Euclidean distance. Finally, we inspect the result manually and repeat the experiment if the correspondences appear to be wrong. The tongue model mesh is shown in Fig. 1, highlighting the vertices selected to correspond with the three tongue coils in the \acEMA data. With these estimated correspondences, we fit the multilinear model to each considered \acEMA data frame of the corpus by minimizing the energy:

 E(→s,→p)=EData(→s,→p)+α ESC(→s)+β EPS(→p) (4)

The data term measures the distances between the selected vertices of the generated mesh and the corresponding coil positions. The speaker consistency term weighted by generates energy if the current speaker parameter differs from the one of the previous time step. The remaining term, the pose smoothness term weighted by fulfills a similar role: It penalizes changes of the pose parameter over time. As a minimizer of this energy is the best compromise between those mentioned assumptions, the fitting results will be close to the data and show smooth transitions over time. The degree of smoothness can be controlled by adjusting the weights and . As the multilinear model can be used to measure the probability of generated shapes, we can also choose how far the results are allowed to deviate from the model mean: We limit the possible values for each entry of the parameters to an interval where , is the mean and the standard deviation of the corresponding entry in the training set of tongue meshes. In order to obtain a minimizer, we use a quasi-Newton solver [Liu1989] that supports limiting the solution to the given intervals.

### Ii-B Multimodal Statistical Parametric Speech Synthesis

\acreset

HTS

The \acHTS framework first presented by \citetZen2005 is a standard statistical parametric speech synthesis system. The architecture comprises four main parts:

1. the parametrization of the signal,

2. the training of the models,

3. the parameter generation, and

4. the signal rendering.

The focus of our study impacts the parametrization (a) and the rendering (d) stages. Therefore, we use the standard training stage (b), described in [Zen2005], and the standard parameter generation algorithms (c), described in [Tokuda2000].

The parametrization of the signal can be performed using any suitable signal processing tool, as long as it is kept consistent with the signal rendering. In the standard procedure, this is generally accomplished by coupling STRAIGHT [Kawahara1999] with a \acMLSA filter [Fukada1992]. First, STRAIGHT is used to extract the spectral envelope, the \acF0, and the aperiodicity. Generally, the \acF0 values are transformed into the logarithmic domain, to be more consistent with human hearing. Since the number of coefficients used of the spectral envelope and the aperiodicity is too high, the \acMLSA filter is used to parametrize these coefficients and to obtain the \acMGC and the \acBAP, respectively.

In this study, we propose to not only consider the parametrization of the acoustic signal but also the parametrization of speech articulation. In previous studies [Ling2009, Ling2010IS, Ling2010SpeCom], \acEMA data was used as the articulatory representation. In the present study, we work towards replacing the \acEMA data by the tongue model parameters. Therefore, our goal is to train on the trajectories of the tongue model parameters using the standard \acHTS framework as presented by \citetZen2005. The training models in \acHTS are \acpHMM, at a phone level, whose observations are composed by decision trees. The leaves of the decisions trees are \acpGMM which are used to produce the parameters at the generation level. The generation level consists of applying the algorithm presented by \citetTokuda2000. Fig. 2 presents the details of the modified architecture.

## Iii Experiments

### Iii-a Multilinear Model

As the database for deriving the multilinear model, we used \acMRI data from the Ultrax project \citepRichmond2012 (11 speakers) and combined it with the data of \citetBaker2011 (1 speaker), which was recorded as part of the Ultrax project, but released separately. In the end, the resulting tongue mesh collection contained, for each speaker, estimated shapes for the phone set \textipa[i, e, E, a, A, 2, O, o, u, 0, @, s, S]. Accordingly, the resulting multilinear model has   and   for the anatomy and tongue pose, respectively. The tongue mesh we used for the template matching was manually extracted from one \acMRI scan, made symmetric to remove some bias towards the original speaker, and finally remeshed to be more isotropic. It consists of vertices, faces, and has a spatial resolution of .

### Iii-B Database

The data used for the experiments in this study is taken from the mngu0 corpus, specifically the “day 1” \acEMA subset [Richmond2011], which contains acoustic recordings, time-aligned phonetic transcriptions, and \acEMA motion capture data (sampled at using a Carstens AG500 articulograph).111From the mngu0 website, http://mngu0.org, we downloaded the following distribution packages: Day1 basic audio data downsampled to (v1.1.0) Day1 basic \acsEMA data, head corrected and unnormalized (v1.1.0) Day1 transcriptions, Festival utterances and ESPS label files (v1.1.1) We selected the “basic” (as opposed to the “normalized”) release variant of the \acEMA data, because it preserves the silent (i.e., non-speech) intervals, as well as the \ac3D nature and true spatial coordinates of the sensor data (after head motion compensation). The \acEMA coil layout for this data is shown in Fig. 3; the coils are explained in Table I.

In order to manipulate the \acEMA data more flexibly, the files were first converted from the binary \acEST format to a JSON structure. Invalid values (i.e., NaN) were replaced by linear interpolation. No further modification, in particular no smoothing, was applied.

From the provided acoustic data, signal parameters were extracted using STRAIGHT [Kawahara1999] with a frame rate of , matching that of the \acEMA data. As we follow the standard \acHTS methodology, we also kept the same parameters. Therefore, our signal parameters are , , and one coefficient for the \acF0.

From the utterances in the data, (, around ) were randomly selected and held back as a test set; the remaining utterances (around ) were used as the training set to build \acHTS synthesis voices. A comparison of phone distributions in the training and test sets shows a satisfactory match (cf. Fig. 4).

### Iii-C Acoustic Synthesis

As a baseline, we first built a conventional \acTTS system using the acoustic data only. This served mainly to validate our voicebuilding process and ensure that the transcriptions provided, and labels generated from them, along with the acoustic signal parameters, were able to generate audio of sufficient quality. Accordingly, we did not undertake a formal subjective listening test, and instead evaluated this baseline experiment using objective measures only.

We synthesized the utterances in the test set using two conditions. The first condition is the standard synthesis process. This condition allows us to evaluate the duration accuracy. For the second condition, we imposed the acoustic phone durations from the provided transcriptions to allow direct comparison with the natural recordings. For the following experiments, we synthesized both conditions as well. The objective evaluation was conducted based on the following metrics.

For the duration evaluation, we calculated the duration \acRMSE at the phone level (in ) between the reference duration and the one synthesized using the first condition.

Considering the other coefficients, we compared the synthesis result (), achieved using the second condition, to the reference () present in the test corpus. As the duration was imposed, we have the same number of frames for the produced utterance and the reference one. To evaluate the \acF0, we used three measures: the \acVUV error rate percentage creftype 6 to check the prediction of the \acF0, the \acRMSE in creftype 7, and the \acRMSE in cent creftype 8. The latter measure focuses on the frames which are voiced in both conditions (original and predicted \acF0). Furthermore, it is a log scale measure adapted to the human perception.

 v(x,y)={0x \& y are both voiced/unvoiced1otherwise (5)
 VUV(r,s)=(T∑t=1v(rt,st)/T)∗100 (6)
 RMSE$Hz$(r,s)= ⎷T∑t=1(rt−st)2/T (7)
 RMSEcent(r,s)= ⎷1200∗T∑t=1(log(rt)−log(st))2/T (8)

Finally, to evaluate the spectral envelope production, we computed the \acMCD between the \acMGC vectors of dimension in :

 d(x,y)=M∑m=2(x(m)−y(m))2 (9)
 MCD(r,s)=10ln10∗√2∗ ⎷T∑t=1d(rt,st)/T (10)

Except for the duration, all parameters were evaluated at the frame level. Based on these measures, we can compare our results to previous studies, such as the one presented by \citetYokomizo2010.

The results of this evaluation are given in Table II and comprise the mean, standard deviation, and confidence interval with a value at . Compared to [Yokomizo2010], we achieved slightly better results, notwithstanding the different dataset. Therefore, we can conclude that our acoustic prediction is consistent with the state of the art in \acHTS.

### Iii-D Combined Acoustic and \acEMA Synthesis

Adopting the paradigm of early multimodal fusion, we combined the acoustic signal parameters with the \ac3D positions of the seven \acEMA coils shown in Table I, increasing the vector size by , to parameters per frame. Using the \acHTS framework, we then built another \acTTS system from this multimodal data.

Synthesizing the test set in this way, we obtained, in addition to the audio, synthetic trajectories of predicted \acEMA coil positions. To evaluate the combined acoustic and \acEMA synthesis, we computed the same objective measures as in Section III-C. We also computed the Euclidean distance in space between the observed and predicted positions for the \acEMA coils. Finally we computed the \acRMSE between the dynamics of the trajectories of the coils using a unit of millimeters per frame (). The results of this evaluation are given in Table III. We see that the differences in the acoustic measures compared to the acoustic-only synthesis (cf. Table II) are negligible.

The comparison between the observed and predicted trajectories for one test utterance is illustrated in Fig. 5. The observed and predicted (synthesized) positions of the three tongue coils are shown in each of the three dimensions in the data, along with the Euclidean distance. Silent intervals and consonants classified as coronal \textipa[t, d, n, l, s, z, S, Z, T, D] and dorsal \textipa[g, k, N], based on the provided phonetic transcription, have been highlighted. This helps visualize the correspondence between gestures of the tongue tip (coil T1) and tongue back (coils T2 and T3) for coronal and dorsal consonants, respectively, and the phonetic units they produce.

Several points merit discussion. First of all, there are large mismatches between the observed and predicted tongue \acEMA coil positions during the silent (pause) intervals at the beginning and end of the utterance. This can be attributed to the fact that the wide range of the speaker’s tongue movements during non-speech intervals are not distinguished in the provided annotations, but invariably labeled with the same pause symbol. However, there are at least two very distinct shapes for the tongue during such silent intervals, including a “rest” and a “ready” position (just before speech is produced), in addition to other complex movements such as swallowing. In the absence of distinct labels corresponding to these positions and movements, none of this silent variation can be captured by the \acpHMM trained on this data; instead, the tongue coils are unsurprisingly predicted to hover around global means.

Secondly, there is noticeable oversmoothing and target extrema are not always quite reached. This can typically be attributed to the \acHMM based synthesis technique, despite the integration of global variance. The dynamics, however, are well represented, and the predicted positional trajectories, as well as their derivatives, match the observed reference quite closely.

The axis appears to suffer from a greater amount of prediction error than the or axes. However, it should be noted that the positional variation along the axis is an order of magnitude smaller than that along the axis. It must also be borne in mind that nearly all of the speech-related movements occur in the mid-sagittal plane, represented by the (anterior/posterior) and (inferior/superior) axes; variation along the axis corresponds to lateral movements, which are infrequent during speech.222Incidentally, the “normalized” release variant of the mngu0 \acEMA dataset follows this rationale and consists of flattened, \ac2D data, with all coil positions projected onto the mid-sagittal plane. Having said that, the axis can serve to illustrate the physical coil locations on the tongue in the “day 1” recording session; to wit, the tongue tip coil is actually attached out of plane, a few millimeters to one side.

The Euclidean distances during speech are in the millimeter range, indicating that the predictions of \acEMA coil positions are accurate to within the precision of the \acEMA measurements themselves. However, there appears to be a certain amount of fluctuation with a more or less regular range and shape. The peaks of this fluctuation appear to correlate with spikes in the rms channels of the provided \acEMA data, which supports the hypotheses that it is either an artifact of the algorithm which calculates the coil positions and orientations from the raw amplitudes \citepStella2012, or measurement noise in the articulograph itself \citepKroos2012, or, conceivably, a combination of both factors. Of course, the noise in the Euclidean distance analysis is a direct consequence of our decision to refrain from smoothing the provided \acEMA data.333Perhaps the rms jitter in the unsmoothed measurements could also be exploited in adaptive \acEMA denoising.

### Iii-E \acEMA Synthesis

While the combined acoustic and \acEMA synthesis produced satisfactory results, the requirement to train the system on a multimodal dataset such as mngu0 represents a significant drawback; compared to the reasonably wide availability of conventional, acoustic databases designed for speech synthesis, the number of suitable articulatory databases is extremely low. Encouraged by the practical equivalence in the evaluation of the acoustic measures described in Sections III-D and III-C, we therefore considered the question of decoupling the \acEMA synthesis completely from the acoustic data. Accordingly, we used the \acHTS framework to build another \acTTS system trained only on the \acEMA data, without the acoustic parameters.

Under this condition, the evaluation of the duration \acRMSE and Euclidean distances between the predicted and observed \acEMA coils, computed using the formula given by creftype 7, is given in Table IV. As we can see, the results are nearly identical to those in Table III, which confirms the validity of this approach. Fig. 6 visualizes the comparison between the observed and predicted trajectories for one test utterance.

### Iii-F Tongue-only \acEMA Synthesis

In order to focus on the tongue in the following section, we first needed to investigate how far the tongue coil \acEMA positions can be predicted in isolation from the remaining \acEMA coils. To this end, we created a modified version of the \acTTS system described in the previous section, by including only the tongue coils (T1, T2, and T3), and excluding the rest of the \acEMA data from the training set.

Table V gives the evaluation of the \acEMA synthesis restricted to the three tongue coils. Comparing these results with those in Table IV, we observe that the values are virtually identical, which confirms the validity of this approach. As before, the comparison between the observed and predicted trajectories for one test utterance is shown in Fig. 7. It should be noted that despite the removal of the \acEMA coil on the lower incisor, some residual jaw motion is implicitly retained in the movements of the tongue coils.

### Iii-G Model-based Tongue Motion Synthesis

At last, having verified that the \acHTS framework can be used to synthesize audio and predict the movements of three tongue \acEMA coils using separate models trained on the mngu0 database, we prepared a new kind of \acTTS system to predict the shape and motion of the entire tongue surface, by integrating the multilinear model into the process.

To this end, we first estimated the anatomical features (cf. Section II-A) of the speaker in the mngu0 dataset as follows: We used the upper incisor coil as a reference and estimated the correspondences between the three tongue coils and the model vertices, chosen as described in Section II-A. During this correspondence optimization, we used . Thus, we limit the admissible values for each entry of the model parameters to the interval where is the mean and the standard deviation of the corresponding model parameter. By using such a small interval, we try to prevent overfitting during this step. Afterwards, we fitted the model to all \acEMA data frames and stored the obtained parameter values. Here, we used the speaker consistency weight and the pose smoothness weight in the fitting energy. Thus, we demanded very smooth transitions in this case and especially penalized changes of the speaker’s anatomy over time. In this step, we used to give the approach some freedom during the fitting. We then averaged all obtained speaker parameters to get an estimate of the considered speaker’s anatomical features.

Next, we again fitted the model to all \acEMA data frames where, this time, we fixed the speaker parameter to the estimated anatomy. We note that this approach causes the multilinear model to behave like a single-speaker \acPCA model. This time, we used the weight to increase the influence of the data term. However, we decided to use this time to motivate the approach to consider more plausible shapes.

We note that the settings for the fitting were selected manually by an expert. Of course, this selection might be optimized for the used \acEMA dataset by performing a thorough analysis.

The pose parameters resulting from this fitting step were taken as the training data, and we used the \acHTS framework to build a new \acTTS system that predicts the tongue model parameter values directly from the input text.

To evaluate the performance of this system against the reference \acEMA data, we extracted the spatial coordinates of the vertices assigned during the adaptation step (see above) to produce synthetic trajectories that served as a virtual surrogate for predicted \acEMA data.

We evaluated this synthetic \acEMA data against the reference as before; Table VI provides the Euclidean distances between the predicted and observed \acEMA coils, and one test utterance is visualized in Fig. 8. It should be noted that the tongue model itself contains a temporal smoothing term, which ensures that a noisy sequence of input frames does not cause the \ac3D mesh to change shape or position too rapidly; however, this extra smoothing contributes to widespread target undershoot in the comparison. Overall, the results of this evaluation are very promising, and we can confirm that as far as possible, with only three surface points on the tongue, the animation of the full tongue appears to closely match the observed reference.

Finally, in order to compare the three experimental \acTTS systems (trained without acoustic data), we analyzed the distribution of Euclidean distances between each system and the observed reference data over the entire test set; the results are shown in Fig. 9. The distances are slightly greater when the non-tongue \acEMA coils are excluded, and greater still when the \acEMA prediction is replaced by the direct synthesis of tongue model parameters. However, overall, the distances remain in the same range, which indicates that the latter approach perform no worse than synthesis of \acEMA data – while adding the full \ac3D tongue surface into the synthesis process.

## Iv Conclusion

In this study, we have presented a new process of synthesizing acoustic speech and synchronized animation of a full \ac3D surface model of the tongue. We used the \acHTS framework with a single-speaker, multimodal articulatory database containing \acEMA motion capture data. First, we demonstrated a conventional, fused multimodal approach, then separated the two modalities while ensuring that the objective evaluation measures remained comparable. Finally, we adapted a multi-linear statistical model of the tongue and integrated it into the \acTTS process, and evaluated its accuracy by comparing the spatial coordinates of vertices on the model surface to the reference \acEMA data from the original speaker’s tongue movements. The results are very encouraging, and we believe that this will enable multimodal \acTTS applications that provide tongue animation with human-like performance.

It should be noted that the acoustic synthesis and predicted phone durations need not come from the same corpus as the one used for training the tongue model parameter synthesis system. Under certain conditions, it would be straightforward to use a different, conventional \acTTS system with speech recordings from a different speaker in combination with this tongue model parameter synthesis, perhaps adapting it in the speaker subspace automatically or by hand, to generate a multimodal \acTTS application with plausible, speech-synchronized tongue motion, without the requirement of having articulatory data available for the target speaker. In this way, it is possible to first synthesize the acoustic speech signal, and to provide the predicted acoustic durations to guide the synthesis of corresponding tongue model parameters, which are then used to render the animation of the \ac3D tongue model in real time.

However, there is clearly more work to be done, and in future research, we intend to refine and improve our system, and to evaluate it using human subjects who will rate it perceptually. Such a study can include intelligibility, such as the contribution of visible tongue movements during degraded, noisy, or absent audible speech. However, we also plan to assess the impact on perceived naturalness by integrating the tongue model into a realistic talking avatar (e.g., \citepTaylor2012, Schabus2014), and investigating the importance of naturalistic tongue movements for the overall impression of such avatars in multimodal spoken interaction scenarios with artificial characters. This may also lead us to model distinct non-speech poses for the tongue, such as separate “rest” and “ready” positions.

Regarding the tongue model integration, we plan to further investigate such factors as the impact of reducing the dimensionality of the model subspaces on synthesis performance, optimizing the vertex correspondence with \acEMA data, improving the fitting results by adjusting the weights for the smoothness terms, and exploring speaker adaptation using volumetric data, such as the \acMRI subset of the mngu0 corpus \citepSteiner2012.

## Acknowledgments

This work was funded by the German Research Foundation (DFG) under grants EXC 284 and SFB 1102. The authors would like to thank Korin Richmond, Phil Hoole, and Simon King for creating and releasing the mngu0 database. Studies such as the one described in this paper would not be possible without such high-quality, open databases.

\printbibliography\todototoc\listoftodos
You are adding the first comment!
How to quickly get a good reply:
• Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
• Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
• Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters


text:
A homogeneous polynomial of degree kis a homogeneous function of degree k, but there are many homogenous functions that are not polynomials. A function f(x, y) in x and y is said to be a homogeneous function of the degree of each term is p. For example: f(x, y) = (x 2 + y 2 – xy) is a homogeneous function of degree 2 where p = 2. Homogeneous Differential Equations Introduction. A differential equation of the form dy/dx = f (x, y)/ g (x, y) is called homogeneous differential equation if f (x, y) and g(x, y) are homogeneous functions of the same degree in x and y. are all homogeneous functions, of degrees three, two and three respectively (verify this assertion). where $$P\left( {x,y} \right)$$ and $$Q\left( {x,y} \right)$$ are homogeneous functions of the same degree. Observe that any homogeneous function $$f\left( {x,y} \right)$$ of degree n can be equivalently written as follows: ... Let us see some examples of solving homogeneous DEs. For example, x 3+ x2y+ xy2 + y x 2+ y is homogeneous of degree 1, as is p x2 + y2. H can represent translation, rotation, stretching or shrinking (scaling), and perspective transformations, and is of the general form H = ax bx cx px ay by cy py az bz cz pz d1 d2 d3 1 (1.1) Thus, given a vector u, its transformation v is represented by v = H u (1.2) Method of solving first order Homogeneous differential equation Also, to say that gis homoge-neous of degree 0 means g(t~x) = g(~x), but this doesn’t necessarily mean gis (e) If f is a homogenous function of degree k and g is a homogenous func-tion of degree l then f g is homogenous of degree k+l and f g is homogenous of degree k l (prove it). H, a 4x4 matrix, will be used to represent a homogeneous transformation. The first question that comes to our mind is what is a homogeneous equation? If fis linearly homogeneous, then the function deﬁned along any ray from the origin is a linear function. (or) Homogeneous differential can be written as dy/dx = F(y/x). A first order Differential Equation is Homogeneous when it can be in this form: dy dx = F( y x) Differential Equations are equations involving a function and one or more of its derivatives.. For example, the differential equation below involves the function $$y$$ and its first derivative $$\dfrac{dy}{dx}$$. Well, let us start with the basics. (f) If f and g are homogenous functions of same degree k then f + g is homogenous of degree k too (prove it). Example: an equation with the function y and its derivative dy dx. Homogenous Function. 2.5 Homogeneous functions Definition Multivariate functions that are “homogeneous” of some degree are often used in economic theory. We know that the differential equation of the first order and of the first degree can be expressed in the form Mdx + Ndy = 0, where M and N are both functions of x and y or constants. January 19, 2014 5-3 (M y N x) = xN M y N x N = x Now, if the left hand side is a function of xalone, say h(x), we can solve for (x) by (x) = e R h(x)dx; and reverse … • Along any ray from the origin, a homogeneous function deﬁnes a power function. Homogeneous Functions | Equations of Order One If the function f(x, y) remains unchanged after replacing x by kx and y by ky, where k is a constant term, then f(x, y) is called a homogeneous function . Example – 8. Similarly, g(x, y) = (x 3 – 3xy 2 + 3x 2 y + y 3) is a homogeneous function of degree 3 where p = 3. Here we look at a special method for solving "Homogeneous Differential Equations" Homogeneous Differential Equations. Homogeneous Differential Equations. Definition of Homogeneous Function A function $$P\left( {x,y} \right)$$ is called a homogeneous function of the degree $$n$$ if the following relationship is valid for all $$t \gt 0:$$

Google Speech-to Text Streaming, Thomas' Cinnamon Raisin Bagel Nutrition Facts, Estates At Bellaire, How To Solve Substitution Cipher Python, Grand I10 Nios Engine Cc, Psalm 5:3 Nlt, Ff3 Best Jobs, Plum Fruit In Urdu, Pc Lights On But No Fans, Rufus Is A Good Boy,

format: plain


text:
# Reconstructing Graphs from Degree Distribution

Given a degree distribution, how fast can we construct a graph that follows the given degree distribution? A link or algorithm sketch would be good. The algorithm should report a "no" incase no graph can be constructed and any one example if multiple graphs can be constructed.

• Welcome! How is the "degree distribution" given? As stochastic distribution, as list of degrees, ...?
– Raphael
May 17 '12 at 13:00
• See Exercise 2.6 here. An algorithm for creating a graph from a given degree-sequence is given. May 17 '12 at 13:41
• To clarify Raphael's comment, when I read degree distribution, I think of a probabilistic distribution on degrees. As utdiscant mentions, degree sequence is probably what you want. If you mean the probabilistic sense, you're probably looking for some randomized construction algorithm that tries to "approximate" the distribution. It doesn't make a lot of sense to me to "report a no" in this setting, though, because I think most graphs will be some sort of outlier? May 17 '12 at 15:03
• And if you actually want to generate a graph with a given degree distribution, then this paper seems to have the trick. It seems the algorithm described in my previous comment, is actually the Havel-Hakimi algorithm in the answer by Wu Yin. May 17 '12 at 15:52

If you mean how to construct such a simple graph (no self loops and no parallel edges), maybe Havel-Hakimi theorem is what you are looking for. You can google it yourself, and the wikipedia page Degree (graph theory) is also helpful.

• thanks. yes wiki page is helpful in this case.. May 18 '12 at 5:26

If the degree distribution is given as a list of degree, then you can do the following, given $n$ nodes with degrees $d_1, ... ,d_n$:

Create a complete graph $K_n$ on $n$-vertices. For each vertex $v_i$ in $K_n$, split it into $d_i$ copies. Split here means, create a number of copies with edges to every vertex $v_i$ has an edge to, but no edges to other copies of $v_i$. If $d_i = 0$ then simply remove the vertex. In the new graph, call these vertices $v_{ij}$ for $1 \leq j \leq d_i$.

Once you are done, you have a very dense graph on $N = d_1 + ... + d_n$ vertices; call this graph $H$. Pick your favorite algorithm for maximum matching (since the graph is so dense, you should probably use one of the fast matrix-multiplication based algorithms) and run it on $H$. This will return a matching $M$. If the matching is not perfect (i.e. if it does not cover every vertexes) then your degree distribution was impossible; so return no.

If you have a perfect matching $M$, then remove all edges not in $M$ from $H$, and then for every $1 \leq i \leq n$ merge the $d_i$ many vertices $v_{i1}, ... , v_{id_i}$ into one vertex $u_i$. Merging two vertices means combining them into one, such that the resulting vertex has edges to every vertex at least one of the original had an edge to. Call the resulting graph $G$; it has the desired degree distribution.

The resulting runtime is $O(N^\omega)$ where $\omega$ is the constant for the fastest matrix-multiplication algorithm (which at the time of writing is about $2.373$). In terms of number of vertices in the resulting graph, in the worst case of degree distribution being dense, we have $O(n^{2\omega})$.

• From your (quite clear) explanation it is not at all clear why matrix multiplication enters the equation.
– Raphael
May 17 '12 at 16:24
• @Raphael matrix multiplication is one of the ways to solve maximum matching and it is the preferred method for dense graphs, since the best version of Edmonds's matching algorithm runs in $O(\sqrt{|V|}|E|)$ which would give $O(N^{2.5})$ for this problem, since $H$ is pretty dense. So if you have access to a good fast matrix multiplication (or working with a matrix-oriented language like Matlab) I would use Mucha and Sankowski's matrix approach to matching. May 17 '12 at 16:29


text:
An official website of the United States government

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

# Multi-Purpose Test Station - PHADES

Credit: Brian Maranville

PHADES (Polarized 3He and Detector Experiment Station) is typically used for the testing of 3He and related optics, detector optimization, and crystal alignment. A pyrolytic graphite monochromator set at a take-off angle of 75 degrees produces a monochromatic beam of 4.1001(5) $${\unicode{x212B}}$$ with fixed energy of 4.8662(6) meV. Flux is relatively low at 2 x 106 $$\frac{n}{cm^2s}$$. The vertical divergence of the neutron beam is also small at about 1 degree. The instrument is simply designed, yet highly versatile. It's primary uses are crystal alignment, 3He neutron spin filter testing/development, and detector testing/development.

A series of collimators are available for use. Located at the beam exit, a 25 cm by 75 cm optical table of variable height may be used to support loads up to 100 kg. A second pyrolytic graphite crystal is available for use as an analyzer. Optional removal of the optical table combined with translation of the secondary spectrometer of over 1 m allows large sample environment equipment to be accommodated. Samples are typically mounted on a standard Huber goniometer with ± 22 degree tilt and ± 10 mm translation. Beam to sample heights are 6 inches (152 mm) and 2.75 inches (70 mm) from the Huber base.

A pencil detector (A4) travels from -7 degrees to +150 degrees, providing a maximum Q of 2.96 $${\unicode{x212B}}$$-1 and corresponding minimum sample reflection d-spacing of 2.123 $${\unicode{x212B}}$$. Water and air lines are available to support sample environment equipment including electromagnets. Software for hkl crystal alignment is available.

To request time on PHADES please submit a short beam time request through the IMS System. Any questions may be sent to the instrument contacts, listed to the right.

Created February 10, 2020, Updated February 11, 2020


text:
# Step-by-step Solution

## Integral of sin()^2

Go
1
2
3
4
5
6
7
8
9
0
x
y
(◻)
◻/◻
2

e
π
ln
log
lim
d/dx
d/dx
>
<
>=
<=
sin
cos
tan
cot
sec
csc

asin
acos
atan
acot
asec
acsc

sinh
cosh
tanh
coth
sech
csch

asinh
acosh
atanh
acoth
asech
acsch

### Videos

$\sin\left(\right)^2x+C_0$

## Step-by-step explanation

Problem to solve:

$\int\sin^2dx$
1

The integral of a constant is equal to the constant times the integral's variable

$\sin\left(\right)^2x$
2

As the integral that we are solving is an indefinite integral, when we finish we must add the constant of integration

$\sin\left(\right)^2x+C_0$

$\sin\left(\right)^2x+C_0$
$\int\sin^2dx$

### Main topic:

Integration by substitution

~ 0.94 seconds


text:
# Simple proof of König's theorem

I have been trying to understand the proof of the König's following theorem on bipartite graphs: "The maximum cardinality of a matching in a graph $$G$$ is equal to the minimum cardinality of a vertex cover.". While working on it, I have found a simpler interpretation of the proof than the one given in Graph Theory book by Reinhard Diestel. Is the proof correct, or am I missing something here?

We can form the minimum vertex cover candidate by selecting the elements of the matchings that are connected to an unmatched element if there is one in the matching, and by selecting the element from the left hand side (or the right hand side, but consistently) otherwise. The selection made in the example graph can be shown to be the minimum vertex since:

1. All the edges in the matching are covered by it since we have constructed this using one element of every matching,
2. All the edges that are connected to the unmatched elements are covered since we have selected the minimum vertex cover according to that. The critical point to emphasize here is that only a single element of a pair can be connected to a one (or many) unmatched element(s), because otherwise we would have an augmenting path that could have been used to form an additional matching which would violate the idea of the maximum matching.

For the K3-3 graph, the algorithm randomly picks a side to select the vertices from the maximum matching to form the minimum vertex cover. This is due to the fact that all elements from both sides are matched. I believe the size of the maximum matching is 3 for K3-3, and it can be seen that minimum vertex cover is of size 3 too.

• There's something I'm not quite clear about in your proof. Would you mind working out another example? Could you take a maximum matching in $K_{3,3}$ and show how your procedure leads to a minimum vertex cover?
– bof
Dec 4 '20 at 15:34
• @bof I have shown the K3,3 case, for which I needed to clarify the text a bit. I have my doubts regarding this proof too, so I would really appreciate if you could show me where I am wrong. Thank you for the ideas! Dec 5 '20 at 9:03
• @AmitRajaraman Yes, that was a mistake, I have fixed it. Thanks for pointing it out... Dec 5 '20 at 9:04


text:
# Can one reduce a problem of unknown complexity to a hard problem to show hardness?

In this paper (page 3 Theorem 1) the authors want to prove that their problem is NP-complete. Their method is as follows. Let their problem be known as $P$. They show that their problem can be written as a $0\text{-}1$ integer program. Then they claim that $0\text{-}1$ integer programs are NP-complete and therefore their problem $P$ is NP-complete.

I find this proof hard to believe. For the problem $P$ to be NP-hard I think one has to reduce the $0\text{-}1$ integer program into an instance of problem $P$ and not the other way around.

Please can someone explain if this proof in the paper is acceptable?

• The proof shows, in a roundabout way, that the problem is in NP, not that the problem is NP hard.
– Joe
Jan 16 '14 at 20:37

You should not believe such a proof (still a claim might be true even if the proof is not). You can also see this by suddenly being able to "prove" silly things. The 2-SAT problem is a special case of the $k$-SAT problem, where each clause has exactly two literals. There are plenty of algorithms that solve 2-SAT in polynomial time, thus 2-SAT is in P. On the other hand, 3-SAT is already NP-complete.

Consider the following "theorem" and "proof".

"Theorem:" 2-SAT is NP-complete. "Proof:" Given an instance of 2-SAT, duplicate a literal in every clause to produce an instance of 3-SAT. Because 3-SAT is NP-complete, we conclude that so is 2-SAT.

We of course know this is nonsense.

• That paper is being cited by many others. So I didn't want to assume it was incorrect just by my instinct. I hope IEEE will correct this.
– Mat
Jan 7 '14 at 13:59
• @triomphe If you think you've discovered an error in a paper, you might want to check out this question on Academia.SE.
– Juho
Jan 7 '14 at 14:12

The proof is not correct. As you've indicated the reduction needs to go in the other direction, i.e. integer programming must be reduced to their problem, not the other way around.


text:
# Gamma correction

Example of CRT gamma correction. The dotted line indicates a linear transfer function (γ=1); the solid line shows how a typical CRT behaves; the dashed line represents the inverse function.
The effect of gamma correction on an image: The original image was taken to varying powers, showing that powers larger than one make the shadows darker, while powers smaller than one make dark regions lighter.

Gamma correction, gamma nonlinearity, gamma encoding, or often simply gamma, is the name of a nonlinear operation used to code and decode luminance or tristimulus values in video or still image systems.[1] Gamma correction is, in the simplest cases, defined by the following power-law expression:

$V_{\text{out}} = A {V_{\text{in}}}^{\gamma}$

where A is a constant and the input and output values are non-negative real values; in the common case of A = 1, inputs and outputs are typically in the range 0–1. A gamma value γ < 1 is sometimes called an encoding gamma, and the process of encoding with this compressive power-law nonlinearity is called gamma compression; conversely a gamma value γ > 1 is called a decoding gamma and the application of the expansive power-law nonlinearity is called gamma expansion.

## Explanation

Gamma encoding of images is required to compensate for properties of human vision, hence to maximize the use of the bits or bandwidth relative to how humans perceive light and color.[1] Human vision, under common illumination conditions (not pitch black nor blindingly bright), follows an approximate gamma or power function. If images are not gamma encoded, they allocate too many bits or too much bandwidth to highlights that humans cannot differentiate, and too few bits/bandwidth to shadow values that humans are sensitive to and would require more bits/bandwidth to maintain the same visual quality.[1][2] Gamma encoding of floating point images is not required (and may be counterproductive), because the floating-point format already provides a piecewise linear approximation of a logarithmic curve.[3]

Although gamma encoding was developed originally to compensate for the input–output characteristic of cathode ray tube (CRT) displays, that is not its main purpose or advantage in modern systems. In CRT displays, the light intensity varies nonlinearly with the electron-gun voltage. Altering the input signal by gamma compression can cancel this nonlinearity, such that the output picture has the intended luminance. However, the gamma characteristics of the display device do not play a factor in the gamma encoding of images and video – they need gamma encoding to maximize the visual quality of the signal, regardless of the gamma characteristics of the display device.[1][2] The similarity of CRT physics to the inverse of gamma encoding needed for video transmission was a combination of luck and engineering, which simplified the electronics in early television sets.[4]

## Generalized gamma

The concept of gamma can be applied to any nonlinear relationship. For the power-law relationship Vout = Vinγ, the curve on a log–log plot is a straight line, with slope everywhere equal to gamma (slope is represented here by the derivative operator):

$\gamma = \frac{\mathrm{d} \log(V_{\text{out}})}{\mathrm{d} \log(V_{\text{in}})}$

That is, gamma can be visualised as the slope of the input–output curve when plotted on logarithmic axes. For a power-law curve, this slope is constant, but the idea can be extended to any type of curve, in which case gamma (strictly speaking, "point gamma"[5]) is defined as the slope of the curve in any particular region.

## In photography

When a photographic film is exposed to light, the result of the exposure can be represented on a graph showing log of exposure on the horizontal axis, and density, or log of transmittance, on the vertical axis. Such curves are known as Hurter–Driffield curves. Since both axes use logarithmic units, the slope of the linear section of the curve is called the gamma of the film. Negative film typically has a gamma less than 1; positive film (slide film, reversal film) typically has a gamma greater than 1.

Photographic film has a much greater ability to record fine differences in shade than can be reproduced on photographic paper. Similarly, most video screens are not capable of displaying the range of brightnesses (dynamic range) that can be captured by typical electronic cameras.[6] For this reason, considerable artistic effort is invested in choosing the reduced form in which the original image should be presented. The gamma correction, or contrast selection, is part of the photographic repertoire used to adjust the reproduced image.

Analogously, digital cameras record light using electronic sensors that usually respond linearly. In the process of rendering linear raw data to conventional RGB data (e.g. for storage into JPEG image format), color space transformations and rendering transformations will be performed. In particular, almost all standard RGB color spaces and file formats use a non-linear encoding (a gamma compression) of the intended intensities of the primary colors of the photographic reproduction; in addition, the intended reproduction is almost always nonlinearly related to the measured scene intensities, via a tone reproduction nonlinearity.

## Windows, Mac, sRGB and TV/video standard gammas

Plot of the sRGB standard gamma-expansion nonlinearity (red), and its local gamma value, slope in log–log space (blue).

In most computer systems, images are encoded with a gamma of about 0.45 and decoded with a gamma of 2.2; a notable exception, until the release of Mac OS X 10.6 (Snow Leopard) in September 2009, were Macintosh computers: they used 0.55 and 1.8 respectively. In any case, binary data in still image files (as JPEG) are explicitly encoded (that is, they carry gamma-encoded values, not linear intensities), as are motion picture files (such as MPEG). The system can optionally further manage both cases, through color management, if a better match to the output device gamma is required.

The sRGB color space standard used with most cameras, PCs, and printers does not use a simple power-law nonlinearity as above, but has a decoding gamma value near 2.2 over much of its range, as shown in the plot to the right. Below a compressed value of 0.04045 or a linear intensity of 0.00313, the curve is linear (encoded value proportional to intensity), so γ = 1. The dashed black curve behind the red curve is a standard γ = 2.2 power-law curve, for comparison.

Output to CRT-based television receivers and monitors does not usually require further gamma correction, since the standard video signals that are transmitted or stored in image files incorporate gamma compression that provides a pleasant image after the gamma expansion of the CRT (it is not the exact inverse). For television signals, the actual gamma values are defined by the video standards (NTSC, PAL or SECAM), and are always fixed and well known values.

## Power law for video display

A gamma characteristic is a power-law relationship that approximates the relationship between the encoded luma in a television system and the actual desired image luminance.

With this nonlinear relationship, equal steps in encoded luminance correspond roughly to subjectively equal steps in brightness. Ebner and Fairchild[7] used an exponent of 0.43 to convert linear intensity into lightness for neutrals; the reciprocal, approximately 2.33 (quite close to the 2.2 figure cited for a typical display subsystem), would provide optimal perceptual encoding of grays. The following illustration shows the difference between a scale with linearly-increasing encoded luminance signal (linear input) and a scale with linearly-increasing intensity (i.e., gamma-corrected) scale (linear output).

 Linear encoding VS = 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Linear intensity I = 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

On most displays (those with gamma of about 2.2), one can observe that the linear-intensity scale has a large jump in perceived brightness between the intensity values 0.0 and 0.1, while the steps at the higher end of the scale are hardly perceptible. The linearly-encoded scale, which has a nonlinearly-increasing intensity, will show much more even steps in perceived brightness.

A cathode ray tube (CRT), for example, converts a video signal to light in a nonlinear way, because the electron gun's intensity (brightness) as a function of applied video voltage is nonlinear. The light intensity I is related to the source voltage VS according to

$I \propto V_{\rm S}{}^{\gamma}$

where γ is the Greek letter gamma. For a CRT, the gamma that relates brightness to voltage is usually in the range 2.35 to 2.55; video look-up tables in computers usually adjust the system gamma to the range 1.8 to 2.2,[1] which is in the region that makes a uniform encoding difference give approximately uniform perceptual brightness difference, as illustrated in the diagram at the top of this section.

For simplicity, consider the example of a monochrome CRT. In this case, when a video signal of 0.5 (representing mid-gray) is fed to the display, the intensity or brightness is about 0.22 (resulting in a dark gray). Pure black (0.0) and pure white (1.0) are the only shades that are unaffected by gamma.

To compensate for this effect, the inverse transfer function (gamma correction) is sometimes applied to the video signal so that the end-to-end response is linear. In other words, the transmitted signal is deliberately distorted so that, after it has been distorted again by the display device, the viewer sees the correct brightness. The inverse of the function above is:

$V_{\rm C} \propto V_{\rm S}{}^{(1/\gamma)}$

where VC is the corrected voltage and VS is the source voltage, for example from an image sensor that converts photocharge linearly to a voltage. In our CRT example 1/γ is 1/2.2 or 0.45.

A color CRT receives three video signals (red, green and blue) and in general each color has its own value of gamma, denoted γR, γG or γB. However, in simple display systems, a single value of γ is used for all three colors.

Other display devices have different values of gamma: for example, a Game Boy Advance display has a gamma between 3 and 4 depending on lighting conditions. In LCDs such as those on laptop computers, the relation between the signal voltage VS and the intensity I is very nonlinear and cannot be described with gamma value. However, such displays apply a correction onto the signal voltage in order to approximately get a standard γ = 2.5 behavior. In NTSC television recording, γ = 2.2.

The power-law function, or its inverse, has a slope of infinity at zero. This leads to problems in converting from and to a gamma colorspace. For this reason most formally defined colorspaces such as sRGB will define a straight-line segment near zero and add raising x + K (where K is a constant) to a power so the curve has continuous slope. This straight line does not represent what the CRT does, but does make the rest of the curve more closely match the effect of ambient light on the CRT. In such expressions the exponent is not the gamma; for instance, the sRGB function uses a power of 2.4 in it, but more closely resembles a power-law function with an exponent of 2.2, without a linear portion.

## Methods to perform display gamma correction in computing

Up to four elements can be manipulated in order to achieve gamma encoding to correct the image to be shown on a typical 2.2- or 1.8-gamma computer display:

• The pixel's intensity values in a given image file; that is, the binary pixel values are stored in the file in such way that they represent the light intensity via gamma-compressed values instead of a linear encoding. This is done systematically with digital video files (as those in a DVD movie), in order to minimize the gamma-decoding step while playing, and maximize image quality for the given storage. Similarly, pixel values in standard image file formats are usually gamma-compensated, either for sRGB gamma (or equivalent, an approximation of typical of legacy monitor gammas), or according to some gamma specified by metadata such as an ICC profile. If the encoding gamma does not match the reproduction system's gamma, further correction may be done, either on display or to create a modified image file with a different profile.
• The rendering software writes gamma-encoded pixel binary values directly to the video memory (when highcolor/truecolor modes are used) or in the CLUT hardware registers (when indexed color modes are used) of the display adapter. They drive Digital-to-Analog Converters (DAC) which output the proportional voltages to the display. For example, when using 24-bit RGB color (8 bits per channel), writing a value of 128 (rounded midpoint of the 0–255 byte range) in video memory it outputs the proportional ≈ 0.5 voltage to the display, which it is shown darker due to the monitor behavior. Alternatively, to achieve ≈ 50% intensity, a gamma-encoded look-up table can be applied to write a value near to 187 instead of 128 by the rendering software.
• Modern display adapters have dedicated calibrating CLUTs, which can be loaded once with the appropriate gamma-correction look-up table in order to modify the encoded signals digitally before the DACs that output voltages to the monitor.[8] Setting up these tables to be correct is called hardware calibration.[9]
• Some modern monitors allow to the user to manipulate their gamma behavior (as if it were merely another brightness/contrast-like setting), encoding the input signals by themselves before they are displayed on screen. This is also a calibration by hardware technique but it is performed on the analog electric signals instead of remapping the digital values, as in the previous cases.

In a correctly calibrated system, each component will have a specified gamma for its input and/or output encodings.[9] Stages may change the gamma to correct for different requirements, and finally the output device will do gamma decoding or correction as needed, to get to a linear intensity domain. All the encoding and correction methods can be arbitrarily superimposed, without mutual knowledge of this fact among the different elements; if done incorrectly, these conversions can lead to highly distorted results, but if done correctly as dictated by standards and conventions will lead to a properly functioning system.

In a typical system, for example from camera through JPEG file to display, the role of gamma correction will involve several cooperating parts. The camera encodes its rendered image into the JPEG file using one of the standard gamma values such as 2.2, for storage and transmission. The display computer may use a color management engine to convert to a different color space (such as older Macintosh's γ = 1.8 color space) before putting pixel values into its video memory. The monitor may do its own gamma correction to match the CRT gamma to that used by the video system. Coordinating the components via standard interfaces with default standard gamma values makes it possible to get such system properly configured.

## Simple monitor tests

To see whether one's computer monitor is properly hardware adjusted and can display shadow detail in sRGB images properly, they should see the left half of the circle in the large black square very faintly (or not at all), but the right half should be clearly visible. If not, one can adjust their monitor's contrast and/or brightness setting. This alters the monitor's perceived gamma. The image is best viewed against a black background.

This procedure is not suitable for calibrating or print-proofing a monitor. It can be useful for making a monitor display sRGB images approximately correctly, on systems in which profiles are not used (for example, the Firefox browser prior to version 3.0 and many others) or in systems that assume untagged source images are in the sRGB colorspace.

On some operating systems running the X Window System, one can set the gamma correction factor (applied to the existing gamma value) by issuing the command xgamma -gamma 0.9 for setting gamma correction factor to 0.9, and xgamma for querying current value of that factor (the default is 1.0). In Macintosh systems, the gamma and other related screen calibrations are made through the OS control panel. Microsoft Windows versions before Windows Vista lack a first-party developed calibration tool.

In the test pattern to the right, the linear intensity of each solid bar is the average of the linear intensities in the surrounding striped dither; therefore, ideally, the solid squares and the dithers should appear equally bright in a properly adjusted sRGB system.

## Terminology

The term intensity refers strictly to the amount of light that is emitted per unit of time and per unit of surface, in units of lux. Note, however, that in many fields of science this quantity is called luminous emittance, as opposed to luminous intensity, which is a different quantity. These distinctions, however, are largely irrelevant to gamma compression, which is applicable to any sort of normalized linear intensity-like scale.

Luminance can mean several things even within the context of video and imaging:

• Luminance is the photometric brightness of an object, taking into account the wavelength-dependent sensitivity of the human eye (in units of cd/m²);
• Luminance (video) is the encoded video "luma" signal, i.e. similar to the signal voltage VS.
• Luminance (relative) is the luminance signal used in a color-space encoding, relative to a white level.

One contrasts "luminance" in the sense of color (no gamma compression) with "luma" in the sense of video (with gamma compression), and denote luminance by Y and luma by Y′, the prime symbol (′) denoting gamma compression.[10] Note that luma is not directly calculated from luminance, it is the (somewhat arbitrary) weighted sum of gamma compressed RGB components.[11]

Likewise, brightness is sometimes applied to various measures, including light levels, though it more properly applies to a subjective visual attribute.

Gamma correction is a type of power law function whose exponent is the Greek letter gamma (γ). It should not be confused with the mathematical Gamma function. The lower case gamma, γ, is a parameter of the former; the upper case letter, Γ, is the name of (and symbol used for) the latter (as in Γ(x)). To use the word "function" in conjunction with gamma correction, one may avoid confusion by saying "generalized power law function."

Without context, a value labeled gamma might be either the encoding or the decoding value. Caution must be taken to correctly interpret the value as that to be applied-to-compensate or to be compensated-by-applying its inverse. In common parlance, in many occasions the decoding value (as 2.2) is employed as if it were the encoding value, instead of its inverse (1/2.2 in this case), which is the real value that must be applied to encode gamma.

## References

1. Charles A. Poynton (2003). Digital Video and HDTV: Algorithms and Interfaces. Morgan Kaufmann. pp. 260, 630. ISBN 1-55860-792-7.
2. ^ a b Charles Poynton (2010). Frequently Questioned Answers about Gamma.
3. ^ Erik Reinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward, and Karol Myszkowski (2010). High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting. Morgan Kaufmann. p. 82. ISBN 9780080957111.
4. ^ McKesson, Jason L. "Linearity and Gamma – Chapter 12. Dynamic Range". Learning Modern 3D Graphics Programming. Retrieved 11 July 2013.
5. ^ R.W.G. Hunt, The Reproduction of Colour, 6th Ed, p48
6. ^ Peter Hodges (2004). An introduction to video and audio measurement (3rd ed.). Elsevier. p. 174. ISBN 978-0-240-80621-1.
7. ^ Fritz Ebner and Mark D Fairchild, "Development and testing of a color space (IPT) with improved hue uniformity," Proceedings of IS&T/SID's Sixth Color Imaging Conference, p 8-13 (1998).
8. ^ SetDeviceGammaRamp, the Win32 API to download arbitrary gamma ramps to display hardware
9. ^ a b Jonathan Sachs (2003). Color Management. Digital Light & Color.
10. ^ Engineering Guideline EG 28, "Annotated Glossary of Essential Terms for Electronic Production," SMPTE, 1993.
11. ^ Charles Poynton, "Digital video and HDTV algorithms and interfaces", 2003.


text:
# DL_DetectAnomalies2_Deploy

Loads a deep learning model and prepares its execution on a specific target device.

### Syntax

void avl::DL_DetectAnomalies2_Deploy
(
const avl::DetectAnomalies2ModelDirectory& inModelDirectory,
const atl::Optional<avl::DeviceKind::Type>& inDeviceType,
const int inDeviceIndex,
avl::DetectAnomalies2ModelId& outModelId
)

### Parameters

Name Type Range Default Description
inModelDirectory const DetectAnomalies2ModelDirectory& A Detect Anomalies 2 model stored in a specific disk directory.
inDeviceType const Optional<DeviceKind::Type>& NIL A type of a device selected for deploying and executing the model. If not set, device depending on version (CPU/GPU) of installed Deep Learning add-on is selected. If not set, device depending on version (CPU/GPU) of installed Deep Learning add-on is selected.
inDeviceIndex const int 0 - 0 An index of a device selected for deploying and executing the model.
outModelId DetectAnomalies2ModelId& Identifier of the deployed model

### Hints

• In most cases, this filter should be placed in the INITIALIZE section.
• Executing this filter may take several seconds.
• This filter should be connected to DL_DetectAnomalies2 through the ModelId ports.
• You can edit the model directly through the inModelDirectory. Another option is to use the Deep Learning Editor application and just copy the path to the created model.

### Remarks

• Passing NIL as inTargetDevice (which is default), is identical to passing DeviceKind::CUDA on GPU version of Deep Learning add-on and DeviceKind::CPU on CPU version on Deep Learning add-on.
• GPU version of Deep Learning add-on supports DeviceKind::CUDA and DeviceKind::CPU as inTargetDevice value.
• CPU version of Deep Learning add-on supports only DeviceKind::CPU as inTargetDevice value.


text:
Here is a simple but surprising fact.

Recall that the Hilbert transform $Hf(x) := p.v. \int f(x-t) \frac{dt}{t}$ is $L^p \to L^p$ bounded for all ${1 < p < \infty}$ (and even $L^1 \to L^{1,\infty}$ bounded, of course). The quadratically modulated Hilbert transform is the operator

$\displaystyle H_q f(x) := p.v. \int \int f(x-t) e^{-i t^2} \frac{dt}{t};$

this operator is also known to be $L^p \to L^p$ bounded for all ${1 < p < \infty}$, but the proof is no corollary of that for $H$, it's a different beast requiring oscillatory integral techniques and almost orthogonality and is due to Ricci and Stein (interestingly though, $H_q$ is also $L^1 \to L^{1,\infty}$ bounded, and this can indeed be obtained by a clever adaptation of Calderón-Zygmund theory due to Chanillo and Christ).

The bilinear Hilbert transform instead is the operator

$\displaystyle BHT(f,g)(x) := p.v. \int \int f(x-t)g(x+t)\frac{dt}{t}$

and it is known, thanks to foundational work of Lacey and Thiele, to be $L^p \times L^q \to L^r$ bounded at least in the range given by $p,q>1, r>2/3$ with the exponents satisfying the Hölder condition $1/p + 1/q = 1/r$ (this condition is strictly necessary to have boundedness, due to the scaling invariance of the operator). This operator has an interesting modulation invariance (corresponding to the fact that its bilinear multiplier is $\mathrm{sgn}(\xi - \eta)$, which is invariant with respect to translations along the diagonal): indeed, if $\mathrm{Mod}_{\theta}$ denotes the modulation operator $\mathrm{Mod}_{\theta} f(x) := e^{- i \theta x} f(x)$, we have

$\displaystyle BHT(\mathrm{Mod}_{\theta}f,\mathrm{Mod}_{\theta}g) = \mathrm{Mod}_{2 \theta} BHT(f,g);$

it is this fact that suggests one should use time-frequency analysis to deal with this operator.
Now, analogously to the linear case, one can consider the quadratically modulated bilinear Hilbert transform, given simply by

$\displaystyle BHT_q(f,g)(x) := p.v. \int \int f(x-t)g(x+t) e^{-i t^2} \frac{dt}{t}.$

One might be tempted to think, by analogy, that this operator is harder to bound than $BHT$ – at least, I would naively think so at first sight. However, due to the particular structure of the bilinear Hilbert transform, the boundedness of $BHT_q$ is a trivial corollary of that of $BHT$! Indeed, this is due to the trivial polynomial identity

$\displaystyle (x+t)^2 + (x-t)^2 = 2x^2 + 2t^2;$

thus if $\mathrm{QMod}_{\theta}$ denotes the quadratic modulation operator $\mathrm{QMod}_{\theta}f(x) = e^{-i \theta x^2} f(x)$ we have

\displaystyle \begin{aligned} BHT_q(f,g)(x) = & \int f(x-t)g(x+t) e^{-it^2} \frac{dt}{t} \\ = & \int f(x-t)g(x+t) e^{ix^2}e^{-i(x+t)^2/2}e^{-i(x-t)^2/2} \frac{dt}{t} \\ = & e^{ix^2}\int e^{-i(x-t)^2/2}f(x-t)e^{-i(x+t)^2/2}g(x+t) \frac{dt}{t} \\ = & \big[ \mathrm{QMod}_{-1} BHT( \mathrm{QMod}_{1/2} f, \mathrm{QMod}_{1/2} g )\big](x). \end{aligned}

Of course this trick is limited to quadratic modulations, so for example already the cubic modulation of $BHT$

is non-trivial to bound (but the boundedness of the cubic modulation of the trilinear Hilbert transform would again be a trivial consequence of the boundedness of the trilinear Hilbert transform itself… too bad we don’t know if it is bounded at all!). Polynomial modulations of bilinear singular integrals (thus a bilinear analogue of the Ricci-Stein work) have been shown to be bounded by Christ, Li, Tao and Thiele in “On multilinear oscillatory integrals, nonsingular and singular“.

UPDATE: Interesting synchronicity just happened: today Dong, Maldague and Villano have uploaded on ArXiv their paper “Special cases of power decay in multilinear oscillatory integrals” in which they extend the work of Christ, Li, Tao and Thiele to some special cases that were left out. Maybe I should check my email for the arXiv digest before posting next time.