A uniform phase representation for the harmonic model in speech synthesis applications

Abstract

Feature-based vocoders, e.g., STRAIGHT, offer a way to manipulate the perceived characteristics of the speech signal in speech transformation and synthesis. For the harmonic model, which provide excellent perceived quality, features for the amplitude parameters already exist (e.g., Line Spectral Frequencies (LSF), Mel-Frequency Cepstral Coefficients (MFCC)). However, because of the wrapping of the phase parameters, phase features are more difficult to design. To randomize the phase of the harmonic model during synthesis, a voicing feature is commonly used, which distinguishes voiced and unvoiced segments. However, voice production allows smooth transitions between voiced/unvoiced states which makes voicing segmentation sometimes tricky to estimate. In this article, two-phase features are suggested to represent the phase of the harmonic model in a uniform way, without voicing decision. The synthesis quality of the resulting vocoder has been evaluated, using subjective listening tests, in the context of resynthesis, pitch scaling, and Hidden Markov Model (HMM)-based synthesis. The experiments show that the suggested signal model is comparable to STRAIGHT or even better in some scenarios. They also reveal some limitations of the harmonic framework itself in the case of high fundamental frequencies.

1Introduction

Parametric speech signal representations are necessary in almost every field of speech technologies: speech and speaker recognition [1],[2], speech and speaker transformation [3], synthesis [4], diarization [5], etc. Each of these fields, however, requires a particular type of parametrization scheme. Thus, while low-dimensional filter bank based Mel-Frequency Cepstral Coefficients (MFCC) [6] are sufficiently accurate for recognition purposes, they are not suitable for speech reconstruction by themselves. Indeed, applications involving spoken outputs, such as speech coding [7], require the speech signals to be represented by a set of features yielding almost transparent analysis/resynthesis. Voice transformation and speech synthesis impose even stricter requirements, since the parametric speech representations they deal with must provide a solid and flexible framework to sculpt all the characteristics of the speech sounds through direct manipulation of the features (see, for instance, [8]-[10]). Interestingly, recent statistical trends are also encouraging research on parametric speech representations with a constant number of parameters and with good mathematical properties [4].

Sinusoidal models represent the speech signal by means of a sum of sinusoids given by their instantaneous frequency, amplitude, and phase [11]. These models have been widely used for speech analysis, resynthesis, and modification [12]-[14]. Sinusoidal models have evolved over the years [3],[15], and recently, the so-called adaptive Harmonic Model (aHM) [16] has also been shown to yield practically transparent analysis/resynthesis and excellent modification performance [17],[18]. Despite the inherent assumption that speech can be represented only by harmonic sinusoidal components, even in unvoiced segments, aHM succeeds at capturing the relevant spectral information and noisy nature of a speech signal and thus, representing the speech signal in a uniform way, without using any voicing decision. As long as the perceptual information carried in the phase is preserved, the uniform way of describing and manipulating signals is a remarkable practical advantage of aHM with respect to alternative models involving an explicit separation between harmonics and noise [19] for two main reasons: (i) locating the voicing boundaries is an error-prone process and (ii) in voice transformation, such a separation implies the need for two independent modification procedures, one for each component [3],[20], which increases the risk that listeners perceive them as two independent output signals.

The features handled by aHM are not directly compatible with methods involving statistical modeling because the amplitude and phase parameters lie on the harmonic grid which is dependent on the fundamental frequency f₀[21]. To avoid this issue, amplitude and phase information have to be isolated from f₀ and translated into independent parameters. However, while amplitude envelopes are relatively easy to obtain through interpolation between sinusoidal amplitudes [22],[23], the representation of phase remains an open problem. Recent attempts of obtaining a consistent phase envelope [24]-[27] provide features which are theoretically valid in voiced time-frequency regions but are not informative in unvoiced ones. Thus, standard speech parametrization systems used in statistical frameworks tend to discard the phase information. Instead, they rely on a minimum-phase component derived from the amplitude envelope, along with complementary parameters related to the degree of harmonicity in different time-frequency regions, such as band aperiodicities [28] or maximum voiced frequency [29].

This article presents a novel phase representation that has been designed to handle, in a uniform manner across time, all the relevant information conveyed by the phase parameters of a full-band aHM model, namely the maximum-phase component and the noisiness. This is done through the following steps: first, aHM analysis [16] is performed to obtain the instantaneous phase from the waveform; then, the minimum-phase term is subtracted from the measured phases, and the local Phase Distortion (PD) [25] is calculated; finally, the short-time mean and standard deviation of the PD are computed in the neighborhood of each frame, the former being highly correlated to the maximum-phase component, and the latter to the degree of noisiness. Among the advantages of this novel approach, we can mention the following: (i) it is valid to analyze signals exhibiting harmonic and noise components that overlap both in time and in frequency and thus, avoiding binary voiced/unvoiced decisions which are error-prone and result in annoying artifacts, especially in synthesis [30],[31]. (ii) Since it helps avoiding an explicit separation between harmonics and noise, it provides a solid and uniform framework for speech manipulation thus, avoiding artifacts near the voicing boundaries [21]. (iii) It can be easily made compatible with statistical frameworks. Moreover, given the continuous nature of the feature streams, the use of Multi-Space Distributions (MSD) [32] can be avoided. In that sense, the involved training and generation procedures can be simplified. In addition to these advantages, the suggested phase representation facilitates the study of the perceptual importance of the maximum-phase component and the degree of noisiness, which are linked to separate features. Indeed, phase perception is still a source of controversy in speech processing [33]-[35].

The next section first summarizes the low-level analysis of the speech signal using the aHM model. Then, the novel phase features based on the mean and standard deviation of PD are described in details, which are followed by the description of the synthesis step. Finally, the evaluation section will show the importance of the features and demonstrate the feasibility of the suggested representation in the context of voice transformation and speech synthesis.

2The adaptive harmonic model

Given a speech waveform s(t), we assume that its continuous fundamental frequency curve f₀(t) can be known a priori, thanks to numerous existing methods. For the experiments described in this article, the STRAIGHT’s f₀ analysis method [10] has been used, which allows fair comparisons during evaluation. The speech waveform is first segmented into analysis frames centered around time instants t _i . For the reason of clarity, a constant step size will be first assumed (e.g., 5 ms). Pitch synchronous analysis will be used later on, for statistical characterization in Section 4. At each time instant t _i , the main goal is to represent the frequency content of each frame using features capturing independent characteristics of the speech signal. For this purpose, in a Blackman window of three pitch periods around each t _i , the aHM model [16] is first used to decompose the analytic signal of s(t) into harmonic frequency components:

$$s_i\left(t\right)=\sum _{h=1}^{H_i}{a}_{i,h}·e^{j\left(h{\varphi }_0\right(t)+{\varphi }_{i,h})}$$

(1)

where i is the frame index, H _i =⌊0.5f _s /f₀(t _i )⌋ is the number of harmonic up to Nyquist frequency in the frame i, f _s denotes the sampling frequency, a_i,h is the real-valued amplitude of the h th harmonic at frame i, ϕ_i,h is the instantaneous phase parameter, and ϕ₀(t) is a real function which adapts the frequency basis of the harmonic model to the waveform frequency modulations [15]:

$${\varphi }_0\left(t\right)=\frac{2\pi }{f_s}\underset{t_i}{\overset{t}{\int }}{f}_0\left(\tau \right)\mathrm{d\tau .}$$

(2)

For this work, the Adaptive Iterative Refinement (AIR) algorithm presented in [16] is used to refine the f₀(t) values and the sinusoidal parameters (a_i,h and ϕ_i,h) are estimated using the Least Squares (LS) solution.

Conversely to the conventional harmonic model [3], the aHM model uses a full-band non-stationary frequency basis. This mainly allows to represent a whole speech recording using a single and continuous harmonic structure during both analysis and synthesis steps [15]. This structural property is very convenient for the phase models and processing used in this work.

Also, in unvoiced segments, assuming that an f₀(t) curve can be obtained without substantial erratic jumps, it has been shown that aHM can represent both voiced and unvoiced segments uniformly, without voicing decision [16]. Given the goal of this work, this property is obviously a necessary prerequisite. Additionally, together with its harmonic tracking algorithm (i.e., AIR), this model provides almost always the most accurate and precise sinusoidal parameters compared to state-of-the-art methods [16]. Eventually, this good accuracy and precision might not be critical for obtaining the results of this article. However, this allows to minimize the influence of the sinusoidal parameter estimation on the results thus, strengthening the link between the suggested phase processing techniques and the results obtained. Finally, like the conventional harmonic model [3], the resynthesis obtained by aHM is almost indistinguishable from the original recording [16]. This ensures that the aforementioned properties come with no perceptual degradation.

Despite the advantages of aHM, the sinusoidal parameters a_i,h, ϕ_i,h, and f_i,h=h f₀(t _i ) lie on the harmonic grid, which is not convenient for manipulation of the perceived characteristics or for statistical modeling. Moreover, the instantaneous phase parameter ϕ_i,h constantly wraps from one instant to the next, which makes its modeling far from straightforward. In the following steps, we aim at building amplitude and phase features which are independent of the harmonic structure and we focus on the modeling of the instantaneous phase.

2.1 A simple representation of the amplitude

For this study, we assumed that the voice production consists of a spectrally flat source and a filter [36]. In frequency domain:

$$S_i\left(f\right)=G_i\left(f\right)·V_i\left(f\right)$$

(3)

where S _i (f) is the Fourier transform of s _i (t), described in hertz for the reason of clarity, G _i (f) is the spectrum of the voice source, and V _i (f) is the vocal tract filter response. Therefore, the harmonic amplitudes a_i,h can be considered as discrete samples of the filter amplitude response |V _i (f)|:

$$\text{if}\left|G_i\right(f\left)\right|=1\forall f\Rightarrow a_{i,h}=\left|V_i\right({\mathit{\text{hf}}}_0\left(t_i\right)\left)\right|.$$

(4)

We also assumed that V _i (f) is minimum-phase so that ∠ V _i (f) is linked to |V _i (f)| through the Hilbert transform [37]. The phase response ∠ V _i (f) can also be retrieved through the imaginary part of the Fourier transform of the minimum-phase cepstrum ${\widehat{v}}_{-}\left(t\right)$ of V _i (f):

$${\widehat{v}}_{-}\left(t\right)=\left\{\begin{array}{cc}0& t<0\\ 2\widehat{v}\left(t\right)& t>0\\ \widehat{v}\left(t\right)& t=0\end{array}\right.$$

(5)

where $\widehat{v}\left(t\right)$ is the real cepstrum of |V _i (f)|, i.e., $\widehat{v}\left(t\right)={\mathcal{F}}^{-1}(log|V_i\left(f\right)\left|\right)$, as described in [37]. Modeling the amplitude envelope is a well investigated subject, and it is out of the scope of this article. In order to estimate |V _i (f)| in a robust and simple way, we used a linear interpolation of a_i,h across frequency, as used in [38], on a discrete scale of 512 frequency bins up to the Nyquist frequency. However, for the reason of clarity, the continuous notation in hertz will be used in the following.

Even though the assumption of spectrally flat source is widely used, it is also known that this hypothesis is basically wrong since the glottal pulses have a low-pass characteristic [39]. Therefore, in this work, |V _i (f)| encompasses the amplitude spectra of both the glottal source and the vocal tract. Nevertheless, using PD, it has been shown that this assumption allows to extract glottal source information which is almost independent of the vocal tract filter [40]. Indeed, this property was critical for estimation of glottal model parameters using phase minimization [25],[40]. For the work presented in this article, this same property ensures that the impact of the vocal tract filter on the phase features representing the source is minimal. On the contrary, the impact of the glottal source on the vocal tract feature is far from negligible, which is not convenient. However, a robust separation of the vocal tract filter and the glottal source is far from straightforward [31],[41]-[44]. Thus, in this work, we chose to favor again robustness and simplicity, in order to focus, beforehand, on the phase features. Interposing a separation process within the presented phase feature extraction can be part of future works.

3Representations of the phase

In this section, we first describe the analytical model of the instantaneous phase used in this work. State-of-the-art phase processing are then described and discussed analytically using this model. Finally, the novel characterization of the short-term statistics of PD is described.

3.1 Theoretical model of the instantaneous phase

In order to represent the instantaneous phase parameter ϕ_i,h, models have been already suggested for phase synchronisation between frames [45],[46] and speech coding [14],[47]. In this work, we suggest to represent the measured ϕ_i,h using a model similar to that in [47]:

whose terms are described here below. In voiced segments, each glottal pulse of the glottal source has a shape which has mainly maximum-phase characteristics [27],[39]. This glottal pulse shape has also a position in time c _i . Speech processing techniques often define c _i as glottal closure instants [48],[49], or as energy local maxima of a residual signal [50],[51], or as pitch pulse onsets [12],[14],[27] for centering windows and to synchronize instantaneous phase parameters.

Even though such a definition is necessary for many approaches, we will show below that it is not necessary when using the Relative Phase Shift (RPS) [24],[33] or PD, which avoids an extra estimation procedure and its potential misestimation errors. In unvoiced segments, one can assume that this shape is basically random for each frame. In Equation ??, the source shape term θ_i,h represents this pulse shape in both voiced and unvoiced segments. Since the analysis windows are not centered on each c _i (i.e., c _i ≠t _i ), a linear phase term is also necessary in order to represent the time delay between t _i (the window’s center) and the position of the source shape c _i . In the literature, assuming the frequency structure is stationarity within a frame, i.e., f₀(t)=f₀(t _i ), this term is often simplified to a term which is linear in both frequency and time, i.e., h(2π f₀(t _i )/f _s )(t _i −c _i ). Conversely, in Equation ??, we use the integral form since the harmonic structure is not stationary in the aHM model.

Finally, according to the voice production Equation 3, the voice source is convolved by the vocal tract filter impulse response. Thus, we add the minimum-phase ∠ V _i (ω) to the model.

The following sections describe the suggested way to characterize ϕ_i,h for speech processing using statistics of PD and using the RPS as an intermediate step.

3.2 From phases to relative phase shift

The linear phase component in Equation ?? constantly wraps the instantaneous phase ϕ_i,h from one time instant to the next. This constitutes a major issue in phase modeling [27].

To alleviate this issue, the RPS has been suggested [33], which is expressed as:

$${\text{RPS}}_{i,h}={\varphi }_{i,h}-h{\varphi }_{i,1.}$$

(7)

The second row of Figure 1 shows an example of RPS computation (the harmonic values have been interpolated on a continuous frequency axis for reason of presentation). To further analyze the results of the RPS computation, one can replace the estimated instantaneous phase parameter ϕ_i,h by its models Equation ??:

$$\begin{array}{ccc}{\text{RPS}}_{i,h}& =& {\theta }_{i,h}+h\frac{2\pi }{f_s}\underset{c_i}{\overset{t_i}{\int }}{f}_0\left(\tau \right)\mathrm{d\tau }+\angle V_i\left({\mathit{\text{hf}}}_0\left(t_i\right)\right)\\ -h·\left({\theta }_{i,1}+\frac{2\pi }{f_s}\underset{c_i}{\overset{t_i}{\int }}{f}_0\left(\tau \right)\mathrm{d\tau }+\angle V_i\left(f_0\right(t_i\left)\right)\right)\end{array}$$

(8)

Figure 1

Examples of computation of RPS and PD, interpolated on a continuous frequency scale. V(f) can be removed prior to RPS or PD computations, thus removing the effects of the formant and focusing on the glottal source.

which reduces to:

$${\text{RPS}}_{i,h}={\theta }_{i,h}-h{\theta }_{i,1}+\left(\angle V_i\left({\mathit{\text{hf}}}_0\left(t_i\right)\right)-\mathrm{h\angle }{V}_i\left(f_0\left(t_i\right)\right)\right)$$

(9)

Equation 9 shows that the RPS computation discards the linear phase terms. It remains only the source shape at each harmonic relative to that of the first harmonic and the contribution of the minimum-phase envelope ∠ V _i (f) relative to that at the first harmonic. In voiced segments, these two remaining terms can be easily assumed to evolve smoothly across time because the shape of the glottal pulse and the vocal tract do so. Therefore, this property of RPS basically solves the issue of phase wrapping.

Additionally, c _i is also discarded in Equation 9 so that there is no need to estimate any GCI or pitch pulse onset. This avoids misestimation of such time instants and the consequences on speech processing techniques.

The RPS can also be computed on the Linear Prediction (LP) residual [33], which removes the minimum-phase contribution of V _i (f). Similarly, let us define:

$${\tilde{\varphi }}_{i,h}={\varphi }_{i,h}-\angle V_i\left({\mathit{\text{hf}}}_0\right(t_i\left)\right)$$

(10)

where ${\tilde{\varphi }}_{i,h}$ is the instantaneous phase where the minimum-phase frequency response corresponding to the amplitude envelope has been removed. Consequently, Equation 9 becomes:

$${\widetilde{\mathrm{\text{RPS}}}}_{i,h}={\tilde{\varphi }}_{i,h}-h{\tilde{\varphi }}_{i,h}={\theta }_{i,h}-h{\theta }_{i,1.}$$

(11)

The third row of Figure 1 shows an example of $\widetilde{\mathrm{\text{RPS}}}$ (with, again, the interpolation on a continuous frequency axis). In Equation 11, only the source shape and the harmonic number h remains. Ideally, we want to extract features from the speech waveform which are independent from each other as much as possible. However, h belongs to the harmonic structure which is already handled by f₀(t) and the property of harmonicity of the model. Therefore, this harmonic number is still inconvenient for characterizing the phase properties independently from f₀(t). Interpolating the ${\widetilde{\mathrm{\text{RPS}}}}_{i,h}$ values on a continuous frequency axis (as depicted in Figure 1) removes the harmonic sampling. However, the harmonic number is still present in the interpolated values. Note also that, h increases the RPS variance towards high frequencies and drowns the variance of θ_i,h into that of h θ_i,1, which is not convenient for characterizing the source shape in mid and high frequencies.

3.3 From relative phase shift to phase distortion

The problem mentioned above can be solved by simply computing the finite difference of ${\widetilde{\mathrm{\text{RPS}}}}_{i,h}$ with respect to h. In the general context of signal processing, the relative phase difference between two frequency components is known as PD, whose perceived characteristics are already studied and known [52]-[54]. In the particular context of speech analysis, we already used the PD for the estimation of glottal model parameters [25],[40] and for emotion valence detection [55]. Moreover, in [40], we have shown that the PD is directly linked to the maximum-phase component of the glottal source. This sole property allows to estimate parameters of glottal models as presented in [25],[40]. Therefore, PD is also a strong correlate of the maximum-phase component of the voice source. The rather complicate definition of PD in [25],[40] is actually equal to:

$$\begin{array}{ccc}{\mathrm{\text{PD}}}_{i,h}& =& \underset{h}{\Delta }{\widetilde{\mathrm{\text{RPS}}}}_{i,h}={\widetilde{\mathrm{\text{RPS}}}}_{i,h+1}-{\widetilde{\mathrm{\text{RPS}}}}_{i,h}\\ =& \left({\tilde{\varphi }}_{i,h+1}-(h+1){\tilde{\varphi }}_{i,1}\right)-\left({\tilde{\varphi }}_{i,h}-h{\tilde{\varphi }}_{i,1}\right)\\ =& {\tilde{\varphi }}_{i,h+1}-{\tilde{\varphi }}_{i,h}-{\tilde{\varphi }}_{i,1}\end{array}$$

(12)

where $\underset{h}{\Delta }$ denotes the finite difference operator. The fifth row of Figure 1 shows an example of PD (with the interpolation on a continuous frequency axis). Basically, the PD measures the phase desynchronization which exists between each sinusoidal component of the voice source. Additionally, this desynchronization is centered on the first harmonic phase, like for the RPS. The finite difference makes also the PD similar to the group delay, whose perceived characteristics have been already studied and demonstrated [56] and whose applications are numerous [48],[57]-[61].

By replacing ϕ_i,h by its model Equation ??, the PD computation leads to:

$$\begin{array}{ccc}{\mathrm{\text{PD}}}_{i,h}& =& {\theta }_{i,h+1}-{\theta }_{i,h}-{\theta }_{i,1.}\end{array}$$

(13)

Since the linear and filter terms cancel, only the source shape terms remain in Equation 13. Equation 13 also shows that the computation of the PD represents the phase desynchronization of the source shape between each harmonic, centered on that of the first harmonic. Compared to Equation 11, the harmonic number h is also removed, as expected, by using the finite frequency difference. Consequently, when h increases, it adds to the RPS measurement but does not influence the PD measurement. For example, when using PD in fourth and fifth rows of Figure 1, one can see red patterns appearing around 1.5 s between 4 and 8 kHz. On the contrary, no clear pattern appears in the same time-frequency region using the RPS (second and third rows). Using RPS, the region concerned actually seems as blurred as in noisy time-frequency regions (e.g., around 1.8 s). This is explained by the presence of the harmonic number h in RPS which increases the wrapping of the phase values.

3.4 Statistical features of the phase distortion

As shown in Equation 13, the phase distortion represents basically the source shape. In voiced segments, the source shape accounts mainly for the shape of the glottal pulse. In unvoiced segments, the time evolution of this shape throughout adjacent frames reproduces the noisiness of the voice source. Therefore, in this section, we suggest to statistically characterize the phase distortion in a short-term window in order to extract a feature related to the shape at a given time and another feature representing the local variation of this shape around that same time. This characterization will allow to manipulate the components of the speech in voice transformation and Hidden Markov Model (HMM)-based synthesis.

We first assume that the information carried in PD is independent of the fundamental frequency. As a consequence, we interpolate PD_i,h on a linear frequency scale (as done for the previous figures), like a phase spectral envelope [62],[63], and thus, removing the harmonic number from the representation. To achieve this phase envelope, we first unwrap PD_i,h and then interpolate it linearly on a discrete scale of 512 frequency bins up to the Nyquist frequency, and thus, PD_i,h becomes PD _i [ k]. Here, the unwrapping function is necessary to avoid meaningless values during the interpolation process. Nevertheless, the resulting PD _i [ k] is still a circular data. Instead of the discrete notation in bins, the continuous notation in hertz will be used in the following descriptions and sections for the reason of clarity, i.e., PD _i [ k]⇔PD _i (f), like for the amplitude spectral envelope V _i (f).

On a frame-by-frame basis in an analysis/synthesis procedure, the sole information carried by PD _i (f) might be sufficient to reconstruct an instantaneous phase which has the same perceived characteristics as those of the instantaneous phase ϕ_i,h. This property has actually been shown through listening tests in [33]. However, through manipulation of PD _i (f), by time scaling, pitch scaling, or statistical modeling, the short-term statistical characteristics of the analyzed voice might not be preserved. For example, stretching PD _i (f) over time would automatically reduce its temporal variance and thus, changing the extent of randomness in the voice, which is not the purpose of a time stretching transformation.

In this article, we suggest to preserve the short-term mean and short-term standard deviation of PD _i (f) in speech processing applications using features that represent these two moments. In order to estimate the mean and standard deviation, we assume that the distribution of PD _i (f) obeys a normal distribution. Moreover, since PD _i (f) is a circular data defined in (−π,π], we make use of the wrapped normal distribution [47],[64].

In order to ensure that the short-term estimate of PD variance is not influenced by the number of periods, it is first necessary to use the same number of time instants t _i in each glottal cycle and not a constant step size as previously assumed so far. In order to have enough values for computing a reliable short-term variance in the following, we used four analysis instants per period:

$$t_i=t_{i-1}+\frac{1}{4}\frac{1}{f_0\left(t_{i-1}\right)}\text{with}{t}_0=0.$$

(14)

3.4.1 The short-term mean of PD

Since the values PD _i (f) are circular data, the wrapped normal distribution [47],[64] has been used in this work to model PD _i (f) over a few periods. The mean is estimated with [64]:

$${\mu }_i\left(f\right)=\underset{i}{\text{mean}}\left({\mathrm{\text{PD}}}_i\left(f\right)\right)=\angle \left(\frac{1}{N}\sum _{n\in B}{e}^{j{\mathrm{\text{PD}}}_n\left(f\right)}\right)$$

(15)

where $B=\left\{i-\frac{N-1}{2},\cdots ,i+\frac{N-1}{2}\right\}$ and we used N=25 frames in this work, which corresponds to six periods. This averaging of PD _i (f) is necessary for separating the randomness characteristics of the phase from its smoothly varying behaviors. Even though six periods might appear to be substantial, it ensures that the mean does not model the randomness of the phase, which has to be modeled by the feature described below.

3.4.2 The short-term standard deviation of PD

According to [64], the standard deviation should be estimated by:

$$\begin{array}{ccc}{\sigma }_i\left(f\right)& =& \underset{i}{\text{std}}\left({\mathrm{\text{PD}}}_i\left(f\right)\right)\\ =& \sqrt{-2log\left|\right.\frac{1}{N}\sum _{n\in B}{e}^{j\left({\mathrm{\text{PD}}}_n\right(f\left)\right)}\left|\right.}\end{array}$$

(16)

where $B=\left\{i-\frac{N-1}{2},\cdots ,i+\frac{N-1}{2}\right\}$. However, in our context of application, two issues arise with Equation 16. Firstly, as shown in Equation 13, PD _i (f) is related to the source shape in voiced segments. One can easily assume that this shape is non-constant and evolve smoothly across time. Thus, over a few periods, PD _i (f) has also a non-constant trend. Figure 2 shows an example of PD _i (2f₀(t)) and the corresponding estimation of σ _i (2f₀(t)). One can see that the waveform is drastically changing from 1 s to ∼1.07 s. This phenomenon is revealed by PD _i (2f₀(t)) which tends towards low phase values.

Figure 2

Example of estimation of the standard deviation of PD, using Equations 16 and 18 .

The σ _i (f) estimate is overestimated by the presence of this trend whereas σ _i (f) is supposed to represent only the noisiness of the voice source. Moreover, the time evolution of μ _i (f) already modeled this trend of the voice source. Consequently, there is no reason to keep this trend in the standard deviation estimate. In Figure 2, one can also see that the estimation of the variance using Equation 16 reaches 0.5 rad in the bottom plot, even though the waveform does not show any noise around 1.07 s. Therefore, to alleviate this problem, we suggest to remove an estimate of the trend prior to the computation of the standard deviation. The trend is first estimated by averaging PD _i (f) over two periods:

$${\stackrel{⌢}{\mathrm{\text{PD}}}}_i\left(f\right)=\angle \left(\frac{1}{M}\sum _{m\in C}{e}^{j{\mathrm{\text{PD}}}_m\left(f\right)}\right)$$

(17)

where $C=\left\{i-\frac{M-1}{2},\cdots ,i+\frac{M-1}{2}\right\}$, with M=9. This trend is then removed before computing the standard deviation:

$$\begin{array}{ccc}{\sigma }_i\left(f\right)& =& \underset{i}{\text{std}}\left({\mathrm{\text{PD}}}_i\left(f\right)-{\stackrel{⌢}{\mathrm{\text{PD}}}}_i\left(f\right)\right)\\ =& \sqrt{-2log\left|\right.\frac{1}{M}\sum _{m\in C}{e}^{j\left({\mathrm{\text{PD}}}_m\right(f)-{\stackrel{⌢}{\mathrm{\text{PD}}}}_m(f\left)\right)}\left|\right.}\end{array}$$

(18)

where $C=\left\{i-\frac{M-1}{2},\cdots ,i+\frac{M-1}{2}\right\}$ and M=9 frames. Using two periods for the standard deviation and six for the mean are motivated by the following reason. A wider window for the standard deviation could cover the end of a noisy segment and the beginning of a voiced segment and thus, overestimating the presence of noise at the beginning of the voiced segment. Therefore, a short window seems necessary to quickly adapt the standard deviation estimate in transients. On the other hand, using a wider window for the mean allows to obtain a more robust estimate of the source shape in transients where harmonic sinusoidal parameters are less reliable than in voiced segments.

In order to have the same number of analysis instants in each period, the step size of the analysis instants was first adapted to f₀(t) (see Equation 14). However, both mean and standard deviation have to be independent from f₀(t) so that each feature represents independent characteristics of the speech signal. Additionally, a variable step size is not desirable for many applications, like in statistical modeling, where a constant step size is necessary. Consequently, prior to any application, μ _i (f) and σ _i (f) features are resampled at new time instants ${\widehat{t}}_i$, with a constant step size, each 5 ms.

Figure 3 shows an example of features extraction.

Figure 3

Example of features extraction: Phase Distortion (PD)’s mean and standard deviation. They characterize mainly the maximum-phase content and the noisiness of the voice source, respectively.

4Synthesis

The analysis steps described above provide, each 5 ms, the features f₀(t _i ), V _i (f), μ _i (f), and σ _i (f). This section describes the method used to resynthesize a full speech signal using these features. This synthesis method is similar to that used originally for the aHM model [16]. Basically, each harmonic track is first synthesized across time, independently of each other, using a sampling rate f _s . The synthetic harmonics are then added up all together, without using any windowing scheme. Since the synthetic signal is bandlimited to the Nyquist frequency, the continuous notation for the time axis will be used in the following, for reason of simplicity (i.e., x[ n]⇔x(t)).

In order to synthesize the continuous amplitude ${â}_h\left(t\right)$ of each harmonic track, the amplitude envelope is first sampled at harmonic frequencies for each time instants ${\widehat{t}}_i$:

$${â}_{i,h}=\left|V_i\left({\mathit{\text{hf}}}_0\left({\widehat{t}}_i\right)\right)\right|.$$

(19)

Then, these anchor values ${â}_{i,h}$ are interpolated across time on a logarithmic scale in order to obtain ${â}_h\left(t\right)$.

In the aHM model, the continuous instantaneous phase ${\widehat{\varphi }}_h\left(t\right)$ was not directly interpolated from the measured instantaneous phase parameters ϕ_i,h because of the linear phase term which is present in ϕ_i,h and does not allow to compute a reliable interpolation. Instead, a relative phase${\check{\varphi }}_{i,h}$ was first computed in order to remove the influence of the linear phase [16]:

$${\check{\varphi }}_{i,h}={\varphi }_{i,h}-h·\frac{2\pi }{f_s}\underset{0}{\overset{t_i}{\int }}{f}_0\left(\tau \right)\mathrm{d\tau }$$

(20)

where the zero for the beginning of the integral means the beginning of the speech signal. Since the linear phase is removed in Equation 20, the relative phase changes smoothly from one time instant to the next if the shape of the signal is also changing smoothly. Thus, any processing of ${\check{\varphi }}_{i,h}$ is better conditioned than a processing of the raw instantaneous phase value ϕ_i,h. This property explains the success of the simple processing techniques presented in [17],[18]. In [16], the relative phases were then interpolated on a continuous time axis using splines, i.e., ${\check{\varphi }}_{i,h}\Rightarrow {\check{\varphi }}_h\left(t\right)$. Finally, the continuous instantaneous phase ${\widehat{\varphi }}_h\left(t\right)$ was recovered by adding back the linear phase previously removed:

$${\widehat{\varphi }}_h\left(t\right)={\check{\varphi }}_h\left(t\right)+h·\frac{2\pi }{f_s}\underset{0}{\overset{t}{\int }}{f}_0\left(\tau \right).$$

(21)

The final synthetic signal $ŝ\left(t\right)$ was generated by summing all the harmonic tracks together:

$$ŝ\left(t\right)=\sum _{h=1}^H{â}_h\left(t\right)e^{j{\widehat{\varphi }}_h\left(t\right)}·{\chi }_{\left[{\mathit{\text{hf}}}_0\left(t\right)<f_s/2\right]}\left(t\right)$$

(22)

where H is the maximum value of all H _i and the indicator function ${\chi }_{\left[{\mathit{\text{hf}}}_0\left(t\right)<f_s/2\right]}\left(t\right)$ discards any harmonic segment whose frequency is higher than Nyquist.

For the analysis/synthesis method suggested in this article, the aHM synthesis summarized above has to be adapted in order to use μ _i (f) and σ _i (f). Looking at Equation 21, the linear phase term can be reconstructed because f₀(t) is preserved through the analysis step. However, the relative phase ${\check{\varphi }}_h\left(t\right)$ is lost. Therefore, the main goal is to reconstruct a synthetic relative phase which has the same perceived characteristics as the original one. For this purpose, we suggest to follow the phase model Equation ?? while using the short-term mean and standard deviation of the phase distortion in order to resynthesis a source shape. First, at each instant ${\widehat{t}}_i$, we synthesize a phase distortion ${\widehat{\mathrm{\text{PD}}}}_{i,h}$ using the wrapped normal distribution [47],[64]:

$${\widehat{\mathrm{\text{PD}}}}_{i,h}=\mathcal{W}\mathcal{N}\left({\mu }_i\right({\mathit{\text{hf}}}_0\left(t_i\right)),{\sigma }_i({\mathit{\text{hf}}}_0\left(t_i\right)\left)\right)$$

(23)

where $\mathcal{W}\mathcal{N}(\mu ,\sigma )$ generates random values which obey a wrapped normal distribution of mean μ and standard deviation σ. Note that, this procedure is similar to known phase randomization techniques [65]-[67]. However, because of the finite difference used to compute PD, our approach is similar to randomizing the group delay and not the instantaneous phase. Because of this difference, the randomization is always centered around the linear phase. Therefore, this approach ensures that the noise component is always merged with the deterministic component, which avoid these two components to be perceived separately. Based on Equation 13, then, we suggest to approximate a source shape using ${\widehat{\mathrm{\text{PD}}}}_{i,h}$. However, because the PD values are centered around θ_i,1 (see Equation 13), this value is lost during the analysis step. Therefore, we assume θ_i,1=0 and thus:

$${\widehat{\theta }}_{i,h}={\underset{h}{\Delta }}^{-1}{\widehat{\mathrm{\text{PD}}}}_{i,h}$$

(24)

where ${\underset{h}{\Delta }}^{-1}$ is the cumulative sum, which compensates for the finite difference in Equation 13. Following the model of the instantaneous phase Equation ??, we finally add the minimum-phase response of the envelope V _i (f) in order to obtain the synthetic relative phase values ${\check{\varphi }}_{i,h}$:

$${\check{\varphi }}_{i,h}={\widehat{\theta }}_{i,h}+\angle V_i\left({\mathit{\text{hf}}}_0\right(t_i\left)\right).$$

(25)

The rest of the synthesis process is identical to that of aHM. The continuous relative phase values are interpolated across time using splines, and the linear phase is added at the end in order to obtain the continuous instantaneous phase ${\widehat{\varphi }}_h\left(t\right)$ Equation 21 which is finally used in Equation 22.

The complete analysis/synthesis procedure is called Harmonic Model + Phase Distortion (HMPD).

4.1 Correction of σ _i (f)

When testing the resynthesis capabilities of the method through informal listening tests, we found that the perceived characteristics of the fricatives were not properly reproduced. Basically, no segments of the signal were fully randomized. After investigation, we found that σ _i (f) was limited in its measure. To illustrate this phenomenon, Figure 4 shows the average of σ _i (f) measured from synthetic signals of known σ _i (f). Synthetic signals of 4s are first generated using the HMPD synthesis method described above with μ _i (f)=0∀i,f. Each synthetic signal uses a different σ _i (f) value between 0 and 2. The HMPD analysis method is then used to reestimate the σ _i (f) values from the synthetic signals. Figure 4 shows the measured σ _i (f) averaged along the 4s. It shows that the measured σ _i (f) hardly reach 1.2, which is not sufficient for reconstructing a sound which is perceived as fully noisy. The following two reasons can explain this issue. Firstly, during analysis, a window of three periods is used to estimate the sinusoidal parameters Equation 1. This window is necessary to obtain a frequency resolution which allows to distinguish harmonic peaks and estimate the sinusoidal parameters with a sufficient accuracy. However, this window also over-smoothes the variance of the parameter estimates across time. Even though these parameters allow reconstruction of the signal on a frame-by-frame basis, this over-smoothing effect does not allow to estimate a short-term variance with sufficient accuracy for proper reconstruction of the phase’s statistics. Secondly, the window size M in Equation 18, which has to be short enough to follow the time evolution of the speech signal, limits also the standard deviation estimate. Note that, this effect appears with or without removing the PD trend ⌢PD _i (f) during analysis and thus, using either Equations 16 or 18.

Figure 4

Measured and corrected σ _i ( f ) from a synthetic signal with a known σ _i ( f ).

To avoid overloading this presentation, we chose to alleviate this issue in a simple manner. We corrected the σ _i (f) values prior to synthesis so that it was given a sufficient randomness when it was greater than an empirical threshold. Through informal listening tests, we found that a forced σ _i (f) value of 2 used above a threshold of 0.75 properly reconstruct the noisiness of fricatives while preserving the voiced segments quality. Figure 4 depicts this correction. Figure 5 shows an example of σ _i (f) after correction, as it is used during the synthesis step. Future works are planned to study the influence of the window sizes and address this issue in a neater way.

Figure 5

Example of correction of the standard deviation estimate of the Phase Distortion (PD) σ _i ( f ). The correction mainly affects the noisy time-frequency regions, where the correction ensures a full randomization.

5Evaluation

This section aims at assessing the quality and versatility of the proposed phase representation. To this end, experiments have been conducted in three different scenarios: resynthesis with no modification (Section 4), pitch scale modification (Section 4), and HMM-based speech synthesis (Section 4).

Even in resynthesis, objective measures such as Signal to Reconstruction Error Ratio (SRER) or PESQ [68] are not suitable for evaluation as they are waveform sensitive. While it is true that the suggested HMPD representation retains the waveform characteristics of the signal, it does not keep its linear phase term: the original linear phase removed between Equations 8 and 9 and the synthetic one added in Equation 21 are not necessarily the same but just have the same derivative, i.e., f₀(t). Consequently, original and synthesized waveforms are not time synchronous. Objective measures are also inconvenient for comparing different configurations of the HMPD vocoder, including those dropping the maximum-phase component given by μ _i (f), in which the shape of the glottal pulse is not preserved.

Therefore, in this paper, all evaluations have been carried out by means of subjective listening tests, as suggested by [69]. All the sounds used in the following tests are available at [70].

5.1 Quality of resynthesis

The first test was designed to evaluate mainly the importance of μ _i (f) and σ _i (f) in terms of perceptual quality. The speech database used in this experiment contained a total of 32 utterances spoken in 16 different languages (two utterances per language, one from a male speaker and one from a female speaker). Such a multilingual database had been thoroughly designed to exhibit a very wide phonetic variability and also an heterogeneous set of speakers. The sampling frequency of the signals, f _s , varied between 16 and 44.1 kHz. All original recordings showed high signal-to-noise ratios as they had been collected from various synthesis databases. The test was conducted through a web-based interface. A total of 43 volunteer listeners were presented with the original recordings of randomly selected signals along with their reconstructed versions using: aHM; the suggested HMPD using both μ _i (f) and σ _i (f); HMPD using σ _i (f) only; the well-known STRAIGHT vocoder, which was used as a hidden anchor. Then, they were asked to grade the quality of these sounds using a 5-points scale [69]. The order of the reconstruction methods was randomized too, and the listenings were made through headphones or earphones. For consistency and to avoid the fatigue of the evaluators, each listener was asked to grade only the voices of two languages (both male and female voices) randomly selected among the 16.

Figure 6 contains the resulting mean opinion scores along with their 95% confidence intervals. The scores have been normalized according to the number of occurrence of each language and to the variance of each listener’s answers, as suggested in [69]. The scores achieved by aHM are consistent with those reported in previous studies [16] and confirm the very good resynthesis potential of this signal model. However, one of the most remarkable observations is the significant performance gap between aHM and HMPD, which gives an idea of the relevance of phase in quality issues. This loss of quality could be partly explained by the following reason. First, the HMPD resynthesis is not time synchronous with that of aHM. Indeed, the synthetic linear phase term used in Equation 21 is not that of the original speech signal. On the one hand, the voiced segments can be delayed or advanced of only a maximum of half of a period. This effect is surely impossible to perceive within a voiced segment, where the amplitude envelope and the phase characteristics are properly decorrelated using the suggested analysis/synthesis procedure. However, on the other hand, this time desynchronization might play a role in highly non-stationary speech segments like in transients from plosive to voiced segments or in creaky voice. Indeed, in such cases, the time amplitude envelope, which is driven by the mean of the spectral amplitude envelope, should be synchronous with the impulses triggered by the linear phase. However, the time desynchronization in HMPD may break this necessary correlation which can easily blur the perception of time events and degrade the overall quality. Note that, this reasoning also holds for the STRAIGHT method where the linear phase is also fully artificial and neither synchronous with the original one. According to informal listening, the creaky voice segments seem, indeed, not properly reconstructed in both HMPD and STRAIGHT.

Figure 6

Subjective assessment of perceived quality by 43 listeners of resynthesis methods, with the 95% confidence intervals.

Among other reasons, the measure of randomness using σ _i (f) might not adapt quickly enough in transients, so that the beginning and the end of voiced segments can be sometimes over randomized. Smoothing techniques and different separation procedures for estimation of μ _i (f) and σ _i (f) should be investigated in the future.

Regarding the relative performance of HMPD- μ σ and HMPD- σ, the average scores indicate that, for the voices used in this experiment, the listeners were not able to perceive any difference between them. This suggests that the contribution of μ _i (f) is not perceptually significant in comparison with that of σ _i (f). Even more, since the link between PD_i,h and the maximum-phase of the glottal pulse has been shown and exploited [25],[40], this suggests that the maximum-phase information is hardly noticeable at this overall quality level. Admittedly, this could also be an indicator that μ _i (f) is not capturing the maximum-phase component properly. In any case, the average results also show that the quality provided by HMPD is at least as good as that of STRAIGHT and better for male voices. Note also that, compared to STRAIGHT, the difference of quality between genders is also clearly reduced using HMPD. In other words, the phase randomization technique suggested in this paper, which exploits σ _i (f), might be a potential improvement and replacement for STRAIGHT’s aperiodicity measures [28].

5.2 Quality of pitch shifting

A second experiment was conducted to check the consistency of HMPD in a more challenging scenario. In that sense, pitch scaling is preferable over time scaling because it can shed light on possible inaccuracies in isolating amplitude or phase information from periodicity information. Therefore, after the analysis step, f₀(t _i ) was multiplied by a factor of 2, or 0.5, in order to shift the pitch of the voice one octave upwards or downwards, respectively. The signals in the database described in Section 4 were manipulated using three different methods: HMPD- μ σ, HMPD- σ, and STRAIGHT. In the case of HMPD, the pitch modification factor was applied to all f₀(t _i ) values, without any distinction between voiced and unvoiced segments, while in STRAIGHT unvoiced segments were obviously kept unvoiced.

Using a web-based interface, 30 listeners gave their pairwise preferences for the three possible combinations of methods using a 5-points scale [69]: strong preference for one method, preference for one method, preference for the other method, strong preference for the other method, or uncertainty. Again, each listener assessed the quality of the upwards and downwards shifts of the recordings of two languages, for one female and one male speaker per language. The individual scores given by the listeners were then aggregated into a single mean score for each method, which shows global preference of one method against all the others.

The results shown in Figure 7 indicate that HMPD- μ σ is less preferred than HMPD- σ. Deeper investigation based on informal listening revealed that a low-frequency resonance could be perceived in some signals after HMPD- μ σ manipulation. This might corresponds to the glottal formant effect [71] which is not properly handled in our manipulation. Indeed, keeping the phase characteristics of the original glottal source, as μ _i (f) is supposed to do, does not make sense after pitch scaling by one octave. Finally, this can also be interpreted as a symptom that μ _i (f) is not reproducing the maximum-phase component properly. In any case, the resulting artifacts make signals more unnatural. By discarding the contribution of μ _i (f), HMPD- σ avoids this issue and achieves a better quality. This is the reason why only HMPD- σ was considered for evaluation in the next Section 4.

Figure 7

Preferences for 30 listeners of upwards and downwards pitch shifts of one octave, with the 95% confidence intervals.

Concerning the comparison between HMPD and STRAIGHT, for upwards pitch shifting, STRAIGHT is clearly preferred over HMPD- σ. However, for downwards shifts, clear preferences are shown for HMPD- σ. Informal listening revealed that for upwards pitch shifting, the speech signals modified by HMPD sound tenser and lack some noisiness. This is due to the inherent limitations of modeling speech exclusively through harmonics: even for an adequate phase variance across time, at high pitch values, the frequency gap between every two consecutive harmonics does not allow a proper reconstruction of noise characteristics. STRAIGHT is not prone to this effect because it uses a wideband noise [28]. This is undoubtedly one issue in HMPD to be solved in future works.

5.3 Quality of statistical parametric speech synthesis

To assess the quality of HMPD- σ in statistical parametric speech synthesis, we built a system based on the HMM-based Speech Synthesis System (HTS) [72] (v2.1.1). HTS learns a correspondence between labels containing phonetic, linguistic, and prosodic information and one/many streams of vectors containing acoustic features. This correspondence is modeled at phone level through five-state left-to-right context-dependent HMMs with explicit state duration distributions. The technology behind this well-known system is explained in detail in [4].

Both HMPD and STRAIGHT were slightly modified to meet the requirements of HTS. In both of them, 39th-order Mel-CEPstral (MCEP) coefficients were used to model the amplitude envelope |V _i (f)| as suggested in [73], the only difference being that in HMPD, these coefficients were obtained from discrete harmonic amplitudes as in [29]. To model the degree of noisiness, the aperiodicity measures provided by STRAIGHT were averaged within five meaningful bands, as detailed in [73], whereas HMPD’s σ _i (f), which takes values in the range [ 0,∞) like |V _i (f)|, was also translated into MCEP coefficients (order 12). For synthesis, the σ _i (f) on linear scale was recovered from the corresponding MCEP coefficients, like the amplitude envelope. Given the importance of pitch artifacts in HMM-based speech synthesis, for a fair comparison, we used the same f₀(t _i ) values for both vocoders, namely those provided by STRAIGHT. In unvoiced segments, the continuous f₀(t) curve required by HMPD- σ was simply obtained by linear interpolation of the non-zero f₀(t _i ) values. The resulting curve was then modeled using continuous HMMs with one Gaussian mixture per state instead of MSD-HMMs, as proposed by [74].

Before presenting the generated utterances to the evaluators, we manually checked that no significant prosodic differences were present between the two vocoders. During synthesis, all parameter streams were generated through the standard maximum likelihood parameter generation procedure with global variance enhancement. Table 1 summarizes the settings used for the streams.

Table 1 Summary of the streams used in the HMM-based synthesis system

We trained models for four different speech databases: one female and one male speaker in Spanish, containing 1.2 and 2K utterances, respectively [75],[76]; and one female and one male speaker in English, containing 1.1 and 2.8K utterances, respectively [77],[78], all with f _s =16 kHz). All the samples using STRAIGHT and HMPD are available at [70]. For the sack of completeness, samples using impulse-based glottal sources (μ _i (f)=0 and σ _i (f)=0 ∀i,f in the whole signal or only in the voiced segments, as often used in the literature as baseline systems [79],[80]) have also been generated and are available on the demonstration page [70]. However, given their very poor quality, they have not been included in the following listening test in order to avoid their potential influence on the results of STRAIGHT and HMPD. Therefore, we conducted a pairwise preference test between STRAIGHT and HMPD only, similar to that of Section 4. For each voice, 31 listeners gave their preference for each method for one synthetic utterance randomly selected among ten.

Figure 8 shows the global mean preferences. Since there are only two systems under comparison, the preferences are symmetrical. The results show no significant difference between the two systems despite the high number of evaluators. Therefore, the quality achieved by HMPD is comparable to that of the state-of-the-art, while it uniformizes both the speech representation and the modeling process by discarding the voicing decision. More importantly, this preliminary experiment is a reliable confirmation that phase variance across time can inspire features that succeed at capturing the time- and frequency-varying degree of noisiness of speech in the aHM framework.

Figure 8

Preferences for 31 listeners about HMM-based synthesis, with the 95% confidence intervals.

Interestingly, the gender dependencies observed in the previous experiments also arise in Figure 8. Indeed, listeners seem to prefer the female voices of STRAIGHT and the male voices of HMPD- σ. As mentioned in Section 4, this phenomenon is due to the inherent limitations of harmonic modeling at high pitch values. Forthcoming works will address this issue.

6Conclusions

In this paper, features based on mean and standard deviation of the PD have been suggested for analysis/synthesis of speech signals, leading to a new HMPD vocoder.

These features avoid voiced and unvoiced segmentation. Thus, the perceived quality of HMPD synthesis is independent of the reliability of a voicing estimator. A first listening test has shown that HMPD resynthesis quality is as good as that of the STRAIGHT vocoder for female voices and better for male voices.

A second preference test about pitch scaling has shown a limitation of HMPD when the harmonics are not dense enough to properly reproduce noise properties (e.g., with high f₀). Future works are planned to address this fundamental issue of the harmonic models. However, a clear preference has been shown for HMPD in downwards shifts, suggesting that additive wideband noise, often used in existing vocoders, is not necessary for low pitched voices. A last test has suggested that the quality of HMPD in HMM-based speech synthesis is similar to that of the state-of-the-art. Therefore, HMPD basically simplifies the signal representation, in terms of uniformity, by removing the voicing decision, without losing, on average, perceived quality.