PLDA in the i-supervector space for text-independent speaker verification

Abstract

In this paper, we advocate the use of the uncompressed form of i-vector and depend on subspace modeling using probabilistic linear discriminant analysis (PLDA) in handling the speaker and session (or channel) variability. An i-vector is a low-dimensional vector containing both speaker and channel information acquired from a speech segment. When PLDA is used on an i-vector, dimension reduction is performed twice: first in the i-vector extraction process and second in the PLDA model. Keeping the full dimensionality of the i-vector in the i-supervector space for PLDA modeling and scoring would avoid unnecessary loss of information. We refer to the uncompressed i-vector as the i-supervector. The drawback in using the i-supervector with PLDA is the inversion of large matrices in the estimation of the full posterior distribution, which we show can be solved rather efficiently by portioning large matrices into smaller blocks. We also introduce the Gaussianized rank-norm, as an alternative to whitening, for feature normalization prior to PLDA modeling. We found that the i-supervector performs better during normalization. A better performance is obtained by combining the i-supervector and i-vector at the score level. Furthermore, we also analyze the computational complexity of the i-supervector system, compared with that of the i-vector, at four different stages of loading matrix estimation, posterior extraction, PLDA modeling, and PLDA scoring.

1 Introduction

Recent research in text-independent speaker verification has been focusing on the problem of compensating the mismatch between training and test speech segments. Such mismatch in most part is due to the variations induced by the transmission channel. There are two fundamental approaches to tackling this problem. The first approach operates at the front-end via the exploration of discriminative information in speech in the form of features (e.g., voice source, spectro-temporal, prosodic, high-level) [1]-[6]. The second approach relies on the effective modeling of speaker characteristic in the classifier design (e.g., GMM-UBM, GMM-SVM, JFA, i-vector, PLDA) [4],[7]-[15]. In this paper, we focus on the speaker modeling.

Over the past few years, many approaches based on the use of Gaussian mixture models (GMM) in a GMM universal background model (GMM-UBM) framework [7] have been proposed to improve the performance of speaker verification system. The GMM-UBM is a generative model in which a speaker model is trained only on data from the same speaker. New criteria have then been developed that allow discriminative learning of generative models. Support vector machine (SVM) is acknowledged as one of the pre-eminent discriminative approaches [16]-[18], and it has been successfully combined with GMM, such as the GMM-SVM [8],[9],[19]-[21]. Nevertheless, approaches based on GMM-SVM are unable to cope well with the channel effects [22],[23]. To compensate for the channel effects, it was shown using the joint factor analysis (JFA) technique that the speaker and channel variability can be confined as two disjoint subspaces in the parameter spaces of GMM [12],[24]. The word ‘joint’ refers to the fact that not only the speaker, but also the channel variability is treated in a single JFA model. However, it was been reported that the channel space obtained by the JFA does contain some residual speaker information [25].

Inspired by the JFA approach, it was shown in [13] that speaker and session variability can be represented by a single subspace referred to as the total variability space. The major motivation for defining such a subspace is to extract a low-dimensional identity vector (i.e., the so-called i-vector) from the feature sequence of a speech segment. The advantage of i-vector is that it represents a speech segment as a fixed-length vector instead of a variable-length sequence of acoustic features. This greatly simplifies the modeling and scoring processes in speaker verification. For instance, we can assume that the i-vector is generated from a Gaussian density [13] instead of the mixture of Gaussian densities as usual in the case of acoustic features [7]. In this regard, linear discriminant analysis (LDA) [13],[26],[27], nuisance attribute projection (NAP) [8],[13],[28], within-class covariance normalization (WCCN) [13],[29],[30], probabilistic LDA (PLDA) [10],[31], and the heavy-tailed PLDA [32] have shown to be effective for such fixed-length data. In this paper, we focus on PLDA with Gaussian prior instead of heavy-tailed prior. It was recently shown in [33] that the advantage of the heavy-tailed assumption diminishes with a simple length normalization on the i-vector before PLDA modeling.

Because the total variability matrix is always a low-rank rectangular matrix, a dimension reduction process is also imposed by the i-vector extractor [12]. In this study, we advocate the use of the uncompressed form of the i-vector. Similar to that in [13], our extractor converts speech sequence into a fixed-length vector but retains its dimensionality in the full supervector space. Modeling of speaker and session variability is then carried out using PLDA, which has shown to be effective in handling high-dimensional data. By doing so, we avoid reducing the dimensionality of the i-vector twice: first in the extraction process and second in the PLDA model. Any dimension reduction procedure will unavoidably discard information. Our intention is therefore to keep the full dimensionality until the scoring stage with PLDA and to investigate the performance of PLDA in the i-supervector space. We refer to the uncompressed form of i-vector as the i-supervector, or the identity supervector, following the nomenclature in [13],[29]. Similar to that in the i-vector extraction, the i-supervector is computed as the posterior mean of a latent variable, but with a much higher dimensionality.

The downside of using i-supervector with PLDA is that we have to deal with the inversion of large matrices. The size of the matrices becomes enormous when more sessions are available for each speaker in the development data^a. One option is to estimate the subspaces in a decoupled manner, which might lead to suboptimal solution [12],[24]. In [34], we showed that the joint estimation of subspaces can be accomplished by partitioning large matrices into smaller blocks, thereby making the inversion and the joint estimation feasible. In this study, we present the same approach with more detail and further refinement. We also look into various normalization methods and introduce the use of the Gaussianized rank-norm for the PLDA. In the experiments, we compare the performance of both i-vector and i-supervector under no normalization and various normalization conditions. Meanwhile, a fusion system that combines the i-vector and i-supervector is presented as well. In addition, we provide an analysis of the computational complexity associated with the i-vector and i-supervector at four different stages: loading matrix estimation, i-vector and i-supervector extraction, PLDA model training, and verification score calculation.

The paper is organized as follows. In Section 2, we introduce the i-vector paradigm, which includes the formulation of the i-vector and i-supervector and its relationship to the classical maximum a posteriori (MAP). Section 3 introduces the probabilistic LDA, where we show that the inversion of a large matrix in PLDA can be solved by exploiting some inherent structure of the precision matrix. Section 4 deals with PLDA scoring and introduces the Gaussianized rank-norm. We present some experimental results in Section 5 and conclude the paper in Section 6.

2 I-vector paradigm

2.1 I-vector extraction

The purpose of i-vector extraction is to represent variable-length utterances with fixed-length low-dimensional vectors. The fundamental assumption is that the feature vector sequence, $\mathcal{O}$, was generated from a session-specific GMM. Furthermore, the mean supervector obtained by stacking the means from all mixtures, m, is constrained to lie in a low-dimensional subspace with origin ℳ as follows:

$$\mathbf{m}=\mathrm{ℳ}+\mathbf{Tx},$$

(1)

where m and ℳ are the mean supervectors of the speaker (and session)-dependent GMM and the UBM, respectively. The subspace spanned by the columns of the matrix T captures the speaker and session variability, and hence the name total variability[13]. The weighted combination of the columns of T, as determined by the latent variable x, gives rise to the mean supervector m with ℳ as an additive factor.

The i-vector extraction process was formulated as the MAP estimation problem [13],[35]. Notice that in (1), the equation is concerned with the construction of the mean supervector m from the parameters $\left\{\begin{array}{cc}\hfill \mathrm{ℳ},\hfill & \hfill \mathbf{T}\hfill \end{array}\right\}$ and the latent variable x. The variable x is unobserved (or latent) as is the supervector. The optimal value of x is determined by the observed sequence $\mathcal{O}$ and is given by the mode (equivalent to the mean in the current case) of the posterior distribution of the latent variable x:

$$\mathit{\varphi }=\underset{\mathbf{x}}{argmax}\left[{\displaystyle \prod _{\mathit{c}=1}^{\mathit{C}}{\displaystyle \prod _{\mathit{t}=1}^{{\mathit{N}}_{\mathit{c}}}\mathcal{N}\left({\mathit{o}}_{\mathit{t}}|{\mathrm{ℳ}}_{\mathit{c}}+{\mathbf{T}}_{\mathit{c}}\mathbf{x},{\mathrm{\Phi }}_{\mathit{c}}\right)}}\right]\mathcal{N}\left(\mathbf{x}|0,\mathbf{I}\right).$$

(2)

The first point to note is that the latent variable $\mathbf{x}~\mathcal{N}\left(0,\mathbf{I}\right)$ is assumed to follow a standard normal prior. The parameters ℳ _c and Φ _c denote the mean vector and covariance matrix of the c-th mixture of the UBM, while N _c indicates the number of frames o _t aligned to each of the C mixtures. Also, we decompose the total variability matrix $\mathbf{T}={\left[\begin{array}{cccc}\hfill {\mathbf{T}}_1^{\mathrm{T}},\hfill & \hfill {\mathbf{T}}_2^{\mathrm{T}},\hfill & \hfill \dots ,\hfill & \hfill {\mathbf{T}}_{\mathit{C}}^{\mathrm{T}}\hfill \end{array}\right]}^{\mathrm{T}}$ to its component matrices, one associated with each Gaussian. Given an observation sequence $\mathcal{O}$, its i-vector representation is given by (2), the solution [13] of which is given by

$${\mathit{\varphi }}_{\mathit{x}}={\mathbf{L}}_{\mathit{x}}^{-1}\left({\displaystyle \sum _{\mathit{c}=1}^{\mathit{C}}}{\mathbf{T}}_{\mathit{c}}^{\mathrm{T}}{\mathrm{\Phi }}_{\mathit{c}}^{-1}{\mathbf{f}}_{\mathit{c}}\right),$$

(3)

where

$${\mathbf{L}}_{\mathit{x}}^{-1}={\left(\mathbf{I}+{\displaystyle \sum _{\mathit{c}=1}^{\mathit{C}}}{\mathit{N}}_{\mathit{c}}{\mathbf{T}}_{\mathit{c}}^{\mathrm{T}}{\mathrm{\Phi }}_{\mathit{c}}^{-1}{\mathbf{T}}_{\mathit{c}}\right)}^{-1}$$

(4)

is the posterior covariance, f _c = ∑ _t  γ_c,to _t − N _c ℳ _c is the centralized first-order statistics [35] for the c-th Gaussian, and γ_c,t denotes the occupancy of vector o _t to the c-th Gaussian. Since T is always a low-rank rectangular matrix, the dimension D of the i-vector is much smaller compared to that of the supervector, i.e., D ≪ C⋅F, where F is the dimensionality of the acoustic feature.

2.2 I-supervector extraction

Consider the case where the latent variable is allowed to grow into the full supervector space, for which D = C⋅F. One straightforward approach to achieving this is by using a CF-by-CF full matrix for T in (1). However, the number of parameters would be enormous, causing difficulty in the training. Another option is to impose a diagonal constraint on the loading matrix as follows:

$$\mathbf{m}=\mathrm{ℳ}+\mathbf{Dz},$$

(5)

where D is now a CF-by-CF diagonal matrix and the latent variable z has the same dimensionality as the mean supervector m. Similar to the variable x in (1), the variable z is unobserved. Given an observed sequence $\mathcal{O}$, we estimate the mode of the posterior distribution as follows:

$$\mathit{\varphi }=\underset{\mathbf{z}}{\mathrm{argmax}}\left[{\displaystyle \prod _{\mathit{c}=1}^{\mathit{C}}}{\displaystyle \prod _{\mathit{t}=1}^{{\mathit{N}}_{\mathit{c}}}}\mathcal{N}\left({\mathit{o}}_{\mathit{t}}|{\mathrm{ℳ}}_{\mathit{c}}+{\mathbf{D}}_{\mathit{c}}{\mathbf{z}}_{\mathit{c}},{\mathrm{\Phi }}_{\mathit{c}}\right)\mathcal{N}\left({\mathbf{z}}_{\mathit{c}}|0,\mathbf{I}\right)\right].$$

(6)

Here, z _c is the sub-vector of z and D _c is the F-by-F sub-matrix corresponding to the mixture c. Notice that such notations are necessary as the likelihood is computed over the acoustic vector o _t . Following the procedure as in [35], it can be shown that solution to (6) is a CF-by-1 supervector:

$${\mathit{\varphi }}_{\mathit{z}}={\mathbf{L}}_{\mathit{z}}^{-1}\left({\mathbf{D}}^{\mathrm{T}}{\mathbf{\Phi }}^{-1}\mathbf{f}\right),$$

(7)

where ${\mathbf{L}}_{\mathit{z}}^{-1}$ is a CF-by-CF diagonal matrix given by

$${\mathbf{L}}_{\mathit{z}}^{-1}={\left(\mathbf{I}+{\mathbf{D}}^{\mathrm{T}}\mathbf{N}{\mathbf{\Phi }}^{-1}\mathbf{D}\right)}^{-1}.$$

(8)

In (7), f is the CF-by-1 supervector obtained by concatenating the f _c from all mixtures (see Figure 1). In (8), N is the CF-by-CF diagonal matrix whose diagonal blocks are N _c I, and Φ is a block diagonal matrix with Φ _c at its diagonal. Recall that N _c and f _c are the occupancy count and centralized first-order statistics extracted using the UBM.

Figure 1

Comparison between i-vector (upper panel) and i-supervector (lower panel) extraction followed by PLDA modeling.

We refer to ϕ _z as the i-supervector analogous to the i-vector since ϕ _z is computed as the posterior mean of a latent variable similar to that in the i-vector extraction, but with a much higher dimensionality. It is worth to note that there exist some subtle differences between the i-supervector extraction and the classical MAP estimation of GMM [36]. In particular, the so-called relevance MAP widely used in the GMM-UBM [7] could be formulated in similar notations. In particular, the mean supervector of the adapted GMM is given by

$${\mathit{\varphi }}_{\mathrm{rel}}=\mathrm{ℳ}+{\left(\mathit{\tau }\mathbf{I}+\mathbf{N}\right)}^{-1}\mathbf{f}.$$

(9)

One could deduce (9) from (7) and (8) by setting D^TD = τ⁻¹Σ and using the results in (5). The parameter τ is referred to as the relevance factor, which is set empirically in the range between 8 and 16 [7]. This is different from that in (7), where the matrix D is trained from a dataset using the EM algorithm in a manner similar to the matrix T for the i-vector. Secondly, the i-supervector is taken as the posterior of the latent variable z which is absent in the relevance MAP formulation.

The i-supervector extractor can be implemented by adopting the diagonal modeling part of the JFA [12],[24] with a slight modification: the diagonal model D is trained per utterance instead of per speaker basis in order to capture both speaker and session variability. Figure 2 summarizes the EM steps. The diag(.) operator sets the off-diagonal elements to zeros, only the diagonal elements are computed in our implementation. Notice that the sufficient statistics {f, N} are session-dependent. We omitted the session index for simplicity.

Figure 2

The EM steps for estimating the loading matrix D for the i-supervector extractor.

2.3 From i-vector to i-supervector

The i-vector extraction is formulated in probabilistic terms based on a latent variable model as in (2), similarly for the case of i-supervector in (6). One obvious benefit is that in addition to obtain the i-vector as the posterior mean ϕ _x of the latent variable x, we could also compute the posterior covariance (4) which quantifies the uncertainty of the estimate and fold in the information in subsequent modeling [37]. Nevertheless, any form of dimension reduction would unavoidably discard information. Following the same latent variable modeling paradigm, we proposed the i-supervector as an uncompressed form of i-vector representation.

Figure 1 compares the i-vector and i-supervector approaches from the extraction process to the subsequent PLDA modeling (recall that the parameter C denotes the number of mixtures in the UBM. F is the size of the acoustic feature vectors. D is the length of the i-vector while the i-supervector has a much higher dimensionality of C·F.). The biggest difference is that there are two rounds of dimension reduction which occurred in the i-vector PLDA system, whereas there is only one time reduction for the case of i-supervector PLDA. In this paper, our motivation is to keep the full dimensionality of the supervector as the input to the PLDA model which has shown to be an efficient model for high-dimensional data [10]. We envisage that more information would be preserved via the use of i-supervector, which could be exploited with the use of PLDA.

3 PLDA modeling in i-supervector space

3.1 Probabilistic LDA

The i-vector and the i-supervector represent a speech segment as a fixed-length vector instead of a variable-length sequence of vectors. Taking the fixed-length vector ϕ _ij as input, PLDA assumes that it is generated from a Gaussian density as follows:

$$\mathit{p}\left({\mathit{\varphi }}_{\mathit{ij}}\right)=\mathcal{N}\left({\mathit{\varphi }}_{\mathit{ij}}|\mathbf{\mu },\mathbf{F}{\mathbf{F}}^{\mathrm{T}}+\mathbf{G}{\mathbf{G}}^{\mathrm{T}}+\mathbf{\Sigma }\right),$$

(10)

where μ denotes the global mean and Γ = FF^T + GG^T + Σ is the covariance matrix. Here, ϕ _ij is i-supervector (or i-vector) representing the j-th session of the i-th speaker. We use ϕ referring either to the i-vector ϕ _x or i-supervector ϕ _z in the subsequent discussion.

The strength of PLDA lies at the modeling of the covariance Γ in structural form as FF^T + GG^T + Σ. To see this, we rewrite (10) as marginal density:

$$\mathit{p}\left({\mathit{\varphi }}_{\mathit{ij}}\right)={\displaystyle \iint }\mathit{p}\left({\mathit{\varphi }}_{\mathit{ij}}|{\mathbf{h}}_{\mathit{i}},{\mathbf{w}}_{\mathit{ij}}\right)\mathcal{N}\left({\mathbf{h}}_{\mathit{i}}|0,\mathbf{I}\right)\mathcal{N}\left({\mathbf{w}}_{\mathit{ij}}|0,\mathbf{I}\right)\mathit{d}{\mathbf{h}}_{\mathit{i}}\mathit{d}{\mathbf{w}}_{\mathit{ij}},$$

(11)

where the conditional density is given by

$$\mathit{p}\left({\mathit{\varphi }}_{\mathit{ij}}|{\mathbf{h}}_{\mathit{i}},{\mathbf{w}}_{\mathit{ij}}\right)=\mathcal{N}\left(\mathit{\varphi }|\mathbf{\mu }+\mathbf{F}{\mathbf{h}}_{\mathit{i}}+\mathbf{G}{\mathbf{w}}_{\mathit{ij}},\mathbf{\Sigma }\right).$$

(12)

In the above equations, h _i is the speaker-specific latent variable pertaining to the i-th speaker, while w _ij is the session-specific latent variable corresponding to the j-th session of the i-th speaker. Both latent variables are assumed to follow a standard Gaussian prior. The low-rank matrices F and G model the subspaces corresponding to speaker and session variability (we denote their rank as N _F and N _G , respectively), while the diagonal matrix Σ covers the remaining variation. From (12), the mean vector of the conditional distribution is given by

$${\mathbf{\mu }}_{\mathit{ij}}=\mathbf{\mu }+\left[\begin{array}{cc}\hfill \mathbf{F},\hfill & \hfill \mathbf{G}\hfill \end{array}\right]\left[\begin{array}{c}\hfill {\mathbf{h}}_{\mathit{i}}\hfill \\ \hfill {\mathbf{w}}_{\mathit{ij}}\hfill \end{array}\right].$$

(13)

Comparing (1) and (13), we see that both i-vector extraction process and the PLDA model involve dimension reduction via a similar form of subspace modeling. This observation motivates us to explore the use of PLDA on i-supervector. The extraction process serves as the front-end which converts a variable-length sequence $\mathcal{O}$ to a fixed-length vector without reducing the dimension. Speaker modeling and channel compensation are then carried out in the original supervector space.

The downside of using i-supervector with PLDA is that we have to deal with large matrices as illustrated in the lower panel of Figure 1. The size of the matrices becomes enormous when more sessions are available for each speaker in the development data. This is typically the case for speaker recognition where the number of utterances per speaker is usually in the range from ten to over a hundred [38],[39]. In the following, we estimate the parameters $\left\{\begin{array}{cccc}\hfill \mathbf{\mu },\hfill & \hfill \mathbf{F},\hfill & \hfill \mathbf{G},\hfill & \hfill \mathbf{\Sigma }\hfill \end{array}\right\}$ of the PLDA model using the expectation maximization (EM) algorithm. We show how large matrices could be partitioned into sub-matrices, thereby making the matrix inversion and EM steps feasible.

3.2 E-step: joint estimation of posterior means

We assume that our development set consists of speech samples from N speakers each having J sessions, though the number of sessions J could be different for each speaker. All the J observations from the i-th speaker are collated to form the compound system [10]:

$$\underset{{\tilde{\mathit{\varphi }}}_{\mathit{i}}}{\underbrace{\left[\begin{array}{c}\hfill {\mathit{\varphi }}_{\mathit{i}1}\hfill \\ \hfill {\mathit{\varphi }}_{\mathit{i}2}\hfill \\ \hfill ⋮\hfill \\ \hfill {\mathit{\varphi }}_{\mathit{iJ}}\hfill \end{array}\right]}}=\underset{\tilde{\mathbf{\mu }}}{\underbrace{\left[\begin{array}{c}\hfill \mathbf{\mu }\hfill \\ \hfill \mathbf{\mu }\hfill \\ \hfill ⋮\hfill \\ \hfill \mathbf{\mu }\hfill \end{array}\right]}}+\underset{\tilde{\mathbf{A}}}{\underbrace{\left[\begin{array}{ccccc}\hfill \mathbf{F}\hfill & \hfill \mathbf{G}\hfill & \hfill 0\hfill & \hfill \cdots \hfill & \hfill 0\hfill \\ \hfill \mathbf{F}\hfill & \hfill 0\hfill & \hfill \mathbf{G}\hfill & \hfill \cdots \hfill & \hfill 0\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill \mathbf{F}\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill \cdots \hfill & \hfill \mathbf{G}\hfill \end{array}\right]}}\underset{{\mathbf{y}}_{\mathit{i}}}{\underbrace{\left[\begin{array}{c}\hfill {\mathbf{h}}_{\mathit{i}}\hfill \\ \hfill {\mathbf{w}}_{\mathit{i}1}\hfill \\ \hfill {\mathbf{w}}_{\mathit{i}2}\hfill \\ \hfill ⋮\hfill \\ \hfill {\mathbf{w}}_{\mathit{iJ}}\hfill \end{array}\right]}}+\underset{{\mathbf{\epsilon }}_{\mathit{i}}}{\underbrace{\left[\begin{array}{c}\hfill {\mathbf{\epsilon }}_{\mathit{i}1}\hfill \\ \hfill {\mathbf{\epsilon }}_{\mathit{i}2}\hfill \\ \hfill ⋮\hfill \\ \hfill {\mathbf{\epsilon }}_{\mathit{iJ}}\hfill \end{array}\right]}}.$$

(14)

Each row in (14) says that each observation ϕ _ij = μ + Fh _i + Gw _ij + ε _ij consists of a speaker-dependent component μ + Fh _i and session-dependent component Gw _ij + ε _ij , where ${\mathbf{\epsilon }}_{\mathit{ij}}~\mathcal{N}\left(0,\mathbf{\Sigma }\right)$ is responsible for the residual variation in (10). In the E-step, we infer the posterior mean of the compound latent variable ${\mathbf{y}}_{\mathit{i}}={\left[\begin{array}{cccc}\hfill {\mathbf{h}}_{\mathit{i}}^{\mathrm{T}},\hfill & \hfill {\mathbf{w}}_{\mathit{i}1}^{\mathrm{T}},\hfill & \hfill \dots ,\hfill & \hfill {\mathbf{w}}_{\mathit{iJ}}^{\mathrm{T}}\hfill \end{array}\right]}^{\mathrm{T}}$ as follows:

$$\mathit{E}\left\{{\mathbf{y}}_{\mathit{i}}\right\}={\mathbf{L}}^{-1}\cdot {\widetilde{\mathbf{{\rm A}}}}^{\mathrm{T}}{\tilde{\mathbf{\Sigma }}}^{-1}\left({\tilde{\mathit{\varphi }}}_{\mathit{i}}-\mathbf{\mu }\right),$$

(15)

where $\tilde{\mathbf{\Sigma }}$ is a block diagonal matrix whose diagonal blocks are Σ, and L⁻¹ is the posterior covariance given by

$${\mathbf{L}}^{-1}={\left[{\widetilde{\mathbf{{\rm A}}}}^{\mathrm{T}}{\tilde{\mathbf{\Sigma }}}^{-1}\widetilde{\mathbf{{\rm A}}}+\mathbf{I}\right]}^{-1}.$$

(16)

The posterior inference involves the inversion of the matrix $={\widetilde{\mathbf{{\rm A}}}}^{\mathrm{T}}{\tilde{\mathbf{\Sigma }}}^{-1}\widetilde{\mathbf{{\rm A}}}+\mathbf{I}$. Following the notations in (14), we could express the matrix inversion as

$${\mathbf{L}}^{-1}={\left[\begin{array}{ccccc}\hfill \mathit{J}{\mathbf{F}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{F}+\mathbf{I}\hfill & \hfill {\mathbf{F}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{G}\hfill & \hfill {\mathbf{F}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{G}\hfill & \hfill \cdots \hfill & \hfill {\mathbf{F}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{G}\hfill \\ \hfill {\mathbf{G}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{F}\hfill & \hfill {\mathbf{G}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{G}+\mathbf{I}\hfill & \hfill 0\hfill & \hfill \cdots \hfill & \hfill 0\hfill \\ \hfill {\mathbf{G}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{F}\hfill & \hfill 0\hfill & \hfill {\mathbf{G}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{G}+\mathbf{I}\hfill & \hfill \cdots \hfill & \hfill 0\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill {\mathbf{G}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{F}\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill \cdots \hfill & \hfill {\mathbf{G}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{G}+\mathbf{I}\hfill \end{array}\right]}^{-1}.$$

(17)

The matrix is large as we consider the joint inference of latent variables $\left\{\begin{array}{cccc}\hfill {\mathbf{h}}_{\mathit{i}},\hfill & \hfill {\mathbf{w}}_{\mathit{i}1},\hfill & \hfill \dots ,\hfill & \hfill {\mathbf{w}}_{\mathit{iJ}}\hfill \end{array}\right\}$ representing a speaker and all sessions from the same speaker. The size of the matrix increases with the number of sessions J, while more sessions are always desirable for more robust parameter estimation. Direct inversion of the matrix becomes intractable.

The precision matrix L possesses a unique structure since all sessions from the same speakers are tied to one speaker-specific latent variable. As depicted in (17) and (18), the matrix L can be partitioned into four sub-matrices: A, B, B^T, and C. Using the partitioned inverse formula[40], the inverse of the matrix L could be obtained as

$${\mathbf{L}}^{-1}={\left[\begin{array}{cc}\hfill \mathbf{A}\hfill & \hfill \mathbf{B}\hfill \\ \hfill {\mathbf{B}}^{\mathrm{T}}\hfill & \hfill \mathbf{C}\hfill \end{array}\right]}^{-1}=\left[\begin{array}{cc}\hfill \mathbf{M}\hfill & \hfill -\mathbf{MB}{\mathbf{C}}^{-1}\hfill \\ \hfill -{\mathbf{C}}^{-1}{\mathbf{B}}^{\mathrm{T}}\mathbf{M}\hfill & \hfill {\mathbf{C}}^{-1}+{\mathbf{C}}^{-1}{\mathbf{B}}^{\mathrm{T}}\mathbf{MB}{\mathbf{C}}^{-1}\hfill \end{array}\right],$$

(18)

Where

$$\mathbf{M}={\left(\mathbf{A}-\mathbf{B}{\mathbf{C}}^{-1}{\mathbf{B}}^{\mathrm{T}}\right)}^{-1}.$$

(19)

The matrix M⁻¹ is known as the Schur complement of L with respect to C[18]. Using these formulae, there are still two matrices to be inverted. The first is C⁻¹ in the left-hand side of (18) and the second is the M in (19). The inversion C⁻¹ is simple as C is block diagonal, where the inversion Q = (G^TΣ⁻¹G + I)⁻¹ can be computed directly from the N _G -by-N _G matrix. Using the notations in (14) and (18), M is given by

$$\mathbf{M}={\left[\mathit{J}{\mathbf{F}}^{\mathrm{T}}\mathbf{JF}+\mathbf{I}\right]}^{-1},$$

(20)

where J is obtained via the matrix inversion lemma:

$$\mathbf{J}={\left[\mathbf{G}{\mathbf{G}}^{\mathrm{T}}+\mathbf{\Sigma }\right]}^{-1}={\mathbf{\Sigma }}^{-1}-{\mathbf{\Sigma }}^{-1}\mathbf{G}{\left({\mathbf{G}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{G}+\mathbf{I}\right)}^{-1}{\mathbf{G}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}.$$

(21)

Using (18) in (15), it can be shown that the posterior mean of the speaker-specific latent variable h _i is given by

$$\mathit{E}\left\{{\mathbf{h}}_{\mathit{i}}\right\}=\mathbf{M}\left[{\mathbf{F}}^{\mathrm{T}}\mathbf{J}\cdot {\displaystyle \sum _{\mathit{j}=1}^{\mathit{J}}}\left({\mathit{\varphi }}_{\mathit{ij}}-\mathbf{\mu }\right)\right],$$

(22)

while the session-specific posterior mean of w _ij could be inferred as

$$\mathit{E}\left\{{\mathbf{w}}_{\mathit{ij}}\right\}=\underset{\mathit{Q}}{\underbrace{{\left({\mathbf{G}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{G}+\mathbf{I}\right)}^{-1}}}{\mathbf{G}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\left[{\mathit{\varphi }}_{\mathit{ij}}-\mathbf{\mu }-\mathbf{F}\cdot \mathit{E}\left\{{\mathbf{h}}_{\mathit{i}}\right\}\right].$$

(23)

One interesting point to note from (23) is that the i-supervector ϕ _ij is first centralized to the global mean μ and the speaker mean F⋅E{h _i } before projection to the session variability space.

From computation perspective, the matrices Q = (G^TΣ⁻¹G + I)⁻¹, Λ = Q⋅G^TΣ⁻¹F, and J could be pre-computed and used for all sessions and speakers in the E-step. The matrix M depends on the number of sessions J per speaker. In the event where J is different for each speaker (which is usually the case), we compute

$${\mathbf{M}}_{\mathit{J}}\equiv {\left[\mathit{J}\cdot {\mathbf{F}}^{\mathrm{T}}\mathbf{JF}+\mathbf{I}\right]}^{-1}=\mathbf{V}{\left[\mathit{J}\cdot \mathbf{E}+\mathbf{I}\right]}^{-1}{\mathbf{V}}^{\mathrm{T}},$$

(24)

where F^TJF = VEV^T is obtained via eigenvalue decomposition in which V is the square matrix of eigenvectors and is E the diagonal matrix of eigenvalues.

3.3 M-step: model estimation

The M-step can also be formulated in terms of sub-matrices. Let ${\tilde{\mathbf{w}}}_{\mathit{ij}}={\left[\begin{array}{cc}\hfill {\mathbf{h}}_{\mathit{i}}^{\mathrm{T}},\hfill & \hfill {\mathbf{w}}_{\mathit{ij}}^{\mathrm{T}}\hfill \end{array}\right]}^{\mathit{{\rm T}}}$ be a compound vector by appending h _i to each session w _ij belonging to the same speaker. We update the loading matrices F and G jointly as follows:

$$\left[\begin{array}{cc}\hfill \widehat{\mathbf{F}},\hfill & \hfill \widehat{\mathbf{G}}\hfill \end{array}\right]=\left\{{\displaystyle \sum _{\mathit{i}=1}^{\mathit{N}}}{\displaystyle \sum _{\mathit{j}=1}^{\mathit{J}}}\left({\mathit{\varphi }}_{\mathit{ij}}-\mathbf{\mu }\right)\mathit{E}\left\{{\tilde{\mathbf{w}}}_{\mathit{ij}}^{\mathrm{T}}\right\}\right\}{\left\{{\displaystyle \sum _{\mathit{i}=1}^{\mathit{N}}}{\displaystyle \sum _{\mathit{j}=1}^{\mathit{J}}}\mathit{E}\left\{{\tilde{\mathbf{w}}}_{\mathit{ij}}{\tilde{\mathbf{w}}}_{\mathit{ij}}^{\mathrm{T}}\right\}\right\}}^{-1},$$

(25)

where $\mathit{E}\left\{{\tilde{\mathbf{w}}}_{\mathit{ij}}^{\mathrm{T}}\right\}$ could be obtained by concatenating the results from (22) and (23). The second moment $\mathit{E}\left\{{\tilde{\mathbf{w}}}_{\mathit{ij}}{\tilde{\mathbf{w}}}_{\mathit{ij}}^{\mathrm{T}}\right\}$ is computed for each individual session and speaker as follows:

$$\mathit{E}\left\{{\tilde{\mathbf{w}}}_{\mathit{ij}}{\tilde{\mathbf{w}}}_{\mathit{ij}}^{\mathrm{T}}\right\}=\left[\begin{array}{cc}\hfill \mathbf{M}\hfill & \hfill -\mathbf{M}{\mathrm{\Lambda }}^{\mathrm{{\rm T}}}\hfill \\ \hfill -\mathrm{\Lambda }\mathbf{M}\hfill & \hfill \mathit{Q}+\mathrm{\Lambda }\mathbf{M}{\mathrm{\Lambda }}^{\mathrm{{\rm T}}}\hfill \end{array}\right]+\mathit{E}\left\{{\tilde{\mathbf{w}}}_{\mathit{ij}}\right\}\left\{{\tilde{\mathbf{w}}}_{\mathit{ij}}^{\mathrm{T}}\right\}$$

(26)

The covariance matrix of the PLDA model could then be updated as

$$\widehat{\mathbf{\Sigma }}=\frac{1}{\mathit{N}\cdot \mathit{J}}{\displaystyle \sum _{\mathit{i}=1}^{\mathit{N}}}{\displaystyle \sum _{\mathit{j}=1}^{\mathit{J}}}\mathrm{diag}\left[\left({\mathit{\varphi }}_{\mathit{ij}}-\mathbf{\mu }\right){\left({\mathit{\varphi }}_{\mathit{ij}}-\mathbf{\mu }\right)}^{\mathrm{T}}-\left[\begin{array}{cc}\hfill \widehat{\mathbf{F}},\hfill & \hfill \widehat{\mathbf{G}}\hfill \end{array}\right]\mathit{E}\left\{{\tilde{\mathbf{w}}}_{\mathit{ij}}\right\}{\left({\mathit{\varphi }}_{\mathit{ij}}-\mathbf{\mu }\right)}^{\mathrm{T}}\right],$$

(27)

where the operator diag(⋅) diagonalizes a matrix by setting the off-diagonal elements to zeros.

4 Likelihood ratio computation

4.1 Model comparison

Speaker verification is a binary classification problem, where a decision has to be made between two hypotheses with respect to a decision threshold. The null hypothesis H₀ says that the test segment is from the target speaker, while the alternative H₁ hypothesizes the opposite. Using the latent variable modeling approach with PLDA, H₀ and H₁ correspond to the models as shown in Figure 3. In the model H₀, {ϕ₁, ϕ₂} belong to the same speaker and hence share the same speaker-specific latent variable h_1,2. On the other hand, {ϕ₁, ϕ₂} belong to different speakers and hence have separate latent variables, h₁ and h₂, in the model H₁. The verification score is calculated as the log-likelihood ratio between two models:

$$\mathit{s}\left({\mathit{\varphi }}_1,{\mathit{\varphi }}_2\right)=log\mathit{p}\left({\mathit{\varphi }}_1,{\mathit{\varphi }}_2|{\mathit{H}}_0\right)-log\mathit{p}\left({\mathit{\varphi }}_1,{\mathit{\varphi }}_2|{\mathit{H}}_1\right),$$

(28)

where the likelihood terms are evaluated using (10) (we shall give more details in the next section). One key feature of the PLDA scoring function in (28) is that no speaker model is built or trained. The verification scores are computed by comparing the likelihood of two different models which describe the relationship between the training and test i-supervectors (or i-vector) through the use of PLDA model.

Figure 3

Using PLDA for verification task.

4.2 PLDA verification score

To solve for (28), we first recognize from Figure 1 that the generative equation for the model H₀ is given by

$$\underset{\tilde{\mathit{\varphi }}}{\underbrace{\left[\begin{array}{l}{\mathit{\varphi }}_1\\ {\mathit{\varphi }}_2\end{array}\right]}}=\underset{\mathbf{\mu }}{\underbrace{\left[\begin{array}{l}\mathbf{\mu }\\ \mathbf{\mu }\end{array}\right]}}+\underset{\tilde{\mathbf{A}}}{\underbrace{\left[\begin{array}{ccc}\hfill \mathbf{F}\hfill & \hfill \mathbf{G}\hfill & \hfill 0\hfill \\ \hfill \mathbf{F}\hfill & \hfill 0\hfill & \hfill \mathbf{G}\hfill \end{array}\right]}}\left[\begin{array}{c}\hfill {\mathbf{h}}_{12}\hfill \\ \hfill {\mathbf{w}}_1\hfill \\ \hfill {\mathbf{w}}_2\hfill \end{array}\right]+\left[\begin{array}{c}\hfill {\mathit{\epsilon }}_1\hfill \\ \hfill {\mathit{\epsilon }}_2\hfill \end{array}\right].$$

(29)

Using the compound form of (29) in (10), we compute the log-likelihood of the model H₀ by

$$\begin{array}{c}\hfill \mathit{logp}\left({\mathit{\varphi }}_1,{\mathit{\varphi }}_2|{\mathit{H}}_0\right)=\mathit{logN}\left(\left.\tilde{\mathit{\varphi }}\right|\tilde{\mathbf{\mu }},\tilde{\mathbf{A}}{\tilde{\mathbf{A}}}^{\mathrm{T}}+\tilde{\sum }\right)\hfill \\ \hfill =-\frac{1}{2}{\left(\tilde{\mathit{\varphi }}-\tilde{\mathbf{\mu }}\right)}^{\mathrm{T}}{\left(\tilde{\mathbf{A}}{\tilde{\mathbf{A}}}^{\mathrm{T}}+\tilde{\sum }\right)}^{-1}\left(\tilde{\mathit{\varphi }}-\tilde{\mathbf{\mu }}\right)-\frac{1}{2}\mathit{log}\left|\tilde{\mathbf{A}}{\tilde{\mathbf{A}}}^{\mathrm{T}}+\tilde{\sum }\right|-\mathit{\alpha log}\left(2\mathit{\pi }\right),\hfill \end{array}$$

(30)

where α = C⋅F for the case of i-supervector while α = D for the case of i-vector. To evaluate the log-likelihood function, we have to solve for the inversion and log-determinant of the following covariance matrix:

$$\left(\widetilde{\mathbf{{\rm A}}}{\widetilde{\mathbf{{\rm A}}}}^{\mathrm{T}}+\tilde{\mathbf{\Sigma }}\right)=\left[\begin{array}{c}\hfill \mathbf{F}\hfill \\ \hfill \mathbf{F}\hfill \end{array}\right]\left[\begin{array}{cc}\hfill {\mathbf{F}}^{\mathrm{T}}\hfill & \hfill {\mathbf{F}}^{\mathrm{T}}\hfill \end{array}\right]+\left[\begin{array}{cc}\hfill \mathbf{G}{\mathbf{G}}^{\mathrm{T}}+\mathbf{\Sigma }\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \mathbf{G}{\mathbf{G}}^{\mathrm{T}}+\mathbf{\Sigma }\hfill \end{array}\right].$$

(31)

The inversion of the above matrix could be obtained by applying twice the matrix inversion lemma. In particular, we first compute J = (GG^T + Σ)⁻¹, the result of which is given by (21), and apply again the matrix inversion lemma on the right-hand side of (31), which leads to

$${\left(\widetilde{\mathbf{{\rm A}}}{\widetilde{\mathbf{{\rm A}}}}^{\mathrm{T}}+\tilde{\mathbf{\Sigma }}\right)}^{-1}=\left[\begin{array}{cc}\hfill \mathbf{J}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \mathbf{J}\hfill \end{array}\right]-\left[\begin{array}{c}\hfill \mathbf{JF}\hfill \\ \hfill \mathbf{JF}\hfill \end{array}\right]\times {\mathbf{M}}_2\times \left[\begin{array}{cc}\hfill {\mathbf{F}}^{\mathrm{T}}\mathbf{J},\hfill & \hfill {\mathbf{F}}^{\mathrm{T}}\mathbf{J}\hfill \end{array}\right],$$

(32)

where M₂ is computed using the solution in (24) by setting J = 2. Now, to solve for the log-determinant of the same matrix in (31), we apply twice the matrix determinant lemma in a much similar way as the matrix inversion. Taking the log of the result leads to

$$\mathit{log}\left|\widetilde{\mathbf{{\rm A}}}{\widetilde{\mathbf{{\rm A}}}}^{\mathrm{T}}+\tilde{\mathbf{\Sigma }}\right|=-2\mathit{log}\left|\mathbf{J}\right|-\mathit{log}\left|{\mathbf{M}}_2\right|.$$

(33)

Using (32) and (33) in (30), we arrive at

$$\begin{array}{c}log\mathit{p}\left({\mathit{\varphi }}_1,{\mathit{\varphi }}_2\left|{\mathit{H}}_0\right.\right)=\frac{1}{2}{\left[{\displaystyle {\sum }_{\mathit{l}=1}^2{\mathbf{F}}^{\mathrm{T}}\mathbf{J}\left({\mathit{\varphi }}_{\mathit{l}}-\mathbf{\mu }\right)}\right]}^{\mathrm{T}}{\mathbf{M}}_2\left[{\displaystyle {\sum }_{\mathit{l}=1}^2{\mathbf{F}}^{\mathrm{T}}\mathbf{J}\left({\mathit{\varphi }}_{\mathit{l}}-\mathbf{\mu }\right)}\right]\\ -\frac{1}{2}{\displaystyle {\sum }_{\mathit{l}=1}^2{\left({\mathit{\varphi }}_{\mathit{l}}-\mathbf{\mu }\right)}^{\mathrm{T}}}\mathrm{J}\left({\mathit{\varphi }}_{\mathit{l}}-\mathbf{\mu }\right)+\frac{1}{2}log\left|{\mathbf{M}}_2\right|+\log \left|\mathbf{J}\right|-\mathit{\alpha }\log \left(2\mathit{\pi }\right).\end{array}$$

(34)

For the alternative hypothesis H₁, we form the following compound equation:

$$\left[\begin{array}{l}{\mathit{\varphi }}_1\\ {\mathit{\varphi }}_2\end{array}\right]=\left[\begin{array}{l}\mathbf{\mu }\\ \mathbf{\mu }\end{array}\right]+\left[\begin{array}{cccc}\hfill \mathbf{F}\hfill & \hfill \mathbf{G}\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill \mathbf{F}\hfill & \hfill \mathbf{G}\hfill \end{array}\right]\left[\begin{array}{c}\hfill {\mathbf{h}}_1\hfill \\ \hfill {\mathbf{w}}_1\hfill \\ \hfill {\mathbf{h}}_2\hfill \\ \hfill {\mathbf{w}}_2\hfill \end{array}\right]+\left[\begin{array}{c}\hfill {\mathit{\epsilon }}_1\hfill \\ \hfill {\mathit{\epsilon }}_2\hfill \end{array}\right].$$

(35)

The first thing to note is that the first and second rows of the system are decoupled and therefore could be treated separately. The log-likelihood of the alternative hypothesis H₁ is therefore given by the following sum of log-likelihoods:

$$\mathit{log}\mathit{p}\left({\mathit{\varphi }}_1,{\mathit{\varphi }}_2|{\mathit{H}}_1\right)={\displaystyle {\sum }_{\mathit{l}=1}^2\mathit{log}\mathcal{N}\left({\mathit{\varphi }}_{\mathit{l}}|\mathbf{\mu },\mathbf{F}{\mathbf{F}}^{\mathrm{T}}+\mathbf{G}{\mathbf{G}}^{\mathrm{T}}+\mathbf{\Sigma }\right)}.$$

(36)

Using a similar approach as for the case of the null hypothesis, it can be shown that the solution to (36) is given by

$$\begin{array}{c}log\mathit{p}\left({\mathit{\varphi }}_1,{\mathit{\varphi }}_2|{\mathit{H}}_1\right)=\frac{1}{2}{\displaystyle {\sum }_{\mathit{l}=1}^2{\left[{\mathbf{F}}^{\mathrm{T}}\mathbf{J}\left({\mathit{\varphi }}_{\mathit{l}}-\mathbf{\mu }\right)\right]}^{\mathrm{T}}}{\mathbf{M}}_1\left[{\mathbf{F}}^{\mathrm{T}}\mathbf{J}\left({\mathit{\varphi }}_{\mathit{l}}-\mathbf{\mu }\right)\right]\\ -\frac{1}{2}{\displaystyle {\sum }_{\mathit{l}=1}^2{\left({\mathit{\varphi }}_{\mathit{l}}-\mathbf{\mu }\right)}^{\mathrm{T}}}\mathbf{J}\left({\mathit{\varphi }}_{\mathit{l}}-\mathbf{\mu }\right)+log\left|{\mathbf{M}}_1\right|+log\left|\mathbf{J}\right|-\mathit{\alpha }log\left(2\mathit{\pi }\right)\end{array}$$

(37)

Using (34) and (37) in (28), canceling out common terms, we arrive at the following log-likelihood ratio score for the verification task:

$$\mathit{s}\left({\mathit{\varphi }}_1,{\mathit{\varphi }}_2\right)=\frac{1}{2}\left[{\displaystyle {\sum }_{\mathit{l}=1}^2{\mathit{\phi }}_{\mathit{l}}^{\mathrm{T}}}\right]{\mathbf{M}}_2\left[{\displaystyle {\sum }_{\mathit{l}=1}^2{\mathit{\phi }}_{\mathit{l}}}\right]-\frac{1}{2}{\displaystyle {\sum }_{\mathit{l}=1}^2{\mathit{\phi }}_{\mathit{l}}^{\mathrm{T}}{\mathbf{M}}_1}{\mathit{\phi }}_{\mathit{l}}+\mathit{K}.$$

(38)

For brevity of notations, we have let

$${\mathit{\phi }}_{\mathit{l}}={\mathbf{F}}^{\mathrm{T}}\mathbf{J}\left({\mathit{\varphi }}_{\mathit{l}}-\mathbf{\mu }\right).$$

(39)

One way to look at (39) is that it centralizes the vector ϕ _l and projects it onto the subspace F where speaker information co-vary the most (i.e., dimension reduction) while de-emphasizing the subspace pertaining to channel variability. In (38), K = log|M₂|/2 − log|M₁| is constant for the given set of parameters $\left\{\begin{array}{ccc}\hfill \mathbf{F},\hfill & \hfill \mathbf{G},\hfill & \hfill \mathbf{\Sigma }\hfill \end{array}\right\}$. Though K diminishes when score normalization is applied, we could calculate the two log-determinant terms easily by using the property of eigenvalue decomposition. In particular, we compute log|M₂| as $-{\displaystyle {\sum }_{\mathit{n}=1}^{{\mathit{N}}_{\mathit{F}}}\mathit{log}\left(2{\mathit{\lambda }}_{\mathit{n}}+1\right)}$ and log|M₁| as $-{\displaystyle {\sum }_{\mathit{n}=1}^{{\mathit{N}}_{\mathit{F}}}\mathit{log}\left({\mathit{\lambda }}_{\mathit{n}}+1\right)}$, where {λ _n : n = 1, 2,…, N _F } are the eigenvalues of the matrix F^TJF (c.f. (24)).

4.3 I-supervector pre-conditioning

Another prerequisite for good performance with PLDA is that the i-supervectors have to follow a normal distribution, as in (10). It has been shown in [33], for the case of i-vector, that whitening followed by length normalization helps toward this goal. However, whitening can never be possible for i-supervector due to data scarcity. To this end, we advocate the use of a Gaussianized version of rank norm [34],[41]. The i-supervector is processed element-wise with warping functions mapping each dimension to a standard Gaussian distribution (instead of uniform distribution as in rank norm). To put it mathematically, let ϕ _l (m), for m = 1, 2,…, CF, denote the elements of the i-supervector ϕ _l . We first get the normalized rank of ϕ _l (m) with respect to a background set B _m as follows:

$${\mathit{r}}_{\mathit{m}}=\frac{\left|\left\{\mathit{b}\in {\mathit{B}}_{\mathit{m}}:\mathit{b}<{\mathit{\varphi }}_{\mathit{l}}\left(\mathit{m}\right)\right\}\right|}{\left|{\mathit{B}}_{\mathit{m}}\right|},$$

(40)

where |⋅| denotes the cardinality of a set. The Gaussianized value is then obtained by using the inverse cumulative density function (CDF) of a standard Gaussian distribution (i.e., the probit function) as follows:

$${\mathit{\varphi }}_{\mathit{l}}\left(\mathit{m}\right)\leftarrow \sqrt{2}{\mathrm{erf}}^{-1}\left(2{\mathit{r}}_{\mathit{m}}-1\right),$$

(41)

where erf⁻¹(⋅) denotes the inverse error function. This can then be followed by length normalization prior to PLDA modeling.

5 Experiment

5.1 Experimental setup

Experiments were carried out on the core task (short2-short3) of NIST SRE08 [42]. We use two well-known metrics in evaluating the performance, namely, equal error rate (EER) and minimum detection cost (MinDCF). Two gender-dependent UBMs consisting of 512 Gaussians were trained using data drawn from the SRE04. Speech parameters were represented by a 54-dimensional vector of mel frequency cepstral coefficients (MFCC) with first and second derivatives appended.

The loading matrices T in (1) and D in (5) were both trained with similar sets of data drawn from Switchboard, SRE04, and SRE05. We use 500 factors for T, while D was a diagonal matrix by definition. The dimensionality of i-vector was therefore 500, while i-supervector is of CF = 27,648 in dimensionality. The rank of the matrices F and G in the PLDA model was set to 300 and 200, respectively, for the case of i-supervector. For i-vector, best result was found with the rank of F set to 300 and using a full matrix for Σ, for which G was no longer required. This observation was consistent with that reported in [32]. Table 1 summarizes all the corpora used to train the UBM, loading matrices T and D, PLDA model, whitening transformation matrix, Gaussianized rank-norm, and the cohort data for s-norm [32].

Table 1 Corpora used for training various components of the system

5.2 Feature and score normalization

Experiments were performed on the so-called det1 (int-int), det4 (int-tel), det5 (tel-mic), and det6 (tel-tel) common conditions as defined in NIST SRE08 short2-short3 core task. The term int refers to interview style recorder over microphone channel. For the det1 common condition, the training and test utterances were both int style of speech. Similar definition applied for other common conditions. The first set of experiments aimed at verifying the effectiveness of PLDA model in the i-supervector space without normalization (raw). Table 2 shows the results. It is evident that the i-supervector system performed much better than i-vector in all the four common conditions for both male and female trials. For the particular case of female trials, the EER for the i-supervector system was lower by 10.27%, 15.46%, 28.42%, and 16.58% in det1, det4, det5, and det6 compared to that of the i-vector system. One possible reason may be that the Gaussian assumption in (10) can be better fulfilled in the i-supervector space with higher dimensionality compared to that of the i-vector.

Table 2 Performance comparison of i-vector and i-supervector on NIST SRE08 core task with no normalization applied

The second set of experiments aimed at investigating the effectiveness of different normalization methods on i-supervector prior to PLDA modeling (i.e., length normalization, whitening, and Gaussianized rank-norm) and also the effects of score normalization (we used s-norm as reported in [32]). For simplicity, we used only telephone data and report the results on det6 (i.e., tel-tel common condition) in Table 3. We observed similar performance for other common conditions. From Table 3, it is clear that length normalization (len) always outperforms raw for both i-vector and i-supervector. Notice that i-vector gains huge improvement from length normalization. For the MALE subset, we observed 20.0% and 6.5% of relative improvement in EER when length normalization was applied on i-vector and i-supervector, respectively. Whitening followed by length normalization (white + len) further improves the performance for i-vector. Similarly, in the case of i-supervector, we used Gaussianized rank-norm followed by length normalization (grank + len) to cope with the high dimensionality. Finally, we also noticed that s-norm gives consistent improvement for both i-vector and i-supervector.

Table 3 Performance comparison of normalization methods on i-vector and i-supervector

5.3 Channel factors in i-supervector space

The low-rank matrices G model the subspace corresponding to channel variability as described in Section 3.1. We evaluated the performance of the i-supervector system at different numbers of channel factors, N _G . Table 4 shows the results for the det6 common condition. We can see that when N _G = 0, which corresponds to a fully diagonal PLDA model, the EER and MinDCF for both of male and female were very poor. A slight increment in the number of channel factors N _G to 10 reduces the EER by 47% and 46% for male and female sets, respectively. Further increment in N _G reduces the EER gradually until it levels off at N _G = 200 after which no further improvement could be attained. We set N _G = 200 for the i-supervector PLDA system in subsequent experiments.

Table 4 Performance of i-supervector PLDA system with different numbers of channel factors, N _G

5.3.1 Performance comparison

In this section, we compared the performance of i-supervector and i-vector under different train-test channel conditions. The PLDA models used for i-vector and i-supervector were the same as described in Section 5.2. In addition, we included microphone data (drawn from SRE05 and SRE06) for the whitening transform, Gaussianized rank-norm, and s-norm to better handle the interview (int) and microphone (mic) channel conditions.

Table 5 shows the results when full normalization (i.e., white + len + snorm for i-vector, grank + len + snorm for i-supervector) was applied. Here, we consider the EER and MinDCF by pooling together the male and female scores. The DET curves under the four common conditions are plotted in Figure 4. Similar to the observation in Section 5.1, i-vector gives better performance than i-supervector for the case with full normalization except in det5 where the i-supervector gives a much lower EER though the MinDCF is slightly worse. This again shows that current normalization strategy (Gaussianized rank-norm followed by length normalization), though effective, has to be further improved. Also shown in Table 5 and Figure 4 are the results by fusing the i-vector and i-supervector systems. The fusion of the two systems gives competitive performance with slightly lower EER and MinDCF across all four common conditions. The two systems were fused at the score level as follows:

$$\mathit{s}=\mathit{\beta }\times {\mathit{s}}_1+\left(1-\mathit{\beta }\right)\times {\mathit{s}}_2$$

(42)

where s₁ and s₂ are i-vector and i-supervector scores, respectively. The fusion weight β is set to 0.5, 0.5, 0.3, and 0.4, respectively, for det1, det4, det5, and det6.

Table 5 Performance comparison under various train-test channel conditions of NIST SRE08 short2-short3 core task

Figure 4

The DET curves of the i-vector and i-supervector evaluated on NIST SRE08 short2-short3 core task. (a) Common condition det1, (b) common condition det4, (c) common condition det5, and (d) common condition det6.

5.4 Computation complexity comparison

The experiments were carried out using the following hardware configuration: Centos 6.4 system, Intel Xeon processor E5-2687w (8-core, 3.1 GHz/core) with 128 GB memory. We compared the total time and the real-time factor of i-supervector and i-vector systems at four different stages, namely, loading matrix estimation, posterior extraction, PLDA modeling, and PLDA scoring. The total variability matrix T in (1) and D in (5) were both trained using a similar set of data drawn from Switchboard, SRE04, and SRE05. Table 6 lists the time of training T and D with ten EM iterations. We can see that it takes about 16.75 h for training T using 348 h of speech, which implies a real-time factor of 0.048. On the contrary, it took only 380 s for training D. Because D is a diagonal matrix, simple vector multiplication can be used instead of large matrix multiplication.

Table 6 Comparison of computational complexity (total time/real time factor) at various stages of implementation

After training the total variability space, we extracted i-vector and i-supervector for all utterances. Table 6 shows the time required for extracting the i-vectors and i-supervectors from the entire SRE04 dataset. The result shows that i-vector extraction consumes much more time than i-supervector. PLDA models were then trained for i-vectors and i-supervectors drawn from Switchboard, SRE04, and SRE05. We can see that training a PLDA model on the i-vector takes much lesser time than for the i-supervector. Finally, we compared the computation requirement for PLDA scoring on the NIST SRE08 short2-short3 core task with 98,776 trials. It can be seen that i-supervector scoring took more time than i-vector mainly due to its comparatively high dimensionality. In summary, the i-supervector system requires less computation at the front-end while the i-vector system is faster at the back-end PLDA.

6 Conclusions

We have introduced the use of the uncompressed form of i-vector (i.e., the i-supervector) for PLDA-based speaker verification. Similar to i-vector, an i-supervector represents a variable-length speech utterance as a fixed-length vector. But different from i-vector, we keep the total variability space having the same dimensionality as the original supervector space. To this end, we showed how manipulation of high-dimensional matrices can be done efficiently in training and scoring with the PLDA model. We also introduced the use of Gaussianized rank-norm for feature normalization prior to PLDA modeling.

Compared to i-vector, we found that i-supervector performs better when no normalization (on both feature and score) was applied. This suggests that the Gaussian assumption imposed by PLDA becomes less stringent and easier to fulfill in the higher dimensional i-supervector space. However, the performance improvement given by the high dimensionality diminishes when full normalization is applied. As such, current normalization strategy, though effective, has to be improved for better performance. This is a point for future work. We also showed that fusion system can give competitive performance compared to either i-vector or i-supervector. Furthermore, we analyzed the computational complexity of the i-supervector system, compared to that of the i-vector, at four different stages, namely, loading matrix estimation, posterior extraction, PLDA modeling, and PLDA scoring. Actually, the results showed that the i-supervector system took much less time than the i-vector system in terms of loading matrix and posterior extraction.

Endnote

^aThe number of sessions is usually limited in face recognition for which PLDA was originally proposed in [10].