Vector Ovector Observation Size Mismatch Between Continuous Agent Rolleragent

Observation Vector

Given an observation vector y from one of the L classes in the training set, one can compute its coefficients αˆ by solving either (8) or (9).

From: Handbook of Statistics , 2013

Handbook of Statistics

Vishal M. Patel , Rama Chellappa , in Handbook of Statistics, 2013

2.1 Sparse representation-based classification

In object recognition, given a set of labeled training samples, the task is to identify the class to which a test sample belongs to. Following Wright et al., (2009), we briefly describe the use of sparse representations for biometric recognition, however, this framework can be applied to a general object recognition problem.

Suppose that we are given $L$ distinct classes and a set of $n$ training images per class. One can extract an N-dimensional vector of features from each of these images. Let ${\mathbf{B}}_k=[{\mathbf{x}}_{k1}\text{,}\dots \text{,}{\mathbf{x}}_{\mathit{kj}}\text{,}\dots \text{,}{\mathbf{x}}_{\mathit{kn}}]$ be an $N\times n$ matrix of features from the kth class, where ${\mathbf{x}}_{\mathit{kj}}$ denote the feature from the jth training image of the kth class. Define a new matrix or dictionary $\mathbf{B}$ , as the concatenation of training samples from all the classes as

$\begin{array}{cc}\hfill \mathbf{B}=& [{\mathbf{B}}_1\text{,}\dots \text{,}{\mathbf{B}}_L]\in {\mathbb{R}}^{N\times (n.L)}\hfill \\ \hfill =& [{\mathbf{x}}_{11}\text{,}\dots \text{,}{\mathbf{x}}_{1n}|{\mathbf{x}}_{21}\text{,}\dots \text{,}{\mathbf{x}}_{2n}|\cdots \cdots |{\mathbf{x}}_{L1}\text{,}\dots \text{,}{\mathbf{x}}_{\mathit{Ln}}]\text{.}\hfill \end{array}$

We consider an observation vector $\mathbf{y}\in {\mathbb{R}}^N$ of unknown class as a linear combination of the training vectors as

(5) $\mathbf{y}={\displaystyle \sum _{i=1}^L}{\displaystyle \sum _{j=1}^n}{\alpha }_{\mathit{ij}}{\mathbf{x}}_{\mathit{ij}}$

with coefficients ${\alpha }_{\mathit{ij}}\in \mathbb{R}$ . The above equation can be written more compactly as

(6) $\mathbf{y}=\mathbf{B}\mathit{\alpha }\text{,}$

where

(7) $\mathit{\alpha }=[{\alpha }_{11}\text{,}\dots \text{,}{\alpha }_{1n}|{\alpha }_{21}\text{,}\dots \text{,}{\alpha }_{2n}|\cdots \cdots |{\alpha }_{L1}\text{,}\dots \text{,}{\alpha }_{\mathit{Ln}}{]}^T$

and ${.}^T$ denotes the transposition operation. We assume that given sufficient training samples of the kth class, ${\mathbf{B}}_k$ , any new test image $\mathbf{y}\in {\mathbb{R}}^N$ that belongs to the same class will lie approximately in the linear span of the training samples from the class $k$ . This implies that most of the coefficients not associated with class $k$ in (7) will be close to zero. Hence, $\mathit{\alpha }$ is be a sparse vector.

In order to represent an observed vector $\mathbf{y}\in {\mathbb{R}}^N$ as a sparse vector $\mathit{\alpha }$ , one needs to solve the system of linear Eq. (6). Typically $L·n\gg N$ and hence the system of linear Eq. (6) is underdetermined and has no unique solution. As mentioned earlier, if $\mathit{\alpha }$ is sparse enough and $\mathbf{B}$ satisfies certain properties, then the sparsest $\mathit{\alpha }$ can be recovered by solving the following optimization problem:

(8) $\hat{\mathit{\alpha }}=\underset{{\mathit{\alpha }}^{\prime }}{\text{argmin}}\Vert {\mathit{\alpha }}^{\prime }{\Vert }_1\text{subject to}\mathbf{y}=\mathbf{B}{\mathit{\alpha }}^{\prime }\text{.}$

When noisy observations are given, Basis Pursuit DeNoising (BPDN) can be used to approximate $\mathit{\alpha }$

(9) $\hat{\mathit{\alpha }}=\underset{{\mathit{\alpha }}^{\prime }}{\text{argmin}}\Vert {\mathit{\alpha }}^{\prime }{\Vert }_1\text{subject to}\Vert \mathbf{y}-\mathbf{B}\mathit{\alpha }\prime {\Vert }_2⩽\epsilon \text{,}$

where we have assumed that the observations are of the following form:

(10) $\mathbf{y}=\mathbf{B}\mathit{\alpha }+\eta$

with $\Vert \eta {\Vert }_2⩽\epsilon$ .

Given an observation vector $\mathbf{y}$ from one of the $L$ classes in the training set, one can compute its coefficients $\hat{\mathit{\alpha }}$ by solving either (8) or (9). One can perform classification based on the fact that high values of the coefficients $\hat{\mathit{\alpha }}$ will be associated with the columns of $\mathbf{B}$ from a single class. This can be done by comparing how well the different parts of the estimated coefficients, $\hat{\mathit{\alpha }}$ , represent $\mathbf{y}$ . The minimum of the representation error or the residual error can then be used to identify the correct class. The residual error of class $k$ is calculated by keeping the coefficients associated with that class and setting the coefficients not associated with class $k$ to zero. This can be done by introducing a characteristic function, ${\mathrm{\Pi }}_k:{\mathbb{R}}^n\to {\mathbb{R}}^n$ , that selects the coefficients associated with the kth class as follows:

(11) $r_k(\mathbf{y})=\Vert \mathbf{y}-\mathbf{B}{\mathrm{\Pi }}_k(\hat{\mathit{\alpha }}){\Vert }_2\text{.}$

Here the vector ${\mathrm{\Pi }}_k$ has value one at locations corresponding to the class $k$ and zero for other entries. The class, $d$ , which is associated with an observed vector, is then declared as the one that produces the smallest approximation error

(12) $d=\underset{k}{\text{argmin}}{r}_k(\mathbf{y})\text{.}$

The sparse representation-based classification method is summarized in Algorithm 1.

Algorithm 1 Sparse representation-based classification (SRC) algorithm

Input: $\mathbf{B}\in {\mathbb{R}}^{N\times (n·L)}\text{,}\mathbf{y}\in {\mathbb{R}}^N$ .

1.

Solve the BP (8) or BPDN (9) problem.

2.

Compute the residual using (11).

3.

Identify $\mathbf{y}$ using (12).

Output: Class label of $\mathbf{y}$ .

For classification, it is important to be able to detect and then reject the test samples of poor quality. To decide whether a given test sample has good quality, one can use the notion of Sparsity Concentration Index (SCI) proposed in Wright et al. (2009). The SCI of a coefficient vector $\mathit{\alpha }\in {\mathbb{R}}^{(L.n)}$ is defined as

(13) $\mathrm{SCI}(\mathit{\alpha })=\frac{\frac{L·\max \Vert {\mathrm{\Pi }}_i(\mathit{\alpha }){\Vert }_1}{\Vert \mathit{\alpha }{\Vert }_1}-1}{L-1}\text{.}$

SCI takes values between 0 and 1. SCI values close to 1 correspond to the case where the test image can be approximately represented by using only images from a single class. The test vector has enough discriminating features of its class, so has high quality. If SCI $=0$ then the coefficients are spread evenly across all classes. So the test vector is not similar to any of the classes and has of poor quality. A threshold can be chosen to reject the images with poor quality. For instance, a test image can be rejected if $\mathrm{SCI}(\hat{\mathit{\alpha }})<\lambda$ and otherwise accepted as valid, where $\lambda$ is some chosen threshold between 0 and 1.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780444538598000084

ASYMPTOTICALLY UNIMPROVABLE SOLUTION OF MULTIVARIATE PROBLEMS

VADIM I. SERDOBOLSKII , in Multiparametric Statistics, 2008

Problem Setting

Let x be an observation vector from an n-dimensional population

with expectation Ex = 0, with fourth moments of all components and a nondegenerate covariance matrix Σ = cov(x, x). A sample

= {x _m } of size N is used to calculate the mean vector $\overline{x}$ and sample covariance matrix

$C=N^{-1}{\displaystyle \sum _{m=1}^N\left({\text{x}}_m-\overline{\text{x}}\right)({\text{x}}_m-\overline{\text{x}}}{)}^T.$

We use the following asymptotical setting. Consider a hypothetical sequence of estimation problems

$\mathfrak{B}=\left\{{(\mathfrak{G},\sum ,N,\mathfrak{X},C,{\widehat{\sum }}^{-1})}_n\right\},n=1,2,\mathrm{\dots },$

where

is a population with the covariance matrix ∑ = cov(x, x),

is a sample of size N from

, ${\widehat{\sum }}^{-1}$ is an estimator Σ⁻¹ calculated as function of the matrix C (we do not write the indexes n for arguments of

). Our problem is to construct the best statistics ${\widehat{\sum }}^{-1}$ .

We begin by consideration of more simple problem of improving estimators of Σ⁻¹ by the introduction of a scalar multiple of C ⁻¹ (shrinkage estimation) for normal populations. Then, we consider a wide class of estimators for a wide class of populations.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780444530493500072

Miscellaneous topics in regression rank tests

Jaroslav Hájek , ... Pranab K. Sen , in Theory of Rank Tests (Second Edition), 1999

PROBLEMS AND COMPLEMENTS TO CHAPTER 10

Section 10.1.

1.

Define the aligned observation vectors ${\hat{X}}_i$ , i = 1,…,n, as in Subsection 10.1.1. and denote their joint distributions under H ₀ and K_n by P_n and Q_n respectively. Use LeCam's lemmas 1o verify that {Q_n } is contiguous to {P_n }.

2.

Use the contiguity result in the preceding problem along with LeCam's third lemma, and extend the joint asymptotic normality of the aligned rank statistics to contiguous alternatives in K_n.

3.

Derive the convergence in formula (10.1.1.14).

4.

Show that ℒ_N , defined by (10.1.1.9), has asymptotically, under K_n , a non-central χ² distribution with p−1 degrees of freedom and non-centrality parameter (10.1.1.17).

5.

Prove that the (p − 1)-multiple of classical ANOVA (variance-ratio) test statistic has asymptotically, under K_n , a non-central ${\chi }^2$ distribution with p − 1 degrees of freedom and non-centrality parameter (10.1.1.18).

6.

Prove the inequality (10.1.1.20). and that the equality sign holds in it only when $\varphi \left(F\left(x\right)\right)\equiv x$ , excepting on a set of null measure.

7.

Prove the asymptotic linearity result (10.1.2.9).

8.

Show that ${\hat{\beta }}_{2n}$ is asymptotically normal, as stated in (10.1.2.11).

9.

Show that the representation (10.1.2.14) holds, and that it implies the asymptotic normality in (10.1.2.11).

10.

Prove the asymptotic result (10.1.2.15).

11.

Prove the asymptotic result in relation (10.1.2.16).

12.

Using the projection method, show (10.1.2.17).

13.

Consider the setup of Subsection 10.1.2, and testing H ₀ : β₁ = 0 against H ₁: β₁ ≠ 0 by means of the statistic ${\widehat{ℒ}}_{n1}$ given by (10.1.2.21). If H ₀ does not hold, i.e. β ≠ 0, show that $n^{-1}{\widehat{ℒ}}_n$ converges in probability to a positive constant, so that ${\hat{L}}_{n1}$ is O_p (n).

14.

Verify that {q_n }, the joint density of {Y_i ; 1 ≤ i ≤ n} under contiguous alternatives {H_n } given by (10.1.2.22), is contiguous to {pn }, the joint density under H _0.

15.

Show that, under {H_n }, $n^{-1/2}{\widehat{ℒ}}_{n1}$ is asymptotically normal with parameters given before and in (10.1.2.24).

16.

Prove that, under {H_n }, the statistic ${\widehat{ℒ}}_{n1}$ has asymptotically the non-central χ² distribution as given in (10.1.2.25) and (10.1.2.20).

17.

Having still the setup of Subsection 10.1.2, prove that the asymptotically optimal aligned rank test uses the score function ϕ₀ given by (10.1.2.27).

18.

(Continuation) Verify the equality (10.1.2.29). i.e. ${\varphi }_0\left(x\right)\equiv \varphi \left(x,f\right)$ , 0 ≤ u ≤ 1.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B978012642350150028X

SPECTRAL THEORY OF SAMPLE COVARIANCE MATRICES

VADIM I. SERDOBOLSKII , in Multiparametric Statistics, 2008

Limit Spectra

We investigate here the limiting behavior of spectral functions for the matrices SandC under the increasing dimension asymptotics. Consider a sequence

= {

_N } of problems

(11) ${\mathfrak{B}}_n={\left(\mathfrak{S},\Sigma ,N,\mathfrak{X},S,C\right)}_{n}n=1,2,\dots$

in which spectral functions of matrices CandS are investigated over samples

of size N from populations

with cov(x,x) = Σ (we do not write out the subscripts in arguments of

_n ). For each problem

_n , we consider functions

$\begin{array}{l}{h}_{0n}\left(t\right)=n^{-1}\text{tr}{\left((I-zS\right)}^{-1},h_n\left(t\right)=n^{-1}\text{tr}{\left((I-zC\right)}^{-1},\\ F_{nS}\left(u\right)=\frac{1}{n}{\displaystyle \sum _{i=1}^n\text{ind}\left({\lambda }_i^S\le u\right),}{F}_{nC}\left(u\right)=\frac{1}{n}{\displaystyle \sum _{i=1}^n\text{ind}\left({\lambda }_i^C\le u\right),}\end{array}$

where ${\lambda }_i^S$ and ${\lambda }_i^C$ are eigenvalues of SandC, respectively,i = 1,2,…,n.

We restrict (11) by the following conditions.

A.

For each n , the observation vectors in

are such that Ex = 0 and the four moments of all components of x exist.

B.

The parameter M does not exceed a constant c ₀, where c ₀does not depend on n. The parameter γ vanishes as n → ∞ in

.

C.

In

,n/N → λ > 0.

D.

In

for each n, the eigenvalues of matrices Σ are located on a segment [c ₁,c ₂], where c ₁> 0 and c ₂does not depend on n, and F _nΣ(u) → F _Σ(u) as n → ∞ almost for any u ≥ 0.

Corollary(of Theorem 2.1). Under Assumptions A-D for any z ∈

, the limit exists $\underset{n\to \infty }{\lim }{h}_n\left(z\right)=h\left(z\right)$ such that

(12) $h\left(z\right)={\displaystyle \int {\left(1-zs\left(z\right)u\right)}^{-1}d{F}_{\sum }\left(u\right),s\left(z\right)=1-\lambda +\lambda h\left(z\right),}$

and for each z, we have

$\underset{n\to \infty }{\lim }\Vert \text{E}{\left(I-zC\right)}^{-1}-{\left(I-zs(z)\sum \right)}^{-1}\Vert \to 0.$

.

Let us investigate the analytical properties of solutions to (12).

Theorem 3.3 . If h(z)satisfies(12),c ₁> 0, λ > 0,andλ ≠ 1,then

1.

|h(z) ≤ α (z) and h(z) is regular near any point z ∈

2.

for any v = Re z > 0 such that $v<v_2=c_1^{-1}{\left(1-\sqrt{\lambda }\right)}^{-2}$ or $v>v_1=c_2^{-1}{\left(1+\sqrt{\lambda }\right)}^{-2},$ we have

$\underset{\epsilon \to +0}{\lim }\text{Im}h\left(v\text{\hspace{0.17em} +}i\epsilon \right)\text{\hspace{0.17em} =\hspace{0.17em} 0;}$

;

3.

ifv ₁≤v≤v ₂, then $0\le \mathrm{Im}h\left(v+i\epsilon \right)\le {\left(c_1\lambda v\right)}^{-1/2}+\omega ,$ where Ω → 0 as ε → + 0;

4.

ifv = Re z > 0 then s(−v) ≥ (1 +c ₂λ |v|)^{− 1};

5.

if |z| → ∞ on the main sheet of the analytical function h(z), then we have

if 0 < λ < 1, then zh(z) = − (1 – λ)_{− 1}Λ_{− 1}+ O(|z|^{− 1}),

if λ = 1, then zh ²(z) = − Λ_{− 1}+ O(|z ^{− ½}|),

if λ > 1, then zs(z) = − β₀+ O(|z|₋),

where β₀is a root of the equation

${\displaystyle \int {\left(1+{\beta }_{0}u\right)}^{-1}d{F}_{\sum }\left(u\right)=1-{\lambda }^{-1}.}$

.

Proof. The existence of the solution to (12) follows from Theorem 2.1. Suppose Im(z) > 0, then |h(z)| α = α (z). To be concise, denote h =h(z). For all u > 0 and z outside the beam z > 0, we have |1 −zsu|^{− 1}. Differentiating h(z) in (12), we prove the regularity of h(z). Define

$b_v=b_v\left(z\right)={\displaystyle \int {\left|1-zs\left(z\right)u\right|}^{-2}{u}^{v}d{F}_{\sum }\left(u\right),v=1,2.}$

.

Let us rewrite (12) in the form

(13) $\left(h-1\right)/s=z{\displaystyle \int u{\left(1-zsu\right)}^{-1}d{F}_{\sum }\left(u\right).}$

It follows that

$\mathrm{Im}\left[\left(h-1\right)/s\right]={\left|s\right|}^{-2}\mathrm{Im}h=b_1\mathrm{Im}z+b_2\lambda {\left|z\right|}^2\mathrm{Im}h.$

Dividing by b ₂, we use the inequality $b_1/b_2\le c_1^{-1}$ . Fix some v = Re z > 0 and tend z = ɛ → +0. It follows that the product

$\left({\left|s\right|}^{-2}{b}_2^{-1}-\lambda v^2\right)\mathrm{Im}h\to 0.$

.

Suppose that Im h does not tend to 0 (υ is fixed). Then, there exists a sequence {z_k } such that, for z _k=v+iɛ_k,h =h(z _k),s =s(z _k), we have Im h → a, where a≠ 0. For these z_k , we obtain ${\left|s\right|}^{-2}{b}_2^{-1}\to \lambda v^2\text{as}{\varepsilon }_k\to +0$ . We apply the Cauchy-Bunyakovskii inequality to (5). It follows that ${{\left|h-1\right|}^2/\left|z_{k}s\right|}^2\le b_2$ . We obtain that |h − 1|²≤ λ⁻+ 0(1) as ɛ_k→ +0. It follows that |s − 1|²≤ λ + 0(1). So the values s are bounded for {z_k }. On the other hand, it follows from (12) that Im h =b ₁Im(zs) =b ₁. We find that $\left(b_1^{-1}-\lambda v\right)\mathrm{Im}h\to 0$ as Im z → 0. But Im h → q ≠ 0 for {z_k }. It follows that $b_1^{-1}\to \lambda v$ . Combining this with the inequality ${\left|s\right|}^{-2}{b}_2^{-1}\to \lambda v^2,$ we find that ${\left|s\right|}^{-2}{b}_2^{-1}-b_1^{-1}v\to 0$ . Note that b ₁is finite for {z_k } and $c_2^{-2}\le b_1{b}_2^{-1}\le c_1^{-1}$ . Substitute the boundaries $\left(1\pm \sqrt{\lambda }\right)+o\left(1\right)$ for |s|. We obtain that $v_1+o\left(1\right)\le v\le v_2+o\left(1\right)$ as ε _k → + 0. We can conclude that Im h → 0 for any positive v outside the interval [v ₁,v ₂]. This proves the second statement of our theorem.

Now suppose v ₁≤v≤v ₂. From (12), we obtain the inequality (Imh)₂< (c ₁ vλ)_{− 1}. But h is bounded. It follows that the quantity (Imh)₂≤ (c ₁ vλ)^{− 1}. The third statement of our theorem is proved.

Further, let v = Re z > 0. Then, the functions hands are real and non-negative. We multiply both parts of (12) by λ. It follows that $\left(h-1\right)/zs\le b_1\le c_2$ . We obtain s≥ (1 +c ₂λ |z|)^{− 1}.

Let us prove the fifth theorem statement. Let λ < 1. For real z → − ∞, the real value of 1 −zsu in (12) tends to infinity.

Consequently,h → 0 and s → 1 − λ. For sufficiently large |Rez|, we have

$h\left(z\right)={\displaystyle \sum _{k=1}^{\infty }{\Lambda }_{-k}{\left(zs\right)}^{-k}},$

where ${\Lambda }_k={\displaystyle \int u^k}d{F}_{\Sigma }\left(u\right)$ . We conclude that

$h\left(z\right)=-{\left(1-\lambda \right)}^{-1}{\Lambda }_{-1}{z}^{-1}+O\left({\left|z\right|}^{-2}\right)$

for real z< 0 and for any z ∈

as |z| → ∞ in view of the properties of the Laurent series. Now let λ = 1. Then h =s. From (12), we obtain that h → 0 as z → − ∞ and h ²= Λ_{− 1}|z|^{− 1}+ O(|z|^{− 2}). Now suppose that Λ > 1,z = −t< 0, and t → ∞. Then, by Lemma 3.6, we have s≥ 0,h≥ 1 − 1/λ, and s → 0.Equation (12)impliest _s→ β₀as is stated in the theorem formulation. This completes the proof of Theorem 2.3.

Remark 7. Under Assumptions A-D for each u≥ 0, the limit exists

(14) $\begin{array}{l}F\left(u\right)=\underset{n\to \infty }{\mathrm{plim}}{F}_{nC}\left(u\right)\\ \text{such}\text{that}{\displaystyle \int {\left(1-zu\right)}^{-1}dF\left(u\right)=h\left(z\right).}\end{array}$

Indeed, to prove the convergence, it is sufficient to cite Corollary 3.2.1 from [22] that states the convergence of {h_nS (z)} and {h_nC (z)} almost surely. By Lemma 3.5, both these sequences converge to the same limit h(z). To prove that the limits of F_nS (u) and F_nC (u) coincide, it suffices to prove the uniqueness of the solution to (12). It can be readily proved if we perform the inverse Stieltjes transformation.

Theorem 3.4. Under Assumptions A-D,

1.

if λ = 0, then F _u=F _Σ(u) almost everywhere for u≥ 0;

2.

if λ > 0 and λ ≠ 0, then F(0) =F(u ₁− 0), where $u_1=c_1{\left(1-\sqrt{\lambda }\right)}^2,$ $u_2=c_2{\left(1+\sqrt{\lambda }\right)}^2,$ and c ₁andc ₂are bounds of the limit spectra Σ;

3.

if y > 0, λ ≠ 1, and u > 0, then the derivative F′ (u) of the function F(u) exists and $F^{\prime }\left(u\right)\le {\pi }^{-1}{\left(c_1\lambda u\right)}^{-1/2}$ ;

Proof. Let λ = 0. Then s(z) = 1. In view of (12), we have

$h\left(z\right)={\displaystyle \int {\left(1-zu\right)}^{-1}dF\left(u\right)={\displaystyle \int {\left(1-zu\right)}^{-1}d{F}_{\sum }\left(u\right)}}.$

At the continuity points of FΣ (u), the derivative

${F^{\prime }}_{\sum }\left(u\right)=\frac{1}{\pi }\underset{\epsilon \to +0}{\lim }\mathrm{Im}\frac{1}{z}h\left(\frac{1}{z}\right)=F^{\prime }\left(u\right),$

wherez =u−iɛ .

Let λ > 0. By Theorem 2.2 for u<u ₁and for u >u ₂(note that u ₁> 0 if λ < 0), the values Im[(u−iɛ)^{− 1} h((u−iɛ)^{− 1})] → 0 as ε → + 0. But we have

(15) $\mathrm{Im}\frac{h\left({\left(u-i\epsilon \right)}^{-1}\right)}{u-i\epsilon }>{\left(2\epsilon \right)}^{-1}\left[F\left(u+\epsilon \right)-F\left(u-\epsilon \right)\right].$

It follows that F′ (u) exists and F′ (u) for 0 <u<u ¹and for u >u ₂. The points of the increase of F(u) can be located only at the point u = 0 or on the segment [u ₁,u ₂]. If λ < 1 and |z| → ∞, we have ${{\displaystyle \int \left(1-zu\right)}}^{-1}dF\left(u\right)\to 0$ and, consequently,F(0) = 0. If λ > 1 and |z| → ∞, then $h\left(z^{-1}\right)z^{-1}\approx \left(1-{\lambda }^{-1}\right)/Z,$ and F(0) =1 − λ^{− 1}. The second statement of our theorem is proved.

Now, let z =v+ ɛ, where v > 0 is fixed and ε → + 0. Then, using (12) we obtain that Im h =b ₁Im(zs). Obviously,

$\left|\mathrm{Im}h\right|\le {\displaystyle \int {\left|1-zsu\right|}^{-1}d{F}_{\sum }\left(u\right)\le \frac{1}{c_1\mathrm{Im}\left(zs\right)}=\frac{b_1}{c_1\mathrm{Im}h}.}$

.

If Im h remains finite, then b ₁→ (λv)^{− 1}. Performing limit transition in (15), we prove the last statement of the theorem.

Theorem 3.5. If Assumptions A-D hold and0 < λ < 1,then for any complex z, z′ outside of the half-axis z > 0,we have

$\left|h\left(z\right)-h\left(z^{\prime }\right)\right|<c_3{\left|z-z^{\prime }\right|}^{\zeta },$

where c ₃ andζ > 0do not depend on z and z′ .

Proof. From (12), we obtain

$\left|h\left(z\right)\right|\le {\lambda }^{-1}\max \left(\lambda ,\left|1-\lambda \right|+2c_1^{-1}{\left|z\right|}^{-1}\right).$

By definition, the function h(z) is differentiable for each z outside the segment

= [v ₁,v ₂],v ₁> 0.

Denote a δ -neighborhood of the segment

by

_δ. If z is outside of

_δ, then the derivative h′ (z) exists and is uniformly bounded. It suffices to prove our theorem for v ∈

₁, where

₁=

_δ− {z: Im z = 0}. Choose $\delta ={\delta }_1=v_1/2$ . Then δ₁< |z| < δ₂, where δ₂does not depend on z. We estimate the absolute value of the derivative h′ (z). For Im ≠ 0, from(15) by the differentiation we obtain

(16) $\left(z^{-1}{y}^{-1}-{\displaystyle \int X^{-2}ud{F}_{\sum }\left(u\right)}\right)h^{\prime }\left(z\right)=\frac{s\left(z\right)}{z\lambda }{\displaystyle \int X^{-2}ud{F}_{\sum }\left(u\right),}$

whereX1= (1 −zs(z)u) ≠ 0. Denote

$\begin{array}{l}\varphi \left(z\right)=\frac{1}{zy}-{\displaystyle \int X^{-2}ud{F}_{\sum }\left(u\right),b_1=}{\displaystyle \int {\left|X\right|}^{-2}}ud{F}_{\sum }\left(u\right),\\ h_1=\mathrm{Im}h\left(z\right),z_0=\mathrm{Re}z,z_1=\mathrm{Im}z,s_0=\mathrm{Re}s\left(z\right),\end{array}$

and let α with subscripts denote constants not depending on z. The right-hand side of (16) is not greater than α₁ b ₁forz ∈

₁and therefore |h′ (z)| < α₂ b ₁|ϕ (z)|^{− 1}.

We consider two cases. Denote α₃= (2δ₂ c ₂)^{− 1}.

At first, let Re s(z) =s ₀< α₃. Using the relation h ₁=b ₁, we obtain that the quantity − Im ϕ (z) equals

$z_1{\left|z\right|}^{-2}{\lambda }^{-1}+2b_1^{-1}{\displaystyle \int {\left|X\right|}^{-4}{u}^2\left(1-z_0{s}_{0}u+z_1\lambda h_{1}u\right)h_{1}d{F}_{\sum }\left(u\right).}$

.

In the integrand here, we have z ₀> 0, 1 −z ₀ s ₀> ½,z ₁ h ₁< 0. From the Cauchy-Bunyakovskii inequality, it follows that

${\displaystyle \int {\left|X\right|}^{-4}{u}^{2}d{F}_{\sum }\left(u\right)\ge b_1^2.}$

.

Hence |Im ϕ (z)| >b ₁ h ₁and |h′ (z)|. Let

$\mathrm{Re}\varphi \left(z\right)={\lambda }^{-1}{z}_0{\left|z\right|}^{-2}-b_1+2{\left[\mathrm{Im}zs\left(z\right)\right]}^2{\displaystyle \int {\left|X\right|}^{-4}{u}^{3}d{F}_{\sum }\left(u\right).}$

.

Definep = λ^{− 1} z ₀|z|^{− 2}−b ₁. We have

$p={\lambda }^{-1}{\left|z\right|}^{-2}{z}_0{z}_1{\left|\mathrm{Im}zs\left(z\right)\right|}^{-1}\left(s_0-\lambda h_1{z}_1/z_0\right).$

Here |h ₁| < α₄,z ₀≥ δ₁≥ 0,s ₀≥ α₃> 0, and we obtain that p > 0 if z ₁< α₆, where α₆= α₃α₅/λ α₄. If zε

₁andz ₁> α₆, then the Hö lder inequality follows from the existence of a uniformly bounded derivative of the analytic function h(z) in a closed domain.

Now let z ∈

,z ₁< α₆,p > 0, and s ₀> α₃> 0. Then, |h′ (z)| ≤ α₇ b ₁|Reϕ (z)|^{− 1}, where

$\begin{array}{l}\mathrm{Re}\varphi \left(z\right)\ge 2{\left(\mathrm{Im}zs\left(z\right)\right)}^2{c}_1{\displaystyle \int {\left|X\right|}^{-4}{u}^{2}d{F}_{\sum }\left(u\right)\ge }\\ \ge 2{\left(\mathrm{Im}zs\left(z\right)\right)}^2{c}_1{b}_1=2c_1{h}_1^2.\end{array}$

.

Substitutingb ₁/Im(zs(z)) =h ₁and taking into account that s ₀> 0, we obtain that $\left|h^{\prime }\left(z\right)\right|\le {\alpha }_7{h}_1^{-2}$ . Thus, for v ∈

_δand 0 <z ₁< α₆for any s ₀, it follows that |h′ (z)| < α₈max $\left(h_1^{-1},h_1^{-2}\right)\le {\alpha }_9{h}_1^{-2}$ . Calculating the derivative along the vertical line we obtain the inequality $h_1^2\left|d{h}_1/dz\right|\le {\alpha }_9,$ whence

$h_1^3\left(z\right)\le h_1^3\left(z^{\prime }\right)+3{\alpha }_9\left|z-z^{\prime }\right|\le {\left(h_1\left(z^{\prime }\right)+{\alpha }_{10}{\left|z-z^{\prime }\right|}^{1/3}\right)}^3$

if Im z. Im z′ > 0. The Hö lder inequality for h ₁= Im h(z) with ζ =1/3 follows. This completes the proof of Theorem 3.5.

Example. Consider limit spectra of matrix Σ of a special form of the "ρ -model" considered first in [63]. It is of a special interest since it admits an analytical solution to the dispersion equation (12). For this model, the limit spectrum of Σ is located on a segment [c ₁,c ₂], where $c_1={\sigma }^2{\left(1-\sqrt{p}\right)}^2$ and $c_2={\sigma }^2{\left(1+\sqrt{p}\right)}^2$ . Its limit spectrum density is

$\frac{d{F}_{\sum }\left(u\right)}{du}=\{\begin{array}{l}{\left(2\pi \rho \right)}^{-1}\left(1-\rho \right)u^{-2}\sqrt{\left(c_2-u\right)\left(u-c_1\right)},c_1\le u\le c_2,\\ 0\text{for}u<c_1\text{and}\text{for}u>c_2.\end{array}$

.

The moments ${\Lambda }_k={\displaystyle \int u^k}d{F}_{\Sigma }\left(u\right)\text{for}k=0,1,2,3,4$ are

$\begin{array}{l}{\Lambda }_0=1,{\Lambda }_1={\sigma }^2\left(1-\rho \right),{\Lambda }_2={\sigma }^4\left(1-\rho \right),\\ {\Lambda }_1={\sigma }^6\left(1-{\rho }^2\right),{\Lambda }_4={\sigma }^8\left(1-\rho \right)\left(1+3\rho +{\rho }^2\right).\end{array}$

If ρ > 0, the integral

$\eta \left(z\right)={{\displaystyle \int \left(1-zu\right)}}^{-1}d{F}_{\Sigma }\left(u\right)=\frac{1+\rho -kz-\sqrt{{\left(1+\rho -kz\right)}^2-4\rho }}{2\rho },$

where k = σ²(1 − ρ). The function η = η (z)satisfies the equation ρ η¹+ (kz− ρ − 1)η + 1 =0. The equation h(z) = η (zs(z)) can be transformed to the equation (h− 1)(1 − ρh) =kzhs, which is quadratic with respect to h =h(z),s = 1 − λ + λh. If λ > 0, its solution is

$h=\frac{1+\rho -k\left(1-\lambda \right)z-\sqrt{{\left(1+\rho -k\left(1-\lambda \right)z\right)}^2-4\left(\rho +kz\lambda \right)}}{2\left(\rho +k\lambda z\right)}.$

.

The moments M _k= (k!)^{− 1} h ^(k)(0) for k = 0,1,2,3 are

$\begin{array}{l}{M}_0=1,M_1={\sigma }^2\left(1-\rho \right),M_2={\sigma }^4\left(1-\rho \right)\left(1+\lambda \left(1-\rho \right)\right),\\ M_3={\sigma }^6\left(1-\rho \right)\left(1+\rho +3\lambda \left(1-\rho \right)+{\lambda }^2{\left(1-\rho \right)}^2\right).\end{array}$

Differentiating the functions of the inverse argument, we find that, in particular, Λ_{− 1}=k ^{− 1}, Λ_{− 2}=k ^{− 2}(1 + ρ),M _{− 1}=k _{− 1}=k ^{− 1}(1 − λ)^{− 1}, M _{− 2}=k ^{− 2}(ρ + λ (1 − ρ))(1 − λ)^{− 3}. The continuous limit spectrum of the matrix C is located on the segment [u ₁,u ₂], where

$u_1={\sigma }^2{\left(1-\sqrt{\lambda +\rho \left(1-\lambda \right)}\right)}^2,u_2={\sigma }^2{\left(1+\sqrt{\lambda +\rho \left(1-\lambda \right)}\right)}^2$

and has the density

$f\left(u\right)=\{\begin{array}{l}\frac{\left(1-\rho \right)\sqrt{\left(u_2-u\right)\left(u-u_1\right)}}{2\pi u\left(\rho u+{\sigma }^2{\left(1-\rho \right)}^{2}y\right)}\text{if}\text{u}\in \left[u_1,u_2\right],\\ 0\text{otherwise}\text{.}\end{array}$

.

If λ > 1, then the function F(u) has a jump 1 − λ^{− 1}at the point u = 0. If λ = 0, then F(u) =F _Σ(u) has a form of a unit step at the point u = σ². The density f(u) satisfies the Hö lder condition with ζ = 1/2.

In a special case when Σ =I and ρ = 0, we obtain the limit spectral density $F^{\prime }\left(u\right)={\left(2\pi \right)}^{-1}{u}^{-2}\sqrt{\left(u_2-u\right)\left(u-u_1\right)}$ foru ₁<u<u ₂, where $u_{2,1}={\left(1\pm \sqrt{\lambda }\right)}^2$ . This "semicircle" law of spectral density was first found by Marchenko and Pastur [43].

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780444530493500060

Volume 4

T. Kourti , in Comprehensive Chemometrics, 2009

4.02.4 Handling Future Observations with Missing Data

Missing measurements are a frequent occurrence in process industries. Therefore, the new observation vector x _new ( Figure 4 ) may have a few elements missing. Latent variable methods that model the process space (PCA, PCR, and PLS) make it possible to infer the corresponding score values of x _new by using the available elements in the vector together with the model built from the training data set.

The fact that process variables are highly correlated and that there is redundancy in process data (i.e., many variables are affected by the same event) makes this possible. Redundancy is beneficial for handling missing data. More details on methods for treatment of missing data in regression can be found in Chapter 3.06.

A variety of algorithms have been suggested ^32,33 to handle missing data, with different degree of complexity: trimmed score method (TRI), single-component projection (SCP), projection to the model plane (PMP) – using PLS or ordinary least squares (PMP_PLS, PMP_OLS), iterative imputation of missing data (II), a method based on the minimization of the squared prediction error (SPE), conditional mean replacement (CMR), trimmed score regression (TSR), and regression on known data (KDR).

Suppose that ${\mathbf{x}}_{\text{new}}^{\text{T}}={\left[\mathbf{x}*\mathbf{x}\ne \right]}^{\text{T}}$ , where without loss of generality we assume that x# is the vector of missing observations. (Following this convention, p ^* and P ^* are loadings corresponding to the known x ^*.) The methods can be seen as different ways to impute values for the missing variables vector, x#. By setting the missing values equal to their expected mean value (i.e., for mean-centered data x#   =   0), we have the TRI method. ³³

SCP is the simplest but also the poorest performing approach: It calculates each of the scores independently and sequentially as ${{\displaystyle \hat{t}}}_i={\text{z}}^{*}{\mathbf{p}}_i^{*}/{\mathbf{p}}_i^{*\text{T}}{\mathbf{p}}_i^{*}$ , where z ^* is x ^* deflated by the first i  −   1 components.

Nelson et al. ³² showed that superior results can be obtained by calculating all of the scores at once by projecting onto the hyperplane formed by the P ^* vectors. In the PMP method, the known x ^* vector is regressed onto the matrix P ^*. Sometimes, depending on the measurements missing, some of the columns of P ^* may become highly correlated and P ^*T P ^* becomes ill-conditioned. It was suggested³² to use PLS, PCR, or regularized least squares regression for the projection.

CMR ³² and TSR ³³ use the known score T matrix from the training data together with the loadings (P ^*) and the available measurements (x ^*) to estimate the score vector. A singularity problem that may arise in CMR may be solved by a procedure suggested by Nelson et al., ³² where the estimated score vector is calculated in two steps: First a parameter β is computed using PLS from T=X ^* β, where T and X ^* _, respectively, represent the score matrix and those columns from the training data set corresponding to known values; then β along with the current available data vector x ^* is used to compute an estimate of the score vector.

In iterative imputation, one may use an initial estimate of the final scores (say, those given by SCP method) to forecast the missing values x̂#, (using their corresponding loadings), then create the new vector and recalculate a score estimate, and iterate until convergence.

Arteaga and Ferrer ³³ presented an extensive study on the various methods. Iterative imputation and SPE methods are equivalent to PMP; KDR is equivalent to CMR. They concluded that based on the best prediction of the missing values, KDR is statistically superior to the other methods. The TSR is practically equivalent to the KDR and has the advantage that a much smaller matrix needs inversion. Additionally, TSR is statistically superior to PMP method.

Before the system is implemented online, there should be a plan for the operators as to how to respond if the values of several variables stop being recorded. For example, if there are three thermocouples in a reactor and one fails, common sense dictates that we can afford to continue the monitoring scheme. On the contrary, if there is only one sensor for a variable uncorrelated with any other, the value for this variable cannot be assessed from the rest of the variables in the system; therefore depending of the importance of this variable, one may not be able to rely on the monitoring scheme until the failed sensor is replaced. This idea was treated quantitatively by Nelson ⁵² and Nelson et al., ⁵³ where they analyzed the uncertainty resulting from missing measurements for the predictions of the values of the Hotelling's T ² and the SPE. Rather than representing an object with missing measurements by a single point, an estimate of the uncertainty regions in the score, Hotteling's T ², and SPE spaces arising from the missing measurements is provided. They suggested measures to distinguish between situations where model performance will continue to be acceptable and situations where it will be unacceptable, and therefore if the missing measurements cannot be recovered the application must be shut down.

Missing data methods did find their way in the industrial applications. In their industrial perspective on implementing online applications of multivariate statistics, Miletic et al. ² emphasize that missing data handling is a necessary feature for both the offline modeling and the online systems and report that are using the methods proposed by Nelson et al. ³²

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780444527011000132

Parameter Estimation of Chaotic Systems Using Density Estimation of Strange Attractors in the State Space

Yasser Shekofteh , ... Sajad Jafari , in Recent Advances in Chaotic Systems and Synchronization, 2019

5.1 The GMM of the Chaotic System

The chaotic system (1) has three variables in the state space. So, the observation vector of its attractor will be formed as v  =   [x, y, z], and we must select D  =   3 as the state space dimension in Eq. (2). To generate the attractor points of the chaotic system (1) as real data, its model has been simulated with parameters a  =   1.0 and b  =   1.0 by a fourth-order Runge-Kutta method with a step size of 10   ms [27,28]. For training data at the first phase, a set of sequential samples of the system (1) including 100,000 samples (equal to 1000   s time length) has been recorded. The initial conditions were set to (−   0.10, −   5.05, −   6.00) as initial conditions of the system (1). Here, we assume that this recorded training data must lead us to estimate unknown parameters of the chaotic system (1), a and b, by minimization of the GMM-based cost function.

Using obtained training data from the chaotic system, we can learn a GMM in order to model the geometry of the attractor in the state space. In other words, the GMM computation fits a parametric model to the distribution of the attractor in the state space. Fig. 5 shows the attractor of the chaotic system (1) in a three-dimensional state space along with its GMM modeling using M  =   64 Gaussian components. In this figure, every three-dimensional ellipsoid corresponds to one of the Gaussian components.

Fig. 5. Plot of the chaotic attractor of the system (1) and its GMM modeling with M  =   64 components in the 3D state space. Here, the parameters of the system (1) are set to a  =   1.0 and b  =   1.0.

As can be seen from Fig. 5, the Gaussian components attempt to cover the attractor in the state space. To show the effect of the number of Gaussian mixtures, in Fig. 6, the attractor of the chaotic system (1) and its GMM models are shown for different values of M  =   16, 32, 48, and 64.

Fig. 6. Plot of the chaotic attractor of the system (1) and its GMM modeling with M  =   16, 32, 48, and 64 components in the 3D state space.

As can be seen from Fig. 6, when we increase the number of Gaussian components, more details of the trajectory of the chaotic attractor can be covered by the added Gaussian components. Therefore, in these experiments, the best GMM modeling of the attractor can be obtained by M  =   64, which shows a precise model of the chaotic attractor. Therefore, by increasing the number of Gaussian components in the GMM, it can cover more complexity of the given time series in its model. The higher value of M can improve the performance of the cost function, but it also increases the computational cost and may be lead to overfitting problems.

In Fig. 7, the information criteria such as AIC, BIC, and the negative of the log-likelihood are considered for the GMM selection problem. It shows that M  =   64 is a good choice for the number of GMM components, because of minimization of the criteria.

Fig. 7. Plot of the information criteria values to select the best GMM.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780128158388000078

Volume 2

G.J. McLachlan , in Comprehensive Chemometrics, 2009

2.30.18 Mixed Feature Data

We consider now the case where some of the feature variables are discrete. That is, the observation vector y _j on the jth entity to be clustered consists of p ₁ discrete variables, represented by the subvector y _1j , in addition to p ₂ continuous variables represented by the subvector y _2j (j  =   1,…,n). The ith component density of the jth observation

$y_j={\left(y_{1j}^{\text{T}},y_{2j}^{\text{T}}\right)}^{\text{T}}$

can then be written as

(84) $f_i(y_i)=f_i\left(y_{1j}\right)f_i\left(y_{2j}|y_{1j}\right)$

The symbol f _i is being used generically here to denote a density where, for discrete random variables, the density is really a probability function.

In discriminant and cluster analyses, it has been found that it is reasonable to proceed by treating the discrete variables as if they are independently distributed within a class or cluster. This is known as the NAIVE assumption. ^49,50 Under this assumption, the ith component-conditional density of the vector y _1j of discrete features is given by

(85) $f_i\left(y_{1j}\right)={\displaystyle \prod _{v=1}^{p_1}}{f}_{iv}\left(y_{1vj}\right)$

where f _iv (y _1vj ) denotes the ith component-conditional density of the vth discrete feature variable y _1vj in y _1j .

If y _1v denotes one of the distinct values taken on by the discrete variable y _1vj , then under Equation (85) the (k  +   1)th update of f _iv (y _1v ) is

(86) $f_{\text{<mi mathvariant="italic" is="true">iv</mi>}}^{(k+1)}(y_{1v})=\frac{{\Sigma }_{j=1}^n{\tau }_i\left(y_j;{\Psi }^{(k)}\right)\delta \left[y_{1vj},y_{1v}\right]+c_1}{{\Sigma }_{j=1}^n{\tau }_i\left(y_j;{\Psi }^{(k)}\right)+c_2}$

where δ[y _1vj , y _1vj ]   =   1 if y _1vj   = y _1v and is zero otherwise, and Ψ^(k) is the current estimate of the vector of all the unknown parameters that now include the probabilities for the discrete variables. In Equation (86), the constants c ₁ and c ₂, which are both equal to zero for the maximum likelihood estimate, can be chosen to limit the effect of zero estimates of f _iv (y _1v ) for rare values y _1v . One choice is c ₂  =   1 and c ₁  =   1/d _v , where d _v is the number of distinct values in the support of y _1vj . ⁴⁹

We can allow for some dependence between the vector y _2j of continuous variables and the discrete-data vector y _1j by adopting the location model as, for example, in Hunt and Jorgensen. ⁵¹ With the location model, f _i ( y _2j ∣ y _1j ) is taken to be multivariate normal with a mean that is allowed to be different for some or all of the different levels of y _1j .

As an alternative to the use of the full mixture model, we may proceed conditionally on the realized values of the discrete feature vector y _1j , as in McLachlan and Chang. ⁵² This leads to the use of the conditional mixture model for the continuous feature vector y _2j ,

(87) $f\left(y_{2j}|y_{1j}\right)={\displaystyle \sum _{i=1}^g}{\pi }_i\left(y_{1j}\right)f_i\left(y_{2j}|y_{1j}\right)$

where π _i ( y _1j ) denotes the conditional probability of ith component membership of the mixture given the discrete data in y _1j . A common model for π _i ( y _1j ) is the logistic model under which

(88) ${\pi }_i(y_{1j})=\frac{\text{exp}\left({\beta }_{i0}+{\beta }_i^{\text{T}}{y}_{1j}\right)}{1+{\Sigma }_{h=1}^{g-1}\text{exp}\left({\beta }_{h0}+{\beta }_h^{\text{T}}{y}_{1j}\right)}$

where ${\mathbf{\beta }}_i={({\beta }_{i1},\dots ,{\beta }_{i{p}_1})}^{\text{T}}\text{for}i=1,\dots ,g-1$ , and

${\pi }_g\left(y_{1j}\right)=1-{\displaystyle \sum _{h-1}^{g-1}}{\pi }_h(y_{1j})$

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780444527011000685

Statistical Control of Measures and Processes

A.J. Ferrer-Riquelme , in Comprehensive Chemometrics, 2009

1.04.9.2.2 PCA-based MSPC: online process monitoring (Phase II)

Once the reference PCA model and the control limits for the multivariate control charts are obtained, new process observations can be monitored online. When a new observation vector z _i is available, after preprocessing it is projected onto the PCA model yielding the scores and the residuals, from which the value of the Hotelling $T_A^2$ and the value of the SPE are calculated. This way, the information contained in the original K variables is summarized in these two indices, which are plotted in the corresponding multivariate $T_A^2$ and SPE control charts. No matter what the number of the original variables K is, only two points have to be plotted on the charts and checked against the control limits. The SPE chart should be checked first. If the points remain below the control limits in both charts, the process is considered to be in control. If a point is detected to be beyond the limits of one of the charts, then a diagnostic approach to isolate the original variables responsible for the out-of-control signal is needed. In PCA-based MSPC, contribution plots ³⁷ are commonly used for this purpose.

Contribution plots can be derived for abnormal points in both charts. If the SPE chart signals a new out-of-control observation, the contribution of each original kth variable to the SPE at this new abnormal observation is given by its corresponding squared residual:

(37) $\text{Cont}\left(\text{SPE;}{x}_{\text{new},k}\right)=e_{\text{new},k}^2={\left(x_{\text{new},k}-x_{\text{new},k}^{*}\right)}^2$

where e _new,k is the residual corresponding to the kth variable in the new observation and $x_{\text{new},k}^{*}$ is the prediction of the kth variable x _new,k from the PCA model.

In case of using the DModX statistic, the contribution of each original kth variable to the DModX is given by ⁴⁴

(38) $\text{Cont}\left(\text{DModX;}{x}_{\text{new},k}\right)=w_k{e}_{\text{new},k}$

where w _k is the square root of the explained sum of squares for the kth variable. Variables with high contributions in this plot should be investigated.

If the abnormal observation is detected by the $T_A^2$ chart, the diagnosis procedure is carried out in two steps: (i) a bar plot of the normalized scores for that observation (t _new,a /λ _a )² is plotted and the ath score with the highest normalized value is selected; (ii) the contribution of each original kth variable to this ath score at this new abnormal observation is given by

(39) $\text{Cont}\left(t_{\text{new},a};x_{\text{new},k}\right)=p_{\text{<mi mathvariant="italic" is="true">ak</mi>}}{x}_{\text{new},k}$

where p _ak is the loading of the kth variable at the ath component. A plot of these contributions is created. Variables on this plot with high contributions but with the same sign as the score should be investigated (contributions of the opposite sign will only make the score smaller). When there are some scores with high normalized values, an overall average contribution per variable can be calculated over all the selected scores. ³⁹

Contribution plots are a powerful tool for fault diagnosis. They provide a list of process variables that contribute numerically to the out-of-control condition (i.e., they are no longer consistent with NOCs), but they do not reveal the actual cause of the fault. Those variables and any variables highly correlated with them should be investigated. Incorporation of technical process knowledge is crucial to diagnose the problem and discover the root causes of the fault.

Apart from the $T_A^2$ and SPE control charts, other charts such as the univariate time-series plots of the scores or scatter score plots can be useful (both in Phase I and II) for detecting and diagnosing out-of-control situations and also for improving process understanding.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B978044452701100096X

Multivariate density estimation

Dag Tjøstheim , ... Bård Støve , in Statistical Modeling Using Local Gaussian Approximation, 2022

9.2.2 Estimation of the joint dependence function

Let $\psi (\cdot ,\mathit{\theta })$ be a parametric family of p-variate density functions. Below ψ is taken to be the multinormal. We recall from Chapter 4 that Hjort and Jones (1996) estimate the unknown density f using the sample ${\mathit{X}}_1,\dots ,{\mathit{X}}_n$ by fitting ψ locally. The local parameter estimate $\hat{\mathit{\theta }}=\hat{\mathit{\theta }}(\mathit{x})$ maximizes the local likelihood function

(9.3) $L({\mathit{X}}_1,\dots ,{\mathit{X}}_n,\mathit{\theta })=L_n(\mathit{\theta },\mathit{x})=n^{-1}\sum _{i=1}^n{K}_{\mathit{B}}({\mathit{X}}_i-\mathit{x})\log \psi ({\mathit{X}}_i,\mathit{\theta })-\int K_{\mathit{B}}(\mathit{y}-\mathit{x})\psi (\mathit{y},\mathit{\theta })\mathrm{\text{d}}\mathit{y},$

where K is a kernel function that integrates to one and is symmetric about the origin, B is a positive definite matrix of bandwidths, and $K_{\mathit{B}}(x)=|\mathit{B}{|}^{-1}K({\mathit{B}}^{-1}\mathit{x})$ , $|\cdot |$ being the determinant. For small bandwidths, the local estimate $\hat{f}(\mathit{x})=\psi (\mathit{x},{\hat{\mathit{\theta }}}_n(\mathit{x}))$ is close to $f(\mathit{x})$ in the limit, because if the bandwidth matrix B is held fixed and $u_j(\cdot ,\mathit{\theta })=\partial /\partial {\theta }_j\log \psi (\cdot ,\mathit{\theta })$ , we have

(9.4) $0=\frac{\partial L_n({\hat{\mathit{\theta }}}_n,\mathit{x})}{\partial {\theta }_j}\stackrel{P}{\to }\int K_B(\mathit{y}-\mathit{x})u_j(\mathit{y},{\mathit{\theta }}_{\mathit{B},K}(\mathit{y}))\{f(\mathit{y})-\psi (\mathit{y},{\mathit{\theta }}_{\mathit{B},K}(\mathit{y}))\}\mathrm{\text{d}}\mathit{y}$

for some value of the parameter ${\mathit{\theta }}_{\mathit{B},K}(\mathit{x})$ toward which ${\hat{\mathit{\theta }}}_n(\mathit{x})$ converges in probability. However, for finite sample sizes, the curse of dimensionality comes into play. The number of coordinates in $\mathit{\theta }=\mathit{\theta }(\mathit{x})$ typically grows with the dimension of x , making the local estimates difficult to obtain at every point in the sample space. One solution might be increasing the bandwidths so that the estimation becomes almost parametric. However, here we propose a different path around the Curse directly exploiting decomposition (9.2). The first step might be choosing a standardized multivariate normal distribution as parametric family in (9.3) for modeling $f_{\mathit{Z}}$ in (9.2) locally:

(9.5) $\psi (\mathit{z},\mathit{\theta })=\psi (\mathit{z},\mathit{R})={(2\pi )}^{-p/2}|\mathit{R}{|}^{-1/2}\exp \{-\frac{1}{2}{\mathit{z}}^T{\mathit{R}}^{-1}\mathit{z}\},$

where R denotes the local correlation matrix. Using a univariate local fit, the local Gaussian expectations and variances in (9.5) are constant and equal to zero and one, respectively, reflecting our knowledge that the margins of the unknown density function $f_{\mathit{Z}}$ are standard normal. However, as $\mathit{B}\to \mathbf{0}$ in the p-variate case, as briefly described in Chapter 4.9, the local mean μ and variance σ in general depend on z . In this chapter, we make the additional assumption in our p-dimensional local Gaussian approximation that $\mathit{\mu }(\mathit{z})\equiv \mathbf{0}$ and ${\mathit{\sigma }}^2(\mathit{z})=\mathbf{1}$ . This is more restrictive than in Chapters 7 and 8, where it was assumed that $\mathit{\mu }=\mathit{\mu }(\mathit{z})=\mathit{\mu }(z_1,z_2)$ and $\mathit{\sigma }\equiv \mathit{\sigma }(\mathit{z})=\mathit{\sigma }(z_1,z_2)$ in the bivariate case, and this more general assumption was crucial in obtaining the local spectral results in Chapter 8.

With this more restrictive assumption that $\mathit{\mu }(\mathit{z})\equiv \mathbf{0}$ and ${\mathit{\sigma }}^2(\mathit{z})\equiv \mathbf{1}$ , we are left with the problem of estimating the pairwise correlations ${\rho }_{ij}$ , $1⩽i<j⩽p$ , in (9.5). Fitting the Gaussian distribution according to the scheme described above results in a local correlation matrix at each point. Specifically, the estimated local correlations are written as ${\hat{\rho }}_{ij}={\hat{\rho }}_{ij}(z_1,\dots ,z_p)$ , $i,j=1,\dots ,p$ , indicating that each parameter depends on all variables. The dependence between variables is captured in the variation of the parameter estimates in the p-dimensional Euclidean space, and its estimate maximizes the local likelihood function (9.3). However, as mentioned, the quality of the estimate deteriorates quickly with the dimension.

If the data were jointly normally distributed, there would be no dimensionality problem, since the entire distribution would be characterized by the global correlation coefficients between pairs of variables, and their empirical counterparts are easily computed from the data. A local Gaussian fit would then coincide with a global fit and result in estimates of the form ${\hat{\rho }}_{ij}={\hat{\rho }}_{ij}(Z_i,Z_j)$ , where the arguments indicate which of the transformed observation variables were used to obtain the estimate. This points to a natural simplification, which we may use to estimate the density $f_{\mathit{Z}}$ , analogous to the additive regression model in Chapter 2.7.1. We allow the local correlations to depend on their own variables only:

(9.6) ${\hat{\rho }}_{ij}(z_1,\dots ,z_p)={\hat{\rho }}_{ij}(z_i,z_j).$

We could also simplify the estimation problem by estimating the local means and variances as functions of "their own" coordinate only: ${\mu }_i(\mathit{z})={\mu }_i(z_i)$ and ${\sigma }_i^2(\mathit{z})={\sigma }_i^2(z_i)$ , but, as mentioned before, here we have chosen the stricter approximation

(9.7) $\mathit{\mu }(\mathit{z})=\mathbf{0}\mathrm{\text{and}}{\mathit{\sigma }}^2(\mathit{z})=\mathbf{1}.$

We refer to Section 9.7 for a further discussion of this point.

The resulting estimation is carried out in four steps:

1.

Estimate the marginal distributions using the logspline method (or the empirical distribution function) and transform each observation vector to pseudo-standard normality as described in the previous subsection.

2.

Estimate the joint density of the transformed data using the Hjort and Jones (1996) local likelihood function (9.3), the standardized normal parametric family (9.5), and simplifications (9.6) and (9.7). In practice, this means fitting the bivariate version of (9.5) to each pair of the transformed variables $(Z_i,Z_j)$ . Put the estimated local correlations into the estimated local correlation matrix: $\hat{\mathit{R}}(\mathit{z})={\{{\hat{\rho }}_{ij}(z_i,z_j)\}}_{i,j=1,\dots ,p}$ .

3.

Let ${\hat{f}}_{\mathit{Z}}(\mathit{z})=\psi (\mathit{z},\hat{\mathit{R}}(\mathit{z}))$ and obtain the final estimate of $f(\mathit{x})$ by replacing $f_{\mathit{Z}}$ with ${\hat{f}}_{\mathit{Z}}$ , and the marginal distribution and density functions with their logspline estimates in (9.2):

(9.8) $\hat{f}(\mathit{x})={\hat{f}}_{\mathit{Z}}({\mathrm{\Phi }}^{-1}({\hat{F}}_1(x_1)),\dots ,{\mathrm{\Phi }}^{-1}({\hat{F}}_p(x_p)))\prod _{i=1}^p\frac{{\hat{f}}_i(x_i)}{\varphi \left({\mathrm{\Phi }}^{-1}({\hat{F}}_i(x_i))\right)}.$

4.

Normalize the density estimate so that it integrates to one.

The existence of population values corresponding to the estimated local correlations is discussed in the following section. It is clear that assumptions (9.6) and (9.7) represent an approximation to most multivariate distributions. The authors are aware of no other distributions than those possessing the Gaussian copula or step functions thereof as in Tjøstheim and Hufthammer (2013) or Chapter 4.3, for which (9.6) and (9.7) are exact properties of the true local correlations. In that case the local correlations are constant or stepwise constant in all its variables. The quality of the LGDE thus depends to a large degree on the severity of assumptions (9.6) and (9.7) on the underlying density. The pairwise assumption is hard to interpret except in general statements about "pairwise dependence structures", and so we proceed in this chapter to explore the impact of (9.6) and (9.7) in practice in Section 9.6 and the subsequent discussion in Section 9.7. Before we do that, we take a closer look at the theoretical foundations of the LGDE.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B978012815861600016X

INTRODUCTION

VADIM I. SERDOBOLSKII , in Multiparametric Statistics, 2008

The Kolmogorov Asymptotics

In 1967, Andrei Nikolaevich Kolmogorov was interested in the dependence of errors of discrimination on sample size. He solved the following problem. Let x be a normal observation vector, and $\overline{x_v}$ be sample averages calculated over samples from population number ν = 1, 2. Suppose that the covariance matrix is the identity matrix. Consider a simplified discriminant function

$g\left(\text{x}\right)={\left({\overline{\text{x}}}_1-{\overline{\text{x}}}_2\right)}^T\left(\text{x−}\left({\overline{\text{x}}}_1+{\overline{\text{x}}}_2\right)/2\right)$

and the classification rule w(x) > 0 against w(x) ≤ 0. This function leads to the probability of errors ${\alpha }_n=\Phi \left(-G/\sqrt{D}\right)$ , where G and D are quadratic functions of sample averages having a noncentral χ² distribution. To isolate principal parts of G and D, Kolmogorov proposed to consider not one statistical problem but a sequence of n-dimensional discriminant problems in which the dimension n increases along with sample sizes N _ν, so that $N_v\to \infty$ and $n/N_v\to {\lambda }_v>0,v=1,2$ . Under these assumptions, he proved that the probability of error α _n converges in probability

(7) $\underset{n\to \infty }{\text{p}\lim }{\alpha }_n=\Phi \left(-\frac{J-{\lambda }_1+{\lambda }_2}{2\sqrt{J+{\lambda }_1+{\lambda }_2}}\right),$

where J is the square of the Euclidean limit "Mahalanobis distance" between centers of populations. This expression is remarkable by that it explicitly shows the dependence of error probability on the dimension and sample sizes. This new asymptotic approach was called the "Kolmogorov asymptotics."

Later, L. D. Meshalkin and the author of this book deduced formula (7) for a wide class of populations under the assumption that the variables are independent and populations approach each other in the parameter space (are contiguous) [45], [46].

In 1970, Yu. N. Blagoveshchenskii and A. D. Deev studied the probability of errors for the standard sample Fisher-Andersen-Wald discriminant function for two populations with unknown common covariance matrix. A. D. Deev used the fact that the probability of error coincides with the distribution function g(x). He obtained an exact asymptotic expansion for the limit of the error probability α. The leading term of this expansion proved to be especially interesting. The limit probability of an error (of the first kind) proved to be

$\alpha =\Phi \left(-\Theta \frac{J-{\lambda }_1+{\lambda }_2}{2\sqrt{J+{\lambda }_1+{\lambda }_2}}\right),$

where the factor $\Theta =\sqrt{1-\lambda }$ , with λ = λ₁λ₂ /(λ₁ + λ₂), accounts for the accumulation of estimation inaccuracies in the process of the covariance matrix inversion. It was called "the Deev formula." This formula was thoroughly investigated numerically, and a good coincidence was demonstrated even for not great n, N.

Note that starting from Deev's formulas, the discrimination errors can be reduced if the rule g(x) > θ against g(x) ≤ θ with θ = (λ₁ — λ₂)/2 ≠ 0 is used. A. D. Deev also noticed [18] that the half-sum of discrimination errors can be further decreased by weighting summands in the discriminant function.

After these investigations, it became obvious that by keeping terms of the order of n/N, one obtains a possibility of using specifically multidimensional effects for the construction of improved discriminant and other procedures of multivariate analysis. The most important conclusion was that traditional consistent methods of multivariate statistical analysis should be improvable, and a new progress in theoretical statistics is possible, aiming at obtaining nearly optimal solutions for fixed samples.

The Kolmogorov asymptotics (increasing dimension asymp–totics [3]) may be considered as a calculation tool for isolating leading terms in case of large dimension. But the principal role of the Kolmogorov asymptotics is that it reveals specific regularities produced by estimation of a large number of parameters. In a series of further publications, this asymptotics was used as a main tool for investigation of essentially many-dimensional phenomena characteristic of high-dimensional statistical analysis. The constant n/N became an acknowledged characteristics in many-dimensional statistics.

In Section 5.1, the Kolmogorov asymptotics is applied for the development of theory allowing to improve the discriminant analysis of vectors of large dimension with independent components. The improvement is achieved by introducing appropriate weights of contributions of independent variables in the discriminant function. These weights are used for the construction of asymptotically unimprovable discriminant procedure. Then, the problem of selection of variables for discrimination is solved, and the optimum selection threshold is found.

But the main success in the development of multiparametric solutions was achieved by combining the Kolmogorov asymptotics with the spectral theory of random matrices developed independently at the end of 20th century in another region.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780444530493500047

whitethentim.blogspot.com

Source: https://www.sciencedirect.com/topics/mathematics/observation-vector

Share this post