{"word_count": 718, "character_count": 6091}
Consider a (stationary) autoregressive process, say of order 2,for some white noise  with variance . Here is a code to generate such a process,&#62; phi1=.25&#62; phi2=.7&#62; n=1000&#62; set.seed(1)&#62; e=rnorm(n)&#62; Z=rep(0,n)&#62; for(t in 3:n) Z[t]=phi1*Z[t-1]+phi2*Z[t-2]+e[t]&#62; Z=Z[800:1000]&#62; n=length(Z)&#62; plot(Z,type="l")Here, we have to estimate two sets of parameters: the autoregressive coefficients, and the variance of the innovation process . Several techniques can be used to estimate those parameters.using least square regressionA natural idea is to see here a regression model, since (if we consider a matrix formulation)Here we can run (conditional) ordinary least squares estimation,&#62; base=data.frame(Y=Z[3:n],X1=Z[2:(n-1)],X2=Z[1:(n-2)])&#62; regression=lm(Y~0+X1+X2,data=base)&#62; summary(regression)Call:lm(formula = Y ~ 0 + X1 + X2, data = base)Residuals:    Min      1Q  Median      3Q     Max -3.0268 -0.7063  0.1065  0.6925  3.2566 Coefficients:   Estimate Std. Error t value Pr(&#62;|t|)    X1  0.23400    0.05463   4.283 2.88e-05 ***X2  0.62863    0.05476  11.479  &#60; 2e-16 ***---Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1Residual standard error: 1.062 on 197 degrees of freedomMultiple R-squared:  0.6349,Adjusted R-squared:  0.6312 F-statistic: 171.3 on 2 and 197 DF,  p-value: &#60; 2.2e-16so we get the following estimators, for the autocorrelation coefficients, and the volatility of the noise&#62; regression$coefficients       X1        X2 0.2339959 0.6286321 &#62; summary(regression)$sigma[1] 1.061839using Yule-Walker equationsAs we've seen in class, we can easily get the following equations for the autocovariance functions,which can also be written (again, using a matrix expression)So we just have to solve a simple linear system of equations. Note that if we divide by the variance, those equations can be written in terms of the autocorrelation functionsThe code is the following&#62; rho1=cor(Z[1:(n-1)],Z[2:n])&#62; rho2=cor(Z[1:(n-2)],Z[3:n])&#62; A=matrix(c(1,rho1,rho1,1),2,2)&#62; b=matrix(c(rho1,rho2),2,1)&#62; (PHI=solve(A,b))          [,1][1,] 0.2256270[2,] 0.6315329Now, we need to extract the estimated innovation process, from this set of parameters&#62; estWN=base$Y-(PHI[1]*base$X1+PHI[2]*base$X2)&#62; sd(estWN)[1] 1.058558This estimator is probably not the best one (we can take into account that we've lost two degrees of freedom), but as a starting point, let us consider this one.An alternative could be to include the variance term in Yule-Walker equations, to get a three dimensional linear equation,It is not much more complicated to solve, actually,&#62; gamma0=var(Z[1:n])&#62; gamma1=var(Z[1:(n-1)],Z[2:n])&#62; gamma2=var(Z[1:(n-2)],Z[3:n])&#62; A=matrix(c(gamma1,gamma0,gamma1,gamma2,gamma1,gamma0,1,0,0),3,3)&#62; b=matrix(c(gamma0,gamma1,gamma2),3,1)&#62; (PHISIGMA=solve(A,b))          [,1][1,] 0.2283151[2,] 0.6283431[3,] 1.1335501using (conditional) likelihood estimatorsFinally, we can assume some distribution for the innovation process. The standard model is a Gaussian model, i.e.has a Gaussian distributionIn that case, the conditional log likelihood (conditional since we set the first two observations here) is&#62; CondLogLik=function(A,TS){+ phi1=A[1];  phi2=A[2]+ sigma=A[3]; L=0+ for(t in 3:length(TS)){+ L=L+dnorm(TS[t],mean=phi1*TS[t-1]++ phi2*TS[t-2],sd=sigma,log=TRUE)}+ return(-L)}Now, we can run standard optimization procedures,&#62; LogL=function(A) CondLogLik(A,TS=Z)&#62; optim(c(0,0,1),LogL)$par[1] 0.2339589 0.6285002 1.0565613$value[1] 293.3042$countsfunction gradient      106       NA $convergence[1] 0$messageNULLIt is also possible to consider a global maximum likelihood optimisation problem, since the variance matrix of vector  has a know form.using (unconditional) likelihood estimatorsThe variance matrix of  is , where autocovariances are not not know, be can easily be computed using a recursive relationship.&#62; library(mnormt)&#62; GlobalLogLik=function(A,TS){+ n=length(TS)+ phi1=A[1];  phi2=A[2]+ sigma=A[3]+ SIG=matrix(0,n,n)+ rho=rep(0,n)+ rho[1]=1+ rho[2]=phi1/(1-phi2)+ for(h in 3:n) rho[h]=phi1*rho[h-1]+phi2*rho[h-2]+ for(i in 1:n){for(j in 1:n){+ SIG[i,j]=rho[abs(i-j)+1]}}+ gamma0=(1-phi2)*sigma^2/((1+phi2)*((1-phi2)^2-phi1^2))+ SIG=gamma0*SIG+ return(dmnorm(TS,rep(0,n),SIG,log=TRUE))}&#62; LogL=function(A) -GlobalLogLik(A,TS=Z)&#62; optim(c(.1,.1,1),LogL)Error in chol.default(x, pivot = FALSE) : Error in pd.solve(varcov, log.det = TRUE) :   x appears to be not positive definiteThe problem is that there is a strong constraint on the pair  to get a stationary process (we are not far away, here, from the border of the triangle, where the process become non stationary). To be more specific (this was mentioned in a previous post), we should havei.e. in a standard matrix form(we can add an additional constraint on the variance parameter, to insure that it will be positive). To run a contrained optimization routine, consider&#62; U=matrix(c(1,0,0,-1,0,1,0,-1,0,0,1,0),4,3)&#62; C=c(0,0,0,-.99999)&#62; constrOptim(c(.1,.1,1),LogL,grad=NULL,ui=U,ci=C)$par[1] 0.2238892 0.6342850 1.0613388$value[1] 297.9202$countsfunction gradient      108       NA $convergence[1] 0$messageNULL$outer.iterations[1] 2$barrier.value[1] 0.000189892(here, to faster, we restrain the parameters so that they will be positive).comparing those estimatesHere, our five estimators are rather close. Let us run more samples to see more precisely how they behave. For the first parameter , we getand for the second one, , we haveThe bias we observe is probably coming from the fact that, with this numerical example, we are not far away from the non-stationary case (the sum of the true parameters should be less than 1, and it is 0.95). When we estimate the parameters, we force them to be inside the triangle, since those parameters can be estimated only if the process is stationary.Observe that the standard-deviation of the innovation process  is here, well estimated,(with clearly some estimators that perform better than others).&#160;