# GLM and correlated variables
I always forget how correlation between features impacts parameter estimation in GLMs. Here, I simulate some data with and without correlation between between features and visualize how this impacts the estimation of the parameters of the model: $\beta$, $\mathrm{cov}(\beta)$, and $t(\beta)$.


```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import statsmodels.api as sm
%matplotlib inline
```

### Simulation variables
For $N=1000$, simulate two variables (let's say, $X_{1}$ and $X_{2}$) with or without correlation (if correlation: $\rho_{12} = 0.7$). We then simulate an outcome variable ($y$) as follows:

\begin{align}
y = X\beta + \epsilon, \epsilon \sim \mathcal{N}(0, 10)
\end{align}

with the true parameters, $\beta$, being $[1, 1]$. We then simulate the data and fit a GLM for 1000 iterations and plot the estimated parameters across for the correlated and uncorrelated data.


```python
N = 1000
true_betas = np.array([0, 1, 1])
true_corr = 0.7
iters = 1000

cov_corr = np.array([
    [1, true_corr],
    [true_corr, 1]
])

cov_uncorr = np.array([
    [1, 0],
    [0, 1]
])

cov = dict(
    corr=cov_corr,
    uncorr=cov_uncorr
)

results = dict(
    corr=dict(
        betas=np.zeros((iters, 2)),
        variances=np.zeros((iters, 2)),
        tvalues=np.zeros((iters, 2))
    ),
    uncorr=dict(
        betas=np.zeros((iters, 2)),
        variances=np.zeros((iters, 2)),
        tvalues=np.zeros((iters, 2))
    )
)

for i in range(iters):
    
    for ii, (covname, covmat) in enumerate(cov.items()):
        data = np.random.multivariate_normal(np.zeros(2), cov=covmat, size=N)
        data = np.c_[np.ones(N), data]
        y = data.dot(effects) + np.random.normal(0, 10, N)
        fit = sm.OLS(y, data).fit()
        results[covname]['betas'][i, :] = fit.params[1:]
        results[covname]['variances'][i, :] = np.diag(fit.cov_params())[1:]
        results[covname]['tvalues'][i, :] = fit.tvalues[1:]
```

Do the actual plotting:


```python
fig, axes = plt.subplots(nrows=3, ncols=2, sharex='row', sharey=False, figsize=(15, 10))

for i, covtype in enumerate(['uncorr', 'corr']):
    
    for ii, stat in enumerate(['betas', 'variances', 'tvalues']):
        
        axes[ii, i].set_title("Cov-type: %s, stat: %s" % (covtype, stat))
        sns.distplot(results[covtype][stat][:, 0], ax=axes[ii, i])
        sns.distplot(results[covtype][stat][:, 1], ax=axes[ii, i])        

sns.despine()
fig.tight_layout()
```

7a76f043079f754babb40bd76b0758565b224756
ipynb
Jupyter Notebook
GLM_correlated_variables.ipynb
lukassnoek/random_notebooks
d7df507ce2b6949726c29de0022aae2d0dc583ac
MIT
2018-05-28T13:45:11.000Z
2021-08-31T11:41:34.000Z
GLM_correlated_variables.ipynb
lukassnoek/random_notebooks
d7df507ce2b6949726c29de0022aae2d0dc583ac
MIT
GLM_correlated_variables.ipynb
lukassnoek/random_notebooks
d7df507ce2b6949726c29de0022aae2d0dc583ac
MIT
2018-05-28T13:46:05.000Z
2018-06-11T15:25:59.000Z
Qwen/Qwen-72B
1. YES
2. YES