How to find the variance co-variance matrix for 4 assets and market returns using the CAPM? I understand the process on excel:
calculate betas, calculate covariance with $Cov(R_{i,t},R_{j,t}) = \beta_{i,m} \cdot \beta_{j,m} \cdot \sigma^2_m$

where m is the market return and i and j are assets.

I'm confused as to what the assumptions are in order to be able to do this?

Can I do this since I have market data 4 US stocks and the S&amp;P500 meaning that I'm only considering systematic risk? 

Please could someone give me the assumptions in order to use the CAPM relation to find a variance covariance matrix?
Sharpe says you can always do a linear approximation to the returns of a family of assets and use that approximation to calculate a covariance matrix. For such an approximation to have economic content, one typically assumes that 1) residuals are uncorrelated with factors and 2) each asset's residuals are uncorrelated with that of any other. The second one has more bite and in particular, is likely not to hold in a one factor model.  

It is worth mentioning that in your equation above, the diagonal entries are not exactly what you write. Instead it is:
$Var(R_{i,t}) = \beta_{i,m}^2 \cdot \sigma^2_m +\sigma^2_\epsilon $
where $\sigma^2_\epsilon $ is the variance of the residual of the linear approximation equation.