Let $\mathfrak g$ be a real simple noncompact Lie algebra. Are there any correspondence between irreducible real representations of $\mathfrak g$ and 
the highest weight correspond to some positive system of restricted roots.


There is a complete classification of the complex representation $\mathfrak g \otimes \mathbb C$. There is a classification of real irreducible representations
using the irreducible representations of $\mathfrak g\otimes \mathbb C$ in 
"Onishchik, Lectures on real semi simple lie algebras and their representations".
But in a paper I read, it is claimed that the real representations can also be classified using highest weight like complex one. But I can not find a reference for that. 



real representation of real semi simple Lie algebra
I'm still unsure what you are looking for (or where you saw the material you recall), but the work of Borel-Tits and also Satake on reductive groups over arbitrary fields including $\mathbb{R}$ might be relevant here. (In characteristic 0 studying the groups is almost equivalent to studying the Lie algebras and their representations.) This line of research doesn't necessarily reveal much that is new about Lie groups, but it does place those groups in a framework which extends to other fields, etc. And it's possible to work over a given field of definition, within the framework of a minimal $k$-parabolic and a maximal $k$-split torus. 


In the foundational work of Borel-Tits [here](http://www.numdam.org/numdam-bin/fitem?id=PMIHES_1965__27__55_0), see especially $\S 12$. There is also a later paper by Tits which deals systematically with representations over arbitrary fields: Representations lineaires irreductibles d’un groupe reductif sur un corps quelconque. *J. Reine Angew. Math.* 247 (1971) 196–220. Still, for most practical purposes it's enough to follow the classical outline of E. Cartan and others, relating real groups or Lie algebras and their representations to the complexifications. 


