What mathematical structure or formalism can be used for modelling strategic investment decisions of agents in the context of competition? I'm trying to analyze the decisions of agents in investing scarce resource like time and money into developing their product/service/offering in the presence of competition. I need to know what kind of mathematics can be used to describe this process, either analytically (preferably) or in the form of an simulation algorithm. Also, I'm working in the context of the Web and the Internet.

For example:

Suppose each agent has two possible strategies: 


Strategy A, or S(A), consists of a high investment cost, which has high returns in the absence of competition or other possible adverse factors. 
Strategy B, or S(B), is a low investment strategy, which comes with smaller returns that are more predictable.


In each possible strategy, there is an option of which products to produce, and also how much to specialize by differentiating these products. Each product has its own cost/benefit profile, i.e. the level of demand for each differentiated version of the product is different, and so is the cost of producing it.

I would like to know how to model situations like the above. 

Specific questions I have in mind are:


Is there an equilibrium that can be reached?
What are the conditions under which each agent would have the incentive to invest as much as possible?
What does the equilibrium look like, in terms of levels of investment and specialization.


Example reference:

"Media, aggregators, and the link economy: Strategic hyperlink formation in content networks" by Chrysanthos Dellarocas, Zsolt Katona, William Rand

I found the above paper very illuminating. I would like to find more work along these lines, but haven't been very successful at tracking down any.
Disclaimer: I am NOT presuming to have an answer to this question but given the lack of any sort of input I am throwing some ideas out there so hold your horses with your downvotes. 

It sounds to me like this is something you can model as a game of incomplete information. As a very very basic set up (see for example Moulin for a detail on games), in a Bayesian form a game specifies: 


The set of $N$ playes
The set of pure strategies $X_{i}$ for each player $i$
The set of types $T_{i}$ for each player $i$
The set of beliefs of each player $i$, represented by a probability distribution $\pi_{i}(.|t_{i})$ over $T_{N \text\i}$: one distribution for each possible type of player $i$
The payoff function $u_{i}(x;t)$ for each player $i$, where $x \in X_{N}$, and $t \in T_{N}$


A Bayesian equilibrium is described by a mixed strategy (combination of strategies) for each player, conditional on his type: $s_{i}(t_{i}) \in \Delta(X_{i})$. The equilibrium property then is $$ \forall i,t_{i} \in T_{i}, \forall s^{'}_{i} \in \Delta(X_{i}): $$ 
$$ \Sigma_{t_{-i}\in T_{N \text\i}} \pi_{i}(t_{-i}|t_{i})u_{i}(s(t);t) \geq \Sigma_{t_{-i}\in T_{N \text\i}} \pi_{i}(t_{-i}|t_{i})u_{i}((s^{'}_{i},s_{-i}(t_{-i}));t) $$

were the following notation is used $$ s_{j}(t)=s_{j}(t_{j}); s(t) \in \Pi_{j \in N \text\i} \Delta(X_{j})$$

Now, if the sets $X_{i}$ and $T_{i}$ are finite, the game possesses at least one Bayesian equilibrium. (This is a direct consequence of Nash's theorem and you can easily find a prove online so I will not get into this). 

Now, this is all very abstract, sort to speak. But applied in a simple way to the example you provided your components are quite straightforward: you have some set of agents who will make an investment decision with some goal in mind (obviously here is to maximize some payoff function), and a stochastic component in the returns that each investment offers. Types may refer to some heterogenity that you might want to impose on your players, and beliefs may refer to their knowledge of states of the world or other player's decisons. Now, you ask whether an equilibrium can be reached. As stated before (and of course this is just to provide you with a basic set up) under certain assumptions the existence of an equilibrium is shown. Now, figuring out that equilibrium, as well as the as the conditions under which each agent would have the incentive to invest as much as possible depends entirely on the complexity of the strategic interaction you model, and on the assumptions you make (among admittedly many others). You can make it as easy (e.g. take to agents with some payoff functions, impose known priors, make it a one shot game, and so on and you will easily find some equilibria, as uninteresting as this may be), or as difficult as you'd like it to be. In general, game theory is concerned with agents that interact, so it seems to me that there is a lot of use for you in this area given what you want to model. As you can see the notion of utility maximization is there, so this is not an issue. 

Now, as far as the paper you mentioned and the general area I am not able to provide any help. But there are many papers out there on investment decisions and competition from that use game theoretical settings (see for example the work of Smit and Ankum, and i'm sure some key search such as game theory investment competition will bring you many more). 

I am aware this is not a satisfying nor complete answer, but I am sure that if you look around at literature even outside your topic you will get a good idea on how you can develop a model (in my own experience this requires a lot of creativity and you often require "inspiration" from the work of others even if it does not necessarily relate to your topic so I would recommend you look at papers in general to gain some insight into the modelling component) 

This other papers may provide an interesting perspective on competition (and investment) and game theory models: 

https://link.springer.com/chapter/10.1007/978-3-319-05951-8_37

https://www.researchgate.net/profile/Lee_Cooper/publication/227442687_Modeling_Asymmetric_Competition/links/09e415140c23842b7f000000.pdf

https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1745-6622.2001.tb00332.x

https://ieeexplore.ieee.org/abstract/document/977512

https://ieeexplore.ieee.org/abstract/document/4438796

I hope this is of some help