Consider the $\mathbb{R}$-vector space of sufficiently nice real-valued functions on the unit square $I^2$, where "sufficiently nice" could be taken to mean any one of a number of things - say continuous for now.


In analogy with matrix multiplication, we can define the product of two such functions $F$ and $G$ as


$$(F\times G)(i,j) = \int\_0^1F(i,t)G(t,j)dt.$$


We can check immediately that this operation is associative (the proof is exactly the same):


$$((F\times G)\times H)(i,j) = \int\_0^1(F\times G)(i,t)H(t,j)dt$$
$$=\int\_0^1\left(\int\_0^1F(i,s)G(s,t)ds \right)H(t,j)dt$$
$$=\int\_0^1F(i,s)\left(\int\_0^1 G(s,t)H(t,j)dt \right)ds$$
$$=\int\_0^1F(i,s)(G\times H)(s,j)ds = (F\times (G\times H))(i,j)$$


Also, $\times$ is obviously bilinear with respect to usual addition of real-valued functions, and hence defines a ring structure on $C(I^2)$ which is considerably different from the usual ring structure (but addition is the same).


Extending the matrix analogy, we see that each $F$ also defines a linear operator $C(I) \to C(I)$ in the usual way, as


$$F(f)(i)=\int\_0^1 F(i,t)f(t) dt$$


for each $f : I \to \mathbb{R}$.


Also, all of this actually generalizes usual matrix multiplication if we subdivide the square $I^2$ into a bunch of small rectangles and let $F$ be constant on each subrectangle, being more or less careful on boundaries.


The only candidate for a unit element for $\times$ is the distribution which has a weight $1$ dirac delta on the diagonal, and is $0$ everywhere else (in other words, the product of the dirac delta with the Kronecker delta! :))


Now my question is: what is this? Is it of any interest, or a mere curiosity? For example, could a notion of "determinant" be assigned to these objects?



Multiplying functions on the unit square as generalized matrices
This is a partial answer concerning the determinants of such objects. While Will Jagy is right that such objects will have an uncountable spectrum, there still may be some hope for assigning a determinant in some cases as follows. First there is one obvious candidate of a trace for such objects: $Tr(F) = \int\_0^1 F(t,t)dt$ (Note that the Identity does not have trace equal the dimension of the matrix under this definition, but this definition parallels your approach of replacing all finite summations with integrals). Then in analogy with finite-dimensional matrix theory, the $k^{th}$ powersum of the spectrum of $F$ should be $Tr(F^k)$. Now using the Newton identities, one can recursively construct the sequence of elementary symmetric functions of the spectrum of $F$.


On the one hand, for ordinary matrices $M$ acting on a vector space $V$ of dimension $d$, the elementary symmetric function $e\_i$ evaluated on the eigenvalues of $M$ is equal to the trace of the operator defined by the action of $M$ on $\Lambda^i V$. In particular, $\Lambda^d V$ is one-dimensional and therefore the action of $M$ on this space is simply a scalar, this scalar is exactly $e\_d = det(M)$.


In your case, your generalized matrices act on an infinite-dimensional function space $V$ and $\Lambda^\infty V$ does not make sense. Nevertheless, combining the earlier definition of the trace of $F$ with the interpretation of the elementary symmetric functions above, we therefore have the following candidate for a determinant of such a generalized matrix $F$ (provided the limit exists):


$Det(F) = \lim\_{i\rightarrow\infty}e\_i(F)$


Where: 


$e\_1(F) = Tr(F)$


$e\_2(F) = \frac{1}{2}\left(e\_1(F)\cdot Tr(F)-Tr(F^2)\right)$


$e\_3(F) = \frac{1}{3}\left(e\_2(F)\cdot Tr(F)-e\_1(F)\cdot Tr(F^2)+Tr(F^3)\right)$


$\ldots$


Of course the difficulty in this approach lies in efficiently calculating the $e\_i(F)$ and showing the above limit exists. Thus you might need to assume some additional constraints to make this approach work.


