**Setting**


In instanton gauge theory, given a $G$-principal bundle $P\to X^4$, the orientability of the moduli space of ASD connections 
$$\mathcal{M}\_k = \{A \in L^{2}\_{k}(X, \Lambda^1 \otimes\mathrm{ad}(P))\ | \ F\_A^+ = 0\}/\mathcal{G}\_{k+1}$$
(the subscript is a Sobolev parameter)
is equivalent to the triviality of the determinant bundle of (the family of) operator(s) 
$$D\_A : L^{2}\_{k}(X, \Lambda^1 \otimes\mathrm{ad}(P)) \to L^{2}\_{k-1}(X,(\Lambda^+ \oplus \Lambda^0)\otimes\mathrm{ad}(P))\\$$
$$D = d\_A^+\oplus d\_A^\*.$$
This is a line bundle over the space of connections modulo gauge $\mathcal{B}\_k$ (suppose we don't have any reducible for simplicity),
$$\det D\to \mathcal{B}\_k$$
where the fiber is given by $\det D\_A := \Lambda^{\dim \ker D\_A} \ker D\_A \otimes \Lambda^{\dim \mathrm{coker} D\_A} \mathrm{coker} D\_A$



> 
> Suppose that $\mathcal{M}\_k$ is smooth. We know that the tangent space at $[A] \in \mathcal{M}\_k $, is given by
>  $$T\_{[A]}\mathcal{M}\_k\simeq \ker d\_A^+/ Im(d\_A)$$
>  thus it seems to me that orientability should be equivalent to the bundle 
>  $$\bigsqcup\_{[A] \in \mathcal{M}\_k}\Lambda^{top}\ker d\_A^+/ Im(d\_A)\to \mathcal{M}\_k$$
>  being trivial.
>  This bundle is a priori different from $\det D$.
>  Indeed, even if $\ker d\_A^+/ Im(d\_A) = \ker D\_A$, in $\det D$ we are tensoring with the top power of the cokernel.
>  What is the point of the cokernel tensor factor?
> 
> 
> 


I know that we want $S\_1 = \ker d\_A|\_{L\_k^2(\Lambda^0\otimes \mathrm{ad}P)}$ and $S\_2 = \mathrm{coker}\ d\_A^+\subset L\_k^2(\Lambda^+\otimes \mathrm{ad}P)$ to be trivial in order to have a smooth neighborhood at $[A]$.
And that $\mathrm{coker}D\_A > S\_1 + S\_2$ but maybe $\mathrm{coker}D\_A$ is not trivial, so we don't have that $\det D\_A = \Lambda^{top}\ker d\_A^+/ Im(d\_A) $ (maybe I am wrong).


**Edit after M. Miller's answer**
In this case $\mathrm{coker}D\_A = S\_1 \oplus S\_2 = \mathrm{coker} d\_A^\* \oplus \mathrm{coker} d\_A^+$ (and consequently $[A] \in \mathcal{M}$ has a smooth neighbourhood iff $\mathrm{coker} D\_A = (0)$). Indeed 
$$S\_1 = \ker d\_A|\_{L\_k^2(\Lambda^0\otimes \mathrm{ad}P)} \simeq \mathrm{coker} \ d\_A^\*.$$
Moreover  

$$(\ker d\_A^\*)^\perp = Im \ d\_A|\_{L\_k^2(\Lambda^0\otimes \mathrm{ad}P)} < \ker d\_A^+$$ therefore $d\_A^+ (\ker d\_A^\*) = Im \ d\_A^+$ and consequently
$$D\_A (\ker d\_A^\*) = Im \ d\_A^+ \quad \Rightarrow Im \ D\_A = Im \ d\_A^+ \oplus Im \ d\_A^\*$$



Orientability of moduli space and determinant bundle of ASD operator
You want a line bundle that's well-defined over the entire space of irreducible connections $\mathcal B^\*$; this is how you see, for instance, that a choice of orientation at one point in the moduli space induces an orientation at every point (even if the moduli space is not connected). Similarly, one proves orientability using this line bundle on the whole configuration space. 


The general recipe, taking a space of Fredholm operators and putting a line bundle over it, is called the determinant line bundle construction and was introduced by Quillen in the context of Riemann surfaces [here](https://mathscinet.ams.org/mathscinet-getitem?mr=0783704). As you say, the fiber above the Fredholm operator $A$ is $\Lambda^\text{max} \text{ker}(A) \otimes \Lambda^{\text{max}} \text{coker}(A)$. 


If you try to define this only using the $\Lambda^{\text{max}} \ker(A)$ factor *you do not get a locally trivial line bundle*, unless you restrict to the locus of surjective operators. Because not every irreducible connection has surjective $(D\_A, d\_A^\*)$, you won't get a line bundle over all of $\mathcal B^\*$. But along the locus of surjective operators, $\det(A) \cong \Lambda^{\text{max}} \ker(A)$.


In particular, the bundle you say is `a priori different from $\det D$' is not: because you have assumed that $\mathcal M$ is smooth, you have assumed that $\text{coker}(D\_A) = 0$ for all $A \in \mathcal M$. (This is precisely what it means for $A$ to be cut out transversely.) So the cokernel factor is not present.


Note that throughout this answer I have restricted to irreducible connections; otherwise some more care should be taken in defining the determinant line bundle (if you're not careful, you would see 'jumps' along the reducible locus.)


