[tex]C[/tex] has a clockwise orientation, so in order to use Green's theorem (which is probably the "theorem" mentioned here), we have
[tex]\displaystyle\int_C\vec F\cdot\mathrm d\vec r=-\iint_D\frac{\partial(xy+x\cos x)}{\partial x}-\frac{\partial(y\cos x-xy\sin x)}{\partial y}\,\mathrm dx\,\mathrm dy[/tex]
where [tex]D[/tex] is the triangle region.
[tex]=\displaystyle\int_0^3\int_0^{12-4x}(\cos x-x\sin x)-(y+(\cos x-x\sin x))\,\mathrm dy\,\mathrm dx[/tex]
[tex]=\displaystyle-\int_0^3\int_0^{12-4x}y\,\mathrm dy\,\mathrm dx=\boxed{-72}[/tex]