We know that in a collision the momentum is conserved, that is:
[tex]\vec{p}_0=\vec{p}_f[/tex]
Since this is vector equation we can divide it in two scalar equations, one for x and for y. Then we have:
[tex]\begin{gathered} p_{0x}=p_{fx} \\ \text{and} \\ p_{0y}=p_{fy} \end{gathered}[/tex]
Then we have for the x direction:
[tex]\begin{gathered} 5.4m=m(2.6)\cos 36.9+1.26mv_o\cos \phi \\ 5.4=2.6\cos 36.9+1.26v_o\cos \phi \end{gathered}[/tex]
and for the y direction:
[tex]\begin{gathered} 0=-m2.6\sin 36.9+1.6mv_o\sin \phi \\ 2.6\sin 36.9=1.6v_o\sin \phi \end{gathered}[/tex]
Hence, we have the system of equations:
[tex]\begin{gathered} 5.4=2.6\cos 36.9+1.26v_o\cos \phi \\ 2.6\sin 36.9=1.6v_o\sin \phi \end{gathered}[/tex]
From the second equation we have:
[tex]v_o=\frac{2.6\sin 36.9}{1.6\sin \phi}[/tex]
Plugging this in the first equation:
[tex]\begin{gathered} 5.4=2.6\cos 36.9+1.26(\frac{2.6\sin36.9}{1.6\sin\phi})\cos \phi \\ 1.26(\frac{2.6\sin36.9}{1.6})\tan \phi=5.4-2.6\cos 36.9 \\ \tan \phi=\frac{1.6(5.4-2.6\cos 36.9)}{(1.26)(2.6\sin 36.9)} \\ \phi=\tan ^{-1}(\frac{1.6(5.4-2.6\cos36.9)}{(1.26)(2.6\sin36.9)}) \\ \phi=69.69 \end{gathered}[/tex]
Now that we know the value of the angle we plug it in the expression for the velocity, then we have:
[tex]\begin{gathered} v_o=\frac{2.6\sin 36.9}{1.6\sin 69.69} \\ v_0=1.04 \end{gathered}[/tex]
Therefore, the magnitude of the final speed of the orange ball is 1.04 m/s and the direction is 69.69°