Presumably, [tex]f(x,y)=y^2+xy-x^2[/tex]. We have
[tex]\begin{cases}f_x=y-2x=0\\f_y=2y+x=0\end{cases}[/tex]
[tex]y=2x\implies 2y+x=4x+x=0\implies x=0\implies y=0[/tex]
so on this region, [tex]f(x,y)[/tex] has only one critical point at (0, 0)[/tex]. Since this point lies on the boundary, we can ignore it for the moment.
Along the boundaries, we have four scenarios to consider:
[tex]x=0\implies f(0,y)=y^2[/tex]
[tex]x=6\implies f(6,y)=y^2+6y-36=(y+3)^2-45[/tex]
[tex]y=0\implies f(x,0)=-x^2[/tex]
[tex]y=6\implies f(x,6)=36+6x-x^2=-(x^2-6x-36)=45-(x-3)^2[/tex]
In the first case, [tex]y^2[/tex] is increasing over [tex]0\le y\le6[/tex], so we'll attain when [tex]y=6[/tex] a value of [tex]f(0,6)=36[/tex].
In the second case, [tex](y+3)^2-45[/tex] attains a minimum when [tex]y=-3[/tex], and would be increasing to either side. This would mean at [tex]y=6[/tex] we get [tex]f(6,6)=36[/tex].
In the third case, [tex]-x^2[/tex] attains a max at [tex]x=0[/tex], which would yield [tex]f(0,0)=0[/tex].
In the fourth case, [tex]45-(x-3)^2[/tex] attains a max at [tex]x=3[/tex], giving a value of [tex]f(3,6)=45[/tex].
This means the absolute maximum of [tex]f(x,y)[/tex] over the square is attained along the boundary [tex]y=6[/tex] when [tex]x=3[/tex], with a maximum value of [tex]f(3,6)=45[/tex].