Two vectors are orthogonal if their scalar product is zero. The scalar product of two vectors is computed as follows:
[tex] (a_1,a_2,\ldots,a_n) \cdot (b_1,b_2,\ldots,b_n) = a_1b_1 + a_2b_2 + \ldots + a_nb_n [/tex]
In other words, you multiply each component of the first vector with the correspondant component of the second vector, and then you sum all these products.
So, we need to compute the two scalar products, and set them to zero:
[tex] (a, 2,b) \cdot (-1, -2, 3) = 0 \iff -a-4+3b = 0 [/tex]
[tex] (a, 2,b) \cdot (3, -2, 6) = 0 \iff 3a-4+6b = 0 [/tex]
We can rewrite the two requests in this linear system:
[tex] \begin{cases} -a+3b = 4\\3a+6b = 4 \end{cases} [/tex]
From the first equation we can deduce [tex] a = 3b-4 [/tex], which substituted in the second gives
[tex] 3a+6b = 4 \to 3(3b-4)+6b = 4 \to 9b-12+6b = 4 \to b = \dfrac{16}{15} [/tex]
If we substitute this value for b in the first equation we have
[tex] -a+3b = 4 \to -a+3\cdot \dfrac{16}{15} = 4 \to -a+\dfrac{16}{5} = 4 \to a = \dfrac{16}{5}-4= \dfrac{-4}{5} [/tex]