Because Ma was stirring the porridge unaware of the voice.
Explanation:
Ba begins to question if the fish was actually talking to him because when he heard the voice, Ma was stirring the porridge and was unaware of the voice. Ba also thought that perhaps his weariness and tiredness was making him imagine things.
After talking to the fish, Ma and Ba felt that Minli was not meant to be found like in the story of secret word and the paper of happiness. They thought that Minli would eventually return once she accomplishes her mission.