I brought Visas into my party for a couple of hours and that was it.
Handmaiden stopped talking to me period. Everytime I spoke to her she said "Leave me alone, I have nothing to say to you"
And because of that, I didn't learn her backstory, or Atris, or anything. Goddamit. She's one of my favorite characters in the game too.