uhhhhhh no.
You need to hit him repeatedly with grabs. Even if its with he same hand.
This can frac up your opponent and really fuck up their perception what the ghost is showing them.

Also somebody tell me what i did wrong here.
I set up a perfect harai goshi and fail terribly. I couldn't get the rotation going.