I don't think the shoulder-drifting-to-back touch was meant as "hey baby". It looked comforting, reassuring, like a peace offering.

That it spooked Ianto so tells me that there's been no reassurance, no comfort; at least not physically expressed. Or that he's not ready to stop punishing himself, and maybe Jack.