Why a whelk...

Another way of looking at Adam is that his original infiltration is that it was at the start, very calculated. Maybe Tosh and Owen's personalities were messed with ON PURPOSE.
He boasted to Jack of how he changed them 'for the better'. By making Tosh confident enough to go for a relationship, Adam is now in a strong alliance with Tosh, the longest-serving member (after Jack of course) - and that nearly worked 'Would You Die For Me' - Tosh very nearly shot Jack for him, and was tempted into not taking the Retcon, as Adam had given her the one thing she wanted - love. By making Owen more introverted and socially awward, Adam is removing Mr Alpha Male from the picture and making himself less open to a challenge. Owen is a young buck, always clashing antlers with the two other males - Adam really didn't need that challenge from Owen if he wanted to keep his cover. By making Jack re-live his childhood memories, he makes the team-leader insecure in himself and it affects his ability to lead - right to the final temptation. Adam just makes sure that he is in the right piece at the right time to assume the role of confidente, as he makes the same assumation that a lot of people make at first glance - Jack and Ianto have a purely sexual liason. Adam has placed himself in two strong alliances with Jack and Tosh, and has neutralised who could be his biggest challenger in Owen. He's cocky, and gets careless.
He then makes two serious mistakes:
No. 1: Gwen - Gwen was on holiday that weekend. Suddenly there's another team member that Adam hasn't accounted for, and he applies a quick fix. The problem is that it is a messy job and he doesn't know that Gwen, unlike the others, actually has a life outside Torchwood, and he doesn't take that in consideration. He's suddenly having to back-peddle when the whole-forgetting-Rhys thing comes to light 'Believe me when I tell you that that is your fiancé.' It's the tiny warning sign that both Jack and Ianto pick up on very quickly as they start to figure out what's happened.
Two: Adam seriously underestimated Ianto. At first impression, Ianto is just the tea-boy, too weak to be a threat or worthy of an alliance - just make him think you've always worked here, job done. As we all know one of the carnal rules of Torchwood is to never assume Ianto is just the tea-boy, and Ianto is the one that is able to figure it out because Adam was too lazy to tie up that loose end - to his cost. Ianto challenges him and proves to be just as deadly a threat as Owen could have been. Adam panics, and is overly brutal and cruel in dealing with him - I would go as far to say sadistic as Adam took great pleasure out of tormenting him, he went too far. And of course, in underestimating Ianto, Adam underestimated his relationship with Jack. I don't think that Adam affected either of them's memories of it that much. He just made Jack preoccupied and Ianto took the sensible action and gave Jack a little space, which Adam takes advantage of. Of course Ianto immediately 'confesses' to Jack and that is where the beauty in this relationship lies (and why Adam is my favourite episode) - the absolute trust and faith they have in each other - Ianto trusts Jack to do the right thing and lock him up, Jack has absolute faith that Ianto is not the monster he thinks he is. Jack and Ianto aren't the touchy-feely couple Gwen and Rhys are but when the storm hits, Jack and Ianto's relationship is strong enough to ride it out.
The memories that define you - Adam messed with the memories that defined Tosh, Owen and Gwen - either delibrately or through carelessness. He didn't mess with the memories that defined Ianto because he didn't think he was important enough to bother with, thus failed to break the strongest alliance in Torchwood (in the Jack/Ianto relationship) and brought about his own downfall as a result.