Merle was very smart (the Gov. is a good influence on him). Putting down his gun as soon as he recognized Glenn made him much less threatening. As soon as Merle saw Glenn, he knew that the group was alive and well so he had a chance to acquire a fresh new source of resources for Woodbury and maybe even get revenge on Rick. Plus maybe his only surviving kin and he could have a nice reunion. Merle controlled himself very well throughout the whole encounter. Given what he just went through, I'm sure he was hoping that here were a nice, soft couple of kids he could take out his Michonne-related frustrations on.

Glenn would have been smart to put him down, but we know that would be seriously out of character for him. I'm sure that Glenn thought: Boy, it would be good to kill Merle. He's crazy dangerous. But he's Daryl's brother. And Daryl is totally broken up about Lori, T-Dog and Carol being dead - I'd better not do anything to drive Daryl over the edge because oh boy do we need him. Add not wanting to kill a human in front of Maggie (who probably hasn't heard anything about Merle) and you have a recipe for total paralysis of the trigger finger.