Ok, I just thought you were wondering who let them in/how did they get in.

 

It is a little perplexing about that scene.
I mean, those 5 or 6 walkers are 'slow' walkers, ie not fast running ones.
So, when they were spotted in the courtyard, the chaos was not very justified. CW citizens should not go into such a panic at all. Just a quick stroll away and/or into a nearby building would suffice. It was not like there were dozens or hundreds of walkers surrouning the town square. They were coming in from 1 direction. And the citizens screamed and ran in all directions.
Perhaps this is just used to setup the death of Sebastian?