Not at all. Remember when Daryl, Sasha, and Abe met the Saviors on the road and the leader of that group used that same phrase, "we usually kill one of you right off the bat." Negan wasn't there either but any survivor from that encounter would walk away with that phrase in their head. Really they are just foreshadowing Lucille by using that phrase repeatedly.

Look at what Jesus says about the negotiations that took place between the Saviors and Gregory. He says "They met with Gregory on behalf of their boss. If Negan were there then that would mean he waited outside while his men went inside somewhere and talked with Gregory. Doesn't really make sense.

 

I'll need to re-watch again, but watching it I got the impression, it was Saviors there on behalf of Negan who were there and killed Rory. Since the phrase "right off the bat" was used by Jesus when discussing them, and was also used by Bud, it might have even been a group of Saviors led by Bud.

 

That's the impression I got.