There's a lot of ways that teaching and DMing have a great deal in common, and one of those key areas is classroom/table management. Keeping the kids engaged and behaviors in line is a skill that can make or break a social skills group, and while a lot of behavioral management is based on building an engaging scenario, there's more to it than that- You're still going to run into behavioral issues, and it's good to prepare for them ahead of time and have codified ways to addressing them. Without further ado, here's the tools I use in my groups.
Behavior tracking on the whiteboard. This one is amazing, especially in larger groups, but only works if you have a co-facilitator. You put two categories on the whiteboard- Positive behaviors and negative behaviors. The positive behaviors include things like 'Raising your hand to talk' or 'Being ready for your turn' or 'showing respect to others,' and the negative ones include 'interrupting others,' or 'delaying game', etc. For the positive behaviors, for each tally when it's noticed, it nets the entire party +25 XP. Negative behaviors, -50. This results in a lot of self policing, and I've actually seen it in action- There was one session where the kids were really rowdy, and little progress was occurring, until one kid hissed, “Look at the board!” and suddenly, they were on task and quiet. Another time, a kid said, “I want us to level today. Guys, let's get a lot of extra XP by being on our game,” and I have to say, they were the best behaved I'd ever seen them.
This works for a number of disruptive behaviors, but I've also seen disrespect happening at the table. Very often it's not intentional, but it still is hurtful. The case that sticks to my mind is one kid saying to a younger, less experienced kid, “What are you thinking? That's stupid!” He didn't mean anything by it, but the kid looked genuinely hurt- I actually didn't witness this happening, but the co-facilitator did and brought it up with me later. In this case, what worked was me talking to the older kid about what he did in private, and how I wanted him to take more of a mentoring role with the younger kids and support them. He did a complete 180, and he now has taken the younger kid under his wing in mentoring him to be a better player. This was a really great outcome, and I think by appealing to this kid's want to be a better person, he was able to turn his behaviors around.
Another thing I see a lot of is kids getting overstimulated. This can manifest in some pretty extreme behaviors, but I always say in a quiet voice, “Hey, do you need to get some water,” or “Hey, you need to take a quick break?” as an option for them to step outside the room and collect themselves. By presenting this option, they either correct the behavior, or realize that they do need to take a few minutes. I had one kid with pretty severe anger issues make a complete turn around by remembering that, if he was about to raise his voice, he needed to leave the room for a few minutes until he found his calm, and has since had zero outbursts, save for one, where he shouted, “Guys it is entirely too loud, and I need you to all be more quiet. And I'm not saying this in character. Thank you.” It worked, the kids quieted down, and we continued on.
That being said, there's still times when unacceptable behavior happens. We've had kids absolutely blow up and get very aggressive, and at that point, it all comes down to deescalation, then after care. Deescalate the situation, then give the kid that blew up space, and focus on making sure all of the kids are ok. Take some time with each kid to talk about it, then talk with the offender and, if there is a way to make it right, establish what needs to be done- usually an apology to the group for their behavior as a starting point. I've seen this work out well once, and fail once- when it worked out, the kid was genuinely repentant and wanted to make things right, and that was that. In the case where it didn't work out, there was a bit of an ego trip involved, and the kid genuinely did not believe he did anything wrong, so he left the group. The big thing to understand is that these things happen, they're really rough, but don't let them destroy the group. Make sure everyone is ok, then move on.
The last thing, and I think this is the most key component, is building relationships. People tend to be more engaged when they're around people they like, so I try to really build a good relationship with all of my kids. Aside from all the organic relationship building that occurs from being in a game together, there's a few formal practices I have. The first is to make sure each kid has a chance to shine every session- and this means creating player/PC specific content in each session plan. This shows that I care about them and want them to be engaged, and that they have a plot hook. It also means that I can say, with all genuineness, “Ahh! Good, you're here. I have something special planned for you.” I also spend the 15 minutes before and after game chatting with the kids and making small talk with them or their parents. This is invaluable time, as it gives you a great amount of insight into what's going on in their lives. Finally, I've started spending 2-5 minutes with each kid at session zero, just chatting with them. Getting them away from the table dynamic and giving your full attention to them gives you a great amount of insight, and is a great way to build up that positive relationship.
Hope this helps you with your table management. I'm still working on other ways to incorporate better ways to manage behaviors around the table, but at this point, here's what has worked the best.