One of the fundamental pieces of RPGs is the concept of rewards- from XP that unlocks additional abilities to gold that can be used to purchase weapons, or magical items or information that helps advance the plot, rewarding your players for playing the game well is a key motivator for continuing play. This is an extremely valuable tool in motivating players to work towards goals in applied RPG groups, but it's important to understand what exactly are effective rewards and how to use them in motivating players towards goals.
Every player comes to a RPG table with a different agenda. Some want to have fun with friends, some want to kill monsters and get cool items, while others want to explore the world and develop their characters. Generally through game you will see what motivates each player, and you can utilize these motivators to reward players behaviors. However, if you are not sure, it's always fine to ask what a player wants to get out of the game, and after a few sessions, they will generally have a very good idea of what they find fun. Once you have that, you can start providing those rewards for behavior you see at the table. This may be something very overt such as rewarding XP for specific behaviors (and having XP penalties for behavior that is not desired), or something more organic, like rewarding a player with a magical item after they've shown considerable growth (or even using a magical item or plot device to encourage socialization.)
But when discussing rewards, it's important to look at what games provide out of the box. In this discussion, I'm going to be referencing Dungeons and Dragons, but most RPGs have some type of system for rewards that this can be generalized to. I'll also be discussing consequences briefly.
Discussing extrinsic rewards refers to things that are specifically measurable in the game, and impact that. These are generally very easy to provide, but you need to ensure that the players see these as worthwhile, and be aware that it can be very easy to overdo it, which may reduce the effectiveness.
Mechanical rewards: When discussing mechanical rewards, I'm talking about XP and gold. Generally, D&D supports providing these for completing quests, but there are many ways to use these rewards around specific behaviors- For example, we use a behavior tracker on a whiteboard for our D&D social skills group that provides bonus XP awards (or penalties) based on behavior. The benefit here is that the group has a very clear connection between their behaviors at the table and the progression of their characters. However, if a player is more motivated around role playing, these bonuses may not be as effective. Encounters that generally support mechanical rewards are generally oriented around combat, but you can apply them fairly broadly across all kinds of situations.
Special Items: Special items can be very tricky, as they can very easily result in an overpowered character, but they can also be a tool to help a player reach a goal (as well as rewarding a character for specific behaviors.) A good example of this was magic goggles I gave to one character that allowed him to see magic energy fields. Suddenly, a normally very quiet kid was put in charge of scanning areas for magical auras, and he quickly had reason to engage with the story, and the social expectation that he would be speaking up whenever they entered a new environment. Other examples include sentient items that only the bearer can communicate with, a single use, overpowered spell, a communication device, or a map/compass. Encounters that reward special items are often based around dungeon delving and puzzles, but they can be a reward for good social encounters as well.
Plot rewards: Many players come to RPGs looking to experience a story, and when you have identified this as a motivator, it can become a very powerful one, as they will do everything in their power to engage with the plot (or derail it as they see fit.) Note that this sometimes results in players who want to derail plots, but even this can be anticipated and utilized, and often times these players will become more focused in making a game interesting over anything else. I've found that such players respond very well to having something really unique and versatile happening to them, such as giving them a curse or mysterious magical item of unknown purpose. A good example of this was a battery pack I gave to a wild magic sorcerer- An arch wizard told him that he needed to wear this to keep his effects under control, but it had a gauge on it that would increase at random intervals. The player became incredibly interested in understanding how it worked, and that became a very good way for him to connect with the rest of the party, as they too were worried when the pressure gauge started going into the red. With players like this, it's important as an applied GM to not step into the mistake of becoming overly attached to your story plans- instead think about the broader group goals, and see the rapidly shifting plot as a tool to achieve that. Other plot rewards include getting a deed to some land, being given a special job, gaining a henchman, or having a part of the character's backstory explored. Plot rewards are very easy to provide for social encounters, but a skilled GM can find places to insert them into dungeon, or even combat encounters.
Social rewards: Many players come to the table just to hang out with their friends, and there's ways to utilize role playing to do this. By creating situations where people can riff off each other and have really good role playing moments, they are getting that desire met. Often times these players can be very useful to the group as a whole, as they tend to want to build group cohesion and do everything they can to make sure everyone is in a good space. By creating space for fun RPing and humor, usually by having social encounters, you can reward your players with a funny conversation or a great opportunity to have a moment in the spotlight.
Sometimes, there is a need for consequences in a game. While the obvious one is killing off a character, I don't think that's an appropriate thing to do in most applied groups. Gary Gygax stated that if a player chooses to wear no armor and wander into a Red Dragon's lair, that is suicide and the character's death is fine. However, in the event of a bad string of dice rolls, it's better to create a consequence that does not end that character's story, but adds some heavy consequence for failure- a physical maiming, loss of prestige, a key item being damaged, etc. While I generally do not like to use punishments in my games, I do enjoy consequences, and I tend to use the 'Rule of Interesting' (a subset of Rule of Cool, where I try to make any consequence something that will ultimately make the game more interesting, while also meeting the need for that consequence.) A good example of this happened in a recent social skills group session, where a very impulsive player had their character approach a box that was containing a very powerful magical artifact and start drawing on it. He was immediately teleported to an alternate realm, and was not able to engage with the group until I shifted scenes to where he was. The player had to work on waiting patiently until the rest of the scene had finished, but then I described what he was seeing, which gave him some clues to help advance the plot, and eventually his character will escape from the shadow realm and be able to explain what he saw. Other examples of consequences can include a fine, magical items breaking, the PC having a time out (getting knocked out or going to jail), or being sued. But note that it is important to make sure that the consequence is used to reinforce the behavior you want to see- if the goal is teamwork and the party is playing a bunch of murder hoboes, but doing amazing teamwork, then no consequence is necessarily needed. Perhaps some adjustment to work on giving their characters some empathy, but if the players could normally not sit down at a table for 2 hours and talk without arguing, but getting their aggression out on townspeople together results in healthy socialization, then it's ultimately hitting that goal.
Finally, a note on death. I never kill PCs (except when I do.) What I mean by this is I will not kill a PC without explicit permission, and usually this is a result of them deciding they no longer want to play a character so they want to kill them off in some interesting or humorous fashion. The one exception of this was a Willy Wonka themed session where, one by one, the party was killed off, but after each death sequence I would narrate the dead PC waking up in a glass tube, with no memory of what had happened over the past hour. Over the next few days, it became obvious that they were all clones, and the group absolutely loved that twist, because while their PCs had absolutely died, their story got to continue in an even weirder way.
As a note, the way you utilize these rewards should be very much based around what type of group you are running. I primarly run social skills groups, but if I were running an educational group, or a more therapeutically oriented group, I would drastically rethink how I would apply rewards.
Anyway, I'd love to get more feedback and thoughts on rewards. Reach out to me on Twitter at @rollforkindness!