We managed to have some success in making our agents unique by adding intrinsic rewards, although the results may have been not as unique as we would have liked. This means we now want to look at modifying the agents' extrinsic rewards in the hopes of making their behavior more unique and ultimately more entertaining for the game. 

The best way for us to start doing that is to look at the SoccerTwos reward functions we described earlier; these are listed here, for reference:

What we want to do now is apply some individualistic modification to the rewards function based on the current character. We will do this by simply chaining the functions with a modification based on the character type, as shown:

  or 
  or 
or 
or 

All we are doing here with these reward functions is simply modifying the reward value by some personality modification. For the girl, we give her a bonus of 1.25 x the rewards, reflecting that she may be excited. The boy is less excited, so we modify his rewards by .95 times, which reduces them slightly. The policeman, who is always calm and in control, remains constant with no rewards modifications.  Finally, we introduce a bit of a wildcard, the half-dead zombie. In order to characterize it as half-dead, we also decrease all of its rewards by half as well.  

You could, of course, modify these functions in any way you please, according to your game mechanics, but it is important to note that the effect of the personality modification you are applying could hinder training. Be sure to take a mental note of that as we get into training this example as well.

A girl, a boy, a zombie, and a policeman enter the soccer field.

Now that we understand the new reward functions, we want to add to our example that it is time to open Unity and code them. This example will require some slight modifications to the C# files, but the code is quite simple and should be readily understood by any programmer with experience of a C-based language.

Open up Unity to the scene we were modifying in the previous example, and follow the next exercise:

1. Locate the RedStriker  agent in the Hierarchy window and select it.

2. From Inspector, click the gear icon beside the Agent Soccer component and, from the Context menu, select Edit Script.  This will open the script and solution in your editor.
3. Add a new enum called PersonRole at the top of the file right after the current enum AgentRole and as shown in the code:

public enum AgentRole
{
  striker,goalie
} //after this line
public enum PersonRole
{
  girl, boy, police, zombie
}

4. This creates a new role, for, in essence, the personality we want to apply to each brain.
5. Add another new variable to the class, as shown:

public AgentRole agentRole; //after this line
public PersonRole playerRole;

6. That adds the new PersonRole to the agent. Now we want to also add the new type to the setup by adding a single line to the InitializeAgent  method, shown here:

public override void InitializeAgent()
{
  base.InitializeAgent();
  agentRenderer = GetComponent<Renderer>();
  rayPer = GetComponent<RayPerception>();
  academy = FindObjectOfType<SoccerAcademy>();
  PlayerState playerState = new PlayerState();
  playerState.agentRB = GetComponent<Rigidbody>();
  agentRB = GetComponent<Rigidbody>();
  agentRB.maxAngularVelocity = 500;
  playerState.startingPos = transform.position;
  playerState.agentScript = this;
  area.playerStates.Add(playerState);
  playerIndex = area.playerStates.IndexOf(playerState);
  playerState.playerIndex = playerIndex;
  playerState.personRole = personRole;  //add this line
}

7. You should likely see an error now in the line. That is because we also need to add the new personRole property to PlayerState. Open the PlayerState class and add the property as shown:

[System.Serializable]
public class PlayerState
{
  public int playerIndex; 
  public Rigidbody agentRB; 
  public Vector3 startingPos; 
  public AgentSoccer agentScript; 
  public float ballPosReward;
  public string position;
  public AgentSoccer.PersonRole personRole { get; set; }  //add me
}

8. You should now be in the SoccerFieldArea.cs file.  Scroll to the RewardOrPunishPlayer method and modify it as shown:

public void RewardOrPunishPlayer(PlayerState ps, float striker, float goalie)
{
  if (ps.agentScript.agentRole == AgentSoccer.AgentRole.striker)
  { 
    RewardOrPunishPerson(ps, striker);  //new line
  }
  if (ps.agentScript.agentRole == AgentSoccer.AgentRole.goalie)
  { 
    RewardOrPunishPerson(ps, striker); //new line
  }
  ps.agentScript.Done(); //all agents need to be reset
}

9. What we are doing here is injecting another reward function, RewardOrPunishPerson, in order to add our extrinsic personality rewards. Next, add a new RewardOrPunishPerson method, as shown:

private void RewardOrPunishPerson(PlayerState ps, float reward)
{
  switch (ps.personRole)
  {
    case AgentSoccer.PersonRole.boy:
      ps.agentScript.AddReward(reward * .95f);
      break;

    case AgentSoccer.PersonRole.girl:
      ps.agentScript.AddReward(reward*1.25f);
      break;

    case AgentSoccer.PersonRole.police:
      ps.agentScript.AddReward(reward);
      break;

    case AgentSoccer.PersonRole.zombie:
      ps.agentScript.AddReward(reward * .5f);
      break;
  }
}

10. That code does exactly what our earlier customized reward functions do. When you are done editing, save all your files and return to the Unity editor. If there are any errors or compiler warnings, they will be shown in the console. If you need to go back and fix any (red) error issues, do so.

As you can see, with very little code, we are able to add our extrinsic personality rewards. You could, of course, enhance this system in any number of ways and even make it more generic and parameter-driven. In the next section, we look to put all this together and get our agents training individually.