You can manage your Friends by tapping the two-person icon at the top of the main screen.

From there, tap the trashcan icon on the row of the player you wish to remove.