Joseph's brothers decided to make money out of him so they sold Joseph as a slave. They even callously ate a meal while Joseph was pleading and crying (Genesis 37: 24-25). Psalm 105 describes how "they afflicted his feet with fetters, he himself was laid in irons."
Now if you're following the story in Genesis 42, you'll discover Joseph's severe emotional anguish upon meeting his harsh, unfaithful brothers again. He was not able to show forgiveness to them right away. As a matter of fact, he didn't reveal his identity and spoke to them harshly. "Where do you come from?" (Gen. 42: 7) and even accused them of being spies (vv. 9-14). Joseph even held hostage one of his brothers while the rest of them returned home to get Benjamin.
Was Joseph taking vengeance on his brothers? Somewhere along the line, Joseph wrestled with bitterness against his brothers. Like all of us, Joseph was also human. I believe he really struggled with his inner turmoil. But what is clear was that Joseph did not exact vengeance against his brothers. He almost did, but he did not. Though still heartwrenching, he chose not to harbor bitterness and unforgiveness towards his brothers.
"You intended to harm me, but God intended it for good to accomplish what is now being done, the saving of many lives." (Genesis 50:20)