This really doesn't seem that difficult. If you're in the left lane, not passing anyone, then move over regardless. If you stay in the left lane just because you like it there, with no one in the right, and then an a-hole comes up behind you at a ridiculous speed and tailgates, now you're both making mistakes.
On the other hand, if the right lane is moving at the speed limit or below with heavy traffic, and you want to pass at, say, 5-10 mph over the limit, IMO it's perfectly fine for you to stay in the left lane as long as it takes until there's a reasonable gap in the right lane for you to pull back in. As long as you're continuously passing the right lane traffic, then it's up to you what speed to travel -- even if someone behind you wants to go even faster.
If you're the type of person who thinks that a person should immediately vacate the left lane even though they're passing people steadily above the limit, just because you want to go even faster, then IMO that's ridiculous.