If I felt like you describe it wouldn't happen twice! I'm lucky with the small band of folk I backpack with - different paces but never an issue. We're all self-contained anyway should I want to go my own way or stop early or (unlikely) go further. It's over a year since I went out on my own. I also enjoy that.