I've been an ardent no towel user for hiking or general travel for a few years (since my general travel is by bicycle or hitchhiking, pack volume and weight is still important). Hiking, I'm happy enough to drip dry, use my buff or a fleece. If its warm enough to wash or swim, these things will dry anyway.
For travel, if I'm showering somewhere, it's usually indoors and therefor warm enough to drip dry. I wear lightweight quick drying clothes, so I can put them on when still a bit damp and it's fine.
I might be tempted to start bringing something for colder trips, where it may become necessary to wash, but it might also be uncomfortably cold to drip dry or sponge myself off with a buff or my fleece. And long hair is a bit of a pain without a towel - but to deal with long hair, you need a fairly "significant" towel.
I mostly just take 2 J cloths.
If anticipating a lot of swimming and sun, I take an old very light very thin cotton sarong .
A sarong sounds quite nice for walking in on hot days with a bit of UV protection