You hit right on the main drawback of a fanny pack. The other drawback is that for a fast draw it takes two hands. You can draw with one but the action is substantially slower (more awkward and fumble prone).
Milt Sparks co. makes a summer special holster that has a thin laminate layer. This layer is protection from persperation when using IWB holster. There are holsters make that have the belt loop sewn to the bottom of the holster. This allows the shirt to be tucked in hiding the pistol. The only signature showing is a loop disguised to look like a key holder loop. The drawing method is much like the 'hackthorn rip'.
The pager pal type of holster seem to work well for smaller pistols.
I'm not a fan of the Thunderwear although I hear that they work well. I find them somewhat uncomfortable. In your scenario, I would investigate one of the above options, ankle carry, or a fanny pack that is reinforced. If you choose the fanny pack method make sure the strap is reinforced, when carrying turn up your awareness, and you are licensed. This would seem to eliminate the problems you brought up.