I think the best summary of how to do this is to take a look at Herb Sutter's three article set on "Minimizing Compile-Time Dependencies". https://herbsutter.com/gotw/
"The Compilation Firewall", or pimpl idiom, is a clever mechanism for getting code out of the header: https://herbsutter.com/gotw/_100/
I use this liberally. Generally, it is a runtime performance issue only on repeated allocations for which you can optimize if needed. Once-only allocations can be ignored when using compiler firewalls.
"The Compilation Firewall", or pimpl idiom, is a clever mechanism for getting code out of the header: https://herbsutter.com/gotw/_100/