Then use greylisting only.

I've never had a case of a false positive complain but yes, they can happen and I don't terribly mind because that means the sender's mail server is blacklisted and they should know.

Also in any case my recommendation is to use a 3rd party mail service, for deliverability reasons mostly.