ASA and IPSEC VPN filtering

I have an ASA and I only want specific IP's to be able to access my ASA to
form an IPSEC peer. I created a rule for the outside interface to only
allow specific peers to be accepted via isakmp, and ESP, but the rule
never gets any hits. Is the ASA like the routers and the ACL's do not
apply to the ASA interfaces itself? Is it possible to filter out what
IP's I want the ASA to respond to via ESP and isakmp via an ACL? Long story
short I am being asked to do this b/c of aggressive mode for my VPN's.

Thoughts?

-- 
Christopher D. Copley
copley.chris_at_gmail.com
Blogs and organic groups at http://www.ccie.net