Can I go behind my Agencies back, and contact their clients directly?

Hi all, first post here.

I'm a start up - looking to get into the freelance world. I'm struggling to find my first clients and my agency doens't seem to be pulling through.

I can't find an answer to my question anywhere so thought I'd post here. Has anyone done this before? Is it a taboo. Are there any consequences in doing this and will my agency catch on and hate me / put a 'bad word' out about me?

Thanks to anyone who replies with some info!

Ash.