I think the biggest thing for me was fear of the unknown. I didn't know what to expect the first time I ever walked into an AMP. Back then, I didn't have the luxury of helpful and informative boards like this one and ASPD. You also have the unreasonable fear of thinking that everyone in the world is watching you walk in.
Lucky for me, my first time, the lady (MILF) was very well versed in taking care of newbs and she proceeded to make me feel very comfortable and took very good care of me. I was hooked from the getgo. Sadly, that isn't always the case.
Another "fear factor" for a lot of guys is the fear of getting "caught" doing something they're not supposed to be doing, especially when there's a SO involved.
But once you get over that initial fear and uncertainty, a whole new world opens up to you.
K.