I addressed this some time ago on the Om Sweet Om Yoga LLC facebook page, but it is worth looking at again.
There are two generic, polar opposite thoughts about Yoga: "it's too easy, so why bother" or "it's too hard, so why bother?" This post is focused on the second thought - why is Yoga so d@#n hard? While there are certainly other aspects to Yoga, most people ask this question in regards to the asanas/postures. Generally speaking, asanas get really hard, awkward, or painful when you aren't ready to move as deeply into them as you do. This isn't a problem with yoga or the postures - they don't demand you carry out the deepest expression of the physical shapes. In fact, Yoga mostly just asks you to be willing to do some work, both inner and outer, and the best way to get comfortable in a complex posture is to first get comfortable in a simpler posture with the same movement theme. And no, you do not need to be flexible to practice Yoga any more than you need to already be strong to lift weights. There's no problem with your body, either - the body isn't something to be overcome in your practice, though practice long enough and your object of focus will move away from the body all the same. A Yoga practice should always rise up to meet you exactly where you are. You do have to exercise some common sense here. If you've never done a minute of asana practice in your life, are not flexible or athletic in any way, and you decide to attend a truly advanced class, virtually nothing but your Relaxation Pose will meet you where you are because you went to a class that was completely inappropriate for your experience level. It's OK - we all make bad decisions sometimes, just own it, do what you comfortably can, and next time try a beginner or all-levels class that will have some viable options for you :)
Often times classes are labeled "all levels" or "multi level" meaning options will be offered at least for beginners and intermediates. It is up to the teacher to offer appropriate options in the first place, but it is generally up to the student to choose what options they will use. Assuming the teacher has done her/his job and offered something for everyone, in these situations the difficulty may be sourced from your own ego, which DOES demand you go deeper and even push yourself beyond your limits. The beginner trying to keep up with the experienced intermediate student on the next mat can easily become overwhelmed and frustrated. Instead of looking inward, you'll be looking for the clock wondering when this painful hell called a Yoga class is going to be over. In a multi level class, try this instead: work to release attachment to an imagined end result of any given posture, and replace that with acknowledgement and acceptance of the mild to moderate resistance/challenge you're feeling (this is valid for the intermediate/advanced student who popped into a beginner class, too). Then, let go of the belief that where you are and what you're feeling should be otherwise. By doing this, you will help yourself find the perfect movement expression for you at that moment. Your postures can still require effort, but you will have taken a huge step toward balancing that effort with ease (think less is more). You'll also send a message to your ego that you're no longer interested in what it's bringing to your practice. As the ego's influence lessens, your practice will be lighter and more productive far beyond the asanas.
And moving beyond the asanas is ultimately a very valid point! What we in the west call "Yoga" generally refers to these physical postures, but asana and meditation do not have to be all that separate. Whether you move a little or a lot in any given posture, if you need to take Child's pose or Savasana for 20 minutes in the middle of a class, or if you just breeze through class with complete and total physical ease, with practice your awareness will move much deeper. Gradually, you will drift beyond the physical body, beyond the thoughts, emotions, the mind in general, and find yourself in Santosha - a state of contentment.
In the mean time, the next time you're waiting for Santosha to find you and you find yourself watching the clock in your Yoga class, just take a break, take a figurative step back, remember that it's not a race, and sometimes you just really need to try things more than once. While each and every asana may not now or ever be for you, Yoga will not limit your efforts, only you can do that.