It may depend on your knees, but I think you should be OK unless you have a condition that doctors have warned you not to aggravate.
I have a rough and grumpy joint in my left knee, the result of various motorcycle accidents and a carefree approach to healing and repair in my youth. Winter mornings are a pain for me, as are early-morning descents of the stairs... The lack of respite I get with fixed is more a blessing than a pain for my fooked knee.
But pedalling a fixie has never made me wince, from a ride to the shops to accompanying my daughter on a 130-mile charity ride.
The key things are these:
1. Get it set up so that it's comfortable (not a professional bike fit, just take advice and look on the internet)
2. Get the gearing right (I use 67-69", many use lower). Take advice from your LBS on this, as optimum gearing varies depending on use, terrain and general fitness as much as it does on preference.
3. As tyred says above, if you have knee worries, keep both brakes fitted.
If you have an open mind about it, you will love riding fixed-gear. Be prepared for a few surprises initially, when you forget where you are and stop pedalling.
The relative silence, the sense of being 'very connected' to the machine and the negligible maintenance will all become addictive.
Good luck!