1) So, what kind of shape am I in -- am I financially ready to retire and live off the earnings of investments etc.?
2) And since refinancing isn't an option since I've paid down most of the interest, should I pay off mortgage now?
1. You appear to be able to comfortably retire. Your monthly expenses are on the high side, but eliminating the $2,500/mo mortgage payment will bring that down to reasonable levels relative to the size of your portfolio. With a $3M portoflio at a conservative 3% withdrawal rate, you should be able to draw $90,000/yr indefinately, and you have other income streams. You will need some sort of bridge plan to get you to medicare elegibility at 65, but with your resources you should be able to afford it.
2. Nothing wrong with paying off the mortgage if you have the resources, and are bond-heavy, not trying to leverage yourself for growth at this point. That said, your logic for not refinancing is wrong. You owe $103K and are paying 5.125% on the outstanding balance. The interest you've paid in the past is irrelevant. So, either pay it off asap, or refi to much lower current market rates.