Welcome to the forum!
Yes, you should have filed Form 8606 each year you put non-deductible contributions into traditional IRA. One for you. A separate one for your wife (if there were contributions in Her IRAs too). Recording this info and sending it to the IRS is what keeps you from paying tax on the money a second time. When you start withdrawing the money from your IRA, you will be given credit for having already paid taxes on some of it. That also happens using Form 8606.
You can get the old Forms 8606 on the internet. Fill out the 09 form first, then 10, then 11, then 12. Each one builds on the previous one. You can send them in all together. There is supposed to be a penalty for filing them late, but I don't believe the IRS has been charging the penalty so far.
And lets say a have a few years that I do qualify and some that I don't , how does the gov. know? and how does one keep track ???
It all happens on that Form 8606. Every time you make a non-deductible contribution, you add it to that year's Form 8606. On the years you do deduct the contributions, you don't add anything to the Form 8606 that year. But you always keep those forms.
When the time comes for taking money out, if half has already been taxed, you only pay tax on half of what you take out. If 25% has been taxed already, you pay tax on the other 75% if you take money out.
Form 8606 is pretty easy. Just follow the instructions.