Its not the Cams that wear out, however I went with the Andrews 54`s good wide torque band and not heavy on your valve springs (actually I changed out the valve springs as well for something less stressed, you should be looking at the inner cam Bearings (Go to Torrintons or equivalent) they can collapse, as you stated the lifters (many good choices S&S/Fueling etc) and also many are upgrading the cam chain Hydraulic tensioners I went with the zippers.
But most of all you need to check the Crank "Run Out", and if above 4 thousands of an inch then check your oil pump!!
If you have 40k without any upgrades you have done well, Don`t wait.
Good Luck.