So as a newbie I have been reading about fine tuning a bow. But what if the bow has no shelf? How would you find out where to put the nock unless its trial and error? Also what would you need to tune besides arrows for a shelfless bow?

The process is much the same........your hand is the shelf. Nocks are set by measuring and fine tuned by trial and error anyway.
Bows don't get tuned much.........brace, nock, silencers. There is much more to tuning arrows shelf or not.

you must have a ref.point of mark on your bow right?than it about the sane like Bjorn said.you will need a lighter spine arrow to get proper flight though.

I like shooting shelfless bows, and maybe every fourth bow I build, I go that way.  You do need a reference.  Most shelfless bows do have a leather riser wrap.  I like to stick a thin wedge of leather under the edge of the wrap.  You can usually pry the edge of the wrap out enough to just slide the wedge in, with a little Barge Cement on it.  If you want to get fancy and be kind to left handed friends, put one on each side.   The wedge isn't enough to be a shelf, but gives a good reference for hand placement and arrow placement simultaneously.   Here's an example:





You can see that I've also inset a 3/8" dot of leather as a rub point.  It stands just proud enough of the wood to give some protection.

nice bow dick.

Oh and one more thing, The bow I have in mind is a 40# horse bow that is 30" with no shelf of course. I have read that horsebows tend to like stiffer arrows, so does stu's calculator thingy work still? I have talked to a bowyer saying that usually the arrow is 10# lighter rated poundage. How true is that for all you experienced archers/bowyers/wizbangs.

I've found most bows without a shelf require a weaker shaft then the weight of the bow.Because it is not even close to center.I had a 55# BBO with no shelf and 45-50# spine arrows shot great out of it.You may have to try a few different spines to find out which spine will work best for you.Your draw length and length of the arrow are other factors too.

With my selfbows I spine arrows about 10# lighter than bow weight. You can weaken the arrow spine also by adding more weight up front or by making the arrows longer. For each inch over 28" you reduce the spine weight by 5# so a 50# spined, 30" arrow will shoot like a 40# spined arrow. I'm not sure of the tip weight values.

Seems like I read a post here somwhere about tuning self arrows (cane or shoot shafts) that sanding them in the center reduced spine. Is this right or wrong?

Stiks

You can reduce the spine on shoot arrows and dowel arrows by sanding the center. By reducing the center of the shaft you lessen the force needed to deflect around the bow. I don't think it will work on cane but cane has a wider spine range than do other shaft materials. My cane arrows shoot well from bows that are 40+# to almost 60#. I also cut natural shafting long...29" to 30" for my 26" draw.