What would cause wear on the outer side of the rest shooting 4 fletch?

Could be several things.  Could be stiff arrows, low brace height, or bad release.

Usually, a too stiff arrow.  Try raising your brace height.  Bill

I concur with Bill and Buckeye, too stiff. I had the same issue a while back, try increasing your brace about 1/8 -1/4" and see what happens. Another thing to listen for is a "whack" sound when you release, the stiff arrow will smack the side of your bow. When you get it right you will hear an immediate difference.

I agree that it sounds to be a stiff arrow issue, I had a Zipper Extreme that I didn't have any arrows for it but shot it a few times in the basement, every time I would shoot that bow there would be this loud WHACK sound & it was because of a stiff arrow issue, 45# bow shooting spined & tuned arrows for my 50-53# bows.

One more thing, if you know the arrows are spined correctly, it might be because you are short drawing.  If you know you are hitting your anchor, it might be because you are not getting your back into it.  If you can, post a video on the shooter's forumn.  There are some very knowledgable shooters on here, if it is a form problem, they will spot it pretty quick.  Bill