I was thinking of just getting the down underwear top and using it as a midlayer, not next to the skin. Anyone done this?

Mr. BCD,
That is exactly how I use mine.  Synthetic next to the skin, down as a mid layer and wool for the outer layer.
At my age, I never walk/hunt fast enough to work up a sweat.  Up here, "you sweat/you die", is a common expression with a lot of history to it.
Best Wishes

I used to wear down, but no matter what I did my back got wet with sweat.  I remember reading an article of a man that dumped his brand new solo wood canvas canoe in the Wisconsin River after Christmas, the temps were near zero and he was miles from anything. He pulled up on shore and was waiting to die, when he realized that he was no longer cold enough to die.  Every bit of clothing he had on except for his rubber thinsulate boots was wool.  So he put his canoe back in the river and paddled home.

That's thinking positively!