I go kind of backwards in terms of cold weather gear.
I go with a thermal base layer, nike or Uniqlo. This is to keep heat close to the body and wick out moisture.
Then, depending on how cold, a thin layer or full sleeve pullover.
And for me, it's the chilly wind that gets to me, so I typically will have a windstopper top. I don't mind being generally cold, but when a cold gust of wind goes right through your clothes and feels like it freezes any moisture you've generated from moving around, it's the worst feeling.