cford, there are two defenses against riding when it's cold outside:
1) Layered insulation
2) Wind blocking
It sounds like you are going in the right direction. My main piece of advice on the insulating layers is to avoid cotton and seek all technical (man-made) fabrics. They wick moisture without causing evaporative cooling.
I have found you don't need all that much insulation on your legs. At 41 degrees, I would wear sports long underwear (it's thin), track wind pants, and my riding pants. I happen to only have mesh pants, so in cold weather I ride with the waterproof liner installed as a wind break.
Up top, I wear sports long underwear, a long-sleeve technical sports pullover, and windbreaking jacket or liner, and an outer windbreaking riding jacket. I'm big on wind protection; all the insulation in the world won't help you if the cold air can penetrate and defeat the trapped air layers.
I also use a Schampa Shielded chin guard
. I put on all my under layers, then the chin guard, then the outer riding jacket.
I don't know if you have insulated riding gloves. They can be expensive. I was once caught in colder than expected weather with non-insulated gloves. I pulled out my low-buck Moto-Centric rain mitts
as a wind break and it made a huge difference! Another low-cost option is thin glove liners.