I ran into this problem this past summer when I got stuck with the night shift....I'd go into work at 9pm (easily in the 70's, no problem), and then have to ride home at 5am(often in the mid 40's). I'd freeze my nuts off going home, especially since with all the peaks/valleys around here it's very common to get foggy/misty mornings.
I normally use Fox MX gloves on the bike as well, which are fine down to the low 60's. I have a pair of insulated/waterproof hi-viz gloves for work, got them at a work wear/safety store for like $30. Youngstown....I think? They're quite warm and wind doesn't penetrate them....after 45min of highway speeds in the 40's my hands would be a bit chilly but not bad.
I'd snap in the windproof liner of my mesh jacket(which otherwise has absolutely zero warmth factor), and wear a hoodie underneath....that usually worked well.
So a small backpack with a hoodie, the jacket liner, and heavier gloves sufficed for me. I've never had the bike out in near or sub freezing temps....don't want to worry about black ice with only two wheels.