I'm in a very similar boat to you - a bit younger but mortgage, 2 kids and 10 years into an ok paid career. I've managed to drop some hours, will do 30/31 in the new year and then will try to pick up part time work on my off days. A lots of ifs and buts as to whether it'll pan out, but it's been on my mind for years so need to give it a go. By not leaving my current job (as much as I would like to), I'll have some financial security and can see if it'll be possible to transition. If not, I'll have learnt some new skills and had some fun with chainsaws.
What I found promising, is if you read the survey arbtalk did last year abut how people got into the industry, and quite a few people who responded started late in life, which shows it's possible if you're willing to put in the effort.
Good luck to you!