I don't see a big problem here - if you love doing env art and are good at it, go for it! Also I don't think you will lose too much of your char modeling knowledge. The important stuff, anatomy, etc. keeps staying in your brain. That doesn't go away. If you want to get into char stuff again, might take a bit practice, but…