I think you need to stop smoothing your mesh and adding control loops. Work on getting the correct shapes and proportions 100% nailed down, then start adding control loops and subdividing your model.
I think it would help you a lot to not focus on the side view so much. Right now it looks like you did the majority of the work from the side and then added some supporting loops. Instead, look at the contours of some real guns and try and mimic that in yours.