As far as I know, loops don't work if you don't have continuous quads. That's kinda the point of loops. On both sides of your problem you have polygons, it only works on quads. From Wikipedia, searching Edge loop (computer graphics) "In a stricter sense an edge loop is defined as set of edges where the loop follows the…
I had this issue the other day, make sure your vertices are all welded. I had a lone vertex hanging out in the middle of an edge and it was messing my swift loop up.