I can't quite tell what is going on but it just looks like faces are popping thru from the other side? That or maybe some verts are not welded. Try adjusting the the amounts and follow where the problem areas are. You may just have to fix it manually because of awkward face normals.
well, i would have said the same, see if everything is welded or
see if there is anything welded which you didnt want to be welded
if everything fails, remake it and see if it gets better.
or you put another edit poly on top of the stack and just edit the verts that stick out...
edit (over 10 years later, as I find this answer very lacking myself):
It looks like your model beneath the "Shell" modifier is unhealthy.
The first step of debugging would be to check whether you got any unwanted holes in your mesh.
Deactivate the "Shell" modifier for a second and go down into the "Edit Poly" modifier.
There you can either hit the "3" ond your keyboard or manually select the "Border" selection.
Generally marquee select the area that is giving you troubles. (Try to avoid the obvious big hole you modeled on purpose)
If nothing gets selected, you are good on this front.
Alternatively just select all the vertices and weld by distance with a very, very low number.
(If you work in a very small scale, sometimes it can be helpful to scale the mesh by factor 1000, resetting the XForm, doing the operation, then scaling back and resetting the XForm once more.)
Now, ten years later, you could also try the "Mesh Clean" modifier.
It might also be that the "Shell" modifier has to interpolate too much because the angles between adjacent faces are too steep. In this case you could try to chamfer the edges that trouble you.
The "Shell" modifier now also has a feature called "straighten corners", which you can activate by checking the tickbox in the modifier's GUI.
In the end, its hard to tell without having access to the actual scene.
If you cant get it fixed properly and you are on a tight time budget, try adding a new "Edit Poly" modifier on top of the shell modifier and then move the wrong vertices by hand.
That said, the fundamental problem probably is a lack of proper topology and / or modeling techniques, so maybe a bit more research in that direction could help in the long run. (I mean this with all due respect, I dont want to diss OP's skills!)
I´d suggest using "Reset Xform" (you can find it in utilities submenu - that little hammer icon above the modifier list) and converting into editable poly. After try using shell modifier and see if it helped. It usually helps when I have such problems - it´s really annoying issue btw., so I feel your pain.
From the pic its kind of hard to tell what is going on, any chance you could post the mesh?
Based on what I see, I don't think its a bug, but that the geometry folds back in on itself in a way that it forces the shell modifier to poke through. If you simplify the geometry around the rim it should work itself out.
Another thing to check for is flipped faces. The shell modifier works on the angle of the vertex normal, so any faces that are flipped will have inverted normals and the shell modifier will go the opposite direction.
Replies
see if there is anything welded which you didnt want to be welded
if everything fails, remake it and see if it gets better.
or you put another edit poly on top of the stack and just edit the verts that stick out...
edit (over 10 years later, as I find this answer very lacking myself):
It looks like your model beneath the "Shell" modifier is unhealthy.
The first step of debugging would be to check whether you got any unwanted holes in your mesh.
Deactivate the "Shell" modifier for a second and go down into the "Edit Poly" modifier.
There you can either hit the "3" ond your keyboard or manually select the "Border" selection.
Generally marquee select the area that is giving you troubles. (Try to avoid the obvious big hole you modeled on purpose)
If nothing gets selected, you are good on this front.
Alternatively just select all the vertices and weld by distance with a very, very low number.
(If you work in a very small scale, sometimes it can be helpful to scale the mesh by factor 1000, resetting the XForm, doing the operation, then scaling back and resetting the XForm once more.)
Now, ten years later, you could also try the "Mesh Clean" modifier.
It might also be that the "Shell" modifier has to interpolate too much because the angles between adjacent faces are too steep. In this case you could try to chamfer the edges that trouble you.
The "Shell" modifier now also has a feature called "straighten corners", which you can activate by checking the tickbox in the modifier's GUI.
In the end, its hard to tell without having access to the actual scene.
If you cant get it fixed properly and you are on a tight time budget, try adding a new "Edit Poly" modifier on top of the shell modifier and then move the wrong vertices by hand.
That said, the fundamental problem probably is a lack of proper topology and / or modeling techniques, so maybe a bit more research in that direction could help in the long run. (I mean this with all due respect, I dont want to diss OP's skills!)
Based on what I see, I don't think its a bug, but that the geometry folds back in on itself in a way that it forces the shell modifier to poke through. If you simplify the geometry around the rim it should work itself out.
Another thing to check for is flipped faces. The shell modifier works on the angle of the vertex normal, so any faces that are flipped will have inverted normals and the shell modifier will go the opposite direction.
It worked!
I am happy I was able to help even ten years after!
I found my post lacking, though, so I did an update.