Trying to get rid of jagged edges
polycounter lvl 9
For the past week I've been trying to figure out a way to get rid of jagged edges from my painted textures. So far I have not succeeded in getting rid of them 
. I've tried subdividing my mesh to a million polies in Zbrush in order to get nice, smooth, crisp, lines and so far it looks good at high polycounts. The problem is that once I create the texture from my polypaint with my subtool at the highest sudivision I get jagged edges after it is baked into a texture. It seems rather pointless to go up to such a high level of subdivision level only to get jagged edges.
I've even upped the resolution to 4k and I still get jagged edges. I've read somewhere online that when you transfer paint information from either baking or from one mesh to another you lose some quality. I've tried painting the jagged edges in Mudbox and I still get the same kind of result.
I was hoping Mudbox would fix it but it didn't with the current UVs that I have. However I found out that if you enlarge your UVs to fill the entire UV tile you get better quality results with painting and no more jagged edges. I tried doing that but once I transferred the paint layers to my original UVs the jagged edges came back again.
I don't know what to do now :poly134:. I am at a loss. Does this have something to do with raster vs vector? Do I have to convert it into a vector image so I can get crisp lines? Maybe my UVs are layed out wrong or something? Did I miss something while exporting my textures?
. I've tried subdividing my mesh to a million polies in Zbrush in order to get nice, smooth, crisp, lines and so far it looks good at high polycounts. The problem is that once I create the texture from my polypaint with my subtool at the highest sudivision I get jagged edges after it is baked into a texture. It seems rather pointless to go up to such a high level of subdivision level only to get jagged edges.I've even upped the resolution to 4k and I still get jagged edges. I've read somewhere online that when you transfer paint information from either baking or from one mesh to another you lose some quality. I've tried painting the jagged edges in Mudbox and I still get the same kind of result.
I was hoping Mudbox would fix it but it didn't with the current UVs that I have. However I found out that if you enlarge your UVs to fill the entire UV tile you get better quality results with painting and no more jagged edges. I tried doing that but once I transferred the paint layers to my original UVs the jagged edges came back again.
I don't know what to do now :poly134:. I am at a loss. Does this have something to do with raster vs vector? Do I have to convert it into a vector image so I can get crisp lines? Maybe my UVs are layed out wrong or something? Did I miss something while exporting my textures?
Replies
here is what i would suggest.
1) redo your uvs. break the sleeves off apply smooth/unwrap. and refit all the pieces to your uv space.
2) you can start your painting on a new 2-4k texture.
i strongly suggest you texture paint in mudbox. you can directly paint on your texture with no vertex painting/transfer necessary. plus many other advantages i will not go into here.
that should give you a fighting chance to get some good paint on your shirt.
Thanks gray for your reply. I've always thought you were not suppose to not break up your uvs further to avoid creating more seams on your mesh. Also this mesh is not the only one I'm doing. It's part of a character model that I'm trying to texture so breaking the shirt uvs down and creating new shells might not be such a good idea. I plan to have everything in one UV tile so space might be an issue.
The UV's could use some proper relaxing however.
The reason you're getting those jaggies is because you have alot of circular surfaces, that are being linearly interpolated across the surface if unwrapped in a 2d environment. Basically, pixels which are in a straight line will show up better in your texture, even at low sizes, vs. those that aren't straight.
Also, in ZB you're painting on polies, so any stretching on the polies = color trying to stretching across the limited numbers of polies, while in MB, you paint on actual texture unwrap, but last I checked, their texture previewer doesn't smooth the texture bleed.
Did you try and test the texture in an actual package, with a proper material/shader setup? How close is the camera going to get? Will there be a normal map? Spec map? Take those into account before worrying about something most people won't see.
trying to find the right number of uv shells is a balance. you have to balance the amount of texture space with the best coverage you can get. once you start packing everything into your uv space you will find you get a tighter pack and more coverage if you break some stuff into smaller tiles. how many varies,
the pixelation your getting is a combination of not enough coverage and your unwrap looks like it needs some smooth/relax.
there is no reason any more to avoid seams if it gives you better coverage. that was a rule when people painted textures in 2d Photoshop and painting over seams was hard. but with 3d painting seams are no longer an issue. i rarely if ever hesitate to cut something into more tiles if it looks like it needs it.
don't be afraid to redo your uvs if it gives you a better result. thats just part of doing good work. i have have had to re-uv stuff many times for various reasons.
I think I forgot to relax the UVs before I started painting the textures. No I haven't really tested the texture in an actual package, with a proper material/shader setup.
I do understand about whether or not to bother with the jaggies since it probably will not be seen because the head and neck are going to be in the way. However it is something that has been bothering me these last few days and I couldn't simply get over it. It just looked ugly and messy and felt like I did a sloppy job with texturing. Even though it is not a big deal I'm sure it's going to come up later in future projects so I guess it's better to deal with issue like these now rather than later.
Maybe you are right about not worrying about as many seams when it comes to creating new UV shells. It's just an old habit of mine while I was attending school and maybe it's time for a change. I was afraid you would say "redo uvs". Although I don't have a problem doing this with tools that make UVing easier nowadays it's just a bit of a tedious, boring, and repetitive job better well spent on the creative side of your project. I could get into doing Ptex but it's not as mainstream as doing the traditional UV mapping texturing for games.
P.S. This shirt I'm working on is for a game and plan to use Marmoset for test renders.