High poly mesh didn't fit in GPU memory (and other issues)
Hi
I'm currently having some problems with Substance Painter I've not had before and they're really confusing me. I'm not sure if this is dependent on the size and scale of the high poly mesh(es), but whenever I try to bake my AO, Thickness or Curvature, I keep getting an error saying 'High poly mesh didn't fit in GPU memory. To try again, disable the GPU raytracing option'. Now, sure, that would fix the error, but I want to understand why I'm getting this issue as other projects have always been fine.
Also, how does raytracing work in Substance Painter? I thought that was more dependent on a GPU that can handle it, such as an RTX card?
I'm guessing that the raytracing option has been a part of Substance Painter for some time now...? As I say, this error has never come up before. Turning it off might fix the error, but I find the AO, Thickness and Curvature bakes take much longer. I've tried monitoring the GPU, but I can't see much activity. I'm using a GTX 1080, which has 8GB of VRAM. I've tried looking under Task Manager and with a hardware utility (though I may not be using the best ones) and I've not seen any activity on the GPU's side. It's hardly used any VRAM either. I've tried CPUID and HWiNFO64 and can't say I've been able to find any benchmarking tool for the GPU. What info is there I haven't seen change or anything.
Some of the other issues I keep getting involve using an FBX and an OBJ. With the FBX, I keep getting errors saying it failed to compute the bitangents and that it needs to recompute them. I tried taking my low poly into Maya and made sure to check all the Normals. They were all locked, so I unlocked them and reapplied any smoothing, either using Smooth Edges or Smooth/Hardened Edges. Not sure if they're supposed to be locked again afterwards...? I export from both Maya and 3ds Max with the Tangents and Binormals options ticked, but that still hasn't prevented the errors from coming up either. I tried asking on the Substance Painter Reddit forum and someone suggested I bake with OBJs instead. So I re-exported the low and high meshes out as OBJs, which has fixed that error, but now my material IDs don't bake correctly. Even though I ticked the option to export materials, the error I get now says 'failed to locate material, creating a new one'. I even tried ticking the option to create a material library file, but same problem. Plus, the file is ridiculously small at 4kb.
Finally, although it's ticked by default, should I leave 'Compute Tangent Space Per Fragment' enabled? What does that do exactly? I looked up the documentation, but I'm still non the wiser. Is that related to setting up your Normals/Tangent Space Normals and Smoothing Groups manually? Or should it be kept enabled if the mesh is going into Unreal?
Thanks.
EDIT: I've just done a small test, and by exporting out as an FBX, and it recomputes the Normals and Bitangents. the Normal bake for one of my meshes has come out perfect. However, if I use an OBJ, and prevent it from recomputing them, but bake comes out bad. I'm lost and confused why that would be. The only thing that comes to mind, and correct me if I'm wrong here, is because I unlocked the Normals and set up the smoothing in Maya, exported it out, and then loaded it into 3ds Max. I've noticed that if I add an Edit Normal modifier after a mesh has come from another package, the Normals show as green. Resetting them turns them blue. Are the Normals/Bitangents being screwed up from one package to another, or is this nothing to worry about?
Replies
There's about 50 questions there.
The memory thing is probably misleading unless you're trying to bake an object with hundreds of millions of triangles. Most likely something else is borked.
Using obj is almost never the answer
Normal issues - triangulate the mesh.
Meshes imported to max will have explicit normals - this is the same as locked normals in Maya. It's a way to try and ensure that normals are preserved.
Compute tangent space per fragment is not well explained in the docs. What you need depends on target engine but it will not dramatically affect the appearance of your normal map.
You don't want to recompute normals - probably ever
Basically...
Something is wrong with the mesh.
Post pictures of it going wrong using a triangulated fbx and without recomputing normals at import.
SP did complain about one part having some tris on the high poly, but it was refusing to bake any of the meshes with AO etc. Even if I deselected everything else except for a smaller part, such as the eyes, it still failed. I did a test bake of only those (set up a new scene in Max and just had eyes_high/eyes_low and exported them out) and there was no issue with baking AO. After some experimenting, I think it was that my high poly was just too much for my GPU. I had to split all of the texture sets up - exported each set in a group as the main high poly and just swapped them in and out to bake each part. That seemed to get around the issue. :)
Yeah, someone on reddit suggested OBJ and I did mention that they can cause issues with various things. I was getting problems. Also, I always triangulate the low poly on export. :) Just not the high.
Despite splitting up the texture sets and getting around the AO issue, it still came up with the 'recompute normals/binormals'. Everything has baked now, but yeah... Still couldn't prevent that error. And I always tick Tangents and Binormals when exporting from Max and Maya. Although, Maya complains about that when combined with Smoothing Groups and Triangulate enabled.
Yeah, I'm getting the same issue again with a different project. And this one doesn't have as many parts. Ergo - it hasn't taken me as long to prep and set up for baking, so I can't even imagine how I could have borked it up. :-\ It just consists of the main body as one texture sheet, the eyes, the head, and the lower body. But straight away, when I did a test bake of just the Normal map - 'Mesh normals are invalid (some values are null) and will be recomputed.' Not sure if it's just a bug with SP. Otherwise, I can't imagine what I could have done to knacker it. 🤔
I moved from 6gb gpu descktop to 16gb GPU laptop. So a huge relief . All those out of memory issues are gone. Couldn't make my desision for a year because I thought laptop GPU would be a crap anyway. Turned out it's not that much of a crap.
Can't imagine to be back working on my desctop again now even having 12 core AMD there and twice more RAM
So my guess a videocard with 16 gb is a main necessary thing now .
Yeah, one day I'll need to. I already had to upgrade my laptop's RAM from 8GB to 16. Can still struggle a bit. As things are, I'm not able to give my PC a full upgrade any time soon. I've been using a GTX 1080 for the past 4-5 years or so. I'd have to get myself a whole new mobo too before I could upgrade my CPU. And I need to for the RAM. I'm still rocking a Haswell chip and DDR3 RAM. 😬
I just upgraded from my 2GB of VRAM to 12GB.
There was the time when no one really cared about VRAM but today I would take every bit I can get after figuring out that substance painter just eats 11GB for one of my character textures in 2k alone. 😬
Memory really isn't the problem here - no mesh is going to eat 8gb of memory.
there's something dirty in the exported model that's upsetting painter - whether that's an authoring issue or an export issue I can't say but it's not running out of GPU memory cos there's too many vertices
Well, I've tried three low poly meshes (projects) and had the same message from SP about the Normals, so I can't hazard a guess what or where the problem is. Whether it's something from Maya or Max... I'll try a very simple mesh, like a cube, and see what happens.
EDIT: Hmm. Didn't come up with a simple cube experiment. That was from 3ds Max.