I'm currently having some problems with Substance Painter I've not had before and they're really confusing me. I'm not sure if this is dependent on the size and scale of the high poly mesh(es), but whenever I try to bake my AO, Thickness or Curvature, I keep getting an error saying 'High poly mesh didn't fit in GPU memory. To try again, disable the GPU raytracing option'. Now, sure, that would fix the error, but I want to understand why I'm getting this issue as other projects have always been fine.
Also, how does raytracing work in Substance Painter? I thought that was more dependent on a GPU that can handle it, such as an RTX card?
I'm guessing that the raytracing option has been a part of Substance Painter for some time now...? As I say, this error has never come up before. Turning it off might fix the error, but I find the AO, Thickness and Curvature bakes take much longer. I've tried monitoring the GPU, but I can't see much activity. I'm using a GTX 1080, which has 8GB of VRAM. I've tried looking under Task Manager and with a hardware utility (though I may not be using the best ones) and I've not seen any activity on the GPU's side. It's hardly used any VRAM either. I've tried CPUID and HWiNFO64 and can't say I've been able to find any benchmarking tool for the GPU. What info is there I haven't seen change or anything.
Some of the other issues I keep getting involve using an FBX and an OBJ. With the FBX, I keep getting errors saying it failed to compute the bitangents and that it needs to recompute them. I tried taking my low poly into Maya and made sure to check all the Normals. They were all locked, so I unlocked them and reapplied any smoothing, either using Smooth Edges or Smooth/Hardened Edges. Not sure if they're supposed to be locked again afterwards...? I export from both Maya and 3ds Max with the Tangents and Binormals options ticked, but that still hasn't prevented the errors from coming up either. I tried asking on the Substance Painter Reddit forum and someone suggested I bake with OBJs instead. So I re-exported the low and high meshes out as OBJs, which has fixed that error, but now my material IDs don't bake correctly. Even though I ticked the option to export materials, the error I get now says 'failed to locate material, creating a new one'. I even tried ticking the option to create a material library file, but same problem. Plus, the file is ridiculously small at 4kb.
Finally, although it's ticked by default, should I leave 'Compute Tangent Space Per Fragment' enabled? What does that do exactly? I looked up the documentation, but I'm still non the wiser. Is that related to setting up your Normals/Tangent Space Normals and Smoothing Groups manually? Or should it be kept enabled if the mesh is going into Unreal?
EDIT: I've just done a small test, and by exporting out as an FBX, and it recomputes the Normals and Bitangents. the Normal bake for one of my meshes has come out perfect. However, if I use an OBJ, and prevent it from recomputing them, but bake comes out bad. I'm lost and confused why that would be. The only thing that comes to mind, and correct me if I'm wrong here, is because I unlocked the Normals and set up the smoothing in Maya, exported it out, and then loaded it into 3ds Max. I've noticed that if I add an Edit Normal modifier after a mesh has come from another package, the Normals show as green. Resetting them turns them blue. Are the Normals/Bitangents being screwed up from one package to another, or is this nothing to worry about?