Home Technical Talk

Paint vertex normals in 3dsmax

sublime tool
Offline / Send Message
Vexod14 sublime tool
Hello there !

I was wondering if one could paint vertex normals in 3dsmax using a brush system, not necessarily the one used to do vertex paint as this might have different needs, I'm thinking about something like hairstyling (in terms of UX), and smooth, pinch, average, weighted, masks types (limit to front faces, limit to selected vertices/faces/edges, etc)

We can transfer normals, weight them, and that's already super cool ! But we still don't really have a direct way to edit them (the edit normals is too painful as is, it's okay for very small edits, but imagine a big rock, or hairstyles...)

Maybe I'm unaware of a fantastic plugin/script which already does the job, If you know one, I'd love to hear out from it  :3

Replies

  • Eric Chadwick
    I’m not aware of any. But I wonder if you could wire something up between vertex paint and data channels.
  • Noren
    Offline / Send Message
    Noren interpolator
    Not aware of any, either.

    Kinematic LAB has a normal texture painter, though, that also can transfer the colors to VrayFur, so maybe you could calculate vertex normals from that:
    https://www.youtube.com/watch?v=Wz0Jy0q3WeU

    Other than that, as a basic idea:
    Have your base mesh, smoothing groups and all.
    Make a copy where you split every hard edge.
    Create an inflated/pushed copy of that.
    Manipulate that copy with brushes (use your first copy as a morph target to restore some bits if necessary, tyConform probably offering the most options for the latter.
    Use mascript to calculate your normals from vertices with identical vertex IDs in your two copies. Main problem would be that it's somewhat visual/intuitive, but not as much as seeing the vertex normals directly. Maybe you could have them drawn interactively and do the rest under the hood, but that most likely would hardly be... interactive, at least with basic scripting.
    Also, there's the problem of getting the vertex normals back to the unsplit mesh, but if I recall correctly, there are options to project vertex normals on a per face basis.
  • Vexod14
    Offline / Send Message
    Vexod14 sublime tool
    Thank you for the answers guys, I'll keep an eye on the topic if a solution comes up, what I want to get is not very clearly stated I think, I'm sorry for that 
  • Noren
    Offline / Send Message
    Noren interpolator
    It's described clear enough, it just doesn't exist (to the best of my knowledge). So if you need something that at least approaches what you want in the short term, you'll probably have to frankenstein something together.
  • Neox
    Offline / Send Message
    Neox grand marshal polycounter
    yeah for flow man creation for like hairstrands i also always hoped for a method that lets me comb and interpolate vertex normals and then bake these down into a flowmap or vertex color. i always hated painting flowmaps in 2d or painter
  • Noren
    Offline / Send Message
    Noren interpolator
    Had a look at Hair and Fur:
    The guides (Display Guides in the Display rollout) are hard coded to 15 segments, but if you scale all of them down enough right at the start and choose the tips as selection method, you can manipulate them reasonably well. "Stand" would be similar to a gradual reset/revert, "Clump" to pinch. 
    You can turn the hair display off or set their amount to zero, but you can also keep them on in order to see a bit better what's going on, albeit interpolated between vertices. Different from the guides, you can set them to one segment at least, so that might help with visualization, too.

    When you are happy with your result, you can then convert the guides to splines and use Normalize Spline with a large enough distance so each of them has two vertices/knots each. Using maxscript to get the vertex normals for your original mesh from that should be feasible. This approach ignores smoothing groups, though.
  • Klunk
    Offline / Send Message
    Klunk quad damage
    We can transfer normals, weight them, and that's already super cool ! 

    haven't you answered your own question? ..... it's not painting the normals as such but you could create a "coarse" mesh, cage as it were, and use that with "transfer normals" to perturb the normal on the target mesh ?
  • Vexod14
    Offline / Send Message
    Vexod14 sublime tool
    I already use that method yes, but for some cases I found it lacks responsiveness. I'd really like being able to "hairstyle" vertices normals using a paint brush, to get those happy accidents, and a more direct fine-tunning control. And I see through all the answers that it's going to be pretty technical (which always fascinates me, in a good way ! I'm not easy with maxscript, I can edit what's already existing and...that's it)
    Thank you for your help anyway, be sure I really appreciate it !
  • Noren
    Offline / Send Message
    Noren interpolator
    @Klunk: The problem with projecting/transferring the actual normals of a mesh is that it's a closed surface (at least usually, maybe you could break things up into many smaller units, but then you'd have to manipulate those somehow).
    Imagine you want to comb your vertex normals sharply in one direction over a larger area: That would be a constant steep slope that would move away from the target surface fairly quickly. Likewise, there might be problems with overhangs or more complicated meshes in general. 
  • Klunk
    Offline / Send Message
    Klunk quad damage
    another alternative method is to use a basic normal map shader - paint into the map with photoshop (rtt a sphere to a plane normal map gives a good palette picker) or a 3d paint app (to give you your "real time" interface) then do a map to map channel (IIRC there a free plug that does this) then do a map channel to normals (there are  free scripts for this) 
    map to vertex color
    p.s. you might need a modified map channel to normals script to use a tangent space normal map :/ ( 2 * (vertcolor - 0.5))
    it works in principle.....

    bitmap texture -> map channel (vertex color) -> normals
  • Neox
    Offline / Send Message
    Neox grand marshal polycounter
    Klunk said:
    another alternative method is to use a basic normal map shader - paint into the map with photoshop (rtt a sphere to a plane normal map gives a good palette picker) or a 3d paint app (to give you your "real time" interface) then do a map to map channel (IIRC there a free plug that does this) then do a map channel to normals (there are  free scripts for this) 
    map to vertex color
    p.s. you might need a modified map channel to normals script to use a tangent space normal map :/ ( 2 * (vertcolor - 0.5))
    it works in principle.....

    bitmap texture -> map channel (vertex color) -> normals

    my idea is actually the other way around, being able to edit/comb these and then baking those into a texture. painting flow maps is just really cumbersome :)

    because on stylized hairstrands its a looooot of strokes, a looooot of blending things, while its just a few vertices/normals to bend
  • Klunk
    Offline / Send Message
    Klunk quad damage
    true, ideally you want some kind of clustering/master normals acting as a vector field within the edit normals modifier, i've done similar with my tree stuff :/
  • poopipe
    Offline / Send Message
    poopipe grand marshal polycounter
    klunk is completely correct about needing a full vector field - the data has to be continuous and smooth or all manner of stupid shit happens in the gaps

    the best "don't need to write a special tool approach" I've found is to use a hair sim, apply a material that writes out the derivatives along each hair (arnold has a node ) and bake the result to a texture.  it'll get you a direction map rather than a flow map but you can get flow from that fairly trivially

  • pxgeek
    Offline / Send Message
    pxgeek interpolator

    Interesting stuff.

    Noren could be on to something. The HF mod even places guides at each vertex.

    Also a shame none of the freeform graphite tools are available for edit normals. I wonder how trivial it would be for AD to implement such a thing, as all the pieces are seemingly there already

  • Noren
    Offline / Send Message
    Noren interpolator
    I was thinking about giving it a go, especially since you probably could also go the other way round and convert your vertex normals to hair guides (again), but I would have to lean heavily on ChatGPT or similar for it if I don't want it to take forever with my rudimentary knowledge base, and I don't know how we feel about that (aside from it obviously being pretty hypocritical).
    I've used it for small personal scripts before with mixed results, but sharing the result (however "hacky") is a whole different question.
  • Vexod14
    Offline / Send Message
    Vexod14 sublime tool
    I really like that idea of using the freeform brushes on hair & fur guides to drive normals, but I'm probably the least skilled scripter here. It's a tool I use a lot to edit organic models like hairstyles, and my interest in being able to freely edit normals goes first into this kind of assets, not sure how relevant it might be for hardsurface then but If one day we have that tool I'll certainly try it on such kind of models, just to see what's possible to achieve artistically ^^

    About using AI to solve a question, I personnally advise against it for it's an opportunity to push ideas, knowledge, research, and intellectual growth which will later help solve other problems. My initial post landed us -should I say you all, not including myself as part of solving it- on a really interesting discussion, and I think that's why we're here on a tech/art driven forum, it's all about human pluging their brains together to form a solution, so taking that issue to an AI is like going out of this mindset. At least that's how I envision it
  • Noren
    Offline / Send Message
    Noren interpolator
    True if AI would have caused you to never ask the question in the first place. My angle is more about execution, though. Should I use a tool to implement a "design" (to aggrandize it) when I lack the skills to do so myself in a reasonable amount of time (or at all)? Granted the outcome will likely be poor, but if I shift the example to art, then the answer is pretty clear. So it's more a question of principle.
  • Vexod14
    Offline / Send Message
    Vexod14 sublime tool
    Of course it is a question of principle, I don't use AI, but should this have been the only reason why I ask my question here... I mean, this would've been questionnable on what motivates me to interact with other users. Else I wouldn't have posted other subjects or learnt anything from this place in my whole career, I like to see ideas emulate, and being shaped by mentors/other inspiring artists, as well as participating to the craft at my own level, should it be small, it's always welcome :)
  • poopipe
    Offline / Send Message
    poopipe grand marshal polycounter
    I write tools for a living and I have an opinion on non-programmers using robots to make tools.

    - just use the robot to build the thing.

    The result will be 100% ass from an engineering perspective but that's not the point - Robots are excellent at prototypes / proof of concepts.

    The ethical concerns around robots writing code are basically nil because programmers want you to copy their stuff.
    We're annoyed about robots because they generate code too fast to review, that's too messy to maintain and the hype has convinced non-technical people that they can solve all their problems with a magic autcomplete and we're the ones left cleaning up the mess left behind. 

Sign In or Register to comment.