Home General Discussion

Rotoscoping polygons: Can it be done?

JordanN
interpolator
Offline / Send Message
JordanN interpolator
Remember Paperman? Disney's latest attempt to revive traditional animation in CG form. I was wondering, is it possible to recreate this technique but with 3D video games?
1aRVFbA.gif

I was thinking the closest attempt could be through a series of texture swaps and perhaps at a fixed camera angle, however would that be too memory intensive?

Replies

  • JacqueChoi
    Offline / Send Message
    JacqueChoi polycounter
    Why would they?


    Why not simply do it like Ni No Kuni?




    What you're talking about is intensive sprite based gameplay. It's hella expensive to make the way you're describing.
  • JordanN
    Offline / Send Message
    JordanN interpolator
    JacqueChoi wrote: »
    Why would they?


    Why not simply do it like Ni No Kuni?




    What you're talking about is intensive sprite based gameplay. It's hella expensive to make the way you're describing.
    The same reason Paperman did it, to achieve an absolute traditionally hand drawn cartoon look.

    I don't see motion capture/ morph target animation doing that and that includes Ni No Kuni.
  • Snader
    Offline / Send Message
    Snader polycounter lvl 15
    What part do you want to replicate exactly? There might very well be a way to approximate what you want but you definitely can't use the same techniques Disney used because
    A) you need it to be realtime instead of prerendered.
    B) you don't have a giant renderfarm like they do, I presume.
    C) I think it's a proprietary method, at least for the moment.
  • JordanN
    Offline / Send Message
    JordanN interpolator
    There's a breakdown of how it was done. It starts with a 3D model, they bring in an animator who does inbetween work which is, draw on top of the polygon frame per frame. They also use computer simulations to properly scale the drawings (for example, if you draw a smiley face on the character's hand, the smiley will stretch/disappear depending on where the hand is facing the camera or when the hand moves).

    The last part is definitely proprietary but I'm interested in the "inbetween work", i.e the actual drawing that shows up on the polygon.

    Here's the video of the complete breakdown:
    http://www.youtube.com/watch?feature=player_detailpage&v=TZJLtujW6FY#t=294s

    I'm aware it's all pre-rendered but I'm asking how to "fake" parts of it in real time.

    I bring up texture swaps because video games don't have any other way to alter a polygon's appearance (that I know of). Texture swaps also mimic this technique because they use pre drawn animated assets (ironic).
  • Snader
    Offline / Send Message
    Snader polycounter lvl 15
    You can achieve the render-or-not-render-this-smiley effect a bit by having two materials and blending them with a sort of rimlight shader, in combination with hand-tweaked mipmaps (simplify as a circle/dot at LOD1, completely gone at LOD2). Stretching should not be an issue since 3D rendering 'automatically' does that for polygons at different angles.
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    It's all possible through shaders and material definition, but even that would be pretty expensive.

    Question is, what would be your platform then? I can see something like working for RT rendering in a game engine (EI: UDK dumping movie shots) without any stuttering, but for anything more interactive, it wouldn't be really optimal.
  • Sandro
    What I understood from that video they expanded character silhouetes and rendered out motion vectors. After that they sketched over certain frames and these motion vectors would 'carry' drawings with them.

    As far as I know engines that use deferred rendering store both normals and X-Y motion vectors in Gbuffer as well as other passes. So, distorting drawings in screen space based on those channels would be quite doable I think.
  • Bigjohn
    Offline / Send Message
    Bigjohn polycounter lvl 11
    The other thing is also that, since Paperman is non-realtime, they had their shots all set beforehand. So the animators could draw on top of a frame. With a game you can't do that. The whole point of a game is that the viewer can move the camera around, and look at the world from a different angle. With something like Paperman, or anything where the camera is static, the shots are known ahead of time, which means an animator can draw on top of it.

    What will you draw on top of if the camera can move all over the place?
  • Sandro
    Nothing. Yet there are plenty of games where camera position is not that freeform. Top-Down rpgs, sidescrollers, fighting games etc.

    I guess you could store a drawing or two per-character per-animation and 'spawn' it when animation hits certain frames on top of character and motion vectors would distort that drawing accordingly.

    Or maybe not, I am speculating :)
  • binopittan
    Bigjohn, the more i think of it the more i think it's posibble, for games with fixed camera angle like resident evil 1.

    3d games not necessarily deed camera freedom, in case like Diablo 3, it can be used to create even more handpainterly looks.

    I was thinking something like SF4 opening [ame="http://www.youtube.com/watch?v=R8SD_M_ccac"]Super Street Fighter 4 - Trailer - Opening - Xbox360/PS3 - YouTube[/ame]
  • Bigjohn
    Offline / Send Message
    Bigjohn polycounter lvl 11
    No, but even then, the whole point of what you wanna do is to have an animator draw on top of a frame. But the content of each frame changes frame-by-frame with the decisions of the player, which is unlike non-realtime where the content of each frame is known ahead of time.

    That means that you couldn't have an animator draw on top of the entire frame. You'd have to have them draw on top of the environment in one pass, then draw on top of the characters (from every angle that's used in-game), then superimpose those two somehow. Then if the environment has anything that is animated, that would have to get separated out into its own pass. Compositing everything together at the end.

    It's possible, but doesn't sound practical, and I think you could accomplish something very similar with real-time shaders.

    Edit:
    You may also want to take a look at Overcoat, which allows you to paint in 3D. We've essentially been doing that in games for a while with stuff like 3D-Coat. But it makes more sense than drawing on top of the entire frame when you can't tell the content of that frame ahead of time.
  • JacqueChoi
    Offline / Send Message
    JacqueChoi polycounter
    I think there might be ways to do it using a very elaborate shader system.

    Might help if nothing has any textures or UV's.


    Obviously being 'Real-Time' the results won't be as elegant but if the market is there I'm sure it's feasible.
  • sprunghunt
    Offline / Send Message
    sprunghunt polycounter
    real time 2d animation isn't practical since it's very hard on the animators wrist ;)

    but have you seen dragons lair? it's an old game that was fully animated by some ex-disney animators.

    dragons-lair-wii-dragons-lair-2.jpg
  • JordanN
    Offline / Send Message
    JordanN interpolator
    Bigjohn wrote: »
    The other thing is also that, since Paperman is non-realtime, they had their shots all set beforehand. So the animators could draw on top of a frame. With a game you can't do that. The whole point of a game is that the viewer can move the camera around, and look at the world from a different angle. With something like Paperman, or anything where the camera is static, the shots are known ahead of time, which means an animator can draw on top of it.

    What will you draw on top of if the camera can move all over the place?
    I addressed this in the first post in which the camera is fixed.

    If it was possible to have the animations running while the camera is moving, that would far exceed my expectations (although someone suggested it's possible to have the animations scale based on where the camera is positioned).

    Anyways, I'll put together a short cartoon to better illustrate the technique in question. Need to rest up first.
  • Bigjohn
    Offline / Send Message
    Bigjohn polycounter lvl 11
    Hmm, I apologize, I must not be explaining myself right.

    I'm not talking about the camera being static. I'm talking about the content of the shot (each frame) being an unknown. In a movie you always know what each frame has before you animate on it. You can't do that in a game because you don't know what the player will do.
  • almighty_gir
    Offline / Send Message
    almighty_gir ngon master
    JordanN wrote: »
    The same reason Paperman did it, to achieve an absolute traditionally hand drawn cartoon look.

    I don't see motion capture/ morph target animation doing that and that includes Ni No Kuni.

    except anybody with eyes can see that Paperman is CG.
  • JordanN
    Offline / Send Message
    JordanN interpolator
    Alright, so I drew this fast (the pixel artifacts were a result of cropping it)
    RFRRUsv.png

    Basically, picture an RPG setting in a town. The character walks over to talk to the NPC. When the person says "Hey Dude" the camera stays there while the animation plays. When the NPC is done talking, the camera is free again.

    How is the animation done? It's based exactly where the camera freezes. So when the person says "Hey, Dude!" the position the character is standing is where the drawings will overlap the polygons.
  • Snader
    Offline / Send Message
    Snader polycounter lvl 15
    So basically you want to have a fixed animation from a fixed camera angle? Just prerender the shot and display it as video?
  • JordanN
    Offline / Send Message
    JordanN interpolator
    Snader wrote: »
    So basically you want to have a fixed animation from a fixed camera angle? Just prerender the shot and display it as video?
    Would that be just for the characters or the whole scene? Because I want the whole scene to be happening in engine in real time.
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    How exactly are you going to draw frame by frame? Or predict character moment?

    I could very easily do what paperman did in sprites for a RPG Maker game, but at that point, would it be even worth it? Small pixelating character walking around with several high quality overlayed sprites? What's the point then? How much movement can I see? Are we going to get rid of physics and other particle system to make things simple? Make a linear gallery shooter?

    Again, this is all possible through shaders, in a limited way of Durer patterns, etc, but without someone drawing frame per frame the 'down to detail' layers, how exactly are we supposed to do that like they did in Paperman? They're real-time projecting stuff with Vector pattens and rasterized layers on an offline product at the end for the quality rendering.

    Also, even if we did something like that "say character moves arm up from hip", then you could blend an overlay sprite texture for something like that, but at what cost? How many textures do you need? are we going 8x8 per height and side angles? A total of 64 expensive high quality textures based upon blend-morph targets per moment for 1 character? And that would be the cheap option for a limited animation.

    It's not feasible, even on a PC.

    EDIT: You'll basically need someone to sit next to you and paint the canvas like in Okami and it's attacks, which is just too much.
  • JordanN
    Offline / Send Message
    JordanN interpolator
    Ace-Angel wrote:
    How exactly are you going to draw frame by frame? Or predict character moment?
    It's based on the already existing character animations (i.e the rig). I would be drawing over the model as it moves frame per frame. These frames would then be played back in the game on top of the model.
    Also, even if we did something like that "say character moves arm up from hip", then you could blend an overlay sprite texture for something like that, but at what cost? How many textures do you need? are we going 8x8 per height and side angles? A total of 64 expensive high quality textures based upon blend-morph targets per moment for 1 character? And that would be the cheap option for a limited animation.
    Since it's traditional animation, it could be anywhere from 10 frames to a thousand. Though mind you, I could cheat by only drawing one set of dialogue instead of animating a whole conversation.

    For example, when the NPC says "Hey dude", only those words will have been animated. The rest of his speech could be a series of audible grunts or gibberish.

    I also meant to answer this earlier.
    Ace-Angel wrote:
    Question is, what would be your platform then?
    Since this is something that hasn't been done before, I would want to target the most powerful platform. So that would be PC followed by PS4 and Xbox 720.
  • Snader
    Offline / Send Message
    Snader polycounter lvl 15
    JordanN wrote: »
    Since it's traditional animation, it could be anywhere from 10 frames to a thousand.

    Well in that case, just render the character frames ahead of time in a giant spritesheet and fill up a couple hundred megabyte? Just prerender the frames similar to how disney does.
  • JordanN
    Offline / Send Message
    JordanN interpolator
    Snader wrote: »
    Well in that case, just render the character frames ahead of time in a giant spritesheet and fill up a couple hundred megabyte? Just prerender the frames similar to how disney does.
    If I fill up the spritesheet, where do they go (as in, how to include them in game)?
Sign In or Register to comment.