Building a GPU powered painting program.

Muzzoid

Hey all.

It's been a while!

I've been working on something cool. I always get frustrated with painting software, it's slow, most of them work in 8 bit, and work in srgb, and the workflows haven't improved really that much for decades.

So I'm working on my own painting program, from scratch, using odin. The only library I'm using is a super fast ui layout library called clay, but rolling everything else myself.

The big trick, is I'll be doing all image manipulation on the gpu in compute shaders. Basically not even holding the image buffers in ram. This of course will mean that this program isn't at all suitable for low end hardware, but I don't care, I have a need for SPEED.

I have a lot more crazy stuff planned that nobody has ever done before, but i'll keep that stuff hush, until i have the core program with basic functionality running.

For now, this is showing the program with a 4096x4096 canvas, and a 500px brush, painting at around 900 FPS on a rtx 4070 ti

(kinda funny trying to show performance in a low fps gif i know)
ow

iam717

Play on crisis meme, but can it do layers?

while also painting on said layers, can it do, i found this to be the test in an alternative app @5k and 2.gb's "mem" starts to pick up rather quickly in said app.  Just mentioning this, looks cool.

Muzzoid

It's not worth comparing the approaches to other software unless you know of anything doing it like this exactly. As far as i can tell this is a fundamentally different approach.

You WILL be limited by vram, as you are using uncompressed half floats, but remember this is a painting program, not a design program. For people like me, I'd rather have a lower layer count, but the silkiest smoothest performance possible, over extreme flexibility. This is an experiment in uncompromising performance.

Vexod14

This is extremely interesting ! As someone who mostly paint on a single layer when doing concepts (like, I prefer picking neighbor color over switching from paint brush to erase tool, as less tools means less shortcuts to keep in mind and more time actually expressing my ideas)
I'll keep an eye on your software ! Great work !

pior

Looking forward to it !
Does the focus on high performance mean that it would also behave well on lower-end GPUs, by limiting the canvas size accordingly for instance ?

Muzzoid

I assume it would scale well, as long as the gpu supports compute shaders, it'll work well.

for reference I'm on a 4070 ti.

If that's running at 1000fps, with the 4kx4k canvas. the canvas is using half floats.

The buffer itself is only 128mb of vram, so you could still have 15 layers if you only had 2 gb of vram. obviously we don't get to have all the vram to ourselves, but for most painting jobs, i can't see why integrated GPU's wouldn't work well.

Right now I'm pretty busy in the day job, but outside of that, I'm working on writing the UI systems. UI is surprisingly one of the bigger bottle necks to rendering this, so I've got to get that part right. But i've never written a UI renderer from scratch so it's taking some effort.

pior

Got it. Well, good luck ! Sounds like an endless amount of possibilities ... and just about as many cans of worms

poopipe

I've had a little play at doing this sort of thing a few times over the years (mostly with geometry and pixel shaders tbf) and I genuinely think it has legs if someone can actually be arsed with it. 

the real beauty of working on 'not bitmaps' is that you can do an awful lot of very silly things that just aren't possible when you're working with static grids of pixels. 
i would expect your problems with integrated GPUs will  be resolution related  - they tend to have plenty of memory, its just slow.

best of luck (especially  with the UI )
also make it cross platform

Muzzoid

"the real beauty of working on 'not bitmaps' is that you can do an awful lot of very silly things that just aren't possible when you're working with static grids of pixels. "

I am using bitmaps though? Pixel operations are still king when it comes to painting apps imo. Things like smudging or sharpening becomes impossible if you move to a particle based/vector system. Particle brushes that push down to pixels will definitely have their use-case though

OK!! progress.

I spent a ton of time getting my UI renderer fast, i got it down to one draw call per scissor, so now it's SUPER fast.

So the entire app is still hitting FPS while painting, using the new UI system.

So now i can start to actually build out the feature set now that the app framework is in a pretty good state. It feels unreal having 0 lag while painting. Obviously a lot of painting software already that good in this simple case, but there is so much performance headroom we'll be able to do some pretty impressive stuff.

Muzzoid

OK i've made Tons of progress.

https://www.youtube.com/watch?v=VyBDAu5b_bY

Crappy 5 min paint sketch.

In the past week i've done all this
- added shader gradients to the ui system
- sdf fonts
- tile based GPU undo (this was hell to make lol)
- added a tone mapper
- added brush scattering
- added power to sliders (makes small values easier to pick)
- added basic saving and loading
- support for arbitrary canvas size
- added blue noise to low accumulation values to stop floating point error building up
- + tons more i've forgotten

I checked github 193, commits over 3 months. But i had a month break when i was in taiwan.

poopipe

- tile based GPU undo (this was hell to make lol

vague explanation plz ? 

Muzzoid

Sure,

In order to make undo work for pixel operations, you need to capture the canvas, store it to a big list, then when the times comes paste it back.

The naive implementation would be to literally store a stack of textures, but seeing as we're completely on gpu,  that will eat up vram fast, especially since we are working in floating point.

Instead you can break the canvas into a grid, and check which pixels were modified and only store tiles that are modified. But GPU's work better with less bindings to textures, so it's better to use large atlases of tiles. So the undo system basically creates atlases to store tiles, and then restore them.

Right now, big brushes eat through your undo tile cache pretty quick, but small brushes are very light weight on it. I'll eventually need to make a system to stream tiles to ram, and in the very long term, we can use state reconstruction from stored input data if we want to go back further.

poopipe

got it - so it's basically like the runtime virtual texture stuff in unreal.  good idea

Muzzoid

Tons of progress this week!

• split out input and rendering into different loops. input is captured at 1000hz, and rendering happens at monitor
• refresh rate.
• caching input to handle the above ^^
• brush presets!
• layers!
• compute shader compositor!
• svg icon rendering to atlas (yet to be used)
• progress on the docking ui system, so we can make custom ui layouts

pior

I am genuinely wondering if high performance can have an impact on some of the (somewhat) unconscious choices one can make when working digitally - the most obvious one being canvas size ...

Muzzoid

I would guess so!

I know that I despise any sort of lag, or anything that breaks creative flow, and I think that really plays into resolution too. I would say complex brush types also play into this. There is software with incredibly brush models, like corel painter, but the software is so slow and laggy, it just never gets used.
On-top of that I think software can give people impressions of being super solid and robust, or it can feel very breakable, which comes from all these factors, like how UI works, does it crash, if it crashes do you lose work, how quickly does it start up. Every-time you inconvenience the user you erode this feeling of the software being your friend. That at least is one of the main principals that I'm using as I build this out.

I've also been looking into techniques to get rid of the last bits of input lag, stuff that is usually only used for high performance competitive games. there may be some magic i can do here.

Btw i did test up to 100 layers of 2000x3000 and it works fine fitting that all into v-ram, i'll probably have to build an unloader system to unload inactive canvases to ram/HDD, but that shouldn't be too bad.

poopipe

a really large canvas sounds a lot like a runtime virtual texture to me - which we've established sounds like your undo system

although - you're probably already doing something like that 

Muzzoid

Because compute shader's already break work up into jobs, we don't need anything fancy to handle large canvases. >10kx10k already works out of the box with nothing special added. But keep in mind every time you double the res, that's 4x the pixels.

You are just going to burn through your undo cache a lot quicker.

poopipe

I was thinking more in the hundreds of thousands but I guess that's not really useful for a painting app. 

I will continue to ask stupid questions in here btw - im really curious about how this is going to pan out. 

Muzzoid

Only time i've ever played with files that big was working with lunar scans from nasa. and even then most of the file size wasn't needed because they were super blurry. it's not really needed aha.

Muzzoid

Here's a dumb fun little feature. When you save it now does a sweep of light across the ui. (it looks better not as a low fps gif)

pior

Hey, why not ! Visual confirmation is always good.

And here's the kicker : shall this flashing happen when the save action is initially triggered, or, shall it happen only once the file is confirmed to be written to disk.

Fun stuff

Muzzoid

It happens on confirmation! If the save times ever creep up, i could even use it as a progress bar.

I've been hard at work! I've built an entire docking system from scratch. A lot of my widgets and panels are not resizing correctly yet, so going through and designing all the panels properly is what next. I'm really trying to eat my veggies and do all this architectural stuff before i go ahead and work on the cool innovative new features.

This has got to be the hardest i've worked on something in a loooong time.

Muzzoid

OK! Tons of progress.

- Full action system, which is the universal system that will let us undo anything we want, and store playback ect. (also means we can eventually think about macros. ect)
- tool system, so now we can switch tools

- modals for settings panels ect
- custom top bar to save space, so no ugly white windows top bar.

- I'm half way through building selection tools ect.

- Compute shader, batched layer compositing, and blend modes.

- new layer panel, with full drag and drop support to reorder layers

- Undo+ redo for camera moves (though it only captures 1 move between different tool usages), Animation of camera when undoing.
- dynamic theme system, you can move a few sliders and get a totally unique color theme. This is from my push to make this as appealing to drawing streamers as possible, and also opens up the possibility of people picking themes based on the painting, just to match vibes.
- a lot of refactoring of code, ect, just trying to avoid future tech debt, so far it's really paying off. Registering a new action is very easy now.

For example:

Most of my work is more on the app framework than the painting app, but i'm seeing an end to that tunnel so i can focus on the painting features

pior

I see it went through a name change already !

pxgeek

Looks really interesting! Reminds me somewhat of Heavypaint.

Looking forward to see where this leads!

Muzzoid

haha yeah im still playing with naming.

Hard round is the name of an art community I run on discord, but it's also... it's just a good name.

pxgeek said:

Looks really interesting! Reminds me somewhat of Heavypaint.

Looking forward to see where this leads!

Heavy paint is cool! Totally different goals though. Heavy paint is actually a mesh painting program under the hood, not raster.
It really focuses on expressive brush strokes and painting in app, and isn't very useful as an image editor.
Procreate and Heavy Paint are the only two apps I've seen successfully rethink how painting apps should work, I'm trying to be the third.

My goals are more, look at why photoshop is still used by painters, solve that problem space, without missing any features, and improve the ux on every aspect I can. I'm aiming for more, everything a riot splash screen artist would want in their arsenal.

pior

Having this specific target (= a software that a splash art illustrator could use to fully replace Photoshop) is definitely interesting. You've obviously thought this through already but here are some thoughts :

- First off I think the canvas control really would have to as close as possible to that of PS, or allowing enough tweaking to closely reproduce the PS input scheme (which itself has a few redundancies between newer hardware accelerated canvas behavior/inputs, and legacy inputs). Spacebar pan, scrubby zoom (ctrl+spacebar), Hand tool, ctrl+-, Alt for color picking, a held kew for rotation ... and perhaps many more I am forgetting. New/better concepts could be introduced too of course but I think seemingly simple things like making sure that the diagonal input for canvas zoom can be tweaked to behave just as expected by the user (as opposed to working in reverse, or cancelling itself out in some cases) is certainly no small task especially since users may come from different software. But perhaps you've already built this !

- As for PS image editing features I suppose you'll need all the basics : reliable implementation of Selection, Transform and Liquify ; support for layers, layer groups and blending modes ; effects like Outer glow, Cast shadow ; and so on. And probably PSD import and export. Basically recreating Photopea/Affinity image editing features ... which sounds like another huge task.

- That said I suppose that code for legacy image processing effects like Gaussian Blur, noise, pixelate and many more is widely available.

- Perhaps a more narrow goal could be to focus purely on raw painting, blending modes, selection and transform, and layers support - with layer effects being a lesser priority ?

-----

Personally I feel like the biggest issue with digital painting is the way tablet display devices like iPads and Cintiqs still can't reproduce the precision of regular drawing tools, forcing the user to zoom into the picture in the most unnatural way (hence constantly loosing sight of composition and perspective), but that's a problem that current hardware just can't solve as it comes from the limitations of pen tracking and display tech. Besides using non-display tablets, having a scanner or webcam ready at all times to acquire paper sketches is really the only way to work around this limitation and I often wish for a more integrated way to leverage this hybrid workflow - like being able to import/acquire scan or webcam data directly from within the painting app. But at the end of the day that's just a minor QOL detail as capturing from a webcam or scanner app is just an alt-tab away.

I also find myself wishing that painting apps were able to behave more like Pureref and Inkscape, in the sense of being an infinite canvas where one can  freely compose many objects into, non-destructively.

Muzzoid

Good thoughts!

"- First off I think the canvas control really would have to as close as possible to that of PS, or allowing enough tweaking to closely reproduce the PS input scheme (which itself has a few redundancies between newer hardware accelerated canvas behavior/inputs, and legacy inputs). Spacebar pan, scrubby zoom (ctrl+spacebar),"

Yes, agreed, i've already replicated much of the canvas navigation. One issue i have with photoshop is that some shortcuts are handled with special code, for example dragging to change brush size.
I'm building what i think is the most capable shortcut system possible. Any slider in the app can be mapped to drag shortcuts. You can have modifier only shortcuts, shortcuts that only work in specific contexts ect. I'm forcing every interaction I code to go through this system and be Interactable, so there shouldn't be edge cases here.

I'll also have radial popup menus, and the ability to popup small redesigned versions of many panels under your mouse while holding a key.

The vision here is to minimize the user ever having to shoot their cursor halfway across the screen to change say the saturation, you should be able to just hold S and drag.

And probably PSD import and export.

This is actually the most questionable, because one of the core concepts I'm introducing is a tone mapped canvas, which means colors behave in a more analog way, and we're working in floating point. So if you were to export a file and go to photoshop i can't guarantee that you'll get the same visual output. I'm also considering a crazy idea for how color blending works with colored alpha too, which would push it harder into the photo-shop can't do this category.
Converting from PS to this app is a much easier proposition however.

This is one of those things that people will need to try to truly get it though i think.

"- As for PS image editing features I suppose you'll need all the basics : reliable implementation of Selection, Transform and Liquify ; support for layers, layer groups and blending modes ; effects like Outer glow, Cast shadow ; and so on. And probably PSD import and export. Basically recreating Photopea/Affinity image editing features ... which sounds like another huge task."

Mostly already in the works, the effects are the only one not on the to-do list.

Also liquify should just be a tool, not a mode, like photoshop seems to think. I just want to hit G and grab some pixels and drag them around, none of this open a new window gaf.

I also think many of these functions can be conceptually simplified without losing power. But a lot of this will come out if I'm smart of not based on user feedback when we get to it.

- That said I suppose that code for legacy image processing effects like Gaussian Blur, noise, pixelate and many more is widely available.

Yep basic image filters are shockingly easy to implement.

Personally I feel like the biggest issue with digital painting is the way tablet display devices like iPads and Cintiqs still can't reproduce the precision of regular drawing tools, forcing the user to zoom into the picture in the most unnatural way (hence constantly loosing sight of composition and perspective),

While I can't improve the hardware you are using, I think we can do a lot better when it comes to showing the user the composition and scale of their drawing while they work.

I love the idea of it working more like pure ref. I'm going to be splitting the concept of a canvas (what you draw on) from a view, which is the camera, you will be able to have reference, multiple canvases all within the one view. And of course color constructor swatches, as I should really bring my legacy software along for the ride.

But also, you should be able to have multiple non destructive views into the same canvas data. for example, you could have another view into the same canvas, but instead of looking at all the color layers, it's just the lineart, and they are both the same file. Representing the UX for this might be a bit tough, but it's something that seems like it would be a big help.


Here's some extra stuff too.
brush preview!

Range sliders (gotta make em prettier)

Lerping color themes!

pxgeek

I really dig your ideas about UX and hotkeys. On that point, I’m reminded of a really old android app I used to use: Cloverpaint. It also allowed color sliders to be keymapped, with a clever design that also let you slide up/down to change the scale range, allowing greater or less granularity on the fly, e.g. value slider.

It was such a thoughtfully designed app, probably meant to specifically solve issues relating to small mobile screens; and yet I still wish big staple apps (zbrush, mari) would take a cue from that. Granted zbrush has its own solution, but damn is it really bad! For all other paint apps that can, I just use a stupidly large color wheel pop-up when I need more subtle changes

poopipe

I was just reminded of very early versions of corel painter (around v4) in which you could attach basically any tool setting to any input - eg. stylus tilt could control color and size while pressure controlled opacity and rotation (or whatever), all configurable on the fly etc.
it was a very long time ago so rose tinted glasses apply but my heart has yearned for that level of brush control ever since
(possible hyperbole)