[WIP][UE4] Visual Music Node Exploration (Mix Universe)

chriszuko

So I have had these exploratory Ideas in my head for awhile and I finally have been sitting down and creating the foundational elements for UE4 actor based music-nodes. RIght now its all visuals and no music, however I figured i'd start this thread to show some of the neat stuff I try out as I go along for a few hours each week. The overarching goal is to get a scene setup that can either play music based off of the nodes or sync up the nodes to a composition of sorts. Either way.. this stuff is super exciting to me. Another note is I am recording as much as I can live on Twitch and as a video locally with separate audio tracks so that I could potentially edit a little series together if people are interested. 

Initial Super Quick test with linking actors together. 

First test at spline-based links with hierarchical instanced static meshes along the splines to allow for further editing the spline paths in the future... Potentially could be done with with spline meshes instead, but I wanted to see how complex things got with HISM's. The answer is it got complicated really quick.

The basic linkage is simple atm.

It just asks for a paren't node to be hooked up to with a limit of only 1 parent per node. I think I'd like to expand this as I go however.

Lastly.. I think this image is funny because it actually really helped me sort out logic on how to animate the HISMs

 

Double lastly, if you have any cool ideas, feedback, or images/videos that could be helpful that would be greatly appreciated and is also highly encouraged due to the fact that I don't want to be in a bubble!  Unless it is like cool bubble effects ...that get bigger and when they pop they drop the bass...  *writes down in notes*

garcellano

What the %$#@ whoa.. Interesting

chriszuko

Haha thanks @garcellano

I messed around a bit with the modular system and getting the links to be less of a pain to manage and a bit more swappable. So nodes and links can do different things! I got a simple sample recorded of a full octave so i could have a full range of notes to work with. In blueprint I plug in the sample .wav file I want to use and then say "start at this position if i want to play G. Start at this position if I wanna play A". With this format I could easily then just change the sample out in FL-studio and then render out a new range of notes to use for other nodes. All of this stuff is slowly preparing the setup of things to allow for a pretty complex and diverse toolset of nodes and samples to make something.. potentially all within unreal O.o    We shall see though.

Here is a vid of it in action.

https://www.youtube.com/watch?v=DOpxb7rcS_s

chriszuko

Here is a little bit of a tour of what I've setup so far and the thought process / techniques of the connected nodes.

https://www.youtube.com/watch?v=euXtpsZjbOY

chriszuko

Finally got a simple beat going and a bunch of other stuff. Audio sync issues were definitely a problem that I had to overcome. I ended up having to make a master actor that controls all of the node initializations based off of a list of queries. So basically a node says "hey i want to initialize in about 1 second from now" Then the master actor says "Well I'll go ahead and try to match that time up approximately with the current sequence because you will be a little late!" It seems to be working pretty darn good for the most part. I definitely will need to test to see what happens when a huge frame spike happens. In theory... it would just end up playing all of the things in the queue at once... which would be really bad lol.

Also, this is totally a thing that I am really happy with. I made some visualizations of what note is going to play next using this material setup.. which the comment layout also looks like a piano XD ( Stupid but I love it! )

https://www.youtube.com/watch?v=S8XFhF0hOFM

Happy new year!

chriszuko

Spent some time to go over how things have come along to make the simple loop test video above. Some tool overview and some synchronization issues and how I solved them. If you are interested in just how some of this stuff is linking together and some of the ways I've decided to do things, this vid should help explain things. Please lemme know if you have any questions or if you have suggestions on what I could do better!

Thanks!  

I am excited as hell now though to make something really fun looking out of one of these nodes!

https://www.youtube.com/watch?v=q3hb2P1ro5k

chriszuko

4.22 Added a baked spectral analysis option. I had to try it out! Here is a video of it using just the audio frequencies to drive the animation vs before I had to do 3 anim curves to make it happen. Much more accurate!
https://www.youtube.com/watch?v=p1EI8gW8yU0

Full Write up here!  
https://www.artstation.com/chriszuko/blog/20gO/using-ue4s-422-new-baked-spectral-analysis

chriszuko

Whew... a year later and now this project looks like this! Still experimental.. still nodes connected to other nodes via links. Starting to take shape into something though! O.O
https://www.youtube.com/watch?v=iKKTJFVKHbE&feature=emb_title

chriszuko

Things are getting a bit more interesting now! I've finally been able to nail down some of the more core systems and focus more on how to actually interact with the system.

Here's an updated video showing some of the interactivity and audio stuffs.
If anyone has any ideas, suggestions, feedback, or questions in general I'd be appreciative of them and be happy to answer!

https://www.youtube.com/watch?v=DgTvE5r_4qQ

Fun screenshot messing with style a bit.

chriszuko

I did a bit more of a write up for the video above going over some of the systems stuff and visuals a little bit.
https://chriszuko.com/project-mix-pre-alpha-demo-video-write-up/

chriszuko

So after a bit more time, I have finalized an Undo-Redo System, and am really starting to nail down some of the editing mechanics.
- Adding Nodes
- Copy Nodes
- Editing Node details including Pitch, Velocity, Volume, Duration, Reverb, Filtering

- Editing Node sequences which alter pitch, volume, and transposition of notes

- Editing Node Samples ( Sounds, Piano, Guitar Synth, Strings, Bass, Kick, Hi Hats, Snares, Crashes )

- Editing Link Details - Changing Start and End nodes, duration, Deleting, create new.

- Editing Link Activation Pattern - Alters when a link can or can't fire which ultimately is where the structure comes from!

I made a thing inside of the thing I am making O.O

Utilizing all this, I was able to finally make something from scratch using only in-game UI which ended up getting me something that I am pretty happy with! Any feedback, questions, critiques are super welcome!

https://www.youtube.com/watch?v=ZP_bVCU_JFs

Some elements you see in here I can illustrate a bit more with a few tinier update videos. Sorry for the wall of video, but I think these are pretty interesting for anyone who is wanting to see some of what the editing looks like.

SEQUENCE NODES.

These things are proving to be a pathway to making complex compositions in a more illustrative and compact way.
https://www.youtube.com/watch?v=NtTB0DXsKYw

Editing Sequence nodes.

You can see that volume, pitch, and note transposition can be adjusted which allows for a pretty complex set of things that the one node can handle.
https://www.youtube.com/watch?v=Gckr1tsogG8

Bringing it all together.

with some realtime MIDI playback on my keyboard for the piano.
https://www.youtube.com/watch?v=eg9zku7_OnM

 The undo redo system.

which took me better part of a month of free time to figure out nicely. I can go into it deeper if someone would like, but it was so nice to finally be able to not have to feel the pain when accidentally deleting a node or making bad adjustments.

https://www.youtube.com/watch?v=0U8WWOrVSzU

What's next?

Well... there are plenty of routes to go from here. I want to get to the next visual bar for this thing. I want nodes to feel like you can guess what they are going to sound like before they even start playing. Right now the colors determine the instrument and the rest is really just displaying volume and duration. So any ideas on how to better that I am extremely interested in!

After that, modulators are going to be what I think brings the "Fun" into the gameplay for people who don't want to create as much. The ability to take something like the composition above and mess with how it sounds by moving modulation nodes around... could be super fun and interesting!

Then finally... actually getting other folks to try it out! O.O 2 years in the making so far .. so that doesn't even feel real to me lol.

I know there hasn't been much action on this thread, but I am truly appreciative of those who have been following in the journey!

Thanks!

chriszuko

Excited to share things are slowly but surely making progress here!

I've posted a newer update video showing a new composition and some of the more details about the features and the entire goal of this project. To think... 2 years ago I started this as a fun little visual experiment... wild.

https://www.youtube.com/watch?v=fEVi0nzh2sM

Here are some screenshots of the game in its current state now! Lots and lots of work with FX and Lighting along with general game-play stuff. I am happy to dive into some of the more technical details or setups or answer any questions anyone may have! Also open to feedback and discussion about certain things if something doesn't look or feel right!

chriszuko

Mix Universe is what this is called now!

So I haven't posted on here regularly, but I just want to throw out there that I have been working on this still and have documented a few things along the way. I used to post on polycount quite regularly awhile back but fell off the bandwagon a bit.

To jump straight to it, here is the official trailer for Mix Universe!

https://www.youtube.com/watch?v=EzdMQLBbWfA

You can try it out right now if you'd like in an early playtest form testing the create mode.

And here is the most recent update for the early playtest.

https://www.youtube.com/watch?v=ukInm4F7aSA

Also some fun screenshots.

As you can see this thing.. somehow evolved into an actual playable musical sandbox. I don't think when I started I thought I would keep at it the way I have, but it keeps me constantly curious solving technical challenges and itching at the part of me that wants to make music fun!

I want to do a write up going over Metasounds, Quartz, and Hooking up the visuals and am closing in doing so hopefully by the end of next month. If there is anything in particular that you'd like to know about please let me know and I can make sure to try and touch on it!

Some fun stats that may or may not make you cry. I've spent around an average of 20 hours a week on this for 3+ years at this point... which means roughly around 1000 hours a year totaling a whopping 3000+ hours of time... As a side project... while working a game dev job... It's become a huge part of my life. There have been many ups and downs and truthfully I do it because I enjoy it and plan on keeping at this particular project for at least another year after early access launch.

Finally I would love feedback if you do end up checking it out!

If you hate it, please tell me why! Hit me!

Appreciated!