Getting close! Everything is ready except for set placement, which I thought was going to be the easiest but turns out to be quite complicated, because I’m building the sets procedurally and dynamically. The reason being I want to frame every shot and compose every frame for maximum cinematic punch. This requires lots of math. :)
So I’ve researched and researched and I’m pretty sure I’m the first person to do this. This being, fully automated animation and special effects. It’s kind of unbelievable that I’m the first one to do this. Certainly someone has been working or has worked on this right? But this is where things like Google-Fu fail, because of the way Google and others rank search results. Obscure blogs or posts about an animation AI would never be found.
If I’m the first though, hoo boy! Could be interesting. Imagine a world where studios don’t create the content or decide what’s made, or have any creative control, they simply provide funding to make the movie look better, especially if there is a novel special effect. They also obviously handle marketing and release still. We also still need model and texture artists so a studio would have lots of those people on staff.
Perhaps like Pixar, everyone will focus on good stories after this thing gains traction. Then again, maybe I can’t figure out this last bit and I quit a a take a job in L.A. Who knows?
You’re making it sound amazing, Guap, but after all this time it’s like someone typing about how their new flying car just met mach 3 in trials and then hit the freeway at a roll doing 100 mph, without the corresponding youtube for the eyes and ears… I mean, Pixar isn’t a success just because they make good stories… they’re backed up by some of the best art direction in the industry…
I’m still struggling to understand what is exactly is automated and what the parameters are. Do you have a white paper or patent filing or anything like that available yet?
It’s like a new videogame engine that’s specifically made to play by itself and record the results. Think of the replay movies on a racing sim but that’s the whole game. Or a Madden game where you have it play itself.
This is not a perfect analogy because it’s not really a game, there is no player or interface besides the script and character information. It takes control of any program that has an API that can procedurally control objects, lights, camera movement and focus, etc. So when it’s all converted over (the prototype works with the Blender API right now) it will be able to “control” Unity, Maya, etc. Anything with an API.
I’ll demo it soon, trust me! Just need to polish it a bit more and make sure it’s working correctly.
Just FYI, there are several AI driven animation tools out there, but they are used (and geared) to control crowds, not exactly what you are trying to do here. But yeah, those Zootropolis crowd scenes? a lot of it is computer generated and AI driven.
Having worked both in the creative aspect of films and as a programmer, I remain skeptical this will have an use for high quality (think Pixar) final production. There’s too much artistry and emotional intelligence driving both an animator’s hand and the director´s framing of a shot and deciding cuts. Small pauses and shifts in expressions in characters, matters of shot progression and unusual approaches…
For example: One thing I think the white paper gets wrong is the ability for the program to react to a change in the script to reshoot one scene. A significant change in one scene would not mean that scene needs to be reshot, it means a lot of the visual approach (shot selection) from previous and later scenes would need to shift, and if the change is significant enough (say it’s the ending scene) the whole movie would have to be shot differently.
The problem is more akin to having a computer write good novels (they do write novels, just boring ones). I would love to be proven wrong, though. And I certainly wish you luck.
Where I do think something like this could make an impact (and money) is if the fast and cheap realm of 3D animation production for news programs, reenactments and the like. Basically there where getting something out is better than getting something really good out (like feature films are).
Also, it could be very useful in the high quality animation industry as a tool to help the creative team come up with a preliminary shot list from a script to evaluate viability (create a preliminary animatic to see if the script flows visually, and maybe even using a small part of the data generated to save time). Something like this could save months of work from highly paid individuals, and could be priced high.
Holy crap Juan, you get it exactly! The strengths and weakenesses, the technical pitfalls, the first uses of it, where it’s needed and not, and what markets we are going after first. That white paper is hopelessly outdated now (that was from Feburary I think). All of your comments above are like a prediction as to the challenges, direction changes, obstacles overcome and pitfalls of 2016.
One of the guys most excited by this was on the engineering team that developed the Lightwave .obj file standard. He and I have had many conversations about vector matrices, large array tracking of vertex meshes/arrays, orientation of said matrices, etc. Around May he convinced me that initially my target audience should be exactly as you described: making quick simple visuals for animatics, script visualizations, data viz, “explainer videos”, etc. One idea he had was that you sell an animatic to every studio that purchases a script(or spec script). For say 3% of the price they paid. So a $100K script would be $3000, and so on. Or you could sell it to all writers on some kind of sliding pay scale (so the amateurs and unknowns don’t have to pay as much as say, a Nolan brother). In this way they’d all have an animatic visual to go along with their script when pitching.
Anyway, I could go on for hours but now you’ve got me all excited again. I’ve been distracted by BF1 and this Mommy Works thing and neglecting this project.
Juan, please private message me, let’s talk! Would love your thoughts. When I get some breathing room I need to write another white paper. Maybe we can collaborate!
Yeah, I don’t see anything in the white paper that gives room for, you know, performance. Which isn’t surprising, performance is really, really hard, even for human actors. But I’m struggling to see how this method can in its near-term future form can produce art much more sophisticated than those Taiwanese news animations.
I had a similar reaction to that bit of the paper. If you’re changing the end of a movie from the protagonists being together to the protagonists breaking up, you’ve got bigger problems than just reshooting a scene.
I’m also curious how the feedback mechanism presents scenes/clips to the user. If you’re truly exploring the possiblity space given various low level parameters like attract/repel states and doing some sort of Monte Carlo simulation-esque process, then you’re going to have thousands (millions? billions?) of alternatives to choose from. How do you pare that down to something manageable by the user? But I guess it’s too early to disclose that sort of detail.
Yes, good points both. So the “attract/repel” AI part is kind of like the “backup” to the scripting part now. So I can “force” the characters to be in a “somewhat mad” state when delivering dialog in the script. Or whatever emotional state (for dialog I actually call this char.dialog_delivery_state.emotion_mad or something like that). If you leave it alone it will simulate the interactions for you. The problem with this is that there is no mid state save right now. It simulates the whole movie front to back. If you let it control things one time the character may be in a different state, triggering different actions, camera angles, etc. Its like its simulating a game in real time and show the results, while recording them.
Buuuuut… you can also, via interface, override any and all or even 100% of those states, actions and camera angles. In fact the main thing I’ll be doing in the coming months is stiching those two parts together. For now I want to get the “script to visual” part working completely, then I’ll swing back to the AI driven interactions part.
Man, this would be so much easier with a partner. I envy those guys that design and program these things with their college roommate or small group of friends. I’d pay good money for that interaction now!
It’s pretty crazy you guys are juggling two startups at the same time. More power to you. One area I thought this could be pretty interesting would be with respect to storyboarding, which my daughter happens to have an interest in. I could see something like this being used to sketch out a VERY rough “this is the idea we are going for” prototype to facilitate understanding in the early stages of projects. But I don’t actually know a lot about how that process traditionally works, so just assumption on my part. I have done a bit of art direction though, and misunderstandings/misinterpretations can be costly in terms of time and money, and I’d think that is every bit as much the case here. I guess I’m thinking of this as sort of a rapid prototyping methodology.
Juan, one thing you’ll like, to address your first point above, was the artisty and subtley of movement. The real art it is to animate something with expression, gentle warmth, quivering rage, bulging eyes, popping neck veins, etc. Of course that subtley and the interaction of that is what makes a place like Pixar, well, Pixar! The absolutely best in their industry. And Ray will not be providing that, at least not while I’m working on it by myself and not until I get funding. But I have a really good way to get there involving big libraries of those subtle interaction poses and matching them to character archetypes. Also, think about a 1 to 100 scale of a smile, 1 being a barely perceptible grin and 100 being a “The Smiler” grin from Transmetropolitan. Even though the pathway to that particular in between pose may be static (that is, the same vertices or groups of vertices are moved around), we can have a library of many types of “smile” poses. When you combine this with a huge differences in character design and look, and even camera angles (no two shots are exactly alike) it’s going to be fairly straightforward to pull off a variety of expressions. Though I personally would still want to hand animate the main characters death scene or first kiss scene or whatever critical scenes there are. If I were Pixar.
But even Pixar could use this, I’m sure. I can think of a million ways. They may have it do branching interactions of a storyline to get a feel of the visuals involved. They may have it do a super complicated or even in just easy dialog scene and then go in and futz with it. Heck, they make even just use it to create animations showing character design prototypes and scripting addons they are working on. The beauty is that it generates a finished animation file, ready to render. You can go in and change that file to your heart’s content. Camera angles, poses, add and delete keyframes, reposition bezer curves or movement pathways or whatever.
The tool can be a starting point, brute force workhorse, or complete animator start to finish. The really neat thing is the ability to iterate scenes, shots, and whole movies. When I get it working.
We work a lot with and within Third Floor on our projects, and in many cases, the pitch/previs shots will match almost one for one with the final product. Not to say that preliminary work isn’t iterated, incredibly so sometimes, and more efficient techniques will always be welcomed, but a crude first pass would be as welcome as handing a furniture craftsman a board full of holes and nails and tell him to build on that. And the beauty of previs is that it isn’t months of work… I’ve been directing some previs for Marvel, and turnaround for my pitches are in the days, not months. The things that help us immensely-- light and modular models, simple rigs, asset and motion libraries that are searchable and well maintained, non-linear sequence editting … ie, all the bits quickly grabbed and ready for the artistic decisions. Whether those decisions can be well made is the entire question here.
The quickest way to kill enthusiasm for a project is to show uninformed clients / executives a rough edit that requires imagination for the finished result… it’s hard to see past the decisions already made, and dangerous if it’s in any way garbled or confused.
Rough animatics for newscasts such as News Direct – absolutely!
Spiffy, that place looks awesome. Really awesome. And you are absolutely right about pre-rigging and libraries of standardized models. I’ve already secured a guy from a large games company in Irvine, CA (essentially where I live now) to handle all of that, maintaining and correctly tagging libraries, etc. Much thought has also been give to a Dewey-like system of categorizing things for use for placement, camera angles, framing, composition, etc. I would love to embed this info in the .3DS and .obj and every other file standard directly. That way no additional info besides say a .obj and .mtl file or whatever. We’ll get there.
No, it is. Pretty much the best designed and most supported via community, with the best libraries (especially for science and math stuff). At least out of the languages I’ve personally programmed in. Just new to it. Also haven’t used an interpreted language in a while. I’ve been in management too long!
Anyway, had no choice initially as Python was chosen as the scripting language for Blender and Maya (along with MEL for Maya, but that’s proprietary).
So it creates the animation file itself. The animation file is like the master project or whatever and keeps track of where everything is, what kind of lighting we are using, the position of everything, even stuff like are we using ray-tracing, what’s the unit of measurement we are using as units (mm or ft), all that stuff. That stuff is all saved in a project file. For example with the program Blender they are in a .blend file. If you make an animation at home, anything from a simple text appearing on screen and moving around to The LEGO Movie it’s in that .blend file. So if some studio gave you that .blend file (if they used Blender, they all use different stuff but Maya by Autodesk is the market leader for COTS animation software suites) you could open it up at home and render that yourself or tweak it to your heart’s content. You have to be a little careful though because each shot is individually composited with depth of field, the camera angle, backdrop, etc. Replacing a character model or changing the lighting (even the direction the light is going) is no big problem. Changing the animation subtly or zooming the camera in or out or up or down or whatever is no big problem either within reasons. But if you say swing the camera around 180• you’re going try seriously screw up the framing and original composition of the shot. Not to mention the background, if the AI did not expect a 180• shot. The tricks you use for something as simple as the backdrop changes drastically. A 180• shot requires a correctly mapped torus shaped background (curved and long). Also if you have a complicated background that whole thing is now out of whack. In those cases it’s better to have the AI redo the the shot (which takes it 1 second) according to those new parameters, then tweak the new shot. Or you could have it build the shot every which way, and then you go watch them all pick the best one you like, deleting the rest. Either way, they computer doesn’t care.
You have to be a little careful about the animation file itself for obvious reasons. It would contain all of the models, and artwork, etc. You’d transfer that securely back and forth.
Another cool idea I had is that a studio could send me an animation file they created, I could take a copy and then run this against it. It could make all kinds of interesting changes to that file then we send it back q d forth like that and collaborate.
This is a little more complicated though because you have to read all of the information via the API. I haven’t tried that yet but so far I’ve only gone the one way, “pushed” I formation from my program into the animation program. I’ve never read it the other way, I’m not even sure you can do that.
