Smart Hulk, New Asgard, and the destruction of a massive warship are just a few of the big moments the team at Industrial Light and Magic worked on throughout their time on Avengers: Endgame. Led by visual effects supervisor Russell Earl, the team had their fair share of time working on some of the blockbuster’s most pivotal sequences. ComicBook.com spoke with both Earl and ILM modeling supervisor Bruce Holcomb to break it all down.
Videos by ComicBook.com
Here’s what Earl and Holcomb had to say about Avengers: Endgame and its stunning visual effects shots.
ComicBook.com: So Avengers: Endgame, what a movie, huh? One of the first parts I really want to talk to you two about is Smart Hulk, one of the biggest characters you guys worked on. With this film, Bruce Banner certainly looks a lot different than he has before. How complicated was building Smart Hulk to previous versions of the Hulk?
Russell Earl: It says it in the movie how this Hulk is the perfect blend of the brains and the brawn, and so, one of the things we did was we obviously went back and looked at the Hulk that we had done in the prior films. We were fortunate in that a lot of the people that were working on this movie had worked on the prior films, like the Ang Lee Hulk movie, the last Avengers movie, and Thor: Ragnarok, so we’ve had a pretty solid history of Hulk at ILM.
The fun part about this guy was it was taking down big muscley Smash Hulk and giving him a little bit more Ruffalo. We looked at all the previous models, and we did comparisons. We were looking at Ruffalo and the key art that we got from the guys at Marvel, and then it was really just taking all of those things and trying to dial in just the right amount of Hulk, keeping the big muscley dude, but then also getting enough of the smarts, or the brains, from Ruffalo in there.
Now, as I understand it, you and the team came up with a new facial capture system just for Smart Hulk.
Earl: That is accurate, yeah. We had our existing system at ILM, which was the SnapSolver system. But one of the things we knew we wanted to do in the beginning was sort of trying to get a system in place that would give animation the ultimate control over the performance. And whereas some of the systems in the past, you could get a good solve, so if you look at it from there’s the solve aspect where you’re solving the performance, then you’re taking that performance and you’re retargeting it onto the Smart Hulk model and then at that point, you’re getting it into animation where you have controls to basically help offer that performance. So we rewrote all three, or we used new tools for all three of those steps.
The solve side of it, we started with our SnapSolver which then evolved into actually by the end of the show, we sort of went through three different solvers. By the end of the show, we were using Disney Research’s animation, which was the first time it had been used with head-mounted cameras in production. What that did was it gave us just a photometric solve, so it was looking at each pixel in the images coming off the head-mounted cameras, using that to solve Ruffalo’s performance. That was a big thing for us because we were going in production and we had started with our SnapSolver and it was giving us results and then we were improving upon that, rewriting that. Basically, take the solver and then look back at the initial head scan data which is done in this thing called Medusa, so you’re constantly checking the mesh that the system is building against actual meshes from real Ruffalo performances.
We were building that system and then it was more towards the end of production, we got Sydney Research to come in and tweak their Anima solver to work for us, which would give us a very solid representation of Mark Ruffalo’s performance. From that step, we re-targeted the animation and brought in a system called Blank, which was spearheaded by our facial techs. That system got rewrote and what that would give us is it would take the solver that we had and when it ran, we would look at the mesh and compare the Ruffalo head to the Ruffalo performance. Once we were happy with that, we would run it through Blank and then Blank would take that data and re-target it onto our Smart Hulk character then we had that performance.
The solver generates the appropriate mesh, then the appropriate mesh must be translated into Hulk, but also at the same time because you’re getting this data because it’s a lot of math, you didn’t always have good animation controls over the system. In this case, you rewrote it so that animators would get a nice clean model that would allow them to dial in and out of Ruffalo’s performance so it could break up the polymers down to the shapes but then also have really solid control over it through a series of deformers. There was also, in addition to the Anima product we would get out of Blank, we had a series of deformers, things for lip compression. All of these things that would help bring that performance to life, so we could stay true on model, stay true to Ruffalo’s performance, but at the same time, if you took his performance and you mapped it onto Smart Hulk or re-targeted onto Smart Hulk, it wouldn’t always feel right. The smile might feel too big and broad or he might look too goofy, so Ruffalo gave a very unique performance. We were capturing that, but at the same time being able to alter and control in animation.
Wow, that sounds great. Now let’s talk about the Battle of New York a little bit. Obviously, there were some shots of things that had been in previous movies that showed up right? When you have two Hulks in one scene, was that something where the previous timeline Hulk you were able to actually use an asset that was built for The Avengers or was that something still built from the ground up for Endgame?
Bruce Holcomb: What was it, I guess five years prior to that film? I don’t remember. We kind of reinvent a lot of our systems per show, so we had to kind of go back and re-engineer the old Hulk a little bit with some new technology for rigging and some of the muscles and some of the shapes. You know, it just looks a little rough and I’m always amazed and forget how much technology that we do have kind of layered on itself to just kind of get these things back in action. So like when you had them both in New York, it was clear to see what was old tech, what was new tech, and then how to control the differences between the two. I think they were only in one very small scene together where they actually shared the camera, right?
Earl: Yeah, it’s funny, it’s also like a second chance to go back because I think we’re all very critical of our work and always want to do a better job. So it’s a chance to say, “Oh yeah, we could have done that a little differently.” We had the technical challenge of like, “OK does this even run anymore? What version of the software are we on?” but then at the same time, as Bruce was saying, you also have like, “Oh well we’ve actually built a better rig for muscles,” or “We built a better eye control and better lip control.”
Holcomb: Yeah, and “Oh my God is this green?”
Earl: The other thing too is that it’s different directors and different teams working on the films, so it’s sort of taking what you liked, and in the case of Smart Hulk, it was we had to, we couldn’t upgrade or change him too much because he had to still fit in with the footage from the prior films.
Holcomb: Which was funny because there were a couple folks here at work that when they saw it, that actually weren’t working on the film, they were like, “What Hulk did you use? Because that wasn’t the Hulk that we did on Avengers 1.” I’m like, “No it’s the same Hulk,” and we just animated him a little bit different, have some kind of more Cro Magnon-y kind of moments with anger and he actually gets to say another line in this film. Like even the people that had built him on Avengers 1 didn’t really think that we were using the same model, which I thought was funny.
Moving to New Asgard, it was filmed on a remote kind of village overseas. How much of New Asgard is actually practical? Was most of the infrastructure like the buildings and roads there, or were a lot of those digitally added?
Earl: So a lot of it was digitally altered. The roads and water and buildings, probably 50% of the buildings were there but I think 100% of the buildings were augmented or modified. So the village was there, I think it was a village in Scotland that was shot, but we went in and we added an Asgardian touch, a little bit of mural and paintwork and we changed all the buildings and we went through all the assets they had used in building Asgard and sort of picked some key characteristics that we could use.
We were also looking at sort of Norwegian fishing village reference and just trying to find that balance of Asgard versus Scandinavian fishing village. So we added a boat or two I think, and we augmented almost all the buildings. Then we brought the hillside in a little bit and we sort of changed the coastline from, but it was great because we had a good location to start with and then we could just build off that to give it the sort of right mix of new and old.
With New Asgard obviously comes Thor’s new look. That was mostly prosthetics right? Was there any digital work involved is his new look?
Earl: No, he ate cheeseburgers nonstop. [laughs]
My kind of hero.
Earl: No, that was a great prosthetic. He was wearing a prosthetic that I believe Legacy did for his suit. There was a big scene down the back that we cleaned up, and then we just cleaned it up in places where it didn’t quite bend right at the elbows and shoulders.
Holcomb: Under the arms.
Earl: Under the arms, yeah. I think we ended up adding a little bit of hair here and there. I know, it was a great prosthetic, it just required some cleaning up. I think we added some jiggle in a couple of spots too.
Very nice. Who decides who gets the jiggle job? Is that a dedicated gig?
Holcomb: The new guy! No, I’m kidding.
Earl: Yeah, he’s got lots of references here. OK, who’s next? [laughs]
And not to forget Korg and Miek. They’re in there playing video games, right? How detailed was the script when it comes to something like this? Did the script explicitly say, “OK, so Miek has a slice of pizza in each hand and another slice stuffed in his face,” or was that on the visual effects team to enter that little fun detail in the movie?
Earl: That was something that sort of fell on us. I think they talked about him, they did talk about him eating pizza but then Kevin Martell, who’s our animation supervisor and his team, they sort of just kind of talked over ideas. Like one of the ideas that made it in there was when he ends up throwing a piece of pizza at the TV, that was something that the animation team came up with. And you know, those guys are such great characters. It was their moment to shine too, Miek in particular, but Korg as well.
The Russos are all about storytelling, so it’s not just doing shots per se, we’re always thinking about, “OK, so what is the story here? What’s going on? What are they doing? And just trying to imbibe as much character as we can into them. Even though they’re on screen for the brief moments, it’s trying to give them a story and give them character and life. So we would try things and sometimes we would do multiple things and send them down to the brothers and down below and the team there.
Holcomb: Whose idea was it for the Waititi outfits? I think that was-
Earl: Oh, Taika Waititi. I think that came from the team down there too, Korg wearing the pineapple jumper. He plays Korg, so someone came up with the idea to put him in this pineapple jumper that he has been known to wear on set, so that’s where Korg’s outfit came from. It’s things like that, you get sort of the initial brief of here’s the shot, they’re sitting there and he’s playing video games, but then it’s like trying to just try things and pitch ideas. The Russos are super collaborative and really open to just seeing the things and trying stuff and it’s never about where the idea originated, it’s always about what’s the best idea and that’s what makes them so great to work with. It’s just that they’re open to ideas and sharing and collaboration and that’s always great on a project like this.
You bring up the collaboration stuff with the pizza choice or the pineapple jumper choice. Was there something that you and the team really wanted to do effects-wise that never made it in the final cut?
Earl: Like Bruce has built a ton of the Iron Mans, yeah I think he’s built almost every Iron Man suit, so he’s always got something up his sleeve.
Holcomb: Trying to force it in there, never gets in there.
Earl: I mean, Bruce you probably have a thousand ideas that never make it in. It’s always trying to just find that balance of, we could probably iterate and keep going on things forever. But yeah, I can’t think of anything in particular on this one. I mean again, they’re so open and we send them stuff and they’re like, “Yeah, great, let’s do that.”
Holcomb: We had one. I thought one that was pretty cool that they initially wanted to do was having War Machine develop this forward shield thing that was going to be like a militaristic thing. We kind of pitched them, I don’t know, I guess around 12 or 13 ideas, and we just couldn’t really get something that separated him from like Iron Man technology. Something that was more believable and more current-day military look to it with the gray color and not just like an energy thing, but in the end, it just didn’t really serve the story time-wise. So they just decided not to go with it.
Earl: I was just going to say, I think that’s a big thing. It’s always about story.
Holcomb: It is, yeah.
Speaking about War Machine, he emerges from the rubble in the third act in this really dope kind of a War Machine-Buster type armor; it’s Iron Patriot colors actually. Was that something that was in the movie from previous concept art stage, or was that another one of the things you tinkered with to get in the movie?
Earl: That was one that they had concept art for pretty early on, I think. We built and, I think we only ended up having one shot that had Iron Patriot, and it was out of focus in the background. What happened in a couple of instances was simply multiple vendors working on the film and they weren’t really sure at times who was going to do what. It was actually early on, we did develop an Iron Patriot model based on their artwork, but ultimately we didn’t end up doing any of those shots, but we did build the outside.
Holcomb: Yeah, it’s always story-driven too, because why is he in an Iron Patriot suit after he’s got, I think he’s got two previous War Machine suits on, he’s got the one from Infinity War when he’s at the beginning of the film, then he switches over to the Mark 5 and then of course […] when he gets crushed by the waterfall or whatever, and then that thing’s damaged. Of course, he’s got no suit after that, but if you kind of watch, it’s kind of blurry, we kind of blurred it out more than I thought we were going to, but you can actually see the Iron Patriot kind of armor just laying there in the water.
Earl: I guess it’s kind of an Easter egg. When we come to Ant-Man and he’s buried in that little pile of rubble and he’s tiny, when he comes out of that, if you look, there’s a close-up on his face, you can sort of see the Iron Patriot suit in the background there. And I think there might have been at one point a storyline of how he gets the Iron Patriot suit, but anyway, it’s there in the background, and then Rhodey ends up with it at the end.
You bring up Ant-Man and then there’s the Quantum Realm suits. They have this morphing Ant Man-esque helmet. Why was the decision ultimately to go completely digital on these suits rather than a practical suit?
Earl: I don’t fully know the decision behind it. I do know that what ends up happening is the suits evolve and change. In this case, I don’t think the design of the suit was fully there when they started shooting the scene, and I think they were shooting multiple scenes at multiple locations. I read some articles about people saying that the suits are digital. We do it all the time ,and we don’t necessarily call it out. Where you know, Cap’s body will be CG, or Black Panther’s body 99% of the time is all CG, Spidey, when Tom’s just there wearing the suit, that’s almost always CG. So I think the decision came once the suits had to go on and off. So you can either choose to have two suits that they would get in and out of. I think it just came down to wanting to have the flexibility to get the design just right, and at the time of shooting wasn’t quite there.
I’m speculating on it. I don’t really know the full story behind it, but it’s something that I think all of the effects houses have gotten so good at is just doing digital costumes that the studio can rely more on it and not necessarily have to make those decisions up front. It just allows a little bit more flexibility and knowing that you’re going to have to build the suit anyway, because it has the sort of nanotech growing out of the time watches that and that’s the thing. If you know you’re going to have to build the suit anyway to get the transitions, then you know you’re going to have to have a photo-real suit in which case that then also lends itself to, oh well if we’re going to have it then we can put it in these other shots.
Then sometimes a lot of the costumes are built. Ant-Man’s obviously a suit that’s built, but we’ve done that in all CG, or like I said, Cap, I think all of them at one point or the other have gone all CG.
Holcomb: It’s a tough call too, because in a lot of their storytelling, especially if the film’s kind of progressive, you always want to have the actor’s full head in the shot, because they’re acting, and you want to see their face, and you want to see them perform. You don’t want to hear a bunch of dialogue behind a mask all the time, so I think we all noticed is as the films went on in time, especially in the case of Iron Man, you wanted to see his helmet either completely disappear or be folded up behind his head, but it had to happen pretty quick within some of these shots where they’re delivering dialogue. I think in the case of the time suit one, I mean where does that suit go? It’s got to come completely off. Russ and I have talked about this through several films. They only give you like two seconds to tell that story of where the helmet goes or where the suit comes off and it just goes so fast, you don’t really have […] too many options.
Earl: Yeah, and I think that’s the thing is we’re always trying to, something, especially with the Russos, started back on Winter Soldier, is always trying to bring some sort of physicality and reality to it. It’s the idea of it’s magical, but it’s got to feel like magic that you’ve photographed, or it’s got to feel like tech that you’ve photographed. So that’s always the balance where it’s got to come off, but at the same time we’re trying to do things like preserve volume or make the pieces feel like they move and shift and something that Bruce is really good at is making it all feel like it’s physically correct and happening. Even though it happens over a couple frames, those little things and paying attention to that detail I think helps sell the idea. Those time suits actually didn’t have helmets when we first started working on them, they looked more like alien tech, the bubble helmets that you see in Guardians of the Galaxy. But then it ended up, the decision was made to go with more Ant Man-like helmets on those suits.
Speaking of the Quantum Realm, this isn’t Marvel’s first time with the Quantum Realm, right? We’ve seen it in a couple movies before, but yet it the look is changed. Was the team using Doctor Strange and Ant-Man and the Wasp as reference or were you guys trying to create your own look and feel on how the Quantum Realm was used?
Earl: We had worked on the Quantum Realm for Ant-Man, and then also again for Ant-Man and the Wasp, so we had done it a couple of times now. It’s one of those things that’s fun to get to go back to because we learned a lot on the first couple movies where we had done it and we knew what worked, what didn’t work, how we would approach it differently. On this one, it was trying to always bring something new to it, but the thing that was interesting was in this one, you want to see places you’ve been before, and again, you don’t have very many frames to tell this story. But you’re trying to tell this story of, “OK, we recognize this, we’ve seen this before, we know where we are.”
But in this case, and it’s something that we’ve always talked about on these movies — are they traveling or are they shrinking? Because of the way the worlds were built is, they basically are scaling, so the fractal geometry they’re flying through is actually physically scaling up as they’re moving through it. We would always have conversations of, are they traveling or are they shrinking? Because the animation guys were trying to move these characters through this massive environment, but the environment was being built on the fly and scaling up as they were going. It was a very good back and forth between animation, effects, and compositing to try to bring all this stuff together and show us something that we’ve seen before but at the same time, give it a little bit different feel.
And again, back to story. It was all about trying to tell the story of okay, they’re coming off the platform through the quantum tunnel, so we’re referencing the quantum tunnel stuff from the van that was in Ant-Man and the Wasp. That was the driving force there and then we were tying into some of the stuff we had done in the prior film on Ant-Man and the Wasp, sort of traveling through the different layers of the Quantum Realm. Before, the big thing was trying to tell that story of “OK, the time GPS is able to guide them on these different paths.” It’s funny because they go by so quickly, but a lot of love and time and effort goes into just trying to give a sneak peek of each of those levels, tell the story that they’re traveling through, and just try to bring something new to it.
When it comes to Dr. Strange, I guess not even Dr. Strange, but all the mystic arts practitioners altogether, they have all sorts of hexes or conjured spells. Is there a library of these spells and hexes you can pick from and alter to fit within the movie or is each one of these shields or platforms or orange rope things, are they all built from the ground up?
Earl: I think the library would be the comic books, and that’s something Bruce is well versed in.
Holcomb: I was actually surprised at how much Dan actually knew. Dan DeLuuw, who’s the visual effects supervisor from Marvel. He knew all the names of all the specific spells and kept pointing us to all the specific designs of which spell looked like what and we adhere to pretty much all that stuff.
Earl: Yeah, we use like Winds of Watoom. But yeah, so Dan would pull a lot of comic book reference and then we’d go do our homework and research the reference that he had. And then looking at that and looking at the imagery, we would just get on calls and we would talk through like, “OK, we like the graphic nature of this, or this needs to be a little more organic,” or we would reference the prior films, and then the effects guys could just go and try different things. So we went through, there’s the Winds of Watoom and when Strange is stuck in the waterfall and he creates the water tornado. There’s that beat. Then there’s also the other beat where he comes in, when he first flies in on land and then he’s got the other spell that’s taking some of the bad guys out and sucks back down to that. All of that is basically referenced straight out of the comic books. Then we just try things and send them down and see what they like and it just sort of iterates and evolves. But yeah, it’s always, I think always trying to bring things back from the comics to just feed the fans and give them something that they, give something new to it but also give them something that harken back to the comic books.
Now Bruce, being such a big comic book fan and if you’re able to name some Dr. Strange spells, is there one character, as a modeling guy, you would love to get your hands on to bring to live-action?
Holcomb: Galactus. Silver Surfer.
Galactus? Very nice.
Holcomb: Yeah, hands down. We did Fantastic Four even though they were Fox properties. We did both the original and Rise of the Silver Surfer there, and then we did a remake, but I don’t think anybody really got the cosmic family correct. I know that I’d really like to see a giant thing on that scale like Galactus show up and start stepping on towns and doing all kinds of weird cosmic-level stuff. I’m very excited to see if they want to go that direction, because to me, that was a lot of fun is whenever the cosmic level came into groups of superheroes that were on Earth. And I don’t know, I’d like to see Namor. I’d like to see if they’re going to bring him up now that we’ve got Aquaman running around, I’d like to see what Marvel’s answer to that will be because I felt like in Aquaman, a lot of the stuff that was underwater to me… I felt like there’s something new that you can do underwater that isn’t so lit up and like a theme park ride type of thing. I want to see something a little darker and scarier and murky and more monsters and stuff down under the water, so yeah, there’s tons of them. I don’t think they’re going to be out of storylines for a long time.
Bruce, I know you’ve worked on pretty much every Marvel movie ILM has done. What was the workload on Endgame compared to previous films? Was there this big sense of urgency? Because I mean, Kevin Feige and the Russos, they’re touting, “Hey, this is the culmination of 22 movies.” Did that add a little bit of pressure or was the workload on this not unlike other properties?
Holcomb: I think for me, kind of being one of the folks that’s on the workforce on that end, you’re always nervous about the big battle, right? Everything up until that moment when that starts becoming reality, usually it feels very capable and a controllable amount of work. But usually when the big battles show up, in most cases, so many elements get added and taken away just to service the story that all of the sudden, a minimal workload all of the sudden turns into maximum workload because you’ve added two more sequences that weren’t there to kind of facilitate the story. So that’s kind of where at least for me things get kind of like a little bit sketchy.
For Endgame itself, I feel like it was very together and controlled. I mean for Infinity War, I felt like pulling off Wakanda in broad daylight ended up being much more of a challenge for us then actually the big end battle in Endgame, just because of the lighting itself. They refer to it as the fog of war and that type of thing and a lot of the reference that they gave us, you can play around a lot with that and hide it and give elements of big wow moments just visually that are kind of like off in the background. But when you’re presented just like with a desert in Africa, and there’s really nothing there except the sun, I felt that was the biggest, bigger accomplishment for us as a team. I really enjoyed seeing a lot of that stuff come to life.
Earl: I was just going to say I think what’s great for us, because having done all four of the Russo Brother films, we started with Winter Soldier and then into Civil War. In Winter Soldier, we had the helicarrier destruction. In Civil War, we had the battle with the digital airport and then that led into the digital battle in Wakanda with thousands of characters and organics in the environment. I think all of those things, one was the foundation for the next. So they sort of grew as each film in the Russos’ Marvel history. Each film was sort of a stepping stone for the next one, and so it gave us a chance to develop our tools for the artists to develop the toolset. It was just the progression and Endgame was sort of the end game to that progression.
Holcomb: Nice one!
Earl: It all sort of built to that moment and it was similar to how our experience with the Russos and working on their films and working with them, it was all the same team so we had developed a short hand, we knew what the expectations would be. Even sometimes when you’re working with new directors or new teams, they sort of say one thing, this is what it’s going to be, and then it ends up being something different or it ends up being just that.
But in this case we knew from prior experience what the expectations were going to be. One of the things that I try to do or we try to do is just make sure that we’re always thinking a couple steps ahead. So like Bruce is saying, the battles are going to change or the assets are going to change, just making sure that we’re working in a smart way that we don’t box ourselves into a corner because we always want to be there to serve the team and the directors and the client and not say “No, we can’t do that.” If they come to us and say, “Oh it’d be great if we had this where we just blow the H Ship up” or we do this or do that, we’re right there to be able to say “Yeah no problem, we can totally do that.:
Holcomb: And quite frankly, that’s the best thing that I’ve experienced at least working with Russ is that on every one of those shows, we were asked to do something completely different visually-wise from the helicarriers blowing each other up in the air and then the airport. We did always have that kind of plan thanks to Russ. For me, I didn’t feel overly stressed about how we’re going to pull things off even though the amount, from my aspect it’s always about the amount of data that you’ve got to collect and organize and you’ve got to alphabetize it and make sure that it’s capable to have in your shot in order to pull it off. But Russ is really good about, especially at the airport, I was always amazed, Russ, how much information that you actually did at an airport to find out what belonged where and what baggage cart belonged over there and that was all good. Very good.
The masses go to the movies and oftentimes they’ll end up referring to like CG or VFX as a whole, right? I know depending on difficulty, this is probably going to change, but say on average, how many people are involved in one shot? At ILM, how big is the team that’s working on just the average shot?
Earl: It depends on the shot. I mean I think there’s the people working on the shot but then there’s people developing assets. We’ve got a great production team at ILM. We probably had 400 people across three studios who were working in San Francisco, London, and Singapore. You’ve got a full team of people that are modeling, that are painting, just developing the assets, and then those assets, of course, all feed the shots. Then you’ve got your facial performance and the teams that are doing that.
I would say at least a half a dozen people. Every shot, probably if you look across all departments and all the people, I mean it’s probably a dozen or so people that are working one shot. And I think we try to, as much as we can, just be as collaborative the best we can, and we’ve got great teams of people. We’ve got great artists and over the years just have gotten really good at dividing and conquering. But it takes a village, I mean there’s a lot of work that goes into it. People say “Oh CGI is ruining movies” or “CGI gets a bad rep.” I forget who said they don’t call it a computer-generated screenplay. Just like the computer isn’t actually generating this stuff, it’s a lot of talented artists working behind the scenes to get it done.
Lots of people working very hard and pouring their heart and soul into getting these movies made and doing it in such a way that fans will enjoy.
*****
What was your favorite shot of Avengers: Endgame? Let us know your thoughts either in the comments below or by tweeting me at @AdamBarnhardt to chat all things MCU!
Avengers: Endgame is now in theaters and will be followed by Spider-Man: Far From Home on July 2nd.
