How the biggest movies are replacing actors and sets with CGI

• This genius Marvel fan theory might change the way you see Avengers: Infinity War
• How Doctor Strange created its mindblowing time-travel scenes
• Here’s how Supreme Leader Snoke was changed for Star Wars: The Last Jedi

The Dementor puppet, built on a smaller scale and shot from underneath in a swimming pool to create its spooky rippling effect? Cute, but in the end replaced with a computer-generated version instead. The hand-made, fully-sized animatronic dragon head, built to produce real flame? A triumph – but still much too unreliable, so animated by a VFX company for the final movie.

Over and over it went, and by the end I had to wonder: what was the point of trying out the practical effects in the first place? In the end, surely they must have just realised it would be easier to call up the men with computers on day one.

But then, that raises a much wider question – because if you start to walk down the path of efficiency and flexibility in filmmaking, what happens when it becomes easier to CG the characters than actually use the actors portraying them? And if the technology was good enough, would audiences even notice the difference?

A few years on, the computerised special effects (usually referred to as VFX) industry is bigger than ever, and there are two things everyone thinks they know about how it all works.

Number one: there’s more CG in movies than ever, but that’s offset by number two: the fact that audiences are savvier about and more discerning of VFX than they ever have been before.

But is that second point true? Well, I’m not so sure. While audiences can easily locate and critique overtly CG characters in big movies – Mark Ruffalo’s Hulk in the Avengers, for example – they may be less aware of exactly how much of these films wasn’t actually shot on a set, or how often the actors they’re seeing onscreen have been replaced digitally.

“People do underestimate [the amount of CG], but I think it’s a good thing!” Framestore VFX supervisor Alexis Wajsbrot, who has worked on films including Thor: Ragnarok and Doctor Strange among many other movies, told me earlier this year.

“I think if people underestimate it means we've done our job correctly. It means that it's seamless and people don't see it. Which is the goal.”

What am I talking about? Well, take a look at this VFX breakdown from effects house Cinesite, who worked on an early fight scene for Avengers: Infinity War (generally, specific segments of modern effects-heavy blockbusters are carved up and given to different effects houses to work on).

While no-one would be surprised to see the alien character Ebony Maw constructed by CG artists, it might be more of a shock that often, characters like Iron Man (Robert Downey Jr) and Spider-Man (Tom Holland) are, at times, entirely animated, even in non-action scenes.

Sure, sometimes Downey Jr and Holland wear motion-capture suits or take their masks off (meaning their disembodied heads are plopped onto CG bodies), but a lot of the time you’re just looking at ones and zeroes onscreen – and given that quite a lot of the spaceship set from this video is also entirely CG, there are times during this film when audiences are essentially just watching a photorealistic animated movie, free from real-life actors except for voiceover.

Put simply, it's entirely possible that Iron Man suit and that Spider-Man costume never existed, and some scenes they’re in were never shot on a camera – and this is far from the only time these sort of substitutions have happened, with Infinity War co-director Joe Russo revealing that in his earlier film Captain America: Civil War the suit worn by Black Panther was also entirely computer-generated.

“[They] really did a ton of work on that outfit,” the director said on the film’s DVD commentary.

“I mean, we had an outfit that we used on set. It’s impossible when you’re talking about an otherworldly outfit like the one that the Panther wears, which has a certain luminescence to it because it’s made of a woven metal.

“We could never afford to construct an outfit like that that an actor or a stunt player could move around in without sweating to death or that would capture the luminescence that we need.

“So what we ended up doing in post is [visual effects house Industrial Light and Magic] came in and painted over Chadwick and the stuntman. The outfit is completely CG.”

Now, it’s not SUCH a mind-blowing discovery that some of the masked heroes in Marvel movies would be easier to CG, even if audiences would be fair to assume they were really actors on a set – but this is just the tip of the iceberg.

Because sometimes, even the recognisable, human faces and bodies of the world’s biggest stars aren’t safe from a CG replacement.

For example, this iconic shot – which shows Chris Hemsworth’s Thor towards the end of 2017’s Thor Ragnarok – isn’t actually of Chris Hemsworth. He performed the landing stunt, and was filmed doing so, but ended up being replaced almost entirely with CG, even in close-up.

There’s maybe a little bit of his face in there, on sufferance, but the rest is a state-of-the-art digi-double, aka a computer-generated twin of the Australian actor created with muscle simulations and animation, checked against the real Hemsworth with a special full-body scanner.

“From the beginning we knew that Thor was going to be heavily replaced by a digi-double,” Wajsbrot explained during a talk he gave at the VFX festival earlier this year.

“Because the lightning is so close, you can't get away with a 2D solution. Or you could, but it would look a lot less cool.

“We are just supposed to do the giant jump, and then switch back to live-action. But I knew I wanted it to be digital from beginning to end, just because I wanted the lighting.”

“Because they paid Chris Hemsworth quite expensively, let's try to keep his face!” he added.

“But we will need to close his eye. So we'll need to replace that at least. We'll need to replace the hair, because there is lighting – well, to be fair, I think there is very little [of his face] we kept on the shot on the end. But we tried! We tried hard to keep the face.”

In short, then, Hemsworth was replaced because it was just easier, and more flexible, to do it in CG – just like all those Harry Potter monsters a few years before – and in the sequence that follows this landing, Wajsbrot revealed that much of the ensuing battle had literally no basis in reality, with a large amount (though not all) of the fighting never filmed at all and instead created digitally. You can see some of these scenes, as well as how they built the Thor digidouble (about halfway through) and the entirely CG body for Cate Blanchett's villain Hela, in the breakdown below.

The process of creating these digi-doubles is common, with films like Star Wars: The Last Jedi and Logan making extensive use of them just recently, but despite their widespread adoption (particularly during action scenes) very few members of the public are aware of them.

The old truism that CG human beings are too uncanny and offputting to be convincing now seems like a relic of another time – and apparently, sometimes even the directors of these massive blockbusters might not be aware of how often their stars are replaced with CG.

“I'm sure the director does not even know that in most of the shots [Thor] is CG,” Wajsbrot said of Ragnarok helmer Taika Waititi.

“Because he wants to use the plate (aka the footage that has been actually filmed), as anyone like a director or VFX supervisor does. Most of the time we want to try to use the plate as much as possible, and we want to shoot something.

“And I think they did good to shoot it. Because we used it as a reference, and we could not have achieved the same result if they did not shoot it, for lighting reference, for everything. And like I said, we used a bit of the face.”

This slightly oppositional relationship between director and VFX house when it comes to CG seems a common dynamic, with visual effects supervisor Mike Mulholland (who works for iconic VFX house Industrial Light and Magic) recalling his own give-and-take with Star Wars: The Last Jedi’s Rian Johnson ahead of that film’s production.

“Rian came in and said the thing that people say, they say we want to do everything practically ≠ only a little bit of CG please,” Mulholland recalled during his own talk at London’s VFX festival earlier this year.

“And it's a story we actually go through quite a few times as visual effects houses, to have these conversations with directors, especially people who are coming to a big franchise film, with a lot of visual effects.”

“It's a bit of a process where we're kind of winning trust, and showing what can and can't be done. And we ended up in a very happy place with Rian, because he kind of really got what we could do.”

Among its 850 VFX shots, The Last Jedi ended using digi-doubles too – during the chase on casino planet Canto Bight, John Boyega and Kelly Marie Tran are subtly switched out when a Fathier ear flaps over them, masking the transition – and in other parts of the story ILM made a few of their own CG substitutions, gently letting down Johnson’s original hopes of using practical models for space battles (as they did in the earliest Star Wars movies) and later replacing the few ship exteriors that were built on set for, once again, better lighting.

“We did a practical shoot on the entire A-Wing,” Mulholland said of one particular ship, which had been built on set.

“End of the day, when we were finished, the bit that remained from the practical photography was just her in her cockpit. And we ended up replacing the entire ship.

“There's nothing wrong with the practical thing, but we just needed to get the lighting continuity to match what was happening in the rest of the sequence.”

“And when you're filming, this was filmed maybe 10 months before we finished the shot? You don't know exactly what's going on lighting wise. So we ended up stripping out the practical, replacing with the CG.”

As before, you can see some of this process in the VFX breakdown below at about 15 seconds in.

These are some very specific examples, cherry-picked from a massive industry that goes far beyond the Marvel Cinematic Universe or Star Wars, but they illustrate a wider truth about CG-heavy modern blockbusters – despite what we may think, a lot of the time we’re not any savvier than the audiences who gasped at a flying Superman in 1978, or whooped at the Millennium Falcon’s first jump into lightspeed.

The only difference between us and them is that we’re rarely wowed by cutting-edge VFX any more, just fooled by it – and in some instances, the best-case scenario for the people working on the shots is that we won’t notice their efforts at all, particularly when it comes to replacing well-known, well-paid faces with digital clones.

“Our goal is for no-one to know that it's digi-double," Wajsbrot told me. "And you learn it only in this VFX talk, or watching a breakdown.”

Of course there’s nothing wrong with any of this, and it’s not some huge conspiracy – VFX breakdowns are readily available to view online, and the people behind them are fairly open about their techniques – but it’s unsettling to realise just how often our own eyes can deceive us, even when we think we’re adept at spotting the hidden tricks in blockbuster movies.

And what happens when it becomes easier, cheaper and more flexible to not use real actors for big films at all? Will Chris Hemsworth be willing to just do a bit of voice work and kick back in his trailer while his digi-double does all the hard work? Will an aged John Boyega be up for a youthful recreation on Star Wars Episode XVIII?

Frankly, it’s enough to make you wish they’d whip out the animatronics, puppets and old-school practical models again, and let us all be more aware of whether we're seeing something real or not - even if it did take a bit more time and money.

Though they’d probably be less convincing playing Thor...

Avengers 4 is released in spring 2019