Moving photos from 'Harry Potter' novels finally become a reality, thanks to 'Photo Wake-Up' technology

Moving photos from the magical world of Harry Potter may soon become a reality. Scientists have now created a new algorithm called Photo Wake-Up that makes it possible to make a person from a 2D image run, walk or jump out of the frame. It also allows users to view the animation in 3D using augmented reality (AR) tools. 

"This is a very hard fundamental problem in computer vision," said Ira Kemelmacher-Shlizerman, an associate professor at the University of Washington in the US.

"The big challenge here is that the input is only from a single camera position, so part of the person is invisible. Our work combines technical advancement on an open problem in the field with artistic creative visualisation," Kemelmacher-Shlizerman said.

Previously, researchers thought it would be impossible to animate a person running out of a single photo.

"There is some previous work that tries to create a 3D character using multiple viewpoints," said Brian Curless, a professor at the University of Washington.

"But you still couldn't bring someone to life and have them run out of a scene, and you couldn't bring AR into it. It was really surprising that we could get some compelling results with using just one photo," said Curless.

Researchers say that the algorithm has numerous applications.

Researchers foresee this algorithm could lead to a new way for gamers to create avatars that actually look like them, a method for visitors to interact with paintings in an art museum or something that lets children bring their drawings to life.

READ | Moving photographs from Harry Potter novels may not be far from reality, thanks to AI

Examples in the research paper include animating the Golden State Warriors' Stephen Curry to run off the court, Paul McCartney to leap off the cover of the "Help!" album and Matisse's "Icarus" to leave his frame.

To make Harry Potter's magical photos a reality, Photo Wake-Up begins identifying a person in an image and making a mask of the body's outline. From there, it matches a 3D template to the subject's body position. 

Then the Photo Wake-Up algorithm projects the 3D person back into 2D and stores 3D information for each pixel: its distance from the camera or artist and how a person's joints are connected together. 

Once the template has been warped to match the person's shape, the algorithm pastes on the texture. The algorithm also generates the back of the person by using information from the image and the 3D template. It stitches the two sides together to make a 3D person turn around.

When the 3D character is ready to run, Photo Wake-Up sets up the background to prevent the character from leave a blank space behind. The algorithm fills in the empty behind the character by borrowing information from other parts of the image.

At the time of publishing this story, Photo Wake-Up works best with photos of people facing forward, and it can animate both artistic creations and photos of real people.

It also handles photos where people's arms are blocking part of their bodies. However, the algorithm is not yet fully capable of animating characters who have their legs crossed or who are blocking large parts of their bodies.

(With PTI inputs)