Nelson St. Rebuild

Once the shoot had taken place and we got the shot we wanted in the can, it was then onto the arduous process of post production. As the hard work had been put in during pre production, the tracking process was relatively easy. The techniques I used can be seen on one of my earlier posts, although for this particular shot I actually ended up using the Nuke X camera tracker instead of Autodesk Matchmover. For some reason Matchmover was having a hard time solving. There were a few small issues with exporting the camera track from Nuke to Maya, but luckily I happened upon a neat little python script that did a good job of exporting it for me. That particular script can be found here: http://www.nukepedia.com/gizmos/python-scripts/import-export/fromnuke2maya-export-nuke-cameras/

So, with the footage tracked and the camera imported into Maya, I set about the rather large job of rebuilding the Nelson street environment. As outlined in an earlier post about Environmental Rebuild, I did this in order to give realistic reflections and shadows onto any geo I placed within the scene.

Using the original footage as a guide, I built the entire area out of polygons and textured then via a combination of camera projection and textures captured during the shoot by my lovely lady Elaine.

This screen grab shows the environment visible to the camera rebuilt within Maya. The green static cameras on the walk way chart the path of the tracked camera and have been used to project frames from the original footage onto the geometry.

This screen grab is the view from the actual camera track, looking at the rebuilt and matched geometry with a mixture of projections (lo res) and textures (hi res).

One this environment was finished it was then a matter of checking the track and the geometry to ensure everything lined up okay. In order to cut down on render time, I added some metallic spheres into the scene and hid all the environmental geometry apart from in the sphere's reflection. This meant that all the computer was needing to render was the sphere's and the reflections in the spheres. Much less computationally intensive.
Once the spheres were rendered it was just a matter of importing the footage into Nuke and comping it back over the original footage.

The screen were a last minute addition within Nuke as I still had the original camera track inside of Nuke. Having the same camera in both programs really saves a lot of time.
All in all I'm pretty happy with how this came out and especially happy with the track within Nuke. Seems the Nuke tracker is far superior to Matchmover.

3D Tracking

In an earlier post I took a quick look at the process of Matchmoving, however in retrospect that probably wasn't the best example of matchmoving or 3D tracking process as the movement was only in 2 dimensions. When talking about matchmoving there is a very important difference between 2D and 3D tracking, namely 3D tracking is a much more annoying. 2D tracking is something that can be performed in programs like After Effects and simply relies on the tracker following certain points around the frame. 3D tracking on the other hand is somewhat more complex a process. 3D tracking points are collected via manual track points or auto track points and used to calculate and rebuild a virtual 3D environment with a 3D package such as Maya.


As I have found throughout this latest exercise, it is vitally important to have as much information about your physical camera as possible. The focal length (length of the lens) and the film back (camera aperture or camera censor size) help turn a nightmare track into something a bit more manageable. That and I might also add that Autodesk Matchmover 2011 is a serious pain in the arse. I downloaded a free student licence of Maya 2012 and Matchmover 2012 kindly from Autodesk with a 36 month licence. Anyone interested the address is http://students.autodesk.com/?nd=download_center Make use of being a student and get as much free shit as possible.


But I digress. After downloading and installing Matchmover 2012 the automatic tracking was a breeze. You'll need to break your footage down into an image sequence, I used a Targa sequence which decompiled pretty quick and good quality with Adobes Media Encoder.  Then it's simply a process of upping the quality of your auto track and the length that tracks will stick to the points. Once the track was finished, I solved the camera create 3D points within the scene and added a coordinates system to tell the 3D package which way is up. This is a very important step and helped to line everything up within the Maya axis.

3D Track points within Matchmover after automatic tracking and camera solve. Not the inverted axis with the Y pointing downwards. (Mistake within MM but rectified in Maya).

Once my camera created in Maya it was simply a matter of recreating the lighting of my environment. I used Maya's IBL (image based lighting) to create my environment from within Mental Ray with a full panoramic photo of my living room. This also created reflections within my geometry to help sell the cup. The shadows and reflections on the table were created with a Background shader. This allows a material to only display specular and reflections within the environment. After a bit of tweaking I rendered it out pretty low res and did some colour grading in AE to help tie it all together.

Left middle - perspective view of cup and background shader geometry. Right middle - IBL environment and camera tracks and image plane.

All in all I'm really pleased with the way it came out. Even though it is a very simple shot and piece of geometry, the actual track and coordinates systems proved the can work perfectly and open the door to some much more complex shots to come. Watch this space!

3D Track with geo and grade

Raw Footage.

Match Moving

First port of call on this magical journey: Match Moving.
So, what the fuck is Match Moving I hear you ask? Good question. Match moving is the process of 3D camera tracking, that is finding fixed points in a live action plate(piece of filed footage) that can be tracked in 3d space in order to create a virtual camera in your production application. Sounds complicated? Not really.
It breaks down like this; In every piece of footage there is a paralax, that is objects closer to the camera move faster and objects further away move slower. You feed a piece of footage into your match moving program, in this case Autodesk Match Mover. The program analyses the footage and finds points to stick too, these are called tracking points. The program will try its hardest to stick to these points, be it a lamp post of tree, something that is static in your footage, although if the footage is moving about violently enough this will need to be rectified by hand.

In this example, you can see the tracking points from the 3d track of this street attaching themselves to fixed points (some..!) to generate a virtual camera for our 3d package.


Once the program has analysed the footage, it is down to you to add a co-ordinates system, that is, tell the program on which axis these points lie and a rough estimate of their distance. The program will the "solve" the shot and create a virtual camera based on the tracking points and the footage fed into it. 


Once the camera has been solved you should end up with something looking a little like this; a virtual camera, exactly copying the movement of its real world counterpart. This will allow you to add geometry to your scene and with some colour correction and a few other tricks get it looking prety good.

That's pretty much the nuts and bolts of match moving. I know all the tech geeks out there will righty be saying that you could quite happily do the same shot with a 2D tracker, but that would defeat the point of looking into matchmoving, that and I couldn't find a 3d Piece of footage...


So, in conclusion here's my animation test into matchmoving. This one is pretty simple, but it's a good place to start. I will be looking into much more complex solves in future containing green screen tracking markers and full 3d solves.

P.S. here's something that has utilised the same technique, that and it's got a break dancing robot which rocks!