The idea of 3D has been around for a long time, but I believe people have failed to give it a fair chance by not looking at it as its own medium, but rather just as enhanced 2D. In this project, I hope to present to you an entirely new way of thinking of 3D that makes movies and videos truly engaging and believable in a way no one has ever done before.
There is a lot of things that can go wrong when making a movie or video in 3D. During this project I have discovered many rules that I believe should not be broken. I would claim previous movies have often broken these rules which makes them unenjoyable to watch in different ways. In this project, I will go through all of them.
What is modern 3D?
There are a lot of ways to view 3D video. Previously, the most popular choices has to use 3D glasses together with a 3D projector or 3D TV. This works fairly well, but it's very expensive and those technologies can't really do anything else, which makes it something most people won't find worth purchasing. Recently, technologies like VR and AR has been becoming more and more promising and rumors like Apple's AR glasses as well as the already well established VR headsets like Oculus or PlayStation VR, all indicate us that we are moving into a new era of technology that might replace the smartphone or even displays completely within the not so distant future.
There are really 2 choices when it comes to viewing something in 3D. One way is true 3D where the viewer actually can move their head in 3D space and get closer or further away from objects. This is incredible and should definitely be used as much as possible. The problem is that all the content has to be computer generated in real-time for this to work. Because of that, there is only a small niche of content that actually can be used and therefore not a viable option. The second option is to use Stereoscopic video. The beauty of stereoscopic video is that the viewer can get a hyper realistic 3D experiences from simple traditional 2D video. This is how it works: first you put 2 videos side-by-side in an a normal 2D video file(each video recorded or rendered 64mm apart to simulate the distance between the human eyes). Then, when you put that file into the headset, those two videos are then separated and projected onto 2 spheres. In VR the position of your eyes are then placed at the center of each of the two spheres. The result gives the viewer a realistic 3D experience where they can look around freely. The drawback is that you can only rotate the head in pitch and yaw directions, tilting too much sideways breaks the illusion. Also, if the viewer is moving their head forward to get closer to something, it won't work as the viewer can't actually move around in the world, even though it feels like it when they are standing still.
To figure out what works and what does not work when using this technique, I worked for many weeks straight with a pair of Oculus Quest VR headset trying every single combination of settings I could come up with. The methods I've been using is rendering images and videos from Blender 3D, testing motion and graphics in Unreal Engine 4 as well as my own homemade 3D camera rig made out of two iPhone 7+, a piece of wood, some rubber bands and some nails.
These are the parameters I tested:
Field of view (viewer) Filed of view (camera) Framerate Motion blur Video resolution Depth of field Camera tilting Camera velocity Camera acceleration Camera angular velocity
Field of view (viewer)
There are two things we have to think about when we're talking about the Field of view(FOV). The first one is about how much of the viewers total FOV we should actually use. In VR, the viewer has the ability to look around themselves 360°, but is this actually something we want? As designers and filmmakers we have to make some decisions for the customer and in this case I found that around 90° horizontal and 45° vertical works well. More than this felt distracting and less than this felt crammed.
Field of view (camera)
The second thing that is important to think about is the FOV of the camera. This is probably the most important thing when making content for stereoscopic video since it completely breaks the illusion if not done correctly. The rule is:
The second thing that is important to think about is the FOV of the camera. This is probably the most important thing when making content for stereoscopic video since it completely breaks the illusion if not done correctly. The rule is:
When you are zooming in, everything in the scene scales up perpendicular to the camera. In a 2D video like this one above, it works.
However, in 3D there is a problem. Everything stretches out horizontally and vertically, but the depth remains the same. This creates this weird effect that you won't notice in 2D, but in 3D, you do. (This video is obviously not in 3D but I changed the camera angle to illustrate the effect.)
That was the most difficult part to explain so if you understood that the rest will be a breeze!
Framerate
For some reason it seems that higher framerates are better and I have no idea of why. I tested both 30fps and 60fps and for some reason, 60fps works great while 30fps is almost unwatchable. I have a suspicion that is has to do with the fact that the viewer can move their head in VR and therefore it has a game-like aspect of it that makes lower framerates feel unresponsive and choppy, but even though the framerate of the video is 30fps, the headset itself ran at 72fps, so it shouldn't matter. Hmm...
Motion blur
I was a bit afraid that motion blur in VR video would be nauseating, but it wasn't. Just like normal 2D movies/videos, a little motion blur is necessary! No motion blur easily feels choppy even though the framerate may be high.
Video resolution
Just crank it up as high as possible really. Similar to normal 2D displays, higher resolution feels crisper but since the screen almost occupies 100% of the viewers field of view in VR, all the pixels "increases" in size leaving them very visible if the resolution hasn't increased as well.
Right now, the resolutions of the headsets may be the limiting factor though. In my tests, I used 4880 x 2440 pixels (2440 x 2440 pixels for each eye) and it was clearly too low. That being said though, another big problem was the resolution of my VR headset, which is 1440x1600. Keep in mind that there is a different between the resolution of the stereoscopic video that you can see through the headset and the resolution of the headset itself.
I can't really tell what resolution would be optimal since the headset I own doesn't support higher resolutions, however, I got a feeling we're not very close of having too many pixels right now.
Depth of field
This one is interesting. Since the 3D effect of stereoscopic video is only an illusion there actually is no depth that will result in a blurry foreground and background. That means, even the things you don't look at will always be in focus which makes the 3D effect significantly less believable. Just like normal 2D cameras we make sure our footage has an appropriate depth of field(similar to the human eye), however I would argue it is a lot more important in 3D since the illusion depends on it.
Camera tilting / velocity / acceleration / angular velocity
It may seem like there is a lot to think about here but luckily there is one rule that works for all of them. If what you see and what you feel doesn't align, some nausea may occur. Let me explain.
If the camera is tilting and is no longer 90° to the ground, the horizon will not be where our body expect it should be. Gravity helps us feel what is up and what is down and now that what we see don't align with that it can be a bit confusing. Moving the camera in a constant velocity is completely fine, just like sitting in a car. However, acceleration and angular acceleration can be problematic because that's something we would expect to feel in our bodies. The same actually occurs when rotating even though it is at a constant speed. The reason has to do with the physics of rotating a volume of mass, but you have probably felt it if you ever been spinning in a chair or something similar.
It's important to note that even though there is a risk with tilting/accelerating/rotating, doesn't mean it shouldn't be done. Just like a rollercoaster, what makes some people cause nausea makes other people excited and thrilled!
These rules are not absolute
I have tested all of these rules thoroughly, but they shouldn't be followed blindly. There are two reasons why:
The first reason is possible errors in my methods that could skew the result. For example, I've only used one pair of VR headsets and I only recorded with one pair of cameras. I haven't recorded and tested a large variety of types of footage(example: action vs dialog). I may also have a personal taste that affects what I find to be enjoyable to watch that others may not. These are only some examples of possible errors.
The second reason has to do with the filmmakers artistic choices. The director of a movie will probably tweak most of these rules so that they can tell the story they want to tell. For example even though zooming in or out may break the believability of the 3D effect, it may be done for comical reasons or perhaps the director chooses to creates elements behind the viewer to create an uneasy feeling in a horror movie, even though I usually wouldn't recommend to do that.
Finals words
This project can definitely be a bit confusing and it's a shame I can't show you what the result looks like over a two dimensional website like this but I hope you still found it interesting.
This is an idea I've been obsessed with for about a year now so I'm very happy to finally have dived into this project and explored the potential of the future of media and entertainment. That being said, it is worth mentioning that this is only the beginning. If this type of content becomes mainstream we will continue to learn and discover new ways of using VR and AR that we might miss now in the early days, but I do believe what I showed you today is one of the first bricks to the fundamentals of what one day might become something truly world-changing.
