The previous blog posts already show our weekly progress well, so here’s some additions to this – like an evidence we have a full shotlist now!
This weekly I spoiled the fun a bit by ringing the alarm bells; if we want 9 minutes of film to be done in 10 weeks, we need to start delivering finals now… and not only a few shots, but more like 10-15 shots per week, 50 seconds every week! The two main bottlenecks to overcome urgently;
– Mastering Cycles renders for big environments (too slow, needs optimizing or baking or just use more background mattes & compo trix)
– Cleaning footage (chroma key, mask out issues)
We’re already putting the brains of several developers together on better keying algorithms. If you like to check on the issue, get the footage from the post below, and check on the blurred stuff especially (like from hair of people in back). How to get this keyed best? Or is just good old handwork with feather masks and painting skills?
/production/first-original-footage-frames-in-4k/
Examples of – quite good already – results from Sergey you can find as the bottom two pictures in the gallery above. Sergey continues working on it, especially to explore ways to smoothen edges in keymattes with keeping good alpha-over properties. I think we need simply something like a “blur inside” node… we’ll see!
-Ton-
Would love to see a second or two for fully keyed frames with a background to see what the results are so far.
Stills look more composed than when things are in motion and moving faster than the mind can pull apart a frame.
That’s a good point, but hard to define how much you can trick people… our trained eyes should accept it a bit, at least.
I witnessed those bells being rung. :P I forsee some late nights around the corner!
I hope you quickly find solutions. It’d be a huge shame to have to drop Cycles from production.
With regards to cleaning up shots by hand; I suppose it may prove a little faster, depending on what needs to be done. Can’t expect a computer to do it all, unfortunately!
All in all, everyone’s done an amazing job with improving Cycles, keying, and the film itself! So, again, it’d be a shame if thnigs have to be dropped, or cut short.
My best wishes to you all during this hurdle!
– Cleaning footage (chrome key, mask out issues)-(seen on BA):
http://inf.ufrgs.br/~eslgastal/SharedMatting/
– Mastering Cycles renders for big environments
http://sglab.kaist.ac.kr/CORR/
I go and in the link i found this page : http://www.alphamatting.com/code.php .
The first link generate this http://www.pasteall.org/pic/32959 from this http://www.pasteall.org/pic/32960 . You may have a look.
So did you download the code and in which software did you compiled/implemented it?
They are written in Mathlab, as the *.m extension says, so it’s pretty easy to execute and there is a powerful image api in Mathlab but you “muss pay ?” for it.
I guess they dont have mathlab at bf that why i test it.
The code does not seem to rely on any fancy Matlab packagaes, so it should be trivially portable to Octave.
-> Or in Scilab : work realy pretty well. A nice combinaison of all that could help a lot
The third link is also a mathlab file : what is intersting is that it take profit (=learn) using a mask file like that http://www.pasteall.org/pic/32968 and produce that http://www.pasteall.org/pic/32969
That’s insane. I can’t understand why they don’t give it a try…
hm.. there a to be some DoF in the CG-Part
Ton, take a look to how primatte works:
http://en.wikipedia.org/wiki/Primatte_chromakey_technology
Its me or someone suppressed the content of this pages today : last time i check there was a full explanation but now all is gone.
One don’t like mango and suppress the interesting parts. Old links before todays modif : http://en.wikipedia.org/w/index.php?title=Primatte_chromakey_technology&oldid=496775754
Damn a conspiracy
What else could it be? But the one who didn’t like Mango also doesn’t like knowledge. Why would someone delete information like that?
Saved to my hard disk. But I already understood fully the algorithm and I could rewrite how it works. There is really silly people trying to control the Earth these days…
Seems that the article has been edited by a slightly privileged user because it violated Wikipedia’s rules. It was way too long and was more of a user’s manual than an article in encyclopedia.
I can agree with that. On the other hand, now the shortened article is pretty useless. So if you people do understand the algorithm (not user interface) well, it would be very kind of you to rewrite it briefly in a pair of paragraphs. And the original article could be moved to Wikibooks.
I’m sorry for being off-topic, but I just didn’t like this conspiracy theory.
But why the day bao says it two hours after ? If it was an other day ok but …
I dont know its perhaps a coincidence.
Well, we both see the same logs. But as I understand it, there first was somebody from 98.169.89.217 who just added a {{Howto}} template, meaning the article was badly written. He did so an hour after Bao’s post, so most probably after reading it. Then, a user called Asav noticed (in my opinion) this template and made the easiest fix to it, namely deleted most of the text.
It’s time for Cycles to get bucket rendering implemented! ;)
Joking aside, facing these problems will get Blender better and better, that’s Mango’s goal so…keep rockin’.
Well you do have Brecht mentioning that he is looking into smarter sampling methods and performance improvements for Cycles, so maybe that will help to an extent.
Perhaps it will help when Cycles starts getting optimizations noise-aware sampling, SIMD, QBVH trees, importance sampling, light priorities, and enhanced solutions for reducing noise in Cycles like changing the glossy blur and clamping value on a per-material basis or even using a map that may or may not be automatically generated.
The spot where I wrote SIMD is supposed to be ‘SIMD instructions’, in case anyone gets confused when seeing that.
Or subdivision surface that adapt the level based to the distance to the camera. Texture can be also shrieked if they are to far away. So a pre pass bake (or distil the scene to improve mesh and texture before cycles render it)
Have you guys tried the cmiVFX tutorial for greenscreens?
http://www.youtube.com/watch?v=0Wyta-hkCmY
Nevermind. Just find out that the tutorial is made by one of you Mango guys….sorry! :p
I think our keying tools are pretty capable already. I’d stop thinking about a silver-bullet keying node and try to improve other tools needed to create good mattes.
Masks are essential to define regions. Looking at the samples you shared so far it looks like you tried to pull a single matte from plates, and that’s pretty useless because a good key for hair might not work for out-of-focus backgrounds, transparencies, etc. (maybe I’m wrong, I didn’t see any screenshot of the nodetrees involved)
I think that just tweaking masks and keying nodes to provide a straight-forward way to combine mattes from regions would solve most of your keying problems.
An extra problem is noise. I tried to pull mattes from your sample images, and I couldn’t get decent results without applying some denoising.
Question is: is that noise part of the source material or is it exaggerated by an incorrect colorspace transform? I’m not sure about the answer, but I wouldn’t expect so much noise on a greenscreen shot by a capable DP with a kickass camera.
Anyway, I wouldn’t stuff a denoise option in the keying nodes. Having specialized denoise nodes would be much better than a simple pre-blur (I’m thinking about wavelet denoising on separated channels, that would be awesome).
IMO adressing those problems you’d be good to pull good mattes from your material.
Oh, and a lightwrap node. Lightwrapping is already doable using existing nodes, but it would be very handy to have it in a single node.
I agree the a denoiser node and a lightwrap node can be really useful.
I also thing that one can improve the edge detection could be improved and used as driver for mask :
http://en.wikipedia.org/wiki/Edge_detection
http://en.wikipedia.org/wiki/Deriche_edge_detector
http://users.polytech.unice.fr/~lingrand/Ens/up/Lesson7and8-segmentation.pdf
http://www.m-hikari.com/ams/ams-password-2008/ams-password29-32-2008/nadernejadAMS29-32-2008.pdf
Sobel and prewitt dont generate a 1 pixel continuous line so an other edge detector can be implemented with a vector output in the node. Then in the outputs nodes in compositor add a vector outpost node where one can choose the mask this edge vector would affect and so in generated a mask that feet the nearest edge.
Gez: we rely on masking a lot, a single key is a combination of different approaches, mixed together with masks. That doesn’t take away the need for a tool that can extract quality mattes to start with.
Ah, good to know. I’d love to see a screenshot of those combined mattes in the compositor context.
Here goes my keying attempt on the image above:
.blend:
https://www.box.com/s/104edb022a1dc440cfdf
Screenshot:
http://img221.imageshack.us/img221/4818/rogperkeying.jpg
At least for just this image… it’s not easy but it can be done :)
Check our “Greenscreen Nightmare” on YouTube, this has helped me with a few really bad keys. I almost always do manual tweaking on my mattes. I had to crunch the red channel once to fix a watermelon helmet someone had on their head in front of a greenscreen.
Sorry, “check out”, not “check our”. It isn’t from me. It is from BlendedPlanet.
Hey, I’ve posted a Keying method (a few hours back) that perhaps gave a better result then the one you’ve shown. It went to the spam, when you can please give it a look and let me know how is looking, my monitor isn’t no
So Cycles is not ready for primetime.
Dont desperate it will surely be ready :)
Well, isn’t keying/roto/mattes more of a compositor/tracker issue rather than Cycles?
Besides, Cycles is still being developed. Have a look at what it does so far and if it covers what you need it to, great, use it. If not, then don’t. “Primetime” is a bit vague…
I tried using a typical three-pass key on the example above (using A003C014_120507_00500.png sRGB image from the 2K release). I got a little better results than the example in this post. You will notice the glasses and hair aren’t as crunchy and some of the facial hair shows up more. The top left corner just needs a small garbage mask.
Here is the composite (https://skydrive.live.com/redir?resid=C0FF37B8A3617BDE!122) and the alpha (https://skydrive.live.com/redir?resid=C0FF37B8A3617BDE!121). The blend file is at http://www.pasteall.org/blend/14401.
The node tree is more complex with the three-pass method, but it would be quick to use. You just need to add these nodes to a shot and tweak the three key nodes (maybe some of the other nodes for slight adjustments).
I used Blender 2.63 revision 47635 to set this up.
May I ask how long do you plan the final film to be? 3 min? 5 min? 10min?
Just wondering.
Personally, I think it is MUCH better to deliver a shorter film at higher quality. Quality over quantity any time :)
DavidT in our Mango thread in blenderartists posted what primatte is able to do. Look to the glasses!
https://skydrive.live.com/?cid=c0ff37b8a3617bde&resid=C0FF37B8A3617BDE!118&id=C0FF37B8A3617BDE!118&sff=1
https://skydrive.live.com/?cid=c0ff37b8a3617bde&resid=C0FF37B8A3617BDE!117&id=C0FF37B8A3617BDE!117&sff=1
DavidT did also a try with blender:
blend file:
http://www.pasteall.org/blend/14401
image:
https://skydrive.live.com/redir?resid=C0FF37B8A3617BDE!122
You often need to shoot green screen components at a high frame rate, which makes for a strobe like fore ground element. Mostly terrible looking.
I wonder if motion estimation, derived from the tracker (planar?), could impart information for a vector blur node? This would smooth movement and make the key edge more natural, if pulling a transparent edge is to hard.
Here’s an example of a blur for alpha mattes; which should “shrink inside”:
Original alpha:
http://www.blender.org/bf/blur-inside1.png
Blur-inside with size 30 pixels:
http://www.blender.org/bf/blur-inside2.png
Used as alpha-over:
http://www.blender.org/bf/blur-inside3.png
Obviously it should be masked to only do left side of picture, but it’s a test to show how alpha blur should behave.
This is simply a couple of ops:
– blur the alpha
– subtract 0.9
– clip with 0
– multiply with 10
– blur again
guess there’s some source code for inspiration there:
http://www.youtube.com/watch?v=kkQ1r5g49d4
It relly looks like this learn-matte technics : i post links for the code above but it’s this one http://www.mathworks.com/matlabcentral/fileexchange/31412
Once you download there is a pdf with explanatories. I guess gimp take this because its said to be the accurate keying tools. And what is cool is that can extract backgrond even if it’s a completely non uniform.
See /development/weekly-june-8/#comment-6333
A mix with the adaptive despill that bao shows above will make something perfect.
Indeed cycle issue is more difficult to handle : i dont knows if that can help.
http://www.cs.cornell.edu/~kb/publications/HVEI12InvitedAbstract.pdf
http://www.cs.cornell.edu/~kb/publications/EGRW02.pdf
http://www.cs.cornell.edu/~kb/publications/TOG99.pdf
http://www.cs.cornell.edu/~kb/publications/SIG12BidirLC.pdf
http://www.cs.cornell.edu/projects/manifolds-sg12/
http://www.cs.unc.edu/cms/publications/dissertations/lauterbach.pdf
I fee. that this would be perfect; could you use the GIMP code provided in Blender?
A good ‘non-alpha’ trick to get hair detail for a key can be rather simple.
First you check the different RGB channels to find out which channel gives you the best detail. If it’s blond hair usually the green channel is the best. If it’s dark hair the Blue channel tends to work well. But going to the green channel is usually the best bet for dark and light hair.
Then you isolate the ‘green’ or ‘blue’ or ‘red’ channel and follow it with a gain and/or lift node. If you want blond hair you adjust the gain until the greenscreen is nearly all white. This will require adjustment because you can blow out the channel and lose detail.
One you have have the ‘blond’ detail that you like add a garbage mask around the hair detail and then add a grade node and adjust the white point to the desired color for the hair. Then screen or add it via a comp to the background.
Then comp the primary key over the ‘blond’ detail comp. Adjust the ‘blond’ comp (via transparency) to taste.
If you want dark hair adjust the lift until you can clearly separate the dark hair from the background. Add a garbage mask around the hair detail you want. Then multiply it over the background via a comp.
Then comp the primary key over the ‘dark’ detail comp. Adjust the ‘dark’ comp to taste.
These non alpha tricks can be very helpful and will give you professional looking composites.
Check out this video from FXGUIDE about comping and keying in Nuke and they go over similar principles.
Non-alpha tricks are very useful and combined with good keying can really make the difference. I feel Blender has a good tool set to allow you to get professional keys. Just with a little bit of now how it can be done.
Also, feathering can be a double edge sword and make your edges look like Vaseline. Be careful.
Keep up the exciting work!
Ira.
Here is the link I forgot to add to my post
http://www.tinyurl.com/nuketricks
Thanks Ira that was very informative, I had not considered reconstruction transparent elements via screen or mult.
What chance Mango use UV pass in compositor and add a cycles renders there?
for this music video I used in an intensive keying in Blender
http://www.luceri.it/2012/05/blender-vfx-in-videoclip/
here, where you can download the blend with my basic settings and with advanced color bleed
I hope can be useful
Wow gillan that is a pretty good key on difficult hair. I will be interested to look at the .blend too.
I’m sure you can apply for helping them. Your movie clip looks cool btw and I like the compositing you make for mango. Just imagine your competence with greater tools than today.
If you got time enough.
The best chroma keys are done on layers, using localized settings for different areas of the images and using masks to layer them.
I can recommend bayesian matting, too. There are better algorithms now, but for chroma-keying it has good quality. It’s not very difficult to program and quite efficient for large images.
This paper can also be useful:
http://research.microsoft.com/apps/pubs/default.aspx?id=73833