Bestand wählen
Merken

# 4–D Capture and 6–D Display

Embed Code
DVD bestellen

#### Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
thanks everyone for being here so I would say
it's like a sort of apologizing on behalf of formation from his emissions that who wanted to be here but unfortunately is stripping about extended by day and something that tomorrow on hopefully that would hinder of those dogs and on and will have a moving you can get the same effect from it so all women talk about today is slightly different from what people think often don't of traditional of of the and the fundamental difference over here is that most of the stuff going on at the group and what I'm going to present today is going to involve is going to be based on geometric optics isn't all their object so it we got about a lot of raisins tolerates propagate in space and and not so since it's a little different from the way we think of holograms in general a traditional events and going to go
over some basic of fundamentals just to make sure everyone understands what I'm talking about when I talk about race so this is a sponsor provoked by the cannot function which basically describes the gradients of the intensity of the light at any point in space in any given direction so the final function because you can imagine in free space you have 3 dimensions for the the any position in space and then you have to dimensions for the direction in which you model described the intensity of light so it becomes a five dimensional function and it's called the plenoptic function of that if you consider the special case where we talking about free space l you can imagine if I fought a point in space over here and I have a going in a certain direction the intensity of the array of light does not change along that direction if there's no clue between those 2 points and this actually means that no 1 dimension of the 5 dimensions actually becomes redundant idea let's just a 4 dimensional function which of which is what we call the light field and the various ways of representing the 40 function but the most of our common 1 is this so far to be playing out of the representation of parametrization that we take 2 planes which are separated by some finite distance and you find a point on 1 plane you find a point on some other pain and join these 2 points and this defines 1 day flight that goes through the space between those 2 things if you take all the points on those planes you can get you can completely describe all rays of light that will between those 2 things so that's the basic idea of what a light field is and how and by this for damaged so if
you look at an object or scene of it you can you can consider all the light rays are is applied that coming out from that scene and since this is a 4 D light field this becomes a 4 dimensional function that you can at that you have coming out from a sea of you can also in this problem and you can consider the forward image of 4 degrees off copy them you have in the illumination that incoming on the scene and together this actually gives you an function which we call the
reflectance fear and this effect is to completely at ease in terms of geometric optics describes how scene Board of Appeal from any given that action on that in any given illumination conditions and if you can capture this complete emission reflectance fear you can re-create than of the objects in any way out on that any illumination conditions and that that actually want to go at all unfortunately did not quite
there yet and what they were presented to you what an report about in the stock is only a much smaller subset and then talking particular about a four-dimensional capture so not the complete 80 register for the capture and the
60 display and again this is all in terms of Jamaica expressed operates so you can obviously use what we have to do a lot of these things but I'm going to talk about some of the techniques that we come up so what
Alexis talk about what the camera that's in in very simple terms and what part of this for the incoming light here captures so if you consider a traditional camera you have a lens and then you have a sense of some distance behind it and how the way images of a point in the scene is that it focuses on the rays coming out of that point onto a single point on the sensor but unfortunately what's happening here is you are losing out on a lot of information that's in coming into the camera and in particular if you think of those who did not know what to do and parameterization that we spoke about our what's happening is losing all the information in 1 of those pain in particular the angular information is completely lost and all you get is the spatial information and this actually happens because all on the the laser pointer but if so on if if you look at all this and you that been collapse all the issues that are coming in with this kind of of actually indicated or summed up and you can you can never recover what the intensity of any of these individual arrays was that information is completely lost when you use a traditional camera so people have come up with these all getting on this problem and it's been Altman federal essentially now this concept of India photography value essentially put microlenses instead all a sensor in the plane that these things are coming together and now and that the stress propagate through and how the sensor at some distance behind if my commences and what this gives us is it gives us a small of macro image behind each of these Michael answers and each of these images is this is this idea that information that otherwise lost in a traditional camera and so by using this kind of a camera which contains of migrants as a right in front of us imaging sensor you can recover the complete 4 dimensional light field that is entering the camera so you can essentially capture all the geometric information that's it coming into the camera in terms of the light all
the product of this kind of a camera was recently built by from our scientists in Central and that actually then cytostatic company based in this and out the what they did was they took 1 of these migrants arrays and put this right in front of the of of order of a standard is the medium for my camera let this gives us something very interesting it's similar to some of the stuff you saw in the last hour except since so did not using for the ground and not using beam optics in this case of you don't need going like you only need cases you can work with importance of daylight and normal light
so 1 example that that they would get is just 1 single snapshot just by capturing 1 image and then just be on processing the data that they acquired from that 1 image that are able to really focus on different planes in that image so for example in the scene you have people standing at different depths and someone just single 1 capture they are able to recover this image and and this the this this and this so I'm just on for them again you can see it's focusing at different planes active at different depth from the camera it so this was done by displacing the my consent right next to the sensor in a traditional camera what I purposely trying to do is how can we do something similar but without using a migrants theory and what we call the technique of doing this kind of stuff on by using just about these kind masks so instead of placing a mark on microlens in front of the camera beat this 1 of these mosques in front of the sensor and you're able to get a similar results which is when a bit in the next so 1 of the
1st is that we did was this this what we call the coded aperture camera in which the open the lens of a camera and the output of a carefully designed must be similar to this 1 which is opaque at certain places and transparent at other places into the approach of the lens so that's the model that we use and you can see this portion that I like I want a constant and let the light go through and the part that a black holes of bacon like doesn't get through so essentially gets throwing away half the light that's entering the camera and it turns out that for another half the light in a very structured and very well-thought-out manner you can actually recover much more information out of the photographic that you capture so in particular that this up but diplomats that says do is it lot I if the scene
is perfectly in focus it actually does not change change anything about so
it the so the seamer perfectly focus it won't change anything at all and the like you have a point source so here an LED it appears the same it's just half as bright as order was 4 I work
at the scene is out of focus be the get these over the traditional camera you would get the use of a focus because it so part book a effect on coming from finite size of the aperture of the lens but because the places must inside the approaches of the circuit
of OK get this man structured book to actually an image of the mask that report in the lens butcher and it turns out by all working the point spread function from all this a circular
this to this where they that structure
of mask pattern we are able to use computational techniques to recover an in focus image from another focus image so for example here the camera arm
is focus instead of focusing on the face it focused on the bottom in the painting but because this picture was taken using this of Moscow and that in the portion of the camera we are actually able to recover an image as if it were focused on the face so you can see that Clinton eyes of and this is something that would be impossible using a traditional camera if you've been have this this if you open up this modification and counter so what
what we saw were here was we use this mask and a lens aperture and this give us more or extra information this is kind of similar to the previous talk where they were on looking at using holograms as an optical element what we are looking at is how can we use these mosques optical element so in some sense it's a similar a simpler problem than the 1 they're looking at but it turns out this gives quite a lot of all flexibility in terms of for program so the thing that we investigated next was what happens when we move the mosque at different planes between the lens and the sensor and in particular what happens when we place the Mask right next to the sensor so the book this mosque which I have you welcome to come and take a look like our and be listed in front all of in front of
the sensor of medium from I cannot the similar to the way our researchers at Stanford I placed the my consent right next to the image and so on what this gives us is
essentially a form of optical heterodyning but without getting into too much of that because of what's the math behind this which is very elegant it's a nice low frequencies of main analysis of how this works but it essentially takes under the information that was otherwise lost by a traditional camera and puts it into the higher frequency regions all the information that is a better sense dust capture and it sort of automation had heterodyning where the mosque acts as the carrier frequency just like your optical of theory of and communication and the sensor acts as tho there and the paradigm of the at that event signal is actually essential sense and then in competition the due at the of the heterodyning lobby modulation recovered the light you I yeah so this is a
for the captured by the modified can have at the sensor of hard this must be state next to it and if you zoom in and you can see from structured and all image as its capture and that's basically the shadow of the mosque as falling on the sensor and I must mention the mass that we use here was a somewhat side of items and out of various frequency of various harmonics
so the as give that so
what but if you look at the click comes from all the image that's capture it has this very special structure and it has these of information I different about I different no scale of spatial and angular frequencies and by simply are running the modulation algorithm on this be able to recover the complete 40 light field and what you see over here is just different views of that light field so by taking just 1 single photo we are able to recover what the scene would look like if the camera had moved slightly and not just a stay more that 1 place in this from just 1 4 on here
is another result there on the left is the image that the captured and on the right to stop issuing body focusing results vary focusing on different planes in the image and again this is from just 1 image and on the left and right here showing of the moon the viewpoints slightly within the lens of itself so you can have slight modifications in the viewpoint of the camera and you can recover what the scene would have image would have looked like if the camera was there and this is all physically correct that's not just L for the shop that that is doing is it's what might would have on on the camera I think was this about additional if it focus of the thing
so that was more from the catches sigh how we can capture the 4 D light field by simply making a very simple modification to a camera and then using some competition the next part of the talk is how we can actually process all we can sure and display this information to the user and uh the thing that we've been looking at is so the six-dimensional displays which you might be wondering what the 60 mean then I alluded to this right at the start of the dog but if you think of the 3 D display that changes as you move your viewpoint on the left or right of that would be a tree display if you if it also changes if you when you move your viewpoint up and down that becomes a 40 display but the other dimensions actually come from the direction of the illumination on the display so that the display change linear illumination on the displayed changes if you move flashlight on the display you see a different things and the 1st thing of before I get to the 60 I want to talk about a 40 display of that it's independent of the viewer position but depends only on the illumination position
so this of video here shows this display that we constructed
and I have some of these over here that you wrote from
the common developed later on and as a flashlight is moved behind it is that you
can see the shadows and cost it's that
formed by this display actually
change and this is off-lattice they just like there's there's nothing else in it but it's actually very similar to the traditional India for about it's just in of actually on inside of just on here is
another example of the same display long where it was placed
on wind up in and add the illumination changes during the day you can see a different view on based on where the light is coming in from our at that point in the day Ch
that so as I said the way this works is very similar to integral photography and we basically have Altsys all of the land because
there are a lenticular sheet on a microlens
arrays and then the POS this mosque in that within the land here bitch the masses again it encodes this for the information on to the sheet it's a very similar
master this except the same in color so that it encodes of in book on a different color going on in the directions 1 of the limitations of a here is you can never have any kind of
reflectance greater than 1 it always has to be less than
1 so the next step was
going to the 68 is that I mentioned earlier and now we want that this paper will react not only to changes in the illumination direction but also changes in the viewpoint unfortunately this is a harder than it appeared that 1st and the is that we discussed before i and v
ended up making justice 7 of small 7 months 7 display so it's 7 by 7 pixels display for those at each pixel over here that you see is actually about that that they can have an in an image of that so that's what
each individual pixel looks like it has sees all migrants it is and of so I think through the answers and then in the in between 2 migrants is over here we have this mask that's space actually encodes all the information it
so that's the optics is actually again similar to integral imaging is just a little many layers of the same thing and the so I'm going to show
you a simple these include them all what this of looks like when viewed from different directions and what we have not been awarded in this display was as illumination direction changes behind the display you get these different back so this is the x and y all the incoming light direction you get to see this display part and then our light is coming from the corresponding direction and based
on the viewing direction we change the color of the pattern that he sees as you move to the right CC image and the other part and as you move that you're going go by
and I'm hoping that you can see some of the the display is that in the middle as move from the left to the right of the color of the displacing and I think screw move closer
and what it showing here is the camera position and what color you should see and hear it showing the
light position behind the display and what happened
you should be saying so right now each pixel costs
about 20 or 30 dollars so you can imagine
how building a large display is not quite feasible yet but maybe another 10 years lost so now it's changing the light illumination direction and you can see the pattern is changing by responding to the like position as they understood the
I think that's that's 1 half of the talk and if there are any question of data now I of a lot of this is not my work and presenting others research here also so I would really like to thank them for the slides and all that work and just applied for a group it's
relatively new growth and what we are trying to do is build the cameras of the future try to come up with this if I try to being designed a found was that they had capture the visual world around us and not just be limited by a traditional camera that is a lens and a sense of the standard and on a date any questions you have you remotely few OK well we do have a few minutes user questions again would you please use the microphone is the somebody yeah thank you Linda little we'll 42 school with the demonstration you shared with light coming through the image of how large have you been able to make those images and that will have large you think you will be able to that that's this is what we have right now and this is as large as we've made them so far annotated and then how big of late how big my as you can get and what how much you want to of been on that and also depends on the printing resolution that you're going to use because that's how many so how much how much quantization level you're going to get in terms of and of but it's very very similar to just traditional integral imaging displays that change when you are moved around them except here via fixed-effects when you move around in the chain and the illumination changes and and so it's not very different from that I this i'm sick it do you get an accommodation and focus in the z direction but not with these displays no with nominal techniques or well you don't necessarily get them because so there are issues with the so there this research in I I'm aware of people trying and arm I forget some in Japan where they tried to do that that kind of thing the problem you end up having is you need to have a very high resolution in the angular dimension and it's not really easy to do that if you're far away because of a portion of the I is so small you it's it's hard to get that kind of accommodation there we have tried some similar experiments and on that and we can get what when it's about that but we can get some sense of accommodation but if you I have a and just using all Michael and that isn't indicators sheets it's very hard to get much accommodation the chair b b jets and so high on June and have you considered by using holographic elements to replace these sort of stacks in 60 display ah yes how we are definitely looking at that and that was a lot of what the previous talk was about using photographic elements as sort of just optical elements and that something you active you can the value of and that few
Objekt <Kategorie>
Soundverarbeitung
Hypermedia
Dualitätstheorie
Subtraktion
Ausbreitungsfunktion
Gruppenkeim
Dateiformat
Druckertreiber
Raum-Zeit
Ereignishorizont
Quick-Sort
Datensichtgerät
Ebene
Lineares Funktional
Fundamentalsatz der Algebra
Punkt
Ortsoperator
Hausdorff-Dimension
Globale Beleuchtung
Selbstrepräsentation
Raum-Zeit
Computeranimation
Richtung
Demoszene <Programmierung>
Objekt <Kategorie>
Freeware
Funktion <Mathematik>
Zustand
Plenoptische Funktion
Zustand
Abstand
Bildgebendes Verfahren
Parametrische Erregung
Soundverarbeitung
Motion Capturing
Vervollständigung <Mathematik>
Spiegelung <Mathematik>
Benutzerfreundlichkeit
Globale Beleuchtung
Gruppenoperation
Term
Computeranimation
Demoszene <Programmierung>
Objekt <Kategorie>
Motion Capturing
Teilmenge
Zustand
Konditionszahl
Technische Optik
Gammafunktion
Ebene
Motion Capturing
Punkt
Datensichtgerät
Ausbreitungsfunktion
Term
Computeranimation
Digitale Photographie
Datensichtgerät
Demoszene <Programmierung>
Integral
Rechter Winkel
Zustand
Plenoptische Funktion
Mereologie
Zustand
Vorlesung/Konferenz
Technische Optik
Information
Abstand
Normalspannung
Makrobefehl
Bildgebendes Verfahren
Array <Informatik>
Formale Grammatik
Resultante
Zentralisator
Ebene
Subtraktion
Bit
Ausnahmebehandlung
Biprodukt
Sommerzeit
Physikalische Theorie
Computeranimation
Anpassung <Mathematik>
Verdeckungsrechnung
Demoszene <Programmierung>
Motion Capturing
Digitalsignal
Schwebung
Rechter Winkel
Normalvektor
Ordnung <Mathematik>
Pixel
Bildgebendes Verfahren
Array <Informatik>
Standardabweichung
Demoszene <Programmierung>
Informationsmodellierung
Code
Mereologie
Ruhmasse
Computeranimation
Digitale Photographie
Funktion <Mathematik>
Punkt
Mathematisierung
Quellcode
Ordnung <Mathematik>
Fokalpunkt
Digitale Photographie
Soundverarbeitung
Lineares Funktional
Offene Menge
Punkt
Besprechung/Interview
Fokalpunkt
Digitale Photographie
Verdeckungsrechnung
Demoszene <Programmierung>
Code
Digitaltechnik
Mereologie
Bildgebendes Verfahren
Verkehrsinformation
Metropolitan area network
Offene Menge
Code
Minimum
Mustersprache
Besprechung/Interview
Datenstruktur
Fokalpunkt
Bildgebendes Verfahren
Digitale Photographie
Verdeckungsrechnung
Ebene
Subtraktion
Bildauflösung
Element <Mathematik>
Term
Technische Optik
Computeranimation
Digitale Photographie
Verdeckungsrechnung
Digitalsignal
Rechter Winkel
Code
Hill-Differentialgleichung
Information
Optimierung
Bildgebendes Verfahren
Telekommunikation
Harmonische Analyse
Decodierung
Mathematisierung
Technische Optik
Physikalische Theorie
Computeranimation
Abschattung
Bildgebendes Verfahren
Analysis
Video Genie
Ruhmasse
Frequenz
Quick-Sort
Ereignishorizont
Dialekt
Sturmsche Kette
Digitale Photographie
Motion Capturing
Software
Information
Technische Optik
Aggregatzustand
Größenordnung
Zentrische Streckung
Domain <Netzwerk>
Sichtenkonzept
Information
Frequenz
Computeranimation
Digitale Photographie
Demoszene <Programmierung>
Algorithmus
Digitale Photographie
Plenoptische Funktion
Information
Schnelle Fourier-Transformation
Datenstruktur
Bildgebendes Verfahren
Ebene
Resultante
Ortsoperator
Datensichtgerät
Hausdorff-Dimension
Mathematisierung
Globale Beleuchtung
Besprechung/Interview
Viewer
Computeranimation
Datensichtgerät
Digitale Photographie
Linearisierung
Richtung
Demoszene <Programmierung>
Netzwerktopologie
Rechter Winkel
Plenoptische Funktion
Mereologie
Information
Bildgebendes Verfahren
Datensichtgerät
Abschattung
Vorlesung/Konferenz
Videokonferenz
Datensichtgerät
Mathematisierung
Sichtenkonzept
Punkt
Globale Beleuchtung
Mathematisierung
Technische Optik
Computeranimation
Datensichtgerät
Inverser Limes
Ruhmasse
Technische Optik
Kantenfärbung
Information
Computeranimation
Gammafunktion
Datensichtgerät
Richtung
Array <Informatik>
Globale Beleuchtung
Mathematisierung
Technische Optik
Extrempunkt
Computeranimation
Datensichtgerät
Richtung
Pixel
Datensichtgerät
Information
p-Block
Raum-Zeit
Bildgebendes Verfahren
Prototyping
Datensichtgerät
Verdeckungsrechnung
Mustersprache
Datenstruktur
Datensichtgerät
Mathematisierung
Globale Beleuchtung
Mereologie
Technische Optik
Technische Optik
Bildgebendes Verfahren
Computeranimation
Datensichtgerät
Richtung
Mustersprache
Displacement Mapping
Datenstruktur
Rechter Winkel
Datensichtgerät
Mustersprache
Besprechung/Interview
Kantenfärbung
Bildgebendes Verfahren
Datensichtgerät
Richtung
Pixel
Ortsoperator
Verschlingung
Datensichtgerät
Besprechung/Interview
Kantenfärbung
Datensichtgerät
Rechenschieber
Ortsoperator
Datensichtgerät
Mustersprache
Globale Beleuchtung
Rechenschieber
Gruppenkeim
Computeranimation
Datensichtgerät
Richtung
Hausdorff-Dimension
Datensichtgerät
Globale Beleuchtung
Mathematisierung
Hochdruck
Keller <Informatik>
Element <Mathematik>
Information
Technische Optik
Richtung
Übergang
Hypermedia
Geometrische Quantisierung
Digitale Photographie
Indexberechnung
Bildgebendes Verfahren
Bildauflösung
Motion Capturing
Systemplattform
Fokalpunkt
Visuelles System
Quick-Sort
Integral
Motion Capturing
Verkettung <Informatik>