IOT in Care
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 17 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/50516 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
5
11
12
17
00:00
Software developerEvent horizonProjective planeGoodness of fitInternet der DingeAreaProcess (computing)QuicksortElectronic program guideIBM PCPlastikkarteRight angleDegree (graph theory)NumberOnline helpMachine learningPoint cloudMainframe computerVideo gameString (computer science)SpeichermodellAnalytic setMoment (mathematics)Public key certificateReal numberFirmwareEnterprise architectureMereologyType theorySelf-organizationLevel (video gaming)Computer hardwareDifferent (Kate Ryan album)CASE <Informatik>BitMusical ensembleComputer scienceLine (geometry)Product (business)Term (mathematics)Virtual realityWorkstation <Musikinstrument>Position operatorCartesian coordinate systemDrop (liquid)Office suiteEccentricity (mathematics)Covering spaceView (database)Virtual machineFitness functionPerturbation theoryMeeting/Interview
06:32
Event horizonSoftware developerDifferent (Kate Ryan album)Multiplication signFitness functionElectronic program guideCoefficient of determinationQuicksortNumberProjective planeRoutingPredictabilityPoint (geometry)Instance (computer science)Moment (mathematics)MereologyVideo gameTerm (mathematics)AreaStress (mechanics)Mathematical analysisVirtual machineView (database)InformationComputer engineeringOrder (biology)Connected spaceBuildingPosition operatorLevel (video gaming)Insertion lossKnotSystem callPhysical lawMachine learningEccentricity (mathematics)
12:57
ArchitectureSystem programmingSoftware developerEvent horizonRight angleBitLink (knot theory)Electronic program guideTouchscreenHeegaard splittingComplex (psychology)MultiplicationQuicksortPredictabilityJunction (traffic)CASE <Informatik>Endliche ModelltheorieProcess (computing)Coefficient of determinationTerm (mathematics)SummierbarkeitGroup actionDecision theoryMachine learningAlgorithmComputer programmingAnalytic setFormal languageVirtual machineInstance (computer science)SatellitePhysical systemArtificial neural networkEvent horizon1 (number)Software testingDrop (liquid)Drag (physics)Product (business)AreaData analysisGoodness of fitSpacetimeSet (mathematics)Different (Kate Ryan album)Moment (mathematics)Order (biology)Time travelNumberMachine visionMetropolitan area networkFront and back endsPoint cloudBit rateTrailComputer architectureMereologyData structureWell-formed formulaCuboidMultiplication signType theoryFitness functionPoint (geometry)Projective planeSlide ruleCartesian coordinate systemLibrary (computing)Standard deviationDebuggerParameter (computer programming)LengthVideoconferencingVideo projectorWeb 2.0Program flowchart
19:26
ArchitectureEvent horizonSoftware developerArmCoprocessorRight angleTransport Layer SecurityPhysical systemDifferent (Kate Ryan album)Speech synthesisCartesian coordinate systemComplex (psychology)Virtual machineAreaPoint cloudGame controllerCuboidTelecommunicationPlastikkarteQuicksortAndroid (robot)SpacetimeRevision controlLevel (video gaming)Software developerLibrary (computing)Service (economics).NET FrameworkOpen sourceDebuggerRemote administrationHypermediaStreaming mediaMulti-core processorConnectivity (graph theory)Moment (mathematics)Set (mathematics)Computer hardwareCASE <Informatik>Electronic program guideDynamical systemCoefficient of determinationFunctional (mathematics)Data managementSemiconductor memoryDegrees of freedom (physics and chemistry)Multiplication signFamilyBuildingSensitivity analysisPoint (geometry)SoftwareOperating systemCodeComplete metric spaceLine (geometry)Process (computing)Pattern languageStandard deviationBitAlpha (investment)TrailGraphics tabletWaveBeta functionEndliche ModelltheorieThermal conductivityPlanningFrequencyNeuroinformatikGame theoryLaptopTopological vector spaceVideoconferencingThree-dimensional spaceFront and back endsInstance (computer science)Disk read-and-write headVirtual realityFrequency responseTerm (mathematics)RoutingComputer programmingMobile appPower (physics)Pattern recognitionStack (abstract data type)InformationMappingMusical ensembleOrientation (vector space)PreprocessorHeat transferMachine visionBus (computing)SmartphoneDigitizingBlock (periodic table)Device driverConnected spaceSingle-precision floating-point formatLatent heatWave packetClient (computing)Row (database)Boom (sailing)Data storage deviceArithmetic meanUniform resource locatorThomas BayesOrder (biology)WindowTriangulation (psychology)WebcamDistanceAngleNoise (electronics)Video gameDependent and independent variablesFacebookPrinciple of maximum entropyTwitterCompilation albumClassical physicsSampling (statistics)Touchscreenoutput1 (number)Real numberGezeitenkraftReflection (mathematics)Software testingProduct (business)Goodness of fitSeries (mathematics)Network topologyHazard (2005 film)Branch (computer science)Integrated development environmentCompass (drafting)Source codeNP-hardGreatest elementJava appletProjective planeSlide ruleAlgorithmFormal languageStudent's t-testObject-oriented programmingStructural loadDegree (graph theory)Field (computer science)Covering spaceValue-added networkTranslation (relic)Message passingOperator (mathematics)Sound effectBit rateDirection (geometry)Event horizonComputer fileFraction (mathematics)Annihilator (ring theory)Dimensional analysisVery-high-bit-rate digital subscriber lineKey (cryptography)MathematicsClosed setExecution unit40 (number)Shared memoryIdentity managementTouch typingSelf-organizationMassCopyright infringementMatching (graph theory)Stress (mechanics)DatabaseServer (computing)Division (mathematics)SpeciesForm (programming)Group actionSheaf (mathematics)SatelliteBoss CorporationSinc functionSocial classFingerprint
Transcript: English(auto-generated)
00:06
Good morning ladies and gentlemen. Welcome to this talk which is around IoT in care and healthcare and care in general. Let me introduce myself. I'm Michael Platt. I work for Microsoft. My title is
00:21
Director of Strategic Projects and I actually work out of Microsoft Corporation in Redmond. I've been part of Microsoft Corporation for about 10, Microsoft Corporation in in the US for the last 15 years and then I was in the Microsoft UK for about five or eight years before that. I've been with Microsoft 23 years. I was with IBM for 20 years before that as well. I'm an engineer.
00:47
In fact when I started they didn't have computer science degrees so I had to do an engineering degree and so my background is engineering and I spent as I say 20 years with IBM doing hardware design starting off with chip design working up through cards through to attached
01:04
processes for mainframes. In fact designed some of the original IBM PC so I've done a lot of hardware side of things. That was a while ago. For the last as I say 20 years I've been working with with Microsoft and I'm now in this really good position because I can pretty much
01:24
do more or less what I like in terms of project areas that I can work into. Once you've been around in a company for long enough and you get quite a lot of latitude as to what you can do and what I've been working on for the last two or three years has been in the area of care and healthcare with a number of different organizations and we partner up with
01:45
with people who have specialist knowledge in an area of disability or of care or so and so forth and we actually then go and develop solutions with them and when I say solutions I mean everything from right down at the the hardware level doing case design chip design right up through
02:03
firmware through how the applications that may be running on the phone or on a node of some sort into the cloud and then back out into machine learning. So I'm going to cover the whole of the the whole of the string from machine learning and business and analytics on one side of things right down into the actual hardware design itself in today's session. I'm going to talk about
02:26
that in a number of different product areas that we're working on and in a number of different areas of healthcare that I'm going to be talking about. So first of all let's talk a little bit about IoT itself. When people think of IoT they think of this machinery manufacturing lines
02:42
and so on and so forth that's the conventional the old IoT and there's a lot of money in that. I mean if you want to to build a business as an enterprise organization there's a lot of very large companies building these sort of IoT solutions and making a lot of money out of doing so. The latest thing of course is IoT in the home that's what we've been seeing for the
03:01
last five or ten years it's you know the nest the automatic devices in the home the cameras etc etc in the home and this is the Microsoft HoloLens. I'm working quite heavily with the HoloLens people so I sort of know quite well how that works. We have a lot of the same problems. One of the interesting things about working in healthcare is that a lot of the
03:23
problems that we have to overcome for disabled people for care people who need care are actually some of the same sort of problems that they have to solve for things like virtual reality holographic reality that sort of that side of things as well so and I'll cover those as well in more detail so but actually I'm more interested not in the home and I'm not interested
03:44
in industry I'm interested in people. That's really how we can help people with technologies is what's interesting to me and these are all projects that I have going at the moment so we start off down on the right hand side the epilepsy there's 400 people die a year
04:02
during epileptic seizures. How can we do a better job with technology of saying when those seizures are going to occur? How can we forecast those using things like the Microsoft band this is actually a Microsoft band it's a motion heart sensor Fitbit type device and we actually have a project going where we pick the data up from this and we can then actually say when somebody's
04:25
had a seizure and then following going forward from that we're hoping to be able to predict when seizures can occur so we give that person a warning you know do not go on the train station for the next five minutes because you're likely to have a seizure yeah and you and if you're on
04:40
the train station it's not a good place for that to happen right so you know that's that's the the one of the areas I'm working on and and you can see very quickly how technology can help these people dramatically they can it can literally save lives we I personally avoid the real health care side of things providing medical type support simply because especially
05:02
in the US which is obviously the biggest market you have to go through FDA approval and FDA approval is a nightmare so if you're thinking about trying to do real medical devices then you have to be aware of of the the certification issues that are associated with that and I can I can give you we are actually doing a few projects which are going through FDA
05:22
certification at the moment if anybody wants to know about that I can tell you about the legalities but I don't want to cover that in this session it's meant to be a technical session right over up on the top right hand corner there is probably the one I'm going to be talking most about today which is which is the guide dogs so that's how do we help blind people how
05:40
can we help blind people to be able to find their way around and to navigate and so on and so forth how can we actually make them actually able to have a much fuller life one of the when I went into this particular one I thought okay what we're trying to do is we're trying to show a blind person how to get from this hotel to this train station or something like that
06:04
right actually that's very useful for them but actually what you don't realize is the the biggest problem they have is just understanding where they are and the fear that's associated with that so I took a blind person to back to his office and I said let me drop drop you
06:22
outside the office and he said no because if you do that I won't know where I am if you drop me off at the station it's I know exactly how to get from the station to my office right but if you just put me outside I won't know where I am relative to the office and so I can't find my way so it's surprising just just simple things like that thinking about things from from a blind
06:43
person's point of view you know where am I and what can we do to make that person more secure in their life to be able to do things in their life they've never done before the vast majority of blind people actually really don't go out that much simply because it is so worrying they have to memorize all the routes they go on they take a dog which is great for actually seeing
07:04
which part of the pavement you should be on and where you're walking and for even crossing a road but it won't the dog won't know where the shop is right so there's there's a lot of interesting issues when you get in the shop how do you know where the thing is that you want in the shop so there's a lot of a lot of interesting issues about how you do navigation
07:21
for blind people another project up on the top left hand side there is for for special needs children so these are children with adhd with learning disabilities with dyslexia those sort of things there's a lot of work around how we can provide therapy to those children by through exercises through targeted learning etc etc but in order to do that we need to know what they're
07:46
doing and how they're what is actually going on with them and again we're doing that in a number of different ways first of all we monitor their movements you can tell a lot by taking the movements and doing a machine analysis machine learning on the movements and even more
08:02
specifically with learning disabilities it's coupled very closely to their eye movement and you can determine how much what they're concentrating on and when they're not concentrating uh based on their eye movements but detecting eye movements is actually quite a difficult thing to do and actually we're building uh some glasses i'll show you these later on some
08:22
glasses which actually detect eye movement you can actually select you can find what the eye movements are what your eyes are doing by by detecting the voltage across your nose interestingly enough so we're doing some work on that how do we uh how do we take their eye movements how take their physical movements using things like uh xbox connect which is very good for
08:42
determining where their limbs are and how fast they're moving their limbs and use that information uh in a machine learning analysis to say this is the right time for them to learn this is the wrong time for them to learn and this is how we best teach them today we teach people by sending them information it doesn't really work with people who have learning disabilities you have to understand when they can learn rather than when we want to teach them
09:04
so again we have a project going with a with a company that that's providing the the the teaching support and we're providing the technical support the one in the middle there is injury so i've done a lot of work in injury prediction this is actually mainly for uh for sportsmen so um it's not so much care but it is in this area so i work with
09:25
realm adrid um the the the uh football club in in spain i'm not a football person actually i'm not a sports person so but so i've worked a lot with realm adrid because they're interested for instance in how you can train somebody to the maximum level so that they don't so that
09:43
they just become perfectly physically fit but you don't go further and injure them so they want to to actually predict when somebody's going to get injured and again by putting uh specialist motion sensors by televising their movements by monitoring what they're doing we're able to actually say this person is going to get injured at this point in time and in fact
10:04
actually we're pretty good last season at determining that realm adrid we're gonna we're going to lose simply because of the fitness of their of their players which we could forecast and then finally the one that that uh that's probably of most interest to me although the guide dogs takes most of my time is actually care in the community so this is old people
10:24
and specifically alzheimer's and dementias patients people have medical uh mental issues and mental problems how can we first of all make their lives much easier they get very isolated this is common with most disabilities actually so how do we improve their connectivity
10:41
how do we improve their their their ability to talk to people and to to work with people using technologies um like skype for instance so you can skype in and and they can skype you on your phone and so when they're when they're lost again very common thing for a dementia patient in early stages go outside they get lost they don't know where they are the ability
11:02
to be able to pick up a phone or something and look at at and see a familiar face is incredibly use incredibly valuable to them and then indeed taking that forward into again monitoring their movements so we know if they're getting enough exercise if they're not they tend to sit in one position all the time which is very bad so we can work out what
11:22
what needs to be done in terms of palliative health care and then moving forward from that actually again there's a lot of evidence that suggests that there are certain treatments that there are certain things that we can provide to uh to dementia patients and outside early Alzheimer's patients which will prevent well not prevent but it will slow down the onset of
11:44
the disease by by exercising their mind and and in particular we're working in the areas of what's called remembrance therapy which is shown to be very helpful for the dementia it's also shown to be good for general depression and also by also in terms of
12:02
things like PTSD trauma traumatic stress syndrome all those things can be supported by this this whole area called remembrance therapy and again we can provide the the uh the support for that by by streaming information to those people into into the right format so that they
12:21
can handle it and they can use it so those are those are the projects i'm actually working on at the moment i have various numbers on different projects um i'm the lead on on the guide dogs project and the architect and also the hardware engineer but and i'm lead on most of these in fact i've got like 15 people on the guide dogs and then there's uh five of us
12:40
on each of the others as well so a large number of people working on this stuff and it's very interesting because we learn a lot we learn a lot about what sensors can do what sensors are needed and how that works in providing both a better lifestyle and also treatment to people that need it so there are a lot of problems you wouldn't believe the number
13:05
of problems there are uh in in these areas when you actually try and get down and it's actually interesting because really it looks very simple but actually the devil's in the details in many cases so for instance um walking down the pavement it's easy you think okay we can navigate
13:20
you can go my phone will do gps you know what it navigates you to the middle of the junction right that's not very good if you're a blind person you do not want to be in the middle of the junction you want to be on the pavement on the side to actually find the right pavement on the side of a junction is really really difficult just think of the complexities you don't necessarily have a right angle jungle a junction you may have a roundabout with multiple
13:43
junctions at different points how do you can tell that person where they which which pavement they should be on and where they get to so there's a ton of stuff we have to look at so this is this is the architecture slide right um this is so so this is one of microsoft's architecture slides you know so we have some devices iot devices at one end and we have
14:03
some back-end systems at the other and then we have we have all this wonderful stuff in the cloud that you can go and buy from us right um you know so we have analytics we have event hubs to connect things we have web jobs this is all part of a thing called azure you can you can go and buy that and uh that's the end of my sales spiel and basically it's
14:21
about storing manipulating the data and uh you know it is stuff that you need without a doubt but to me it's it's pretty much the infrastructure the things that interest me are the sensors on the front end and the back end systems and on the back end so i'm going to very briefly talk about the back end and then i'm going to focus in on the on the front end on the sensors
14:41
so in the back end nowadays it's all about machine learning has anybody seen this before this is actually a thing called azure machine learning and it's a way of building uh a complete oh that's sorry about the quality there something's gone wrong with the sort of on my screen i'm sorry the projectors but basically this each one of these is in a little box you know and they connect up um and what you do is you actually drag and drop a machine
15:04
learning type system so what machine learning is about is basically taking a large amount of data running it through a set of algorithms changing the parameters of those algorithms and seeing which algorithms gives you a best fit against a known good piece of data right so
15:22
what you have to do is you bring in some data at the top you scrub out the bad ones because most of the time in machine learning in data analysis in general is cleaning the data up and then in here we have i have three different machine learning algorithms i have a neural network on the left oh sorry two actually and i have a decision jungle on the right what i do is i take the data i split it i use the split data partly to train the model
15:46
okay and then the the bit some of the data that's held back i then use to score the model right so what you do is you train the model then you see how good a job you did with the data that you've held back and then you do that i'm doing that here for two different models and at the bottom i can say which is the best model in terms of the the way of actually
16:03
modeling that data so with this you can then start to do prediction right so you take the historical data and you can say okay you know what i know this person's going to get injured after this length of time oh by the way i know that this person is going to have an epileptic fit because i've taken the data over the last five years and we actually not a very good example
16:22
really because that's a really hard one uh the the injury one was pretty easy um even things like you know you can forecast when a formula one car uh is going to run out of out of fuel and it's going around the track and so there's a lot of different ways you can use this one of the key things is you need a lot of data in order to be able to get this to work and i
16:42
mean a lot of data basically if you haven't got at least two years of data two years data already in there then really it's not going to learn terribly well especially if you get into things like neural networks and deep machine learning deep learning you really really need a lot of data so unless you've got something like vision data then these systems don't work too well
17:05
we're that's why at the moment i i spend a year doing this stuff it's great fun um huge demand for people who can do this uh by the way um as i say this this is this is our product it's called azure um the two main technologies in this space is a thing called
17:21
r hands up who's ahead of r as a language yeah okay so and the other one is python the thing called python pandas so a standard python language with a panda set of libraries on the back of it personally i'm a python pandas man you can actually embed the both r and and uh python pandas algorithms into into this and it gives you a very easy way of of understanding
17:43
what machine learning is doing and it's very quick as well all runs in the cloud and it's you can get free application you can get a free free access to that as well so that's really nice in order to be able to do the prediction but as soon as you start getting into this in a big way you realize i really need a lot more data and i need a lot better
18:01
data to be able to do this and that's where i got to about a year ago it's like yeah you know what i'd like to be able to help predict what people are going to do but in order to do that i need to be much better at at collecting data so that's why i moved into the into um into the uh into the sensor side of things once you've got the data you can do some really nice
18:20
pretty things this is uh this is Microsoft bi business intelligence dashboards you can do lots and lots of really and this is drag and drop stuff right there's no program involved it's it looks really pretty uh in general i find it doesn't do quite what i want but ignoring that um it gives you an idea of what can be done so um
18:40
that's talked very briefly about the back end about how we do analytics let's move on and back to actually um uh how how we can help a specific case which is this is the guide dog's case right um you see how long have we got i'll just um play a little bit of a video i suspected that didn't work this is actually the the the architecture behind
19:31
guide dog system so the guide dog system basically what it allows you to do is allows you to walk around and it tells you where you are it tells you how to get somewhere and
19:40
you can do voice commands right one of the interesting things for a blind person is this is pretty useless because how do you know where anything is right in fact on this particular phone it's pretty difficult to tell which way up it is even right so you know a blind person has a lot of trouble with a smartphone right um and and so you know to say oh well you can look
20:02
it up on the maps right you use google maps or whatever to do navigation that sort of doesn't work for them really so there are front ends that you attached uh narrators they're called which will actually read what's on the screen in a rather mechanical voice and it goes down it says everything on the screen which is slow and tedious but it gives you an idea so how can we do better than that you know what is it that we can do that that will improve that
20:24
experience and and and make it much easier for them to fight to both find where they are find where they're going to both outside and inside so it's indoor location and that sort of side of things so this gives you an idea of if you're going to do that this is the complexity just to the front end so this is not the back end and this isn't actually any
20:42
of the central piece which is holding the the large amounts of data this gives you an idea of all the things you have to need to do you need to be able to do speech text to speech and speech to text you need to pull the telemetry data back because that's why you need the machine learning you need to be able to do routing and you're best doing that in the cloud to be honest rather than on the phone um mapping these things at the bottom here sitting in the
21:04
cloud you need to be able to map get the maps back and be able to work out where you are then you need data sources that go on top of the map so you have a base map but then you things that are specialists for them right because they need information about um you know things that they're going to trip over uh hazards trees actually a real problem uh and they're not
21:23
on maps but tree you know branch coming out is a real problem for a for a for a blind person funnily enough the dogs can see that normally um so there's a whole bunch of different things and then finally personalization data so they need to be able to say where they've been and what routes they have and we need to hold that and then we have a whole series of
21:41
microservices in the middle here again up in the cloud which provide real data back to an application and the application we have running on the phone because we need to connect to the phone so you know the conventional wisdom is hey you know what this is great we've got the phone we'll use the phone and hey presto we can connect our sensors to the phone or the phone's got the sensors on and then uh we can then send that data back it really
22:04
doesn't work very well like that because there are a lot of different phones out there and and i've i've got windows phone up there but the two main ones of course for android and and ios and it's a nightmare to work with them but most phone application systems are built to put things on the screen you know if you want to build another twitter client or you want
22:24
to put up a facebook client these are wonderful devices and they're built for that they're built to have a ui which you can actually put up on the screen very simply very quickly it works well unfortunately it doesn't work so well if you're doing that with voice for instance because voice is seen by something that the operating system controls and therefore you shouldn't have
22:43
any control of and and so typically first of all it's it's locked down and in many many cases when you try and start working with sensors on phones you can't get the thing you want right and it's been amazing to me how often that that we've run into the problems and not
23:00
just not just i'm not actually the best by without a doubt is android i mean android you can pretty much go and do what you like but the problem with that is you pretty much can go and file the phone up really easily you know you straight away you start getting into battery life problems or performance problems because you suck all the power out of the thing right we've been there done that right so the middle bit there goes on the phones we actually again we have
23:25
to support not just the two or three phones we have you have to think about what am i going to do supporting all the old phones you know a lot of people have phones already what are we going to do about supporting new phones that come out what are we going to do about you know connectivity bluetooth well okay everybody says let's use bluetooth you know what apple won't let
23:44
you connect on bluetooth there's a thing called mfi made for a made for a ipod which means that if you use a bluetooth device and this isn't completely correct but it gives you an idea if you use a bluetooth device you have to put a chip in your device before it will attach to an apple and an ios phone right so you know it means that if you want to improve
24:05
the sound quality then you have to get into a whole load of issues around licensing with apple which is not somewhere you want to go i can tell you especially if you're microsoft so you know there's a whole load of issues to do with the phone indeed just what language you use this is the the classic for all developers hey what language should we program in
24:22
right you know especially if you're doing application programming should it be objective c should it be swift should it be java should it be you know dotnet or c sharp you know or so should it be c++ we actually did it in the end in c++ because most of the stuff we're doing is very low level so actually we did most of the work in in their level c++ libraries
24:44
you'll see there and then most again how do we connect how do we communicate we tended to do that with hctp we sort of use rest for everything because there are issues with that as well right so as soon as you get into doing just straight hctp you've got to start thinking about messaging systems there are two there are two main messaging systems out there today that are
25:06
used it's sort of open source world one is amqp and the other is mqtt mqtt came from ibm mqp came from came from the banks we support amqp we also support mqtt after fashion but again
25:24
the devil's in the details it's like we support amqp 10 but most of the systems out there today most of the clients out there today are amqp 0.9 right so it's like and of course needless to say they're not the same so you know that's how we do the connectivity but every time you drill into the technology you'll find especially when you're trying to do low level stuff and sensor
25:44
based stuff you'll find very very quickly there's something that doesn't quite work the way you expect it to work we have a whole bunch of services in in this machine it's in the phone itself a couple i want to call out we have our own ui specific front end which does audio
26:00
we actually do 3d audio hands up anybody here's heard 3d audio true 3d audio so it's pretty new it's been around for a long time but not many people implement it but it's certainly picking up a lot now because of virtual reality so virtual reality you know you can turn around and see everything but of course the sound stays in one place whereas when you actually turn around
26:20
and look at things the sound moves around right and so they have this concept called 3d sound and what that does is it actually works with a thing called an hrtf head transfer head transfer response function which actually maps what your head looks like in space so your head in space first of all it it it affects the what the audio coming back to it it bounces off bounces off
26:45
your ears and so on and so forth it also depending on where you're pointing obviously the sound if the sound's coming from over here gets to this ear later than this ear your your ears are really really good at detecting those things all right you know you think oh hey i can i can listen to stereo music i'm really good you know what for understanding three-dimensional
27:03
space your ears are really fantastic and and a blind person's ears are even better a blind person can actually walk down the street and i've worked with a lot of blind people they will actually use things like a woman in high heels to a blind person a woman in high heels is a wonderful thing please wear high heels everybody right because it clicks all right and the click
27:25
comes back off all the all the walls around them and they can navigate where they are from from the noise coming back there are actually people that can just make noise themselves and use it like a dolphin like an echolocation system so that they have really really good sound
27:41
systems so we need to give them a really good 3d audio environment to understand that means however that we have to have really high fidelity sound so you hear so standard quality audio from a from a phone is in the you know it's in the three or four kilohertz right
28:01
high definition voice is around about seven or eight kilohertz that's the sort of level you have to get to in terms of frequency in order to be able to do voice recognition by the way then you go on from there and so stereo an old person like me i won't hear much above 12 kilohertz right young people like you lot will probably hear up 13 to 15 kilohertz
28:23
maybe 16 kilohertz if you're really young and you think that's it no no we're working at 40 kilohertz with blind people in order to get good 3d audio now they're not hearing 40 kilohertz but they hear the reflection of the 40 kilohertz waves so then you have to go okay how do we
28:40
get that sound into their ears a lot of issues around that and i'll talk about that in detail later on because that's my sensor talk um so those that gives you an idea we've also got to talk to other senses location we need to know where they are how are we going to do that gps is an obvious way doesn't work inside right so how are we going to do that so you can see that there's a lot of stuff at the back end which i'm not talking about there's a lot of stuff
29:02
at the front end and actually incidentally in the middle here there's also a lot of stuff that you have to think about when you're building a real system right so let me talk about the sensors themselves it's just some sample samples senses and what i want to do is is i'm going to go through all the different things that you need to think about so for instance movement
29:21
right i've already talked about the uses of movement and i've talked a little bit about the microsoft band um this thing has a these things all of these things and your phone have a mem sensor in them it's a motion detection sensor they're either six or nine degrees of freedom in general um six is much easier and six is basically a gyroscope and accelerometer in three dimensions right and what you have to do is you have to take the data from all and
29:45
sorry the other three of the compass so it's a magnetometer these have magnetometers in them as well and so you need to pick the data up from all three lots of sensors which give you nine degrees they're called lined off nine degrees of freedom sensors and that then gives you the ability to know when when somebody's moving and where they're fake to a point up to
30:05
where they're facing the problem is the sensors all drift and they need calibration all the time right um it's a fact of life right they it's a mechanic actually especially the compass is like a little ferrite bead they're actually fairly mechanical devices in the in the sensors themselves and they have to be calibrated so you have to what's called sensor fusion
30:24
and sensor fusion is when you take this was on the previous slide is sensor fusion is when you take each of the inputs from all the three of them you take what's happened in the past and you basically calculate the whole thing together in a very large algorithm which which determines how much they're drifting and how much they're not and then you correct for
30:41
that there are to my knowledge like three companies that do it there's one open source set of code out there which i can't understand it was written by a phd student about 10 years ago and nobody else has ever been able to replicate it and there's two companies really that do it one is invents this invents sense sorry and the other is bosh and by the way the best library is the bosh libraries in my opinion if you're doing motion detection they're the people
31:04
to go with although their sensors that their sensors not not not not quite the same i use the inventor sensors as it happens so and this is this is something you'll very quickly find when you get to work with sensors is know your manufacturer because because you're going to get
31:20
to know them really well by the time the project's finished right um the next thing is location hey that's easy it's gps gps doesn't work inside right that's a problem so there's lots of different ways of doing this and the classic that everybody says is hey we'll put beacons up ever since i've been sorry ever since apple came out with with the whole iBeacon thing
31:41
it's like we'll just put a beacon up and that will all work no it doesn't um there are a lot of problems first of all beacons were designed for proximity only they're not designed for you can use them for triangulation they're not terribly accurate but you can do a reasonable job of it actually what they are good for is finding your reference points for calibration
32:02
all right so if you want to calibrate something then then they're useful for that but the big problem with beacons is actually the management of them all right so if you're actually going out in real life and doing this with real people you need to put the beacons where those people are going then you need to put them in the shopping malls you need to put them in stores and we've done that we had them in a big shopping mall in Reading for blind people we also
32:27
did it with Tesco's one of the UK's biggest retailers and we put beacons through their store it's a nightmare you have to manage them people move them people actually strangely if not many get stolen the biggest problem is the batteries fail right and quite quickly too
32:43
irrespective of what manufacturers tell you their batteries don't last that long right and then you say okay we'll have to wire in the stores don't want to wire things in i can tell you now all right the second is that they get moved and it happens all the time and then you have to go back and you have to recap remap the stupid things finally as i say occasionally people
33:03
steal them i have no idea why um but but just managing them and knowing where they are it's a big problem so the answer is beacons and it doesn't work it's not a real answer i can tell speaking to somebody who's been out there we actually put beacons on buses which is one of the cases funnily enough where it actually works buses and trains are good because nobody
33:23
steals them off buses because you put them up with the driver um their buses are maintained so you can get the batteries changed fairly easily and it's really nice for blind people because they actually get a signal from the beacon before the bus gets to them in fact before the bus even comes inside so they really like it because when they're at the bus stop the first
33:41
thing they the first person the bus that knows the the bus is coming is actually the blind person and by the way he knows which bus it is and where it goes to because we give all that information so you actually are able to this is the nice thing about is you not only can you replicate what sighted and non-disabled people do you can go beyond and you can actually give them a lot more information right how was i so um orientation i talked a little bit
34:05
you need to know not only where you are but you also need to know which way you're facing you've got a compass these things have got compasses in them they're not terribly accurate they drift um we're still sort of coming to terms with how we get the best accuracy out of out of a compass to be honest at the moment all right so that that's around
34:24
the location side of things the next thing is is audio how do we get the sound into in to somebody right and again a lot of issues and challenges first of all we as i said we we need to give them this three-dimensional sound we we need to give them this very high fidelity three-dimensional sound very few people work with three-dimensional sound there's a company
34:42
in denmark that has a set of libraries called am3d they're great people to work with and they actually do have a set of 3d libraries for all the different products written in c as it happens and we also have our own internal one as well which is done with research it's being used on the hololens and so that gives us a very high fidelity sound but we
35:05
have to get it to the ears as well we can actually wire it and that's how we do it today so what we actually did in testing is we use these things right and this is from a company it's called the intelligent headset and it's come from a company that owns jabra the headphone people and in here there's a gps in the top for location so the gp in this bulge here
35:24
there's actually a gps receiver there's actually a processor it's actually a ti processor that happens inside the headband plus some batteries and it has a motion detector as well inside this as well so it actually detects which way you're facing it detects it doesn't
35:42
detect comp sorry it doesn't detect compass it hasn't got a magnetometer but it does detect motion so it can see when you turn your head and it knows where your location is and it has very high quality speakers all connected through bluetooth and bluetooth le this is a dual system it's both bluetooth at classic and bluetooth le i hate bluetooth classic the pairing for blind
36:03
people in particular pairing is a big problem and so i try and do everything with le and and i say these use le as well so we actually tried these i mean we did a big field trial the end of last year and we gave these to a lot of blind people we actually made there are actually some modifications to these so they can hear the ambient sound as well it's a really bad idea
36:24
to block blind people's ears right because they can't hear the car that's coming here so we actually made some modifications which made these able to hear ambient sound even so even though they were getting a pretty good external ambient sound they didn't like them just the physical act of covering up their ears is absolutely terrifying for them they just
36:44
feel really vulnerable when you close cover up their ears because they've suddenly lost their whole sense or they feel as though they have even if they haven't so actually that didn't work it worked in terms of what we wanted to do but it didn't work in terms of what was acceptable to them right but it as did it did allow us to get this this high fidelity sound to them
37:03
so we then tried this these are these are bone conduction oops which are just broken i take all these things to pieces and then of course they they break very easily because i don't pull them back together properly so basically this is a bone conduction headset right and it actually clips on here and it's actually uh it's meant for somebody who's running that doesn't want
37:23
to obscure their ears so a lot of this stuff has has other applications in in sports uh you know in gaming for instance this head detect these earphones are actually made for gamers who want to follow what's happening on their game in 3d sound um these didn't work because they actually they cut most a lot of the high frequency sounds by using bone conduction through
37:44
your jaw then you you actually lose a lot of your high frequency sound so your audio response is gone down they're also quite big and this is something we had a lot we don't want big fun so this is actually what we're using we're using a system like this but i'm not showing you how the i'm not going to show you how the uh the front end the audio system
38:01
works because a we're still working on it and and b i think it's quite clever so it gives you but anyway it gives you an idea of how you can do audio out and the sort of have two headphones two microphones in the headsets if you're outside on a windy day it doesn't work all right it just doesn't work if you're trying to do voice recognition signal
38:25
to noise ratio has to be above about 16 db in order for speech recognition to work and if you're in a windy day you just can't get there even if you have a microphone well you can with a boom like this you can do it all right but if you have a microphone in in
38:41
in the line won't work you the speech record just won't work what we found and and so we actually do have in this headset we do have microphones as well we know they're not going to work so what we've actually done is we've put them inside a handset like this okay and there's a speech recognition this is actually an xbox one right there's a there's a microphone high high quality
39:02
digital microphone in this so you actually speak into this because what we found was that coming back to one of the issues is that people can't use a smartphone because you're blind right so actually they want buttons and also to be honest you don't want to be walking around downtown redding or london or i don't know what i was probably all right i walked through
39:23
ours last night it was nice but certainly there's a lot of streets you don't want to be walking around with a 600 phone in your hand and being blind right because you know you've got a white stick in one hand you've got a 600 phone in the other and you're in the dark street late at night you can see what's going to happen you know the idea is not to get people mugged with the system you know it's fairly important so what we do is we give them a a little
39:45
handset like this it's actually got a ti processor in it we're using the actually the guys outside we're showing it off one of these things sense tag we put that in here and that gives them the ability to keep the phone in their pocket and just clip on and touch on this
40:01
when we did the trials before we did the trials we didn't think that was going to be very important we didn't put a lot into it it was really really important to uh to the users themselves they actually i didn't show you the video in the video you can see they're using it all the time they're holding on to it even though they've got a dog or a stick they're holding on to it because it's their connectivity it's their control they feel in
40:24
control if they have it in their hands so we actually realize that this thing they do hold this all the time and so to actually put a microphone in it so they can just talk into it it makes it really easy and and indeed that's exactly what we have with this one which you bring the ability to actually talk into it so but you have to again have get a very high quality
40:45
voice and the final version actually has a a wind baffle on it so we can get up to the sort of quality levels that we need in order to to do the voice again all sorts of issues because today classic bluetooth does not allow you to stream to one stream audio to one and
41:02
receive voice from another doesn't happen right in fact classic bluetooth today won't even give only gives you like a four kilohertz voice stream back right so it's not even good enough even if the microphones were good it's not good enough to do decent speech detection on bluetooth you have to use a thing called voice hd which takes you up to seven kilohertz
41:24
very few systems today support voice hd so you get into all sorts of nightmares with that and in fact what we're doing is we're doing voice over voice over ability bluetooth le so we're actually digitizing the voice and then sending it back over codec through through the
41:42
through the device back up into the cloud so we can do our voice recognition so that gives you an idea of some of the complexities just in that alone some of the other things that we that are one of the other things that that's come up a lot and sort of feeds into this of where am i is um is is video so it is a real problem knowing where somebody is inside and
42:07
the way most people are sort of tracked i think a lot of people have gone down the beacon route and decided that's not so good and now most people are actually trying to use video of some form or other and that's certainly how how we're going as well indeed that's how
42:22
the hololens works too so you know how do we actually provide video back um and then from the video not trying to do like detection of the whole room because that's pointless but actually to be able to detect points in the room which is actually fairly easy and then then from that detection know where you are from from the line so give you an
42:42
example that sounds a bit woolly to me so give you an example in this room you're looking this way it's actually there's a couple of features that are going to be really easy to find right these walls yeah now you you have a model of the room we've got models actually most buildings now you know google ourselves most the internals of most buildings are now
43:01
modeled they hold plans for them the plans have been digitized we've got most of the plans so we sort of know what this room physically looks like and we can pull that down right we then know we can detect these corners all right and from the detection you then say okay i know where i know the angle to that line the angle to that line so i can detect
43:21
by simple triangulation where i am all right we haven't built it you haven't got it to work yet um we're getting there and and i believe that the the whole area of of doing location will be through uh through video in the future because that implies you have some sort of camera all right and nobody's going to walk around with a you know i've actually seen it done right
43:43
so it's strapped to video camera it's trapped at the phone to somebody's head right yeah one of the other thing about blind people in fact disability people in general they don't want to look like idiots any more than you or i do they want to look like normal people they do not want a phone strapped to their forehead right um or indeed walk around with these on so
44:04
you know one of the things we're working on for for the location is these right and they look like reasonably like glasses they're not stylish because you know got them made in made cheaply so and i'm not a designer by any manner of means but the interesting thing about these is they actually have a process they have a camera in the front right in the middle there there's a
44:24
camera it's a pinhole camera you can hardly see it all right um and what it does it feeds to a pcb to a processor on here in one side the other side's actually got a got a battery in it okay so and this pcb's got both the camera processor and the um
44:44
and the bluetooth actually what we do is we do some pre-processing in this uh to do the vision um and then we just send back what's called cloud points if you try and send all so the first thing is that hey you know what we'll have a camera and we'll just stream the data back yeah that's gonna work your battery's gonna last about 30 minutes less with this battery
45:04
it's 100 milliamp hour battery right so you know but that's the first thing hey just stream it all back and we'll do it all on the phone that means you have to do some pre-processing and actually you need quite a hard heavy duty processor to do that and that's what we're doing in this so it's have we got it to work yet not quite but we're getting there um and of course
45:22
once you've got that then it can be used for lots of other things if you've got the headset on uh it's the other thing we're looking at is okay then you can actually add in the earpieces as well so it becomes gives you 3d audio as well it's got a camera on it blind people not too fussed about uh about having uh glasses on but uh you know they they
45:41
don't mind that so much and of course this one's actually also got the uh the motion detector in it so i can actually tell which way somebody's which somebody's way somebody's facing from that so that that's us that's in terms of sense as i missed anything eye tracking oh um one of the interesting things that's going on and i'm i'm sort of toying with i haven't actually built
46:01
it yet um is that if you have this or these on i'm not gonna put them on because they're broken um this is not our working one it gives you points here very close to the to the to the head um and you can actually build this out so it has a pad against the the side of your head here and you can then pick up um the uh the brain waves the alpha and the beta brain waves
46:24
from from the inside of the pads here which allows you to do things like eye tracking epilepsy detection again suddenly the the the gold standard for detecting epilepsy is through brain patterns it's well understood how to do that if we can track the brain patterns we can get a really good job at monitoring somebody's brain um so there's a lot of different ways
46:44
once you've got the device on them to start using that device in other ways okay so so let's talk a bit about the actual sensors you then need to get the data back typically there's three ways of doing it you know bluetooth bluetooth le or uh or using ethernet standard sorry same wi-fi trouble is um especially wi-fi uses a lot of power
47:05
so you know how do you keep the power down um they're actually the uh interesting enough to chips and the hardware cost is much the same for all of this i don't know if anybody's seen this but this is a thing called nate 266 from china uh and it's actually a complete ethernet
47:20
uh wi-fi stack on a strip on a on a little chip complete wi-fi so it's got the whole of stack on here the processors on it plus all the radios plus the antenna plus the rf on this right so this is a complete wi-fi device uh enabled device it's actually two bucks
47:40
you can go and buy these for two bucks buy them off ebay for two bucks they're that cheap all right you know i'll talk about it in a bit more detail once the chinese get involved prices go down really big um and so that's certainly a way of doing it my the problem again is to do with battery life so i tend to use bluetooth i actually use as it happens the ti
48:01
chip so shout out for the ti people and the ti people in the room there you go why do i use the ti chips because you know what working with the chinese stuff i can't speak mandarin and working with the chinese stuff is really hard um but a lot of stuff's going on in china uh in terms of hardware you should really be following that because what's happening
48:22
in china will impact us it's impacting everything uh and it will continue to do so so i've talked i've talked a bit about uh about the headphones um oh yeah this i didn't talk about one of the other things that's interesting is okay so if you actually want to control so so in terms of
48:41
care old people watch tv that's sort of what they do right um tvs have a massively complex remote control they can't use it wouldn't it be nice if we could give them a really simple remote control yeah and by the way that's the sort of thing we've built but we've built it for bluetooth tvs are not bluetooth enabled this thing actually interesting enough is a 20
49:01
20 device it converts from uh bluetooth to um to to to infrared so it's an infrared blaster um again chinese and it's a really good way of being able to control your tv for instance from your phone right because you can just load up an application that allows you to control your phone your air conditioning whatever else from it the other two things i wanted to show because we're
49:23
using this with with elderly people you know we've all met elderly people you know you take them your laptop in and say hey you know what look you can run skype on your laptop and they go laptop what's that you know and it's like oh we got a long way here um wouldn't it be nice if you could just get a box which you plug in the back of the tv um actually it's a full
49:43
computer it's really cheap so you're not having to worry about it it's fixed it's a single function is just going to run their skype for them so he's just going to have a webcam on it run their skype so that they can skype on their tv and they don't even have to know it's a computer well you know what nowadays they have them right so i don't know if you've seen this so this is actually a full pc all right this is actually runs windows 10 it's chinese
50:05
it's kind of called me go all right um it's like i think this is about this is 16 gig it's a low end process bay trail processor obviously 16 gig of memory it's got three or four usb ports hdmi you plug it in the back of the tv turn it on usb pad you don't need a
50:23
special power supply it's just a standard like your phone usb charged um it's got a sd card in it for more memory hey presto it's a really nice single chip single blocks device for embedded an embedded solution in a for elderly people and for care 50 bucks
50:43
one off 50 bucks off ebay now 50 bucks is too much for you and i shouldn't really be telling you this um you can actually get an android version right so this is actually again this is a chinese thing it's an android version the same thing so it's actually got a processor in here uh it's again a 16 gig device runs full android on it again all the same sort of setup
51:04
sd card all the rest of it just plug it in it works uh it's got wi-fi obviously for communications um this is 23 bucks one off a 23 processor you know and actually that works works quite well too so we'll you know you for the care side of things we're putting these into
51:23
into the into people's rooms in the home they all have a tv we just put a box like this in but then we can run skype we can run the the other applications that we want we can run them in place that people don't have to worry about it and those are the different controls
51:42
now this is talking about processors this the one on the left is the little wi-fi chip i was talking about the one on the right is texas instruments guys i hope you notice i got it up there i'm not getting paid for that by the way um the one in the middle is a strange looking beast what is that it's this all right it's actually this this is a processor
52:06
no big surprises there but actually this is a full uh a full phone processor okay so this is actually dual core uh arm seven as it happens processor it has two cameras it has bluetooth
52:23
classic bluetooth le it has wi-fi it has uh it has full 3g so it has both uh voice audio edge and 3g radios in it um it has a loudspeaker it has microphones it has usb port it has uh 12
52:43
gig of memory and an sd slot in that for 60 bucks it's actually 40 bucks one off quantity 40 from china all right so all of a sudden you go well actually you know what i can put something like this into a device like this for 40 i can make this device
53:02
intelligent i can run the whole operating system on here and then i don't have to worry about the fact that these things are changing every day right because management of these things not easy so what i'm saying here is a lot is happening in at the low end of the hardware prices are coming down function features and functions are going up be aware of that because
53:21
it changes the dynamics and it changes what you can do both as in my case with the with the elderly and with the with the disabled people but also will change into into more general population and and that's where i wanted to end up really it was it's around a lot of this stuff goes on in china and we don't see it right because most of us don't speak chinese mandarin
53:46
right and so it's really difficult to track what's going on in in china we know we think we know how china works so you go to a company like foxconn and microsoft and apple and everybody else we all get these things made in china all your hardware is made in hot china really
54:01
but it's made by companies like foxconn and we give them the designs they build it to our spec and so on and so forth but there is a whole different industry out there in china and it's called shanzai and it's a completely different way of building hardware and it's how the chinese do it and it's how they can build something like this for 40 bucks right because first of all
54:25
they use a set of chips that we don't know about hands up who's heard of uh rock chip ti guy loves fries right media tech media tech ships more processors than anybody else in the world i think at the moment it's
54:43
not it's about right anyway they certainly ship a lot of processors all right all the chinese phones all run media tech chips right this has got a media tech chip in it right everything media tech chips are really really cheap which means the chinese can pick up very very very
55:03
cheap components and unfortunately we're locked out because you can't read the chinese and the way they actually do it is that they provide reference designs so go to media media tech they provide a complete reference design like ti i've done with the with the sense tag and all the code associated with it so as a manufacturer all you have to do is pick it up and put it into
55:23
put it onto the onto the pcb make the pcb's and there are thousands of companies in china that make the pcb's for nothing assemble it and put it into a box of your making and that's this whole shanzai because what the chinese actually do is that they actually just copy they copy one another they the the manufacturer will produce a reference design
55:44
and then all the manufacturers will just copy that and then somebody will go and they'll say you know what we think we can build a a better headset give you an idea this is actually a chinese copy of an english headset there's a company it's american company called aftershocks made these right and the chinese did a copy of it it looks identical actually i'm surprised
56:03
you know but the the u.s one is like 80 bucks and this is like 30 bucks right how do they do that because they're using their own manufacturing techniques they're also using their own chips and their own technologies in these things and so the bottom line here is
56:20
if you want to if you want to do anything in hardware or or in sensors indeed then it's really best if you do it the chinese way because if you do it the western way you will find it is extremely difficult to compete and even if you do it with the shanzai way of doing things yeah what you are what they're bad at is they're bad at the software okay so especially the embedded software if they as soon as you can so as soon as you get away
56:46
from the manufacturer's software they become very ragged very quickly right and so it is there's a huge opportunity in my opinion to build systems on the hardware that's coming out of china at the moment and out of these really low-cost chips and actually build new systems
57:04
new things on top of that which attack price points that we can't get to by using the western ways of doing things so and and i believe especially in my market which is extremely price sensitive blind people they don't have a lot of money in general in fact most disabled people they don't have a huge amount of money so you know cost is very very important to them
57:24
and so how they can get how we can provide these things to them at the lowest cost you know and the quickest and easiest way is obviously of great importance to me which is why you know i've been looking at right down at this this whole bottom end of of the shanzai stack all right so that's about it i'm very happy i've got one minute left so if there's any
57:46
questions i'm happy to take them otherwise i'll be around afterwards i'll be outside having lunch as well very happy to take any questions both on uh you know the whole area of disabilities of care it's as you can tell it's something that's close to my heart um you know i have my mother
58:02
actually has dementia quite badly um and you know it's it's heartbreaking really i just hope it never happens to anybody and you know you have it never never have anybody in your family who has any disabilities because it is it is a very difficult thing to deal with and so it's very close to me and i'm very happy that i can use my the skills i've built over you know a
58:25
lifetime to actually then go and do these things and help these people who are desperate need of the technology that we have any questions in that case i'll let you go to your lunch thank you very much indeed for your time and attention