Project Lighthouse and stuff we learnt along the way

Video in TIB AV-Portal: Project Lighthouse and stuff we learnt along the way

Formal Metadata

Project Lighthouse and stuff we learnt along the way
A low-cost device to help blind people live independently
Title of Series
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Year

Content Metadata

Subject Area
Let's see how we can build small IoT devices that can help blind people intheir daily life. The Word Health Organization estimates that more than 250 million peoplesuffer from vision impairment, 36 millions of them being entirely blind. Inmany cases, their impairment prevents them from living independently. Tocomplicate things further, about 90% of them are estimated to live in low-income situations. Project Lighthouse was started by Mozilla to try and find low-costtechnological solutions that can help vision-impaired people live and functionon their own. To this date, we have produced several prototypes designed toaid users in a variety of situations. Let's look at some of them. This will be a relatively low-tech presentation.
Point (geometry) Slide rule Focus (optics) Demo (music) Plotter Multiplication sign Projective plane Set (mathematics) Bit Total S.A. Branch (computer science) Product (business) Product (business) Element (mathematics) Connected space Prototype Befehlsprozessor Computer hardware Point cloud Video game Internet der Dinge Prototype Personal area network
Source code Open source Software developer Online help Instance (computer science) Machine vision CAN bus Semiconductor memory Personal digital assistant Different (Kate Ryan album) Computer hardware Computer hardware Video game Object (grammar) Resultant
Presentation of a group Demo (music) Code Software developer Multiplication sign Bit Number Revision control Medical imaging Prototype Arithmetic mean Software Iteration Right angle Software testing
Mobile app Presentation of a group Multiplication sign Software developer Bit Line (geometry) Mereology Product (business) Product (business) Prototype Estimator Process (computing) Blog Factory (trading post) Computer hardware Video game Divisor Process (computing) HTTP cookie Form factor (electronics) Prototype World Wide Web Consortium
Machine vision Slide rule Trail Graphics tablet Open source Code Length Multiplication sign Connectivity (graph theory) Real number Execution unit 1 (number) Virtual machine Online help Mereology Machine vision Number Usability Facebook Connected space Prototype Iteration Single-precision floating-point format Personal digital assistant Computer hardware Software testing Computing platform Algorithm Demo (music) Binary code Shared memory Planning Line (geometry) Unit testing Demoscene Software Personal digital assistant Computer hardware Internet service provider Chain Faktorenanalyse Smartphone Right angle Object (grammar) Musical ensemble Computational visualistics Prototype Speichermodell
Existence Open source Real number Projective plane Physicalism Instance (computer science) Distance Product (business) Number Frequency Prototype Software Internet forum Iteration Personal digital assistant Factory (trading post) Touch typing Computer hardware Self-organization Faktorenanalyse Software testing Prototype
Trail Mobile app Prototype Software Phase transition Smartphone Speichermodell
oh my name is David I was working on a
project called project lighthouse which I usually plan to introduce to you today things were a bit complicated there are a few interesting plot twists between the time I submitted this talk than the time I finished the slides than the time of when I'm presenting but so let's start with the plot twist the prototype has broken at some point so I'm not going to make a demo of this but so what's project lighthouse project lighthouse was a project of Mozilla's connected devices branch dedicated to helping blind people live in their daily life this talk at some point I considered renaming it how not to die when you're preparing a hardware prototype which is pretty ironic given that connected devices died two days ago so I'm still going to present so about us we devices was a set of internal nonprofit startups inside Mozilla the idea when it worked was to have great ideas make great prototypes try to convince people to give us money to turn the prototypes into actual products and hopefully to make products that will help users connected devices has produced a total of zero products which might be somewhat related to the fact that we don't exist anymore [Laughter] at some point and since we were the rebels we were still using the Dyne user and not the new funky weird logo of Mozilla this might be the last time you see the diners are tell you what this is about I mean what project light house is about because initially project light house was about we know how to use cameras we know how to program this kind of thing can't we do something not creepy because if you look at the use of cameras in connected devices in IOT these days just about everything is about watching you do something and telling someone on the cloud exactly what you're doing which is incidentally one of the reasons while company why some many big companies are interested in this for strategic reasons for insurances and I mean there are many companies that have interested in knowing what you're doing at home either to sell you things or to charge you for things you're doing and that's one of the reasons why there is so much focus and so much money involved and invested in Internet of Things these days it's not just because it's fun it's also because being creepy is profitable but we wanted to do something not creepy so what can we do with the camera a Raspberry Pi and a microphone and a speaker that happened to draw more juice than the element ation of the Raspberry Pi can provide which is pretty much why I cannot do the demo today do something
useful so sometimes the solution meets the problem because there turn out to be some people who have interest in having a camera and something interesting done with the camera some people the life can be changed with camera so according to the World Health Organization there were close to 300 million users sorry people who live with a sight impairment about 30 million of them or entirely blind but still that's many people who have trouble in everyday life recognizing objects for instance this means that in shops these people have lots of trouble they need help to find out exactly what they're buying and since shops tend to at least in big shops tend to change the place of things regularly to force people to revisit everything if your sight visually impaired you're in trouble because they are actively working against you against all your brain or muscle memory they are trying to force you to not know where you are anymore there are basically basically the solution is find help so there are a few devices that I know of essentially two specialized devices to help people in this kind of situation they both are pretty much more than 2,000 for something that actually contains this kind of hardware so the idea is I mean that's worth about 30 maybe 40 I don't remember but that's the battery so the idea is can we do something open source open the hardware and chip and we set out to try that so more specifically
we wanted to help visually impaired people recognize household eat items because if you have two packs of the same size it's pretty hard to know well if you have two cans one is coke and one is Fanta you have no way of knowing which one is which if you don't see them but in that case is pretty easy for camera to make the difference other things are much harder but we try to see if we could do that so we actually got good results with the for the one above and pretty bad results with the one below but we only had a few weeks of development so so we get little
software to developers oh sorry yeah lots of open CV - so - developers working for a few weeks time and we
actually managed to users so I have pictures of actual users but I'm pretty sure that I do not have the right to put them in an image so I used super villains instead so we have 24 test users so we actually tested something that was a slightly more refined version of this this was prototype number one there is a prototype number two that still is somewhere in California and probably going to stay there so I could not show you it would have been a bit easier to make the demo because it does not rely on so many outside stuff and so I call this presentation lessons learned so lesson number one nerd if it hasn't been tested by users it doesn't work it just because it's cool just because it has code just because we had fun developing it doesn't mean that it works so we were lucky after relatively few either eight iterations we got something nice so that was prototype to
our lessons were yes not all of these 24 users who are interested but enough of them were interested that it was worth doing they thought it would change their life not everything in their life but at least shopping which is an important part of life if you need to you know eat so we're supposed to work on making on keeping it cheap making it simpler and have a better form factor cookie so that was making a prototype how do we get from deer there to a product
so we put some designer to design some stuff we put a factory and that's pretty much not it not how it works it's a bit more complicated so for the documents and on a blog that details a little bit more the process from the time you get the hardware prototype and a good industrial design to the time you can actually ship it so forget marketing on the top forget the app development at the top you still get all these three four lines and our estimates were that they could take about six month time to get done could actually be worse when I wrote this slides I was preparing to tell you that it's probably going to take more than six months in fact it going to take forever so I was right so the question here is what do you do are you going to sit down during six months and do nothing how can you not waste your time so again this lectures lecture sorry this presentation is about lessons learned so we actually came up with a solution and it's actually simple
one you are building a device you want to test it so test things if possible with devices that people already have most people these days have a smartphone or tablet some have a small TV so if you can find a device that your users already have you can paralyze things you can have someone possibly you working on the hardware finalizing the hardware getting the providers getting the supply chain getting everything together testing actual hardware components etc etc something that's going to take you insane amount amount of efforts and time and you can also work on the software at the same time which is the things that some of us do best yes
so in this case we set out to use a device that blind users have and this device is called an iPhone it turns out that a huge number of we actually have no numbers of blind users not visually impaired users but for blind users it turns out that many of them have iPhones in the US it's like 85% of those who have a smartphone have an iPhone and many of them have a smartphone I don't remember the exact numbers so this means that we got ourselves working on a third prototype that was that we all fold them we are in open-source stuff if you want to develop for open-source for this let's just say that I have seen many platforms on which developing open source software is nice simple easy stuff works this is not one of them Apple is going to extreme lengths to make sure that everything is locked down and you have to go through them for everything so part of the lessons learned yes you need to go where your users are for testing if you can avoid that platform it's worth trying so we actually only got three weeks of work on that so most 35 weeks most of the time for me was spent actually getting Xcode to work getting unit testing to work and trying to actually release binaries we have not succeeded at that yet everything is locked down really tight so it would be a great time to discuss the next steps what have we done we have an early prototype of that I could actually do a move that let's see if it works so [Music] you don't hear anything do you bunch of people behind the camera oh gosh completely so it was trying to find out which was the interesting object in the scene and apparently the interesting because also the microphone is dying so let's pretend that I have a very nice demo what should the next steps be okay fair enough make the demo work that would be a pretty good next step what should the other next steps me at that stage at that stage assuming that we have software that works nicely assuming that we are still on track to get hardware oh I'm actually we do have many nice hardware designs I did not include them in the slides because I think they involve commercial partners so that we probably do not have the right to put them here it doesn't matter anymore but hey so at this stage our next steps would be ok so make it work for real the prototype apparently doesn't work at for them so opening issue doesn't work at for them so we have been working so far on making some basic computer vision algorithms obviously making better computer vision algorithms would be helpful but in uniting is once since we have the ability to prototype on the this ugly stuff we can iterate and paralyze so people working on improving the computer vision people working with actual users on improving the usability and the accessibility people working with users not necessarily the same ones on coming up with new features we were also planning to go further because at this stage this is an IOT session I was talking about tea but there was no I there is no I in team so one of our plans was to make it social not in the bad way let's add Facebook everywhere but in the good way as in sometimes people need more help than just identifying something easy to identify sometimes people need help to find oh gosh I'm wounded does it look bad one of my breasts feels large you know sick I mean that's a pretty good one and that's the kind of reasons for which you need more help than just what the Machine can help so we wanted to help people share their questions with people they trusts and possibly in cases that were more much boarder share questions with just about any volunteer so we were in parallel starting to work on a platform for crowdsourcing questions from people with sight impairments such as what is that object what am I doing here going to be released ever because there is no single line of code written yet as I said if you don't want to die to avoid multiplying devices real devices but you can again fake stuff using existing devices once you have software running I
mean one of our industrial designs for
instance was actually a necklace relatively heavy necklace containing well basically this and that's something that we can fake easily if we have software we can easily make prototypes that don't have to go through a factory flat you can pick a real necklace or anything that fits around your neck try it and attach stuff that's not sufficient for commercialization but that's more than sufficient to find out whether it's too heavy for users or whether users actually manage to point the camera in the right place because we actually got tests and just actually say that when people point cameras very well that's that was a very good surprise they have difficulties judging the distance so it's sometimes it's too close but it's Center and that's good because that's something that needed to be tested anyway if you want to avoid relying on having a finished hardware device before you can iterate and one of the ways to do it is to have fake device so just something rigged together using a device for which you already have software so one thing you do not need to forget if you are working and if you want to somehow release a hardware device is that it does not stop there you probably have lots of legal things to cover that we are not used to in the open source world in this case people could possibly have accidents because of this physical accidents people trying to use it to identify 25 medications for instance could end up with the wrong medication or they could add a period with rat poison eating rat poison because we had told us that it was beans red beans something like that that's pretty tricky so we of course we want to make sure that the software works but many of these things are have real-life implications which among other things involve lawyers in this case we also you should also have been in touch with a number of people such as organizations helping blind and sight impaired people because that was the best way to get in touch with our users one thing that's often overlooked is the Kickstarter Kickstarter people generally see it as a way to raise money the secret of Kickstarter is that this is not true Kickstarter is actually a marketing tool used by companies to measure interest in a product before they develop it if people are willing to invest ten thousand dollars of ten million dollars that depends on the product and this means that you can sell the product I know of a number of cases in which Kickstarter has been used to sell a product that had no kind of any kind of existence in the real world just to measure whether it was the world is worth worth it was worth trying to build such a product some of our potential partners did that for instance and we had surprises so again if you're trying to build something and turn it from hobby to something that you can distribute Kickstarter is a pretty good tool for that not just for raising money before to check if it's worth doing you can find all these software prototypes of project lighthouse on github and we have a discourse forum if you want to I don't know sympathize with us thank you very much for listening
and if you have questions I will be happy to try an answer yes so what's going to happen with the prototypes of lighthouse will they be released so the software prototypes are on github already I mean we need to put a license because we just forgot that in the rush but on github the hardware design prototypes I don't know I would like us to release them I do not know if they belong entirely to us if they belong entirely to us I think we can release them if they don't we probably can't yes yes so the question is if so many blind people and vision-impaired people have smartphones why not just make an app we also considered that yes that was a separate track I am I'm an IOT session so I was concentrating on the IOT stuff but yes we considered continuing the app track past the prototype phase okay great thank you very much


  280 ms - page object


AV-Portal 3.20.2 (36f6df173ce4850b467c9cb7af359cf1cdaed247)