How (not) to build autonomous robots
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 254 | |
| Author | ||
| License | CC Attribution 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/53197 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Point (geometry)AreaRoboticsThomas BayesVulnerability (computing)BuildingAutonomic computingComputer animationLecture/Conference
00:57
RoboticsPrototypeIterationService (economics)Product (business)Different (Kate Ryan album)Line (geometry)Electric generatorMultiplication signShared memoryLecture/Conference
01:59
Office suiteDifferent (Kate Ryan album)BitVideo gameRoboticsComputing platformGroup actionMultiplication signWebsiteFigurate numberStorage area networkBuildingUltraviolet photoelectron spectroscopyForm (programming)WordSimultaneous localization and mapping
03:32
BitRoboticsPrototypeMultiplication signTerm (mathematics)Order (biology)VideoconferencingShape (magazine)System callLecture/Conference
04:44
Product (business)Dependent and independent variablesRoboticsFeedbackDifferent (Kate Ryan album)Multiplication signCuboidShape (magazine)Virtual machineGoodness of fitMereologyKey (cryptography)IterationStorage area networkLecture/Conference
06:54
Product (business)Different (Kate Ryan album)RoboticsIterationStudent's t-testBuildingComputing platformLogical constantProduct (business)Multiplication signAreaSocial classLine (geometry)Wage labourProcess (computing)Visualization (computer graphics)Order (biology)Endliche ModelltheorieParallel portMetreLimit (category theory)CuboidKey (cryptography)Scaling (geometry)Sound effectMathematicsService (economics)Machine visionBitInheritance (object-oriented programming)Polygon meshMultiplicationOffice suiteComputer animation
10:44
IterationTelecommunicationNP-hardComputer hardwareRoboticsMachine visionFrequencyIterationMereologyMultiplication signComputer hardwareProduct (business)NP-hardRevision controlInheritance (object-oriented programming)Video gameCASE <Informatik>TelecommunicationBoundary value problemSoftwareGroup actionScaling (geometry)Execution unitPlastikkarteOrder (biology)NumberPlanningInformationSign (mathematics)WordLecture/Conference
13:36
PlanningMultiplication signHypermediaDifferent (Kate Ryan album)BitDichotomy
14:21
Flow separationInheritance (object-oriented programming)CoprocessorEntire functionConnectivity (graph theory)BuildingMaizeMereology
15:07
MereologyDirection (geometry)BuildingProduct (business)HoaxValue-added networkDiscrete element methodComputer animation
15:48
Multiplication signMetreGame controllerDesign by contractQuicksortFactory (trading post)YouTubePoint (geometry)Instance (computer science)RoboticsTelecommunicationVideoconferencingChainTerm (mathematics)Wind tunnelProcess (computing)Fiber (mathematics)Scaling (geometry)Virtual machineCASE <Informatik>MereologyPhysical lawPerspective (visual)Selectivity (electronic)Black boxInheritance (object-oriented programming)PrototypeSinc functionShift operatorBitSerial portConnectivity (graph theory)Goodness of fitCasting (performing arts)Different (Kate Ryan album)Standard deviationPay televisionGroup actionSpring (hydrology)BuildingArithmetic meanDigital photographyDivisorEmailCovering spaceAddress spaceComputer animationLecture/Conference
21:42
RoboticsOrder (biology)Design by contractMereologySoftwareQuicksortData conversionScaling (geometry)BitFactory (trading post)Virtual machineLiquidControl flowMobile appMultiplication signTranslation (relic)Row (database)CASE <Informatik>Computer fileStapeldateiDifferent (Kate Ryan album)Process (computing)Connectivity (graph theory)Hacker (term)CuboidSeries (mathematics)DivisorGoogolInheritance (object-oriented programming)MathematicsFerry CorstenPower (physics)Prisoner's dilemmaTerm (mathematics)Lambda calculusVotingVery-high-bit-rate digital subscriber lineComputer simulationLecture/Conference
27:30
Hill differential equationMultiplication signRoboticsObject (grammar)TelecommunicationInheritance (object-oriented programming)MetreGreatest elementEndliche ModelltheorieDirection (geometry)Right angleGame controllerGroup actionScaling (geometry)QuicksortIntegrated development environmentMappingSystem callSocket-SchnittstellePhysicalismVideoconferencingLevel (video gaming)BuildingMedical imagingWave packetDivisorTouchscreenPoint (geometry)2 (number)NeuroinformatikShared memorySoftwareDesign by contractWebsiteComputing platformBit rateJava appletIterationDifferent (Kate Ryan album)Service (economics)Graph coloring19 (number)Video gameReal numberDistanceDependent and independent variablesWeb 2.0Line (geometry)Connectivity (graph theory)Hacker (term)Goodness of fitStreaming mediaInferenceSimultaneous localization and mappingLoop (music)Set (mathematics)Artificial neural networkMusical ensembleHeegaard splittingPoint cloudRegular graphLecture/Conference
33:54
Condition numberGrand Unified TheoryMultiplication signLevel (video gaming)Figurate numberSoftwareAssociative propertyMathematicsMereologyRoboticsStudent's t-testOffice suiteAreaBitLecture/Conference
34:44
Content (media)Hand fanMIDIUser interfaceGroup actionOrder (biology)Online helpProcess (computing)BitRankingLine (geometry)FeedbackTerm (mathematics)RoboticsFactory (trading post)Arithmetic progressionComputer animationLecture/Conference
36:02
Sound effectGraphics tabletGame controller2 (number)RoboticsPersonal digital assistantTheoryCrash (computing)Term (mathematics)Different (Kate Ryan album)Type theoryParallel portOnline helpAlgorithmPattern languageDevice driverPressureReliefSet (mathematics)Arithmetic meanGroup actionPhysical systemMereologyLecture/Conference
37:56
Vector potentialExpandierender GraphStudent's t-testPunched cardRoboticsOperator (mathematics)Multiplication signOrder (biology)Connected spaceSystem callInheritance (object-oriented programming)Product (business)Vector potentialGoogolSoftware maintenanceInteractive televisionPlanningQuicksortDirection (geometry)Focus (optics)Mechanism designMeasurementGroup actionVideoconferencingSet (mathematics)Different (Kate Ryan album)CASE <Informatik>AlgorithmBitProcess (computing)FamilyNP-hardStress (mechanics)MereologyRoundness (object)Right angleComputer hardwareTransformation (genetics)Atomic numberFigurate numberData managementDataflowThermal expansionPressureSpeech synthesisTerm (mathematics)Core dumpCollisionGreatest elementPoint (geometry)2 (number)WhiteboardMatching (graph theory)Transportation theory (mathematics)Row (database)Computer fileLecture/ConferenceMeeting/Interview
43:04
Drop (liquid)Drag (physics)Numerical analysisString (computer science)Standard deviationAnalog-to-digital converterMetadataLogical constantCodeNeuroinformatikComputer fileMultiplication signTwitterXMLComputer animation
43:47
Cartesian coordinate systemMultiplication signBitoutputDifferent (Kate Ryan album)Regulator geneService (economics)CodeClosed setMilitary baseAndroid (robot)Public key certificateLevel (video gaming)Mobile appSoftwareSoftware frameworkIterationTwitterInternetworkingSoftware developerPoint cloudEmailGoodness of fitMathematicsEndliche ModelltheorieRemote administrationProduct (business)Order (biology)Boom (sailing)CASE <Informatik>Revision controlComputer hardwareExpected valueRoboticsComputing platformJoystickFitness functionGraphics processing unitMereologyCondition numberLink (knot theory)Connected spaceTerm (mathematics)Point (geometry)NeuroinformatikLie groupMusical ensembleLattice (order)Medical imagingHand fanExtreme programmingTheory of relativityResultantRoutingAddressing modeLecture/Conference
49:11
AreaRoboticsMedical imagingOrder (biology)SpacetimeNumberService (economics)Multiplication signKeyboard shortcutFocus (optics)Metropolitan area networkCartesian coordinate systemFactory (trading post)Streaming mediaDifferent (Kate Ryan album)Product (business)Form (programming)FacebookModule (mathematics)TwitterPublic key certificateVirtuelles privates NetzwerkBitIntegrated development environmentEmailConnected spaceVector potentialAutonomic computingNetiquetteAuthorizationMereologyTheory of relativityParameter (computer programming)Term (mathematics)FamilyStudent's t-testPosition operatorComputer clusterRight angleTraffic reportingProcess (computing)BuildingGame controllerElectronic mailing listLecture/Conference
54:36
Computer animation
Transcript: English(auto-generated)
00:21
weak spot for LEDs and completely abstains from HDMI adapters these days. He wanted to share with us the experience of attempting to build delivery robots in the past 2.5 years in the Bay Area and so yeah, let's give him a big welcome.
00:47
Thank you, Michael. So just show of hands, who here has built robots before? Well, it's quite a few people. What about autonomous robots? Anybody built autonomous robots? Still quite a few people. Well, today I'm going to be sharing with you the story
01:05
of how not to build autonomous robots. Over the course of the past two and a half years, together with my team, we built the world's largest robotic delivery infrastructure. We went from a concept sketch to a commercially viable service running in three cities. We've had lots of successes and one or two
01:22
failures. So over the course of the next 45 minutes, I'm going to be sharing with you a couple of different stories. First of all, I'm going to briefly introduce myself and I'm going to share the story of how we built robots, the different prototypes we had, the different iterations that we tried. I'm going to jump on manufacturing. We actually went to China and scaled up
01:42
our manufacturing, our production line. I'm going to share with you the story of how we did that. And finally, I'm going to talk about AI and all the magic that is artificial intelligence. So we'll be able to see how we were able to crack that puzzle. So without further ado, let's do the introduction. So this is me right here. I like to build things. I built my first website when I was 11 and I
02:06
built my first business when I was 13. It was an iPhone repair business that I was running out of my bedroom. I've been really, really passionate about building things and over the course of many years, I built a couple of different startups. One of them was a food delivery platform. We ended up
02:20
running three different cities and doing hundreds of deliveries a day by the time I was 19. So I got to experience startups pretty early on. I've been really enjoying that time. After this food delivery startup failed, I went to some cryptocurrency startups and then went to work for big corporations. And that was actually very boring. I adorned my office with some
02:42
supplementary graphics. After a while, I got a little bit bored of this corporate life. It wasn't really for me. So I decided to get a one-way ticket to San Francisco. So I ended up in San Francisco staying on a friend's couch, not really knowing anybody. And I was really fortunate to be introduced to an incredible group of people. And over the course of about two
03:02
and a half years, we started to take a concept, a sketch that we had, and we built up a robot. At first, it was something that barely even worked, but then we gradually got to something that worked a little bit better and better and better. After a while, we actually managed to build a whole fleet of robots. I think at the peak, we had 150 robots. So it was a really, really cool
03:23
experience. And during that time, I got to meet the lieutenant governor of California, how to figure out how to do manufacturing in China, and most importantly, work with an incredible team, who had a lot of fun with building these robots. So yeah, that's a little bit about me and what we were building. And maybe now we can jump into how not to build robots. So this
03:44
is our very first robot. It's a really small prototype we built. It was basically a shopping basket on wheels. There's a RC car there below. There's a shopping basket, and there's an Arduino Raspberry Pi. It barely worked, honestly. It was really, really hacky. And what ended up happening is that most of the time, we
04:00
just dropped off the robot in front of the customer. They literally just dropped it in front of their door, just to see if they would order food with robots. The answer was overwhelmingly yes, so we decided to spend some more time building out technology. There's a small, I don't know if you can see it here, yeah, there's a small orange holder. That's actually a phone holder. So our very first prototype, it had a phone sitting on top
04:23
of it, doing a video call, so that somebody can remotely control it from Colombia. We really started off small, really humble, just to see if it would work. And that's something that we do a lot of, just being really resourceful in terms of trying out things. After about a year of this, we moved on to
04:41
something that looks a little bit more like this. So we started playing around with the shape. We started playing around with the design. We noticed that people responded really positively to faces and to like things that look like people. So we actually built in a face. So we took this little animation that we built and we put it onto the robot. And this is actually really,
05:01
really positive. We had a lot of good responses from the community, a lot of great feedback. And what we've seen is that people really love to have robots that are kind of friendly. There was another company that deployed robots that looked like vending machines or almost like tanks in San Francisco. And they got banned really, really quickly. So we decided that we would do our best
05:21
to make sure our robots were as friendly as possible instead of threatening and scary. So that was a very important part of it. After another year, we ended up scaling up our production and we went to China to manufacture robots. And here, this is what we ended up doing. Actually a
05:41
robot, we built it entirely from scratch. We built it on chassis, our own cabin, our own compute module, just about everything. It was a really cool experience. That was me.
06:05
So yeah, that's a robot. This is the one we were rolling around the past six months. And we also had some failures in between, as you saw previously, this one. So we actually tried a couple of different concepts. So this was one of this was a Kiwi trike. We thought that maybe we can figure out how to have
06:23
robots do part of the delivery and then trikes do another part of the delivery. We also tried to do restaurant robots. We had robots that are sitting in the restaurant and bring food out from the counter to your doorstep. But what ended up happening is that it was actually pretty inefficient and people would wait a really long time for their deliveries. So it was very
06:42
important for us to try a lot of different things. We tried this robot, the Kiwi trike, did not quite work out as we expected. We tried a restaurant robot. We tried a box that would sit behind our robot. We tried a hub that would have like a bunch of different robots inside of it. So we really really tried a lot. And with every iteration we constantly tried new techniques. We
07:02
constantly tried new manufacturing methods. We really tried just about everything to see if we can make it work. And what we ended up building is a platform that was really loved by people. We built a platform that students adored. It was our primary demographic. We were delivering to college campuses and students really loved our products. We actually had people dress up as
07:23
Halloween costumes. We had entire classes go for Halloween in the kibibot costumes. So it was really really cool stuff. Had a lot of great support, a lot of trust from the community as well. And that's like coming back to the design, that aspect of having a friendly robot that meshes seamlessly within the fabric of a community is like super super important. We've seen
07:42
other robots around and they were maybe not as friendly. Maybe they looked a little scary. Maybe they had something that was a bit off or maybe a little too industrial. But having like a friendly robot that could become a meme, that was something truly revolutionary. Something that really changed the landscape. And as a matter of fact like these kibibots are the only robots
08:00
that are deployed somewhere in the world where they coexist day to day with a community, with people. Like you have some limited deployments of robots here and there. Maybe have a room bed home or something like that. But you don't have any large-scale deployment where you have robots and people living in the same city all the time. So of course it took us a while to figure out what
08:21
to do and how to do it. At first one of our models was to have robots deliver the entire meal. Like go from the restaurant all the way to the customer and we would have a robot do that delivery. Turns out it was pretty inefficient. People would wait like 60 minutes, 90 minutes for their delivery. And we realized that maybe automating all that was not the most
08:40
efficient approach. So what we instead did is a multi-model approach where we had people and robots. This is actually a really cool visualization that my team came up with. The blue lines are robots. So these are robots roaming around our Berkeley coverage area. And then the yellow lines are people. So how this would work is that people would go to restaurants, they pick up the food and they take it to a cluster. So they take it to a cluster where you
09:04
had a bunch of robots, they load it into the robot and then the robot would actually do the last few hundred meters to your doorstep. And because we were able to do this, we were able to go and build a platform that handled hundreds of orders a day with very very few people. I mean labor costs are really high for delivery. You'd be paying somewhere between five and thirteen
09:24
dollars to get a meal delivered in the US. And as a student that's like super expensive. It's not something that you can afford to do every day. And also there is a pretty big shortage of people who want to do this job in the first place. The churn is really high, people are leaving all the time because they don't like to like sit in the car all day and just deliver food. So
09:40
that's why we have this parallel like this multi-model approach where people are like biking around, they're enjoying their time outside and the robots are actually doing all the boring stuff like the waiting. So the robot would go up to your doorstep and it would wait for you to put on your pants, your shoes and to actually walk outside. So that way we were able to change the dynamic, we were able to change our deficiency from like one or
10:01
two deliveries an hour, as you would have for like a traditional delivery service, to as much as 15 deliveries an hour per person. So it made the delivery far more affordable and we were able to offer delivery at just one dollar a delivery, which is a cost that changes completely the way people approach delivery. And in fact if we look at our top 20% of users, they were
10:22
ordering over 14 times a week. So they were very very happy that they could get whatever they wanted very quickly. Of course not everybody was super happy so we did have some people that didn't fully appreciate the magic that is the KiwiBot. So we didn't have one person try to steal it, but they didn't get
10:41
away with it. We found them pretty quickly and they hid it in the trunk. Not a very smart move. We ended up finding it with GPS and also triangulating the Wi-Fi. So this guy decided to steal it because he doesn't like robots. I don't know why, but he was clearly very passionate about that topic and he
11:05
stole it and now he's in jail. So yeah, don't steal robots. So maybe some conclusions from our robot part, like from building robots, from figuring out like what to do and what not to do. A really important thing that we do a lot
11:20
in software but maybe not as much in hardware is iteration. Like we iterated through three major revisions and like lots of small revisions during a really small period of time. It was really interesting to see like that transition every single time we try something new we try it maybe for like 20 robots at a time. Like not our whole fleet, we just try for a small portion of our
11:41
fleet and that we were able to iterate really quickly and see what sensors work, what cameras work and just to see what we could do in order to grow the product. So it was very important to iterate. Communication. Communication is absolutely fundamental and not only communication like in the company or anything but more importantly communication with your
12:02
community. Because we weren't just building a product in isolation, we were building a product for people who live in a city who have an established life and we're kind of intruding into their life by bringing in a new product that takes their sidewalks. So communicating what we're doing, showing them what this is and what this robot does is super important. Actually
12:21
very early on our designs had no text on it, they had like no information, it was just like a basket case on an RC car and people were really confused. The police were like, hey what is this? So we had to add a lot of communication, we had to like put food delivery on the robots really clearly, we had to add a license plate with like a phone number that somebody could reach out to us. So communication is very very important when it comes to
12:44
robots. Also scaling hardware is hard, super hard. I mean it was crazy when we first started it was just Arduinos and Raspberry Pis and that did not scale really well. Like sure we could have maybe 10 or 20 units at once but then how do you handle updates, how do you handle kind of just weird things that
13:04
happen all the time. So it was really challenging to do this, we actually killed a bunch of SD cards, didn't really know you could destroy SD cards but you can and we learned a lot of things about hardware pushing it beyond its normal boundaries. So yeah, iteration, super important, communication is key, like getting buy-in from your community and scaling
13:23
hardware is super super hard. That's something we actually figured out how to solve by going into China. So how to do or how not to do manufacturing. So as every China story goes I hopped on a plane and I ended up in China and
13:44
it's really interesting to see because like you have this perception of China from the media, you have this idea of what it would look like but the reality is it doesn't look anything like what you would expect. It was a completely different world. It was at the same time Blade Runner and like the most modern city in the world and it was truly an awesome experience. I highly
14:05
recommend anybody who has the opportunity to go in and explore the world but of course the culture is a little bit different. We were surprised to see some things happening there. It was a weird dichotomy between communism and consumerism. So it's kind of interesting to see that sometimes but the
14:24
reason why we came to China is for manufacturing and there's no better place for that than Shenzhen. In Shenzhen you have Huashanbei. It's this huge market. It's a market that spans several city blocks and you can actually find anything and everything you want. We were able to get components super
14:43
quickly, super easily and you can spend days just walking through a single building finding different things. There were entire city blocks dedicated to like just LEDs or just connectors or just processors. It was absolutely crazy. You could really really really get lost inside of these mazes and what
15:02
was really incredible to see and something I've never seen anywhere else in the world is just how easy it is to get hardware, to get things, to get parts. It was super easy to just go in and get something and you could get it at one piece, two pieces or a thousand pieces like instantly and if you're
15:20
seeing anywhere else in the world that's super hard to do. So just by the virtue you're actually able to prototype things, you're able to build things incredibly fast. You're able to go in, you're able to commission a PCB and get all the parts almost instantly which is not something you see anywhere else in the world. And also a lot of the manufacturers have their booths here so these would be direct booths from the manufacturers so you
15:42
can say go up to them start talking to them and ask like hey can you make this product this specific way? Can you do it how I want it? And they'll be like sure why not and they'll do it for you. So it was really really valuable to just learn from these people, from the vendors here, from the manufacturers about like how to build things and it was actually really surprising to see
16:01
everything they have in stock. Two years ago we built an R-Escillation here that covered a tunnel with LEDs. We covered one of the tunnels at 34C3 with LEDs and we used this tiny tiny chip. It was a $5 ESP-266 chip that basically was able to control all your LEDs. And over the course of five
16:21
years up to that point I spent a lot of time figuring out how to build it myself. I played with Raspberry Pis, I played with PCA controllers over serial and like I finally managed to get a prototype to work but it was super clunky, super expensive and it wasn't very reliable. Then I go to China and I find that it's available there and much better quality, much cheaper, much
16:43
faster. So it was a really really interesting shift in perspective. It's something you can appreciate when you're abroad even if you're browsing like eBay or AliExpress it's kind of hard to appreciate just how much selection you have and how you can find just about any tool, anything you need to find. So it's really really incredible. But these markets are cool but
17:03
what was even cooler are the factories and during our course in China we were able to visit a lot of factories. All these factories they were super welcoming, they always loved having you over, they invited you to really really luxurious dinners, we have way too much food and it was a feast and celebration
17:21
every time. Actually relationships are super super important in China like a lot of people in the West like they have contracts and they say like okay this is the terms of the contracts. Well China you do sort of have contracts but they don't matter as much as relationships. Like when you have a relationship with a manufacturer you have to like always like go to dinner with them, drink beer, smoke, go to KTV like it's a really involved
17:42
relationship and you're only able to have a good communication based on that relationship because if you don't have a relationship they kind of forget about you and we actually had a couple of instances where manufacturers ghosted us like they had a critical component and they just stopped answering our emails, they stopped answering our WeChat, they just completely ignored us and for some pieces they were completely irreplaceable like we
18:01
could not just go out and find another factory to produce a specific part the way we want it and the only way you can ensure that this doesn't happen is by really explicitly making sure that you have good communication, a good relationship with that manufacturers. It's super super important. This is one of the factories we worked with. It's really crazy. I mean we went there and we're just absolutely blown away by the
18:24
scale of everything and also blown away by how manual everything is. There's actually audio here. Everything was super manual people were just like there with minimal or no protective equipment whatsoever just like building things that look like they were made by robots or machines but they were in reality just built by people with the hands which is super crazy to see and
18:46
there were a lot of Blade Runner-esque designs really bizarre contraptions there in this factory. This is our fiberglass factory. The way we built our casing was actually prototyping it first in carbon fiber
19:01
sorry fiberglass and then moving on to a mold in carbon fiber and actually Scotty he made a really cool video on YouTube so if you search for hockey stick factory on YouTube you can see a huge video where my buddy Scotty actually goes with me to this factory to discover how they make this mold and how they make these carbon fiber things. It was actually really
19:21
crazy to see it was cheaper to make a carbon fiber mold than it was to make a plastic mold so since the tolerances were a little bit different since like the process was a little bit simpler you were able to make a mold that was very very strong and very undestructible without necessarily having to have all of that expense up front for like a plastic mold. So yeah that
19:42
was our fiberglass factory really exciting stuff really crazy scale. These folks like the first night we came there we arrived at like 8 p.m. and there was a hundred people in the factory just like working at 8 p.m. Really crazy to see. This is another factory we worked with so this was a metal factory it was actually really really really interesting to see how they
20:02
built all these things and at one hand you can build super complex things you can build super complex designs but on the other hand we got surprised a couple of times by being unable to manufacture really simple designs and it took us a while to get a grasp of like okay so we can make really complex metal that's bent but as soon as we add a well to aluminum you
20:23
start to have a big big problem so we had to like change a lot of our designs we had to really adapt to the way things were being made in China and sometimes you could adapt yourself but like at an insane cost so it was better to adapt to the way things were being done there so again very very interesting to see how things are done and no protective
20:43
equipment this is like a two-ton press and his hands are millimeters away from it so yeah it's a different world out there very very different another factory we visited was a PCB factory so this one actually has a really interesting story this factory is not in Shenzhen it's just across the border
21:02
from Shenzhen the city actually passed a law a couple years ago that has very very strict environmental policies so you're no longer able to do PCB manufacturing inside the city anymore so we actually had to drive for a couple of hours outside of the city and over there was huge plant and this was kind of semi automated semi hand handmade sorry where parts of the
21:25
process were done by hand as you see here but then parts of the process were done with the machine so they have this giant machine which is basically like a black box you can't really see inside of it but you had a bunch of chemicals and just like take a PCB and just like move it forward through a chain so it's really interesting to see and this factory also
21:42
had a really quick turnaround they had a three hour turnaround if you paid a premium and like standard it was 24 hours you could also ask them to do PCBA so you can actually get them to assemble the PCB for you and we ended up doing that for some of our PCBs we'd like give them a bill of materials and we'd give them our designs and then they manufacture it we
22:01
actually got in a little bit of a situation with that because we sent them some designs we sent them some parts that we wanted to put in our PCB and it turns out that one of these parts was unavailable and they didn't tell that to us until it was almost Chinese New Year so we had to scramble a bit to find another solution it was very interesting to see how you would
22:20
deal with these factories there were some even cooler factories I think the coolest factory visited was a battery factory where they made lithium ion and lithium polymer batteries it was almost entirely automated you had giant films of things going into a machine and then you had all sorts of liquids and powders it was like combined together it was super super cool they
22:42
didn't allow us to film it unfortunately there may be only a dozen such factories in the world so they're very protective about their technology but the scale of how quickly they're manufacturing these batteries was just incredible they were manufacture them at a crazy crazy scale so all these factories are cool but actually building things is even cooler so we
23:01
ended up partnering with a contract manufacturer I was really fortunate to find one through my network otherwise I would have been totally lost a couple of days before I ended up going to China I found a contract manufacturer that liked to work a startup so like small-scale people and we ended up working with them to build our first batch of 50 robots it was really interesting to see how different our designs were to what they expected so
23:24
they expected things are really ready they were very explicit very clearly specified but we didn't have that the difference between manufacturing in the u.s. for example against China is that in the u.s. like it's a super long process and the back and forth takes super long just to get an
23:40
idea of what kind of files they need whereas in China you're able to like sit down directly with the engineer with like the person in charge and you can figure out what they need and they can help you out instantly actually just here I just want to show you one thing so this is my designer Alejandro and he was translating from English to Chinese with his phone with a Google Translate and it works surprisingly well Google
24:01
Translate actually is not blocked in China for some reason we were able to communicate almost all the time with that also WeChat has like a built-in Translate feature so WeChat is like this universal app that every in China uses and has this built-in translation feature that can translate new text automatically so it's really really cool to see how that worked one
24:22
question that we get commonly asked is like how do we find our manufacturers how do we build these relationships so about 20% of that was through Alibaba so our fiberglass manufacturer we quoted like 30 different manufacturers with the cheapest one of course was far more expensive than we expected and we ended up working with them 20% like for
24:44
example our chassis it was built with companies that we already had a relationship with so we were just able to continue working with them and then 60% was through just references so just like the network just sort of getting to know people and talking to them and saying like oh hey who did you use for this or this or like how did you make these PCBs or just
25:02
getting a conversation going so having that kind of network was really really helpful in order to build these robots so as you can actually see over here our design this is what we had when we came into China when we left we had our own compute module like super sophisticated but this was like a Raspberry Pi a PixHawk and a voltage converter like a DC to DC converter
25:25
that's pretty much it as you could see it's not very reliable it would break a lot so it took us quite a while to translate this into something that was manufacturable so thanks to the dedication of my incredible team that we were able to do that and we kind of did not know we were doing so
25:44
we ended up having all of our parts and all the components ready just days before Chinese New Year so we actually had to do all of the assembly ourselves we didn't have any Chinese workers who could help us do it so this is our team just assembling things in the factory like one or two days
26:01
before Chinese New Year so that was very very interesting we kind of hacked or tried to hack Chinese New Year we assembled all of our robots literally days if not hours before Chinese New Year and we shipped them out and everything was great except our robots got stuck in customs we had a trademark on our box and the customs agent they open the box and saw more
26:24
trademarks on some parts we have 3d printed parts and they're like no this is not gonna go through without the proper paperwork so our robots got stuck for three weeks in China which was really fun a little problematic so yeah those kind of things happen you have to be ready for it after we
26:41
received our robots in California we had to spend another like maybe one or two months refinishing them redoing some parts tweaking them flashing them so it was still a lot of work to get them to work the pieces we shipped out of China was maybe just like a case with most of the electronics and but
27:00
not all of it so we still have to do a lot of tweaking over back home and of course all this wouldn't have been possible without an incredible team so I was really fortunate to be with some really really passionate people some people who would work four months in a row continuously without firstly taking any breaks we had plenty of opportunities to go and take like the high-speed rail go to Shanghai or even Tokyo but we all stayed in Shenzhen
27:24
and spent a lot of time together building these robots it was a really arduous journey so maybe some conclusions for scaling manufacturing some of the failures we've had in relationships I mean relationships are super important like super super important China far more important the
27:41
contracts if you're able to have a good line of communication with your manufacturer that really really helps out if you don't things don't go bad I've had manufacturers that ghosted us we have manufacturers that completely ignored us or manufacturers that just like replace components because they just felt like it so relationships super important don't hack
28:01
Chinese New Year we tried it doesn't work it's a thing China just shuts down for like two or three weeks so it's really really important to respect that people buy tickets to go to their hometowns like months in advance and they're not going to move it for just like some pesky thing that you're building especially if it's like some small-scale thing so yeah don't try to hack Chinese New Year did not work out well for us also do
28:24
with the team while I was in China I saw a couple of solo entrepreneurs try to build their own thing and it was super super hard super stressful having a team is really great especially if like a foreign place where you don't really know anybody having that team there together to support you is super super important especially since you can multitask you can like split
28:42
responsibilities and do something together it's a really really important aspect so that's how we manufactured some of the failures we've had now let's talk about how not to build AI so as we all know AI is magic right just as blockchain and IOT in the cloud it's it's absolutely magic right well the
29:04
reality is it's not that magic so we decided to have a very pragmatic approach to AI we said let's not do anything crazy let's just make something that works so our very first iteration of our robot was this this is like the
29:20
control panel for a robot it was super simple we had a video call coming in from the robot on the left over there it's like literally an iframe super simple stuff and on the right we had a map on the bottom we had some controls you can move the robot forwards backwards it was very very simple it barely worked on the robot we had an Arduino Raspberry Pi all running like in Python and then the serverless Java a communicating
29:41
over of web sockets but this barely worked so we decided okay what can we do maybe we can build an autonomous robot maybe we can build something that would work entirely by itself we actually did that so we built a robot that could go entirely by itself it was fully autonomous and it was actually really
30:00
cool the way we built it is we had a pretty beefy computer inside we had Nvidia Jetson tx2 on that we were running Ross and instead of Ross were running TensorFlow and a couple of other technologies we had YOLO for object detection and some other cool tech that I'm not entirely familiar with since I didn't write that code but over here what the robot did is it looked
30:21
at objects so it was detecting objects it was also measuring the distance to the objects and it also had an inference neural network and you can see that on the top left of the screen here basically based on trained data it would know where not to drive into and it would try to plot a path based on 12 different directions it could go into so it had 12 directions and it
30:43
would go in the direction which had the highest probability of not colliding with somebody or something and this worked okay we were able to get like 99% autonomy but the problem is like since we're doing a commercially viable delivery service that's like offering deliveries to regular people and it's
31:01
not doing something in the lab we really had to do something that worked all the time and the challenge with this is we still needed to have people in the loop we still have to have people who looked at the robot to make sure it would actually not crash and what happens if you have something that's fully autonomous and people assume it works well when it doesn't work well instead of looking at the screen and being ready to take over they're
31:23
just looking at the phone and Instagram so this approach wasn't the best one and instead we decided to use supervision approach so we spent a lot of time building this so this is our supervisors console and it's actually a really really cool platform it's a platform that allows you to
31:40
connect to a robot and the robot streams to you video over web RTC or like the 4G network and you're able to control it over web sockets so the way to work is you would have a supervisor that just sets waypoints for the robot to follow so the supervisor would just like click on the image and he or she would tell the robot to like move I know 10 meters at a time so typically they'd set waypoints every 5 to 10 seconds it was
32:02
a very interesting approach we tried a couple of different approaches we try to do SLAM that really did not work out for us it took too much resources and it didn't give us a significant gain we tried other things as well we tried we tried traffic light detection so we tried traffic light detection there are
32:20
some amazing models available online some great github repos the problem is yes they do work on a very clean data set but when you actually have data and we actually have a real-life scenario where you have like glare you have rain you have weird situations you have homeless people like it doesn't really translate that well in the real world so we kind of struggled with that
32:43
instead we actually had a more middle-ground approach so we are able to detect traffic lights really well but we're not able to detect the color really well or the which kind of signal is giving so instead of what we do over here this automatically zooms in to traffic light so it's very easy to see this video actually that you're seeing is transmitted over a very
33:03
low frame rate very low bit rate as well like I think we're doing 480p at a hundred kilobits a second so it's very very low bit rate and when the robot isn't moving we actually make it go black and white and even lower frame rate so that it doesn't waste resources so yeah it's pretty cool
33:23
stuff over here on the top left we actually have our latencies so we managed to build the infrastructure that allowed us to supervise these robots from Colombia for like 200 milliseconds in less than 200 milliseconds so it's like a blink of an eye it was a really really cool technology it worked over 4G and we did a lot to optimize that we had also
33:43
map over here so this map is really really cool a lot of people ask us like hey did you do mapping did you map out your environment did you need to have something there before you came into a new place and well the
34:00
instead is we actually map out the network conditions so we wouldn't neck map out the network conditions of city and we'd say okay these areas like over here this is like high latency we should avoid those areas because the robot could get stuck there and it's actually very easy to see the network conditions changed continuously like you didn't have the same network conditions every day all day all year they'd actually change
34:23
every few hours so it's something that took us a while to figure out so of two or three robots per supervisor in Colombia and we had like just a bunch of people typically students who just be working part-time and they were sitting in an office in Colombia doing this of course the
34:43
press found out about this and they wrote a very small bit of text in this article and saying like oh Kiwi hires Colombians and pays them $2 an hour and people were really frustrated about that we had a lot of interesting feedback about that but what was interesting to see is that this
35:04
technology actually helps people in Colombia if you're there it's a third world country is a developing country you can get a job at a factory you can get a job at like a textile shop you can get a job maybe McDonald's but there aren't that many tech jobs per se the biggest employer in the country is a phone support company so like when you call in to support land
35:24
it's support line you get connected to Colombia sometimes and that's the biggest employer in the country so in order to get like a tech job it's really really hard and giving people the ability to like go and supervise robots it's something that helped them get something on their CV it helped them step up has helped them learn a little bit more about the
35:41
technology and help them progress in terms of their careers our lead AI guy he actually started off as a supervisor and he went up through the ranks and then he ended up leading the AI and robotics team so it was really interesting and really inspiring to see how that transition happened and we managed to get our technology to work so well that we can do this so we were
36:12
able to get it to work with up to eight seconds latency which meant that you can control it literally from anywhere in the world so even from like an airplane above the Pacific Ocean so it was a really really
36:21
interesting experience and we really try to make it simple so in conclusion for AI we realized that the best approach was to keep it simple we tried a lot a lot of different approaches like we tried the traffic light detection we tried to yellow pad detection so I didn't mention that so in
36:41
Berkeley you have these accessibility ramps and you have yellow pad so that blind people can actually like feel them and see them easier so we built the algorithms detect that and we thought that okay maybe if the robot is stuck in the middle of the intersection you can automatically detect this yellow pad and like navigate to it it's in the purse that worked in
37:00
theory and practice did not quite work we tried segmentation so that was an approach that works okay but some weird things broke it so for example any lamp posts or bicycle posts they were like crash the robot because it didn't see it so yeah keeping it simple was the best approach really not going too
37:21
crazy and the approach we ended up going in the end was to have it more of like a driver assist type like a parallel approach parallel autonomy approach where our robots would help people the same way that cars would help people stay in lanes or have cruise control or like with parking assistance that's kind of the approach we're having I think long term it is going to be
37:41
possible to build a robots more autonomous their companies like starship that have some interesting ideas about how to solve that but I don't think it's quite something that could be scaled to every city just yet another really important thing is the lab does not equal the real world so there were many many great examples of fantastic research papers from some
38:03
great groups and they were great with very polished very clean data sets but they did not work when you deployed them on a hundred robots they were all different that all had slightly different camera calibrations that all had slightly different hardware it all had slightly different chassis it did not
38:21
really translate as well so these algorithms these lab best-case scenarios really need to be modified a little bit what else yeah one thing maybe jumping back to the keep it simple we decided to put in a very simple safety mechanism so the robot actually breaks if it sees something
38:40
within 50 centimeters in front of it so it's kind of like a last measure precaution as you saw before there's the video like you can supervise the robot from anywhere in the world with a lot of latency but having this 50 centimeter like hard brake actually saves us in case the robot loses connectivity or the supervisor is no longer able to supervise a robot so it's always breaking 50 centimeters away from any like collision with like
39:04
a baby or a car or whatever so the approach we really thought about is like how can we expand human potential there's a lot of talk about like AI taking jobs or AI like replacing people's roles but we sort of kind of try to do that and it didn't work like we try to build robots that were
39:22
fully autonomous that went from the restaurant to your door and that didn't work people were waiting a really long time these robots required and singing them out of the maintenance so we ended up going for an approach that was far more parallel autonomy where these robots were like helping people to do more same way the supervisors are getting these assistive technologies were they able to set a waypoint to do the path
39:44
planning and the robot does the motion planning on board we also have the couriers who would just load food into the robots instead of the robots picking up food from the restaurant directly so really expanding human potential I think that's where it's at and over the course of the past century we've seen a lot of examples of this like we've seen operators of
40:01
elevators like before elevators had operators who would make it go up or down and now they're fully automated we had switchboard operators who were the connect phone calls now we can make a phone call to anywhere in the world instantly for free so we're seeing this transformation of work transformation of the way things are done and I think this is just the start
40:20
the way I see these robots is really meshing into the fabric of our societies and solving physical transportation like sure you can move bits from anywhere to anywhere in the world but can you move atoms it's really expensive to do that it's really hard to do that that's where I see robots expanding human potential so conclusions what we did was really cool
40:46
and I think it was a cool experience one thing that we realize is that tech isn't the hardest part right we like spend a lot of time figuring how to build something but figuring what to build is sometimes very important as
41:02
well and I don't think we spent enough time asking ourselves that question we kind of went in all sorts of directions we didn't focus as much on making the best product possible we kind of tried things that were really weird and not well thought out so like having that more long-term thinking like thinking what should we build is very important because like how you
41:21
can just like look up a tutorial on Google and figuring out how to build a robots it's not the end of the world one really important thing for us was interaction so interacting with people figuring out how to make the door open when you actually receive your food super hard that's super super challenging to do or actually the only robot that opens the door for you
41:41
other companies like starship for example they have a button that unlocks a solenoid so it's like the experience is not quite there you have to bend it down you have to figure out if the door actually opens so we spent a lot of time a lot of effort in order to optimize that experience to make as smooth as possible for people also one thing we didn't figure out is financing come back to it in a second that was really really hard to do as
42:04
well so like tech you know not the hardest financing figure out like manage cash flow super important but I think the most important thing is to work with a great team if you're gonna be spending a lot of time with people who you eat live and breathe with it's really important to choose a team that you really connect with and then share the same passion as you do
42:24
because you could be miserable making an amazing amount of money but if you're with a really crappy team with a high turnover it's really boring I was really fortunate to work with one of the best teams in the world and with the course of the past two and a half years we managed to do quite a lot and just last month we actually got an article in the New York Times so
42:41
there's a really big accomplishment for our team and we got to share with our families my mom was really proud so a lot of great traction and a lot of great coverage but unfortunately actually round of money so we kind of things so I decided to leave and start my own thing instead of doing robots I
43:03
decided to do data so now I'm actually focusing more on building a tool that helps you tell stories with data so this is Glint this is a data storytelling tool you're able to drag in some files and it tells you the story of your data without you having to write any code so my hope for
43:22
this is to allow anybody in the world without any knowledge about how to wrangle data how to clean data how to analyze data to be able to tell stories with their data directly from their computers I'm a managing a tool where you can say oh in December there were xx visitors to Congress or last
43:41
summer we had xx sales and automatically fill that for you that's kind of what I'm thinking about if you want to join the effort there is a github I'm more than happy to have any contributors and if you have any questions or comments we're happy to answer on Twitter or here in person thank you so as usual
44:15
feel free to line up in front of the microphones or at your question to the signal angel over there that already has one it's all the way down go ahead
44:27
okay here's a user of your service who apparently got an email from you that announced some changes so he's wondering what's up what you're going to what you're planning to do there whether you're continuing your service or closing shop yeah it's unclear we ran out of funds so I think the CEO is
44:45
still trying to figure out what to do with that I wish him the best of luck but I ended up leaving with a lot of other people so we have like 50 people in November now we have like 10 people left in the country so it's very ambiguous what's happening but yeah I left I prefer audio okay no it's works
45:07
but I'm little bit confused because you are presenting a 1970s concept of a manipulator because a robot is something that works by itself a manipulator somebody who has some joysticks and most things so it's nothing special you just have a interlinked internet link for
45:23
manipulator and in the 70s there were cables so what's the special thing yeah that's a good question I think the magic here is connecting everything together figuring out first of all how to build these robots how to build a reliable connection and how to build a platform that works and as I mentioned like the how that's not that interesting it's more of the what you
45:42
build is that experience were you able to order anything you want and any time and get it delivered in under 30 minutes virtually for free so for example evil people could just buy a remote control car put a bomb in it drive under a police car and Mike boom it's the same same use case you deliver something by a lot of pollution there yeah you talked about
46:06
quickly and rapidly and that's very good model for conceptual stage and software were you in the stage where you were leasing your hardware with iterations because usually a big stack of certification has to come in between
46:23
so I'm not entirely sure are you asking if we got certified at every single release I suppose yeah what level of likes recertification was it totally released so you had to meet like regulations for each iteration yeah absolutely we didn't really get certified because we're not building a
46:42
hardware product for consumers so we're not selling it to anybody we're operating ourselves so we don't fit under the same kind of requirements however we did have to have some permits and part of the conditions of these permits was that we had to meet some expectations but they're very very basic and they weren't rigid like an FCC or a CE certification for
47:05
another question from the Internet why did you develop different applications for Android and iOS for the consumer application I haven't got any more details we just did I mean we had first iOS application I mean 80% of our
47:25
customers were using iOS so we really spent a lot of effort like polishing that iOS experience making sure that worked and at one point our Android app was working super badly so we decided to kill it and everybody was really really pissed off extremely pissed off so we actually reintroduced it and we
47:44
started catching up with features to the iOS version internally all of our apps are built in react and react native so we had like a common framework for all of our internal apps but we didn't have that experience we're expecting the quality of the experience that we're expecting from a consumer app using react that's why we had two different code bases yeah have
48:10
you tried different methods regarding perception for example LIDAR, radar and what are your conclusions from that yeah we tried LIDAR we tried the cheap
48:23
LIDAR we didn't try like a really high end LIDAR so the challenge with having like point clouds is that you have to compute you have to spend a lot of time computing we were using a relatively low power device and it was running from batteries so we didn't have the luxury of having like 10 GPUs in the trunk of a car for example so that was one approach one question
48:41
another question is how much does it cost so LIDAR is they can cost ten thousand one hundred thousand dollars our bill of materials is around two and a half thousand the last versions are two and a half thousand so all of our sensors were very minimal in terms of what sensors we tried we tried a lot of different sensors we tried ultrasonic sensors we tried near for
49:03
near field infrared sensors we tried other sensors yeah we tried a lot of different sensors we ended up just going with cameras so we have cameras we have six cameras on board all them are like full HD we stitch them into an image on our compute module and then the supervisor decides which
49:24
portion of the image they want stream so they can like manipulate with their keyboard to see which portion of the image is streamed so we don't stream the whole image we just import of it the really important part for us was to make something that's viable that can be used commercially so I'm sure LIDAR is really cool but I'm not seeing any commercial deployments of LIDAR
49:41
based Thomas vehicles or robots yet thank you yeah yeah you've tried out many different concepts how to do it and you and you saw that your company ran out of money do you still believe in the business concept of the robots
50:01
delivering packages of food who knows I think I think it was a great learning experience we learned a lot we had a great team and I think we'll see some concept of robots maybe not exactly what we were building maybe something a little bit different but I think it's a little bit inevitable especially with the rise of self-driving cars maybe we'll have cars delivering packages
50:21
instead of robots I'm not entirely sure what would look like I could tell you Amazon they bought one of our competitors dispatch labs so they're making a big bet on this there are two delivery companies in the u.s. Postmates and DoorDash that are building products internally also for with delivery robots and also the company's like FedEx are also building delivery robots and then we have companies like starship for example which are
50:42
building robots and doing b2b with companies all over the world so yeah I think we'll see some form of delivery robots I don't know if it's gonna be what we had or what somebody else is gonna have whether any safety certifications you have to satisfy in order to operate around people know so
51:07
well the thing is in the u.s. like it's kind of just do whatever you want it's very different from Germany you can kind of just do things and you can do them until you get in trouble so we kind of had that approach don't ask for permission ask for forgiveness we ended up having to have a permit in the
51:22
cities we operate in but it was very simple it was like okay you have to have lights you have to have a phone number and you cannot go in these areas that was essentially all the authorization all the permitting and certification that we had wanted to ask did you try other markets like
51:43
autonomous drive is very hard even way more than manage it fully so like perhaps elderly care like you could use this robots and etiquette where we have a controlled environment where everything is the same did you search after other markets where it's less yes it's a great question yeah there is a lot
52:06
of potential for markets like elderly care for example also for mail delivery for applications inside of factories we had a couple different medical companies that reached out to us from like hey we want to move items move packages inside of our facilities so we did have a lot of interest we try to
52:22
keep a focus on the consumer space like really building a consumer experience that worked out before branching out into these more b2b approaches we are elderly character to be one of them I think one important thing about elderly care and services like Meals on Wheels for example is that human contact so I think people who are maybe not seeing as much of
52:40
their family of the relatives they really cherish that connection to get from people who deliver them food so I think it's a multifaceted approach they have to have you have a couple different considerations with these kind of services for the elderly for example what kind of personality do
53:01
Chinese entrepreneurs have I think as I mentioned like it's really important to have relationships so they were very interesting they were very deeply in belief of their government they had nothing bad to say about it they believe that it would bring them everything the best possible even though
53:22
they still try to access Facebook and Twitter with VPNs so they were very very loyal to their governments they were very very diligent if they committed to something they would usually deliver on that they really wanted to make sure you had a good experience and also what we saw for example with building up these relationships like the first few times
53:41
we talked they would try everything to impress us so we got taken to these ridiculously expensive restaurants to make sure that we were welcomed well and make sure everything was right I had an interesting episode earlier this year I was gonna go to Burning Man and then all of a sudden one of my colleagues had an argument with my manufacturer about whether Hong
54:03
Kong is another country or not and I ended up having to go to China to deal with our manufacturer instead of going to Burning Man to make sure we're aligned in terms of our beliefs sometimes it's really delicate you cannot like talk too much about the government there you can't talk too much about politics it's best to stick to business and yeah focus on
54:24
building a product I guess this was it thank you so much