Hi, I'm Aman Mathur, and I'd like to talk about ParaCosm. It is a framework we've developed that enables the systematic testing of autonomous vehicles in simulation. In the here and now, we have many cars that feature partial autonomy or driver assistance features.

Fully autonomous vehicles, though, or level 5 autonomy, is coming soon. The testing of these vehicles, therefore, is quite an important topic. There have been some fatal crashes involving these vehicles in the recent past. In fact, just a few months back, in one of the first ever live-streamed autonomous car races,

a contestant randomly crashed into the wall. Rest assured, there are some very well-placed public safety concerns related to these vehicles.

If you take a step back, these are extremely complex machines. They're supposed to navigate amidst other vehicles and traffic. They're expected to detect, recognize, and follow road and traffic signals. And they're expected to work whether it's day or nighttime, even during adverse weather conditions like heavy rain or snow.

Now, real-world tests are necessary for verifying the safety of these machines. However, these are time-consuming and expensive. We do not have precise control over all aspects of the real world, and even if a problematic scenario is found, it is difficult to recreate it. Therefore, there is a need for quick and systematic testing of autonomous vehicles

in a wide variety of situations and environmental conditions. As you may have already guessed, we already have a solution to this problem. Simulations. All aspects of a simulation can be precisely controlled, and many tests can therefore be quickly performed.

Traditional simulators such as Gazebo for Ross have fallen behind in terms of visual realism, so we have shifted to game engines now. Popular examples are Carla and AirSim. These both work on the Unreal game engine and provide some pre-built environments.

There are also works that use the GTA V game engine, and Scenic is one example. One common problem though is these offer environments which are fixed, and only small nominal changes are possible to the environment or the behavior of actors within them. It is possible to handcraft some new environments using these individual pieces of 3D assets,

but this is rather painstaking and time-consuming. This brings me to our contribution. We propose Paracosm, a framework that enables the easy design of parameterized environments and test cases.

A test configuration in Paracosm consists of a composition of the autonomous vehicle, other actors such as vehicles, pedestrians, etc. The world, which encapsulates the environmental conditions, and road segments.

Alongside this, there can be monitors that evaluate the qualitative and quantitative properties of a test. The system under test, which can be a black box, it interacts with or within the test configuration.

Additionally, we propose some test generation strategies and a related coverage criteria. All of this happens inside a simulator, which is the Unity game engine for us, but we also support outputs to other formats such as OpenDrive so that this can be simulated elsewhere.

What this gets us is the possibility to design tests quickly and easily. For example, we can set up a short program with a test vehicle, a pedestrian, a simple world with dim lighting, let's say to start with a straight road segment, and we can check for example if this car stops for the pedestrian crossing the road or not.

The simulation proceeds and we in fact see that the car does stop. We can very easily do the same test with different light conditions or four lanes instead of two.

Paracosm is based on a synchronous reactive programming model. This means that components such as road segments, vehicles, or pedestrians receive streams of input and produce streams of outputs over time. Components also have an appearance such as the 3D mesh, a physics model which includes information on the mass, joints, etc.

and a behavior which dictates how they act in the world. Now let's start by looking at a simple example which would demonstrate the expressivity and simplicity of our framework. We start with two test variables, light and nLanes.

Light can take any value between 0.2 and 1 and nLanes can be 2 or 4. We obviously want to use these variables to generate many environmental variations. Next we have a world which is responsible for environmental conditions

and we pass it light for the light intensity and give it a constant zero fog. Next we make a straight road segment of length 100 and pass it nLanes for the number of lanes.

Next we place our autonomous vehicle. We place it on the road segment R on lane 1 at a distance of 0.1 normalized on the length of this road. We give it a model and a controller here. We're doing a very simple test. We have a convolutional neural network which would take simple RGB camera images and provide steering and throttle commands.

This goes here. Next we have a pedestrian. Just to give you a flavor for writing reactive behaviors, let me show you what this pedestrian's behavior looks like. This is it. Quite simple. What it does is it looks at the car position and whenever it changes,

it checks whether it is less than a certain distance away from this car. And if this is the case, it starts crossing the road. Now this may not be a very smart pedestrian, but it is a smart test. Next we have the collision monitor and we give it the autonomous vehicle B.

Internally what it does, it just checks if the collider for this vehicle B is empty. Finally we run this test. We pass the road segment vehicle into the environment. The test parameters are the light and end lanes. We have a collision monitor and we do this for 100 test iterations.

Now let's look at a snapshot of a few iterations of our program. As you remember, our test parameters were the number of lanes and light intensity. Our test generation strategy therefore assigns different values for these

and we get different environments in each iteration. Notice the collision. Our collision monitor registered this and we have the exact parameter values and environmental conditions that led to that collision. Moving on, though we have something quite straightforward here,

it is obviously possible to make something more complex. We support components such as cross intersections, tree intersections and roundabouts. And using simple composition and iteration we can get a complicated urban setting such as this here. Next we used simple RGB cameras but it is obviously possible to have depth cameras

or even specifications of cameras like varying focal lengths for example. We already alluded to the fact that the pedestrian's behavior can depend on the test vehicle's actions but in general, actors' actions can be linked together using a reactive framework

to create more complicated emergent behaviors. Moving on, what is critical to having useful and varied tests is the test generation strategy. Our simple example just had two test parameters but it is quite obvious to see that we may want to have several test parameters in general.

For a holistic analysis of the system under test, we need to sample from a large parameter space well. Now there are two kinds of test parameters. First is continuance, for example the light here. Other examples could be the focal length, the max speed of the car, the coefficient of friction on the road, etc.

And then we have discrete parameters. The end lanes was our example initially but it could also be colors or makes of cars, driving direction, etc. A coverage criteria needs to take both of these kinds of parameters into account.

For continuous parameters, we want to ensure that there are no large caps in the parameter space. That is, the samples have a low dispersion epsilon. We cannot sample forever so we want that for a fixed number of samples, there are no large holes.

Random sampling in fact does leave large holes when the number of samples is fixed. We therefore use quasi-monte-cardo sequences, specifically the Harpen sequence here. Next, for discrete parameters, we use k-wise testing. So instead of testing all combinations of discrete parameter values,

we try and test combinations of k discrete parameters. These k-wise covering families are quite successful at finding bugs in a wide variety of applications with just a few samples. Moreover, random sampling generates these with high probability.

So epsilon from continuous parameters and k from discrete parameters gives us our coverage criteria. We can also support other test generation strategies such as those based on cost functions. For this, we first sample using a high coverage strategy and then do a few iterations of local search around the most interesting tests.

An example would be finding higher speed collisions near already high speed collisions, or finding new collisions near narrow escapes. The monitors, as we saw before, can evaluate cost functions for these.

Let's look at a few case studies now. To start with, we have a study which is very similar to the one, to the program we built earlier. Instead, we parameterize the walking speed of the pedestrian on the x-axis

and the distance from the car when the pedestrians start walking on the y-axis. We compare random and Harpen sampling here. For random sampling, as you can see, there are some large empty spaces in the parameter space. Things seem much better for the Harpen sequence. It's much more uniformly sampling the parameter space.

On the same test, we can use local search to find collisions with higher speed. The baseline is 100 samples of the Harpen sequence, as before. And for local search, what we do is we start with 85 samples of the Harpen sequence and then we do 15 searches around the neighborhood of already high speed collisions that we observed before.

We find two times more failures. This is not surprising because we're already sampling close to failing cases. But we do reach our target of finding a higher speed collision. Similarly, we can use local search to find new and unique failing cases

around situations that are near misses. For the same case, we start with 85 Harpen sequences and do local search around almost failing cases so the distance between the autonomous vehicle and the pedestrian is very small.

We're able to find 30% more failures. These are unique failures, though, which was our target. As you've seen, writing tests in ParaCosm is quite simple. I'd now like to give you a sense of the variety of tests that can be written using ParaCosm.

We have a few examples here. So, first we have a test vehicle following a lead car on the same lane over many test iterations. We're varying the fog intensity and the color of the lead car in each test iteration. It is easy to observe the variations in the behavior of this vehicle over different conditions.

Next, here what we're trying to do is make a turn on a T-intersection in a similar high fog situation. Our car is having problems. Here we have a car in an urban environment.

There are many cars on the same lane as us and there are some cars coming from the other direction. As you can see, our car also has problems here. Now, none of the case studies that I just showed you were a whole lot larger than our initial example in terms of lines of code.

I have just focused on closed loop simulations thus far. That is our autonomous vehicle senses and operates inside our virtual world. It is also possible to extract dash cam images from the simulator and test other things separately.

For example, road segmentation or vehicle detection. We're able to find some unique bad cases which are very similar but different from the nominal scenarios for road segmentation here. So a bad case here and a good case here and a bad case here for vehicle detection and a good case here again.

We've done many more tests and these are all available in our paper. I'd like to conclude now. We proposed a framework that enables the quick design of test configurations. Our test generation strategy offers a good coverage over the continuous and discrete parameter space.

All of this helps in useful analysis of autonomous vehicle behavior. So what's next? Something interesting would be to look at how failing tests can be improved and if invariance can be specified as parameters during training.

Of course, the training being done inside a simulation. Thanks a lot for attending our talk. Please do read our paper available by scanning this QR code here for more information. I'm looking forward to your questions.