All right now. I am amplified good morning everybody. My name is Stuart Halloween. We're going to talk about simulation testing with simulant If you're watching the video later, and it looks like I'm in bright sunlight. It's because I sort of am I'd like to start and we're in this sports arena. I don't think we've been rowdy enough

In here yet so far so what I'd like for you to do is I'm going to count to three and then I would like you to really quietly State the name of a testing tool that you've used before and I'm going to listen really carefully and try to gather everything that you say

so one two three All right, so now let's try that again This time I want you to shout it loud enough that everybody everywhere else in the venue is wondering What the heck is going on over here one two three? Wow That is a lot of different testing Frameworks

I'm going to start by talking about a style of testing with which you're probably familiar which is called example based testing and The idea behind example based testing is that this works better with the USB stuck into the computer it turns out

So the idea behind example based testing is You have a scenario that you want to create and so the syntax here is Ruby, but it doesn't really matter And in fact one of the points. I want to make here is that it this approach is fairly language agnostic

And so the idea is that you have some setup that puts the system in a state under which you can do a test Here I'm going to create a new bowling object Then we exercise the system with some inputs so in this case. I am going to roll Missing all the pins 20 times in a row so 20 times roll of zero not a very good bowler here

I'm going to execute with those inputs I'm going to get back some kind of output that is interesting to me and Then I'm going to validate that that output is correct Now I'm going to show you the same style of testing in a radically different syntax enclosure

So this is actually a test from closures test suite verifying that math works And you'll see that it has all of the same elements minus one from the previous Example based test the only of the only thing it's missing is there's no setup, and so I'll make the side point here that

functional programming has that advantage is that in your test suites you need no setup because Basically in order to set up the world to test plus you just have to sort of assume reality Right there is no particular setup that needs to be done, but everything else then proceeds kind of as before So you have some inputs this particular source code form allows you to have kind of a table of inputs

You have execution with those inputs, so we're adding together a bunch of different things you have outputs and you have validation I'm verifying that the sum of all these inputs is equal to the outputs and

Most of the testing that I have seen people do in software systems Takes the form of this example based testing, and so if you're accustomed to doing Unit testing or functional testing or acceptance testing if you're a couple Interest into a test driven development or behavior in development all of these

Styles of testing have that basic mindset in common you have an example in mind you set up the world for that example And you then execute that example, and then you validate that that example is correct and in fact There's a whole cottage industry of test specific tools around making this go You have fixtures and stubs and mocks and all kinds of

Test runners and things that flash up red and green lights and so on and so forth that make this style of testing go I don't really want to write tests this way I want all these components. I want inputs execution outputs and validation, but I want to tease those apart and

So instead of doing them all in one place I want to separate them, and I want to add one element which is typically present in testing but implicit which is a mental model and So the idea in simulation testing is that you start with a model and this is not the model of your application

This is a model of the use of your application All right, so you're modeling user behavior and in most tests This is in your mind right so when I was doing those tests before I was imagining someone's going to go bowling I was imagining that someone was going to add two numbers together now we're going to be explicit about that and

We're going to call those users of the system agents and so with our model We'll think about what kinds of agents we have in our system. What kinds of things they're likely to do to the system and Rather than having a single example We are going to generate hundreds or thousands or millions or billions of these interactions and store them in a database and

so this is going to take the form of something along the lines of At time t plus five seconds in the execution of my system a customer logs in At time t plus 30 seconds 400 customers try to log in simultaneously at some further You know time down the road five of those customers browse to this page and six of those customers make an order and so forth

We're then going to use that model To generate an actual stream of inputs to the system. That's time stamped so for every one of those Interactions each one of those 500 guys trying to log in there's going to be an action guy number seven tries to log in

You know time X and store all those guys in a database Then to execute the test we're going to have a driver program again coordinated through a database that Maps all that action onto processes notice that this is about testing at scale So I'm going to show you a demo in a few minutes where I run this really small

I'm going to run a bunch of processes on my laptop, and they're all going to participate in this test But the idea is you take this to same test and scale it horizontally So because your systems are probably going to scale horizontally and they're going to run Across you know a bigger set than just a single machine So you get these kinds of tests working in the small and then you spread them out

Then we're going to capture outputs and these outputs are the actual side effects that your system has in the world Probably the biggest and most important output is going to be something gets written to databases somewhere But it could be stuff that gets written to queues it could be files that are created it also could be the sort of

Operational stuff that's not really user facing output so this could be logs or metrics or those kinds of things we're going to collect them all and Wait for it put them in a database So a database in some ways you could say you know simulation testing is about what happens if

You wrote tests, and you knew how to use databases, and you took advantage of that at every step in this game and Then finally validation is going to take the form of database queries So instead of assertions you're going to query the database and say oh, I just ran You know six days of simulated use of my system with ten times the number of customers we expect to have on day one

I didn't want to have to wait six days To for that to run so I ran it with time speeded up, so I ran that six days of simulation in an hour Which means that we actually put our system through? Substantially more load then we think it would encounter with or then we expect it's going to encounter in the real

Execution of the system and then come back and do a database query that says okay We had a bunch of users we knew they were all going to try to log in how many succeeded in logging in How many succeeded in placing orders and the questions that you're going to ask in these kinds of? Validations are going to start simple They're going to start not that more come not that much more complex than an assertion in the unit test right if we had

A million entities in our system that tried to log in we might want to verify that a million entities Succeeded in logging in but as you get more advanced in this you're going to do things that are more elaborate So you're going to run the same test suite again, and you're going to kick the power cord out of one of the computers

Or you're going to simulate that in software with a chaos monkey right so you're going to have an entity that roams through your system Causing failures while you're doing this form of testing and now your assertions are no longer exact match Right if the system had been working as anticipated a million users would have succeeded in logging in however We had a seven minute outage In the system that was caused by I kicked out the power cord of both the the available box and the failover box

now you have to ask how many users should have succeeded in logging in and That those queries are going to get into statistical models and probabilistic results, but they can be very valuable so

Everything I'm going to talk about today is really Idea space this is an idea about how to test systems nothing about it is Language specific and nothing about it is database specific and in fact a lot of the original Thinking that went into the style of testing was done in the design of very large-scale

Systems that were originally written in C and using SQL Server, and then were eventually ported to C sharp and using SQL Server I have implemented an open-source version of these ideas and The concrete realization of that has a very specific choice of language and database

I use closure as a programming language, and I used the atomic as the database I Gave an intro to closure talk Earlier today where I was very gentle about closures capabilities and features from the perspective of not Melting people who are coming to it from the first time

When I show you the code here, it's not going to be gentle it is going to be An expert user of closure and atomic using it in anger to kick ass and do this job And so one of the one of the you know decisions that we had to make when we have sample apps for closure We have sample apps for the atomic. This is not a sample app right. This is not intended to

Be approachable even necessarily it's great if it is approachable, but that wasn't a goal But I want you to keep separate in your mind The code you're going to see from the ideas because the ideas are incredibly simple And you can and I hope somebody will Take this and implement an open source approach in C

Sharp and SQL Server or an open source approach in Java and Postgres to doing these same things so We'll go through this in pieces This is the basic schema that I use in simulant so the top third of the screen here is the model partition those are

Entities that describe models, and you'll notice that the model entity has almost nothing on it Because this is the generic framework on which you build in a minute. We'll add specific things to the model the middle is descriptions of tests so translations of those models into time-based activity streams So a test has a one-to-end relationship with agents and an agent has a one-to-end

Relationships with actions that take place at a particular time again notice that this part of the job is totally generic It doesn't say what kind of system. I'm testing what kind of actions those agents are going to be performing It doesn't say whether those agents represent individual users or they represent rolled-up aggregate users

Or they represent other processes that are going to interact with the system And then finally the sim itself is the record of the execution of a simulation now let's drill in a little bit we're going to look at a specific sample and Our sample is going to be and this is hello world simulant style

to give you an idea it would probably take you a half day to a day to work through hello world and Understand it in detail so this this is an object the objective of hello world is to be big enough to see all the pieces That being said it's also incredibly small because hello world simulates traders and these traders are really boring

They don't have names they don't have addresses and They don't have anything that they trade other than arbitrary units and the way that they trade arbitrary units doesn't actually have any exchange rate On them or anything it's it's it's dumbed down but it is Transactional and basically what these traders do is they come along and they hand units of whatever it is that they have to somebody Else in the system and so the goal is to have several thousands of these traders with an initial number of units

Handing units to other people and at the end of the game. What should be true Physics right matter should not be created or destroyed so at the end of this game one of our Objective outcomes is that all the resources in the system are still there, and we didn't have a failure of transactionality

The activity that we're then going to have Based on these agents is going to add to the actions and notice again We're just extending the model of simulant itself We're now going to have actions and the actions as I said are very basic the actions are transfer from which agent is Giving something away transfer to which agent is receiving transfer amount how much are they going to get?

We're going to run a test that takes four hours to run, and you guys are going to watch and because of that The simulant has a clock speed adjuster, so we can say simulate four hours of interaction But do it a lot quicker than that so we don't all have to wait around for four hours

There is a service entity type in simulant which allows you to manage the lifecycle for external systems So this allows you to so it's not like all those ideas from unit testing don't make sense right we still need setup and tear Down we need it kind of more at scale here This could be across multiple processes and multiple systems So the database is keeping a record of what code

Needs to be used to set up the system, and what code needs to be used to set down tear down various processes Another piece of this game that is not obvious when you first start doing it is that after you've run these tests for a While you have a database of the history of your tests So you can say I want to go back in time to three months ago, and say what was my test like then?

Not just what was the code? What were the inputs? What were the outputs? What were the validations that we ran? What did they tell us and a? Big piece of making that useful is going to be and I want to be able to get back to it Right if so if we see a problem, and we go back and look at our test suite from three months ago We'd like to get back to the code

Fortunately most of us are now using source control so we can store pointers into source control along with the test suite Um I use git so simulant supports git all these things are pluggable so you can make it support Whatever you like the important thing here is that you record enough information that you could reconstruct the source code

Based on that information so with git that is a URI of a repository or one or more Repositories and the sha of a particular commit that says I did this work against a particular commit, so Let's go do it So I am going to

Sync up the displays So I can see what you're seeing so I'm gonna go through all of what we just said in code

So the window at the top is a closure REPL where I'm going to be executing code and the window at the bottom is Going to be showing the results of that execution So I'm going to load the code require is loading the namespace of the various things that I need and In this particular case my simulation is going to be running against

The atomic database that's running on my local box I'm not distributing or running at scale at this point, and I'm going to connect to that database I'm now going to load the schema, and I'm broke breaking out loading the schema into three parts And you'll have at least two if not three separate schemas that you load

There's the schema for single in itself, which is the core infrastructure. There's the schema, so I'll load that I'll try to load that what's going on here Uh-oh the transactors dead

What's going on here lots of transactions timing out all right? I will connect to the memory database

Just so we don't have to go troubleshoot that oh Create that database load the schema Now in this particular case It's a bit meta because the application that I'm testing is also written using datomic

So I've got the datomic database for simulation, but I also have a datomic database under test There's nothing about simulation that precludes Doing things a different way right you could say I'm going to use datomic as the simulation database But the system under test is SQL server the system under test is oracle But because I'm doing that I can sort of double dip and I can actually load the schema of the application

Into my simulation database which is going to make it smarter because it needs to talk about the application And one good way to talk about the application is to have access to its schema I'm now going to create a model, so here's my model of the world and let me

Give us a little bit more room here my model of the world is that we're going to have a hundred traders they're going to start with a thousand each and their mean trade amount is going to be a hundred and They're going to trade on average once per hour And so this is where you sit down with domain experts of your system and

Model out not just the system itself, but model out the use of the system. What are people going to do with our system and This is vastly more capable than having an example that covers one use case right unit testing says I'm going to do one thing I'm going to say there's one happy path through my system where this happens this also automatically is going to allow us to introduce parallelism

Right once I have a model. That's like this. I can spread it out and run it and have it be parallel So I'm going to take that model, and I'm going to inject that model into the database. I'm then Going to create an activity model, so this is the actual test so this says taking that model generate a test

With a particular set of settings in again. This is the hello world version there could be lots more Bells and whistles here about how the test is going to use that model in this case the only thing I'm specifying is the Now if you're following along, it's useful to keep the math in your head So start holding some numbers up in your head. We got a hundred traders

They trade an average of once per hour, and we're going to run a simulation for four hours. How many trades should we see? 100 traders they trade once per hour for four hours equals 400 approximately will it be exactly 400 No, but if it's 10 or a trillion we're going to know something went wrong with the system so now I'm going to

Again, not use this database because it seems to be worked. I'm now going to create the system That's going to run the simulation, and I'm going to specify that I want to run the simulation with 10 processes even though

I'm going to run it locally on this box I still want to get some parallelism going so I'm going to ask for 10 processes to do the job and I'm going to spell my uris correctly No, really, I am Now I'm going to associate the code base

For this test so I have a helper function that scrapes get from the command line and pulls out the code base You can see what that data actually looks like So there's a data that says This repository type is get this is the commit shop from which we're running so we don't lose track of that later

I'm also going to create an action log Action log is less core to the model, and I'm not sure I'm even happy with it But everybody who's writing these tests ends up wanting to keep track of things that happen Above and Beyond what their own system records Right so your own system stuff goes in the database because the user bought something or whatever

But I also end up wanting to record other things and so the action log is my experiment with Having a thing that's associated with each action that lets you hang extra things in the database after it's done the big one is elapsed time Right so the one that it supports out of the box is I want to easily go back and be able to ask questions

About how long did it take to do each part of the job? So I'm going to create the action log. I'm going to create a clock This is a four-hour test, but I'm going to run it 960 times faster than real-time I'm now going to wait for it to finish So now I'm waiting on ten separate processes running this test, and we should be done by now there we go

Now having done that I'm going to grab the value of the database One of the reasons why the atomic is particularly suited for doing this job is that the database is a value so I can hold On to an old value of the database and continue to play with it even after the database moves on I'm going to grab the database of the simulation, so this is the guy that's helping me do the test

And I'm going to grab the database of trading which is the system. I'm testing now I want to make sure that the count of actions seems reasonable, so I'm going to execute a datalog query I'm going to join from test through to agent and Then I'm going to join from agent through to actions, and then I'm going to count those guys

439 actions, what was our expectation? Hopefully? 400 that's tolerable that's believable I'm now going to check the amounts of the actions, so this is a more substantial join I'm going to join from test through agents from agent to action from action to find only actions of the type that I'm looking for

And then from action to the amount of the action There's 439 action amounts. That's a hard sanity check Right if there are 439 actions they had better each have an amount so that can be a hard assertion

And then what should the mean of these action amounts be? Approximately 100 104 Alright, it seems pretty good. I now want to check the trades so now I'm going to query not against my test database, but against what actually happened so I'm going to go and find all the trade transactions in the system find their amounts and

Actually find out when they happened although we're not going to use that information right now. I'm going to count those How many should there be? This is where when we run the test and we simulate everything going okay in our system There should be a one-to-one matchup right we attempted to make 439 trades how many trades are we going to succeed in making in the actual system and

Because I ran the system without any stress or pain It's going to match up and the mean of the trade amounts had better be exactly what the mean that we attempted was so That feels good Now I want to check the trader balances, so I'm going to grab all the traders I'm gonna count them. There's a hundred that's another good hard sanity check We said we wanted a hundred traders do we actually have a hundred traders and

then I'm going to grab their balances and I'm going to assert when I add them all together that I get a hundred thousand because there's a hundred traders They all started with a thousand. This is testing the acid transactionality of the system Right because if we you know we created or destroyed matter something would go wrong here

And we're good now it turns out that there is a There was a miscommunication between the development team that wrote the app and the people that are testing it and I was both But I kind of have a split personality that way now when I was writing the app I wanted to make some dumb mistake that the system could check and so when I was writing the app

I had this presumption that nobody's balance was allowed to go negative, but I didn't actually implement that in the system Right so the system has no safety check for balance is going to go negative and so I can filter the trader Balances and pull out all the negative ones, and it's going to turn out. I got lucky if we run the simulation again in

Fact I will say I got extremely lucky here in that I've done this demo publicly a dozen times, and it has never Actually succeeded with this volume What's going to happen though if I run this if I turn the knob up on scale and run it ten times bigger? What's going to happen eventually? I'm going to get back a negative balance and

More importantly it's the kind of test that catches the things that you weren't thinking about not the things that you were thinking about One of the things that's fundamentally broken about a unit testing dominated strategy to testing is That you only do specific examples you're not generating a bunch of different possible inputs, but another thing that's broken about it Is you're usually asking?

Joe developer to do two things simultaneously Think about how something should work and think about how to check that something should work And I call that the sort of the dirty mirror problem because if I'm misconceived how it's supposed to work What am I going to do? I'm going to misconceive how to test that it worked I'm going to write a test that shows you a green light when it's not working

And I that is a real problem on systems that I've worked on is the tests are passing But everything's actually wrong and so one of the nice things about stepping further and further away from just you know Tip-for-tat checking of things is that you're more and more likely to catch something in fact when I first wrote this demo I wrote it at a cut while sitting at a conference that closure cons year and a half ago

I first wrote this I had a mistake in a data log query and my mistake in the data log query Was causing me Basically data log queries returned sets And I didn't have enough things in the return value to actually distinguish all the unique things

I was bringing back only the price I was only bringing back only the prices and so if there were two prices that came out Exactly the same they got folded into each other so my query wasn't returning the right amount of stuff Of course at this level the test went all to hell when the query wasn't returning the right amount of stuff It's like wait a minute the number of stuff. That's coming back out doesn't match the number of stuff. That's coming in

I didn't have to assert that I just went through the exercise of looking at this and asking you You know what's sane in the system and something seemed terribly wrong, and I was able to track it down Now I want to test a non-functional requirement, so here we have a metric on trade times This is a really substantial join, so I'm going to take

A test and join to sim I'm going to join to agents. I'm going to find through the agents Actions, and then I'm going to go into the action log. This is why I wanted the action log in the database I'm going to join into the action log and find The nanoseconds that each trade took I'm going to run those rules through an actual query

that finds those guys and That time I just asserted that the count matched up Let's actually jump in there a little bit 439 So there are all the actual trade times in nanoseconds, and then I'm going to do an aggregate query So I'm finding average of those we do the same query again, and I'm also using here

I'm actually dividing by a million when I get out because I want to think in milliseconds not in nanoseconds and we can look at that mean trade time milliseconds and verify That I mean trade time in milliseconds is actually 200 microseconds because I was using the in-memory database transactions are really fast

And it's well below my target threshold now the thing about this is once I have this in play I can come back later and Go to a database and this UI does not fit on here very well and

Run a query and say find me all the simulations in the system, so there's a simulation Go and grab that entity And I'm going to just shrink Shrink this out of the way shrink this out of the way and I can tool around in here and explore my system and verify

You know this system ran ten processes all the processes Completed correctly and I can go and say you know what I want to go and look at a

Particular run of the test and then go and look at that guys You know see what attributes this guy had this is actually the trading so it has the trades in it So I could find all the trader initial balances. I could say give me all the traders in the system

Of course I can't see the results because I hid that Phooey I'm gonna grab this bar at the bottom again, so here's all the traders in the system and then from a particular trader

Again this works nicely on a bigger screen. I can go find that trader and then Go and look Who all his trades were to and from?

Right you have a database right you can go and sort of explore around in here when something goes wrong and what this allows You to do is to run Realistic load of your system capture all the stuff you did to make that load Capture the specifics of that load capture the specifics of that result and then come back and ask questions later the separation across time is really big so imagine that you get a call from the field and

somebody says We we have this bug in the system. We're seeing symptom X and you scratch your head from it You think oh my god We don't have any validation anywhere that would prevent X from happening And I have a suspicion I know exactly where that happened you go

You don't write a test you don't write any code you go into a database query against the test you ran a week ago or a month ago and say yes, I've reproduced the problem and You actually had already reproduced the problem. You know months ago You just didn't know you had done it yet because you didn't write the validation so this allow allows a temporal separation of

designing your tests running your tests and Validating the results it also allows you to say you know what a year from now I'm going to be five times smarter at validating results but I'm going to have a year's worth of database of Existing tests and results and be able to go back and run better validations later I use that excuse all the time when I'm working on the atomic with which I always tell them

I'm going to write really good validations in the future since Since I know I have the ability that the leeway to go and do it that way Let's go back unmirrored and now have to navigate my way

So there's a lot of parts behind the scenes here. I'm going to go briefly through One of them there's a library enclosure called data dot generators which facilitates Translating your Usage model into actual data to put in the system so the objectives for data generators were to generate all sorts of data with different distributions and be predictable and

The way it works is that the generator functions shadow functions of the same name enclosure So if there's a function enclosure long That does something then there's going to be a function in generators called long that makes a long and the default numeric

Generators are uniform on the range of the type the other defaults had to be arbitrary so closure has a function called string I made a generator called string We have to have an argument about what it meant to be a purely random string so but the idea is that generator sort of shows you how to do that, and then you can extend it and

You'll see this as we get into some examples The big thing here is that it uses a random number generator That is driven by a predictable dynamic binding so you can run the test and deliberately run it again with exactly the same data Even if you've generated billions of things or you can change that seed and generate different data So here's what the API looks like

Gen short makes a short something in the range of a short gen uniform lets you set your own range So give me a something uniformly distributed 0 to 10 gen rand nth lets you pick So platform range any range you want pull things out of a collection

There are collection generators So you can generate a list of shorts when you generate a list of shorts the default sizing is fairly small So when you ask for a list you're going to get a small list Or you can pass a second argument that says how long you want the list to be That can be a number give me a list of size 2 or it can in itself be a function which says give

Me a list based on the distribution of this function and then you can start to compose things together, so you can say gen one of and Say choose with equal weight from these data sources. Give me either a long or keyword Or you can say choose in a weighted fashion from data sources so give me ten times out of eleven give me a long and one time out of eleven give me a keyword and

Then there's just for giggles. There's gen scalar which gens any scalar type known to the system and Gen collection which gens any collection type known to the system. That's not typically those two are not typically that useful

In a domain where you actually kind of know what kind of data you're going to send on the other hand They're really trick for testing serialization frameworks Right because you can just say you know make anything you want and see what's going to happen on Top of that there is a library called test generative test generative you know I'm going to skip test generative, and we're going to go and talk about

Data log So I want to go back and talk about how to read those queries. We were looking at because it's useful Test generative is also very cool the idea behind test generative is that you You don't test by example, but you test by generated inputs, and so you declare this function

Would take a generated input of this type and a generated input of that type John is your talk going to include that sort Of stuff yes, so I think there's a talk here here next it's going to talk about generative stuff So I'm going to I'm going to leave that in more capable hands So I want to go very quickly through data log syntax

Just so you can follow the queries that you're going to see in simulant because the goal here is to work at a higher level Of abstraction than imperative assertion right we want to be able to work with queries and so a query form looks like this. It's got constraints So where says I'm going to constrain what we're going to get it has inputs there can be more than one input So another reason that we picked atomic for this job is that datomic can query with join across databases

So there's the ability to say I'm gonna have a database over here which represents my simulation a database over here Which is the system of under test and then we can join across those when getting answers The end clause allows you to name those inputs so for every input There's going to be a name in the end clause that can then be used to refer to it and

Then find tells you what variables to return Variables start with a question mark these are things to be filled in by the query engine Constants are anything else constants include Things you know like strings and numbers they also include literal types for things that don't usually have literal types So we want to have writable literals for uu IDs ur eyes and dates and so those all have their own literals

Here's a really tiny database that will query against just so you can see how some of this works So the atomic is a universal schema database Everything has the same shape in any attribute value this database says there's some person in the world number 42

Whose email is jdo that person has made two orders where those orders are representation somewhere else in the system Data pattern this is going to constrain the results and bind variables So this says some entity a customer has an email address And we don't know what it is and the constant will then bind the will then

Limit the results returned, so if we run this particular query Against our database. It's going to find two pairs 42 jdo 43 Jane because those two things are allowed to vary All the different slots can be fixed so if I make the entity position slot fixed 42 email I'm now going to find only that guy right because I've now bound entity and attribute, and I'm just looking for value

You can do schema queries, so you can make the attribute slot variable so now it says What are all the keys that customer has? So that query 42 attribute is going to find the attributes that the customer has and of course usually if you want their attributes you

Want their values as well so 42 attribute value would be tell me everything about entity 42 all of this Doing everything entity attribute value is designed to allow us to have an incredibly flexible set of data Right as opposed to a relational database where you make a table and you have to figure out what the names of keys and you

Know what the names of columns are on your table everything has the same column names Right entity attribute value if you want to look at it that way the where clause then specifies a pattern and The fine clause tells you what to return so this is a select star query right here This says find all the customers in the system

This is a join and this is where data log really shines From a learnability and readability perspective if the same variable appears in two places you get a join So this is a select all the customers who've made orders Right they have to actually have orders to show up and then from an API perspective It's just a function call that takes that data structure followed by typically a database

When you add an in clause you get the ability to specify more than one input so here We're gonna say look in the database and a particular email and that second argument. There is now parameterized query

right so the database is the first argument and email is the second and Question mark email even though it's a variable. It's now bound coming in right. We've found that coming in So you have the database and the email the database argument is typically elided Right most of you are used to querying against a single database at a time

Teutonic lets you query against multiple databases, but you don't you still often want to query against one database So you can elide that argument entirely leave it out the dollar sign and leave it out of the where clause You can do predicates so limit the price to be greater than 50 So now I'm saying find all the items in the system that cost more than 50

You can call functions So here we have a function called shipping that takes a zip code and await and Returns the cost to ship somewhere those functions are written in JVM language of your choice right now. It's Java or closure but this is actually calling out to code that you write in your own language and

that will take those inputs and return this output and That leads to the ability to make queries like this one And I want to go through this in a little detail because it really drives home The expressiveness of the query language so the the made-up scenario here is That a customer in Montana in the US ordered some beef jerky and the beef jerky cost seven dollars

But it cost ten dollars to ship it to him how many of you have ever ordered something where the shipping cost is more than The price of the item it just has kind of pisses you off, right? And so your manager comes in and pounds on your desk and says I would like to know How often that's likely to happen in our system so in order to know when that's going to happen you have to ask a

Question where are the customers so this part of the query says find all the customers Join through their addresses to get their zip codes right now I know where all the customers are then it says tell me about all the products in our system, so Join through product find me all the products, and I need to know two things about the product I need to know its price because at the end of the game

I want to see if it costs more or less than shipping it and I want to know its weight because I want to Pass that to the calculator Then I want to say call shipping estimate passing in the zip and wait getting back the ship cost Then I want to constrain this query to return only the things where the shipping cost is greater than the price and The thing that I actually want to return is not the price

But the customer product pairs who are the customers and products within our system for which this would be true now If you have a big system this query is a monster right because it's select star on two different tables logically But it's doable you can do it you have to wait a while

And where does this query run? Because the database is a value this query does not run in the database engine this query runs in your application code So you can stand up an extra box to run this query right if it's really expensive you can scale this out horizontally so my my aim in life is

to have The QA process on systems that I work on be built around executing these kinds of queries against simulations as opposed to the sort of assert and pray approach to things now Please go and build something like this in dotnet. Please go and build something like this in Java

I'll tell you why I use closure and atomic and how it helped But you should be able to take these ideas and and pick the analogous concepts there In closure the big help was multi methods So what I multi methods enclosure allow you to do ad hoc polymorphism so polymorphism without types

If you're doing ad hoc polymorphism you don't need types if you don't need types You can use the data that comes out of the database raw Which means I don't need to do any entity mapping so that gets rid of a lot of code Right because I'm able to work directly with the tape of the data and still have polymorphism

I don't want to live in a world where I have to work with data without polymorphism I buy that part of a polymorphism good The other things that helped enclosure The laziness of closure was really beneficial for processing these long streams of inputs which potentially won't fit in memory Right so when you're running the set scale the actual inputs to the system might not even fit in memory And so the laziness is really nice for that and then the agents in this

We I use the word agent in the simulation to talk about somebody in the system That has a one-to-one mapping and an actual language construct enclosure called agents It almost makes you think that the guy that created closure had simulation testing in mind Or had built his own simulation testing frameworks prior to building closure which he had

So so those are really good fit features as a result of these features. How many lines of code? Do you think the whole framework is? So it's it manages multiple databases it has models it has Generators it has actors and actions and all that

700 lines of code all in it's pretty good on the datomics side Datomic helped us By having an open schema if I wanted to build this with a relational store Then every time I added domain specific stuff. I would have to join from the simulant stuff to the domain specific stuff

Right because it's not in the simulant schema, or I would have to make a brand new schema for each one But I'm just able to layer in new stuff on the existing entity so that was really big Data log is great and the fact that I can extend data log with code written in my own language Means that whenever data log is pissing me off. I can just skip it and just work with collections

So that was really helpful, and then obviously Being functional and having a model for time is a big win here because that's exactly what we're trying to do in our testing Right we're trying to have a model of time in our testing not just a you know We try one thing and verify that it works And then the multi database queries obviously a big help when you start working on larger systems

So you want to adopt this at home? You don't have to use closure you don't have to use the atomic you just have to use the ideas And the ideas are pretty straightforward all right have a model of use for your system Use that model to generate actors and streams of actions, and then have a player that plays those actions through your system

Do not throw out your example-based tests Right those are still good You should still use them it will take a while to get comfortable using this and even if you might throw out some of Your example-based tests once you're comfortable don't don't do that early right take your time and go easy with this

Our experience at relevance teaching consultants in relevance how to use this and Take it out into the field is that comfort with the model comes in about a week? So if you were to sit down on a project and be able to work eight hours a day for five days doing this Then at the end of those five days You would be ready to be cut loose and devise your own models and and so forth give or take so

It's not nothing right. It's not you know learn this in five minutes, but it's not it's not you know infinitely far away either And then finally I would say that Simulation requires time and thought and it's time and thought that is either not done with other testing approaches or is implicit

Right so when you sit down and write the bowling game where somebody bowls you know 20 Complete misses in a row you're imagining that that's a possible user in the system And so those example-based tests have a mental model in mind that doesn't get encoded one of the biggest benefits of Simulation testing is actually forcing you to document that model

Right if you did that and didn't run the tests it would still be worth it And that's worth emphasizing if you did that on a whiteboard and didn't run the test how many of you work on systems? Where you don't have anywhere you could point and say this describes the different kinds of users in our system

And how they interact and at what volume and when and how if you're doing you know? Agile development you probably do have story cards that are broken out by roles So you probably have part of it right you probably have who are the different kinds of users of my system? but I know a lot of people who are going to production with systems where they don't have a Good handle on how their systems are going to behave at scale nor do they have even a sort of first order of

Magnitude approximation of what the scale of different things is going to be and so just having that conversation Even if you never use it to then generate test data and even if you never run that through a player To simulate load against your system is in itself valuable And if you go back to that slide where I took example based testing and broke it apart

It's really about that separation right having all those things together Creates a really on non reusable test suite right manufacturer one bespoke input do some stuff check that It's okay leave right those those create you know hundreds and eventually thousands and tens of thousands of little islands

That can't be maintained Simulation on the other hand takes more time up front Right you're going to have to put in some effort to design these models But once you have these models you're going to be able to move really quickly And I think that you know we use simulation to test atomic and we use simulation And relevance on some customer projects and those projects have been absolutely thrilled

With the agility of their system once they had this kind of testing in place, and we're able to say okay What's going to happen when we have 10x the volume we have today? What are things going to look like that when we get into that situation? So the references are all available online and the most important link is this one

There's a list of all my presentations. You can download the sample code there You can find simulant and also the slides that I have here today, and I have left. I believe 10 minutes for questions I'm hoping this was like an Alien-spacecraft landing in your yard for at least some people because I can say that when I when rich first showed me this stuff

It was definitely an alien spacecraft landing in my yard, so so it took some time for me to get used to questions So everyone's going to go do this then How many people in here are C-Sharp developers?

2-thirds Java developers other Yes, so the question is do you just always run this and then explore and add a search the the fact of it

It's really up to you right by having by having everything be persistent in your test suite right your model is persistent Your inputs are persistent the results are persistent you can you can dial that depend on depending on what you're trying to do

when we were using simulation for room key When Craig was doing that work they had they had a fairly specific job and fairly specific targets So they knew how many events they were going to have to process They knew what kind of hardware they were going to want to run on so they had fairly specific

Questions and fairly specific validations, and so they were able to lean more on you know. I know what I'm asking right I know what kinds of things I'm looking for on the other hand when I'm using it to test atomic I mean the atomic is a database I mean it can go in a million different directions and so my Exploration tends to be more open-ended at the beginning and it's at the repl and it's trying things out

And and the thing that I really miss and we're working on it is This is the prototype, which you can't see and now you can see it, but it's covered up right this doesn't work super well

Okay, oops. I make it kind of smaller like it might look on a bigger monitor, so you can see the idea But what I miss is having gooey to more gooey tools to support this now having said that because everything that I'm doing is

Manipulating databases, I don't need a special-purpose testing tool right I can use a General-purpose database navigation tool that lets me enter ad hoc queries and lets me navigate around so that in itself is really valuable I think one of the mistakes that we've made in testing in this industry is that our testing tools are too bespoke

They're too particular to testing and testing is really just another kind of program And so if your test uses data, where do you put data? In a database, but I see time and time again You know people putting their test fixtures in text files and in your recording things and doing this You know very sort of hand-built thing now part of it is because we're the developers

And so we are the creators and the consumers of test suites And so we don't have as much reality inflicted on us by customers as we do elsewhere But it's kind of a shame right because we have great tools for manipulating data Right Michaels, you know sequel server and Oracle are much better tools for manipulating data than sort of by hand and text files And so I would say that you know

One possible lesson to take from this even if you don't do something like simulation testing is just ask the question of your testing Framework is this really specific to testing is this a more general problem. That's amenable to more general tools other questions yes

Absolutely so the question was if I've got something that already exists I didn't write it with closure and atomic, but I want to use this how do I write the driver program?

Right how do I write the thing that takes the inputs? Yeah, absolutely You know I've joked that you could go so far as to like drive selenium or something like this I think that would be the road to madness It would be really difficult to do But you want to drive you want to drive your model data layer with this you want to drive your controller layer You don't want to I don't think you want to be driving the UI rendering and stuff with this

I think you should test that a different way, but yeah on the room key project I don't think they actually interacted with the code directly. I think that they in fact did hit rest endpoints So the generator was hitting rest endpoints another interesting thing about this though. It's a great example of the benefit of separation I have said here's the notion of a model Here's the notion of the agents and the actors and then there's a player that plays that into the system

You could write more than one player Right you could write a player that plays it in at the code level and you could write a separate let player that plays it In over HTTP rest and you could write a third player that played it in through selenium Right that and it's a great example of the kind of thing. How are you what would you do with an example based test? That worked like that right you can't you can't pivot like that with that kind of test suite

So so having that ability to pivot and and change now having said that writing the player is also not nothing Right that's that's its own its own job So I'm not saying that you're going to write you know one of these tests and have something up and useful in a day It takes longer than that

Yes well

So so the question is you know going back to the small example now browsing through the result set And there's ten traders and not a hundred traders, so when you have when you've run a simulation like this you have all the inputs in hand so you can ask a question like Give me a list of all the inputs the time stamp list of everything that was going to happen you can then say take that time stamp list and

Cross correlate it with what happened and say okay only ten happened Right or only ten got in the system which ten were they were they the first ten? Maybe we hit an exception and we fell over and we didn't do any more was it every seventh one Well that would tell me something different and all of those questions are asked how with a query

Right that's the big deal right being able to go back and explore and ask those questions with a database query is huge Right you can ask those questions by yourself. You can do that, but it doesn't uh it usually takes longer To sort of sort those things out. Yeah, you still have to have a theory right you have to use the scientific method And say okay, what information can I see and what does that tell me the the thing about simulation testing is you have more information?

Typically than after another kind of test and you have more leverage over that information because you have a query language So so it's still up to you to do something good with that information time for one more question Yes

So datomic does not have a book at present I would say the best resource is the documentation site datomic.com and then blog datomic.com is rolling You know the story of that stuff that being said I would caution somebody you know Don't don't learn three things three new things all at once

So if you don't know closure, and you don't know datomic, and you've never done simulation testing Set your initial objective to be one of the three not all of the three because that's a big mouthful Right to take on all at once right do do a little bit of simulation testing in a platform you already know or Learn a little bit of datomic, but don't try to do this with it yet. You know take take it step by step

So I'll stick around outside after if people are interested in this I'm super excited about it, and it has been a lot of fun Working with simulation. It's also been terrific. This is my last talk at NDC my first time here I've had a great time, and hope I will be back. Please make sure to drop off the evaluation cards on your way out

Thank you