We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

NDC Oslo 2013 - Lightning talks 1

00:00

Formal Metadata

Title
NDC Oslo 2013 - Lightning talks 1
Title of Series
Number of Parts
150
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Talk 1: Cross-platform mobile gaming with MonoGame - Rune Andreas Grimstad Talk 2: Teach the kids to code - Simen Sommerfeldt Talk 3: Learn from the history - Thomas Pedersen Talk 4: Polyglot persistence - Ole-Martin Mørk Talk 5: What we have learned aboutSharePoint 2013 and Enterprise Search - Tallak Hellebust
31
59
Thumbnail
1:00:41
89
Thumbnail
1:00:33
90
Thumbnail
1:00:33
102
Game theoryUsabilityCodeVideo gameDynamical systemGravitationImplementationInformationComputer clusterGame theoryComputer programmingType theoryValuation (algebra)Macro (computer science)WindowFlow separationForm (programming)CompilerLine (geometry)Structural loadLoop (music)Cross-platformSampling (statistics)Link (knot theory)Revision controlPlanningMatching (graph theory)CASE <Informatik>Strategy gameAbstractionInstance (computer science)Musical ensembleSocial classCuboidVariety (linguistics)Set (mathematics)Open setRoundness (object)Focus (optics)EstimatorFile formatReflection (mathematics)Software frameworkWorkstation <Musikinstrument>Open sourceWebsiteTexture mappingDifferent (Kate Ryan album)Calculation.NET FrameworkNeuroinformatikComputing platformTouchscreenContent (media)Object (grammar)Image resolutionMultiplication signRight angleService (economics)Copyright infringementInterface (computing)Android (robot)Mobile WebSoftware developerCombinational logicCodeSoftware testingAnalytic continuationPerformance appraisalMereologyVisualization (computer graphics)Goodness of fitProcess (computing)outputOnline helpXNA Framework
Computer animation
FreewareCommodore VIC-20Dependent and independent variablesProcess (computing)Inheritance (object-oriented programming)CodeBitSound effectMultiplication signComputer animation
Digital signalService (economics)Enterprise architectureData managementCodeLattice (order)CodeCodeCommodore VIC-20Goodness of fitSource codeProduct (business)40 (number)DemosceneComputer animation
Digital signalCodeLocal ringEvent horizonAreaMusical ensembleService (economics)CodeDampingInformation and communications technologySelf-organizationStaff (military)Computer programmingType theoryDigitizingLevel (video gaming)Integrated development environmentConnected spaceUniverse (mathematics)Local GroupGroup actionLocal ringVisualization (computer graphics)AreaFamilyExterior algebraOnline helpStudent's t-testLattice (order)HierarchyEvent horizonWhiteboardWebsiteDifferent (Kate Ryan album)NeuroinformatikSinc functionDesign by contract1 (number)Digital electronicsProgrammer (hardware)CausalityPhysical lawTheoryGoodness of fitSystem administratorRoboticsInheritance (object-oriented programming)Cycle (graph theory)Right angleTwitterFacebookComputer animation
SimulationFile formatComputer animation
Physical systemFile formatPhysical systemCodeMereologyError messageSoftware developerFlow separationMultiplication signComputer animation
Message passingCodeComputer programmingSoftware maintenanceGoodness of fitError messageRight angleMetropolitan area networkComputer animationMeeting/Interview
CodeComputer programmingSoftwareInformation technology consultingClient (computing)Computer architectureSoftware developerComputer animation
Function (mathematics)File formatObject (grammar)Software testingSource codeTask (computing)Revision controlSheaf (mathematics)Latent heatComputer fileData modelError messageSpacetimeCodeLoop (music)Endliche ModelltheorieGoodness of fitNP-hardNeuroinformatikBlogCommitment schemeOffice suiteCoroutineCodeLoginSource codeComputer animation
SineCodeFinite element methodOffice suiteCodeTask (computing)Revision controlGoodness of fitError messageSource codeBackupComputer fileDomain nameCommitment schemeMultiplication signFreewareSoftware developerSoftware testingCodierung <Programmierung>Online helpRight angleOffice suiteCoroutine
ArchitectureDatabaseNumberDatabaseData managementGraph (mathematics)Order (biology)Computer architectureRelational databaseTheory of relativityInformation technology consultingTask (computing)Physical systemScaling (geometry)Operator (mathematics)Process (computing)Shared memoryPoint (geometry)CuboidData storage deviceCartesian coordinate systemSearch engine (computing)View (database)Endliche ModelltheorieKey (cryptography)Domain nameObject (grammar)Multiplication signFreewareVulnerability (computing)Figurate numberCodeWellenwiderstand <Strömungsmechanik>Goodness of fitCommunications protocolContent management systemComputer animation
Search engine (computing)NumberRelational databaseDatabaseGraph (mathematics)Information technology consultingDatabaseGraph (mathematics)MathematicsOrder (biology)Computer architectureRelational databaseTheory of relativityStatisticsComputer programmingType theoryInformation technology consultingProduct (business)SynchronizationHard disk driveQuery languageProcess (computing)Data storage deviceCartesian coordinate systemLatent heatData warehouseSearch engine (computing)Traffic reportingEvent horizonKey (cryptography)HookingSinc functionMultiplication signPublic key certificateComputer hardwareProgramming languageLibrary (computing)Task (computing)State of matterForm (programming)Coma BerenicesPhysical systemProjective planeBasis <Mathematik>Source codeDifferent (Kate Ryan album)Computer animationProgram flowchart
Computer hardwareServer (computing)Query languageComputer networkVirtual machinePoint (geometry)Virtual realityRead-only memoryPhysical systemMiniDiscDatabaseStructural loadLocal ringRAIDStorage area networkTime domainGame controllerCountingBefehlsprozessorSanitary sewerQuantumAutomationChemical equationCache (computing)Computer virusData structureComputer hardwareImplementationEnterprise architectureDatabaseInformationMathematicsData centerSoftwareSemiconductor memoryProduct (business)Software testingIntegrated development environmentTask (computing)Subject indexingFlow separationMathematical optimizationAnalytic continuationBefehlsprozessorForm (programming)Binary codeDebuggerFunctional (mathematics)Limit (category theory)Structural loadPower (physics)Maxima and minimaMereologyPhysical systemExecution unitResultantCore dumpSubsetVisualization (computer graphics)Query languageRevision controlGoodness of fitServer (computing)Process (computing)Connectivity (graph theory)System administratorAsynchronous Transfer ModeAdditionPoint (geometry)Logistic distributionCuboidLastteilungSet (mathematics)Web pageMiddlewareSource codeComputer fileDomain nameMiniDiscContent (media)Multiplication signWeb crawler2 (number)Cache (computing)VirtualizationVirtueller ServerWeb 2.0Game controllerDefault (computer science)Software developer1 (number)Fault-tolerant systemWindowComputer configurationAnalytic setForcing (mathematics)Moment (mathematics)TheoryProjective planeVirtual machineScaling (geometry)AreaBusiness modelPlanningMatching (graph theory)CASE <Informatik>Metropolitan area networkCarry (arithmetic)Proper mapCartesian coordinate systemCasting (performing arts)Lattice (order)Greatest elementBit rateSingle-precision floating-point formatForestService (economics)Office suiteAntivirus softwareWeb-DesignerComputer animation
Pairwise comparisonMultiplication signComputer animation
Transcript: English(auto-generated)
Okay? I did? Yeah. Okay, welcome everybody. Before I start, I want to remind you to fill out the evaluation forms when we're done. I'm going to talk about cross-platform mobile gaming with MonoGame.
MonoGame is an open source framework and I'm gonna spend the next 10 minutes telling you what it is, what you can do, how you can use it, and give a few examples. But first, why am I here? My name is Arun Andreas Kimst. I work at Capgemini as a developer, intranime,
and I work with .NET, with the .NET framework, C-sharp, most of the time. I've been programming since I was eight years old, so it's been a while, and I always loved computer games. So, computer games and that stuff is something I always liked to make and not because I want to publish the games,
but because I think it's very good learning working with games. So, MonoGame, MonoGame is an open source cross-platform implementation of the XNA framework. XNA is a game framework made by Microsoft for Windows, the Xbox, and Windows Phone 7.
And MonoGame, it's a more or less faithful implementation of the same framework. You can use it to make 2D games, 3D games, and it runs anywhere, more or less anywhere.
Of course, you can run it on desktop with Windows, Mac, Linux. It runs on mobile devices, iOS, Android, Windows Phone, Windows RT. It runs on Raspberry Pi, on PlayStation Vita. And if we remember that this is XNA, you also have the Xbox. So, by implementing a game in MonoGame,
you reach a wide variety of platforms. Now, how do you get started? Starting is easy. All you need is a computer, Windows, and Visual Studio. You need Visual Studio to compile content, even though you want your game to run on a Mac or Linux.
You will need a Windows PC yet. The guys making the framework are working on a separate content compiler. Content is sounds, textures, other graphics, music, that type of content for your game. And it needs to be compiled so that it's formatted
for the device you're running on. And iPhone use a different graphic format than a Windows box, for instance. If you want your game to run on the iPhone, you will need the tools from Xamarin, Xamarin iOS.
And for Android, you will need Xamarin Android. They also have a Mac version called Xamarin Mac. This is a version of the open source Mono framework that is .NET running on the other platforms. If you want your game to run on a Windows Phone, Windows RT, then you will need a Windows 8 PC.
And with that, you need some knowledge, of course. It's based on XNA, so the example code is available everywhere on the net. And there are framework that helps you build games, 2D games, 3D games, UI, all that kind of stuff. But you need basic programming knowledge, of course,
and you need to know some math, vectors, matrices, that kind of stuff. Using MonoGame is, it's also quite easy. I'm gonna give you the shortest how-to ever. I have six minutes left.
In MonoGame, and in XNA for that case, you have the concept of a game class. The game class implements your game. You have a game loop, an update method, and a draw method that is called continuously. And the update method is where you perform AI calculations,
all that stuff, and the draw method draws your game to the screen, and that's it. You have an initialized method where you can set up the screen resolution, connect to servers, anything you need to do, and load content where you load your textures, load sound if you need that.
And you can use the unload content method to clean up. Usually, you don't care, a game is gonna close. Just a few tips on making games. Remember to focus on one platform first. It's much easier. I prefer to work on the Windows phone,
since the tooling is so great. And when my game is running, it's usable, then I port it to, say, the iPhone and Android. And usually, it's done in maybe an hour, less than that. You can use more or less exactly the same code on all platforms.
If you want to use, say, the GPS or accelerometer or the camera on your phone or mobile device, then a good trick is to make an abstraction of it. For example, I have a game where I want to use gravity. I want to know what's down. And then I define an interface as the gravity service
that always tells me the way what is down. And then on my phone, on the platforms, I implement that service. And that implementation, I can make it fit the platform. So the accelerometer on the Android device
doesn't behave exactly like on the Windows phone device. But using abstractions and interfaces, I can use more or less exactly the same code anyhow. And since we are mobile platforms, remember to think performance. Always think performance.
Try to not create objects all the time. Try to reuse them. Don't use reflection. It's slow, you know that. But link is also terribly slow. So don't use that and try to pre-calculate stuff. Don't do as little as possible for each time your game loop runs.
And of course, remember good coding practices like test your code, separate concerns, and so on and so on. You know it. Now, and the final challenge, of course, finishing the game, that's the hard part. And I can't help you with that. That's your job.
I want to give you a few examples of what MonoGame can do. Because when I heard of MonoGame the first time, I thought, okay, it's for simple games. And you can, of course, make simple games with it, but you can also make quite impressive games. There is a game called Bastion.
It's a role-playing game that runs on the Xbox or Mac and Linux, and it's a best seller. They're selling it loads of licenses. It's story-driven, lots of graphics, lots of music, looks incredible, and it's really fun to play. Then you have Fez.
It's a combination of a 2D and 3D platform game. You have to see it to understand how it works. It's brilliant. It also runs on Windows and XNA using MonoGame. And finally, Skulls of the Shogun. It's a turn-based strategy game,
and a completely different type of game. It also looks really nice, and it runs on all kinds of Windows devices, Windows Phone, Windows 8, Windows RT, and again, the Xbox. Now, if you guys want to know more, I'm staying here for the rest of the conference.
You can take contact on Twitter, email, whatever you want. I love to talk gaming. And just a few resources. MonoGame.net is the site where the guys publish information about games, tutorials, all that kind of stuff.
Since this is open source, you can find it on GitHub, download, clone it. On GitHub, they have lots of resources regarding sample games and everything. So it's recommended. And this is based on XNA, so more or less,
all XNA tutorials are valid for MonoGame. So just Google them. You can, there is enough. Well, that's everything. Thank you very much for listening.
Hello, can you hear me? Well, you look like a friendly lot. Are you parents? Anyone? Raise hands. Children between eight and 18, keep them up. Now, keep them up.
Who are you going to go back and establish code clubs? Exactly, very wonderful. It's not very hard, actually. I'm going to tell you a little bit how to do it. And we try to remove all the obstacles that you might meet. My name is Simon Sonfert. I work as CTO at Bove in Oslo. And I'm going to tell you a little bit about why we started to start this movement
and what we're trying to accomplish. It's not very polite to plug your employer, but I had to give them a big, big thank you for allowing me to do this in my daytime since it exploded after about seven in April. And also, I would like to extend a big thank you
to Fugamut Wigling, who hosted this workshop for the children two days ago. Did any of you see that, go there? It was wonderful. We even got T-shirts for the children with code club on it. And real badges, so they felt like they were in the conference.
And also, Linda Sondvik, she established Code Club in England. And they are in 750 cities in England. So she helps us. So what's the problem? Sources within the EU says that IT represents 5% of GDP in Europe. That's not a lot, is it?
But it represents 25% of the growth in Europe and 40% of productivity growth. So IT is in all the companies, all the government, everywhere, you can't really extract it. It's in there.
I was lucky to have a Commodore 64, which really forced me to learn coding because that's the only way I could do anything useful with it. The problem is that with the advent, all these good user interfaces, all the people, all the children have been removed from the possibility to know that they can even start tinkering on what's behind the scenes.
It's like a car. You don't know that you can look under the bonnet and see that there's something you can do there. So they're becoming digital leaves. Now, if they want to change something, if they want to not just starting to make playlists in Spotify, but actually want to make
another music streaming service, how would they go about that? How can we actually have more recruits to pay our pensions in the IT industry? Well, the thing is that the way the children will actually experience this will be more like this. I'm sorry, Dave. I'm afraid I can't do that.
So what can we do to help them? I think that the government should be teaching, should be actually establishing programming in the schools. I mean, the code clubs are a good thing because you teach children to program, but it will be in areas where the people
are well-educated and resourceful. And I will actually just keep on with the social differences. And what do the government do? It's a wonderful quote. So I sit on the board, advisory board
of the local polytechnic, HÖI OA, engineering school in Oslo. And they were complaining about not having enough qualified student recruits. And I also sit on the board of Data Ferenin in Oslo. And we established just a local group to teach local kids the code. We thought just to have that in Oslo.
And then it really, really took off. It was like I just flipped the switch and people came running because everybody seemed to have this in mind. And I got with me Torg Eir Waterhouse, who is a very resourceful person from Ikotenorga, ICT Norway. And we run this together. It's a movement.
And what we want to do is to make the children be ready to, like I said, master and create things with digital technology, not just playlists in Spotify. So what do we do to accomplish that? Well, we arrange local code clubs or help people do that. And we help schools to teach programming.
Since April, we have been establishing contracts with 20 schools in Norway who will start teaching programming. And we arrange family coding events like I showed you the picture of first there. So you may ask, what to teach? Well, in the ages from eight to about 12,
you can start with visual environments like Scratch. Anyone try that? It's very nice. Even if you're not a programmer, as a parent, you can start using it. And CODO, which is Microsoft's alternative. And for the elder ones, you can use Python, which I think is the best to teach real programming.
And it was made to be easy to learn. Now, for those of you who put your hands up, it's not really difficult to teach programming because there's full coursework available. We've been allowed to translate all the sessions from Code Club in England. We have a wonderful cooperation with computing at school,
which is aimed at teaching at teachers in England. And they've allowed us to take all the material for all the levels and just use it. And be our mentors. And the Code Academy also allows us to just translate and use everything. They're very, very caring and very, yeah, just use everything, I can help you. So no problem for those of you who will start Code Clubs.
And we have become 750 people since April the 2nd. So we must have hit the nerve, in a sense. And we're established groups in, I think, 19 different cities. So you may ask, what type of organization are we?
We're not really an organization. We don't have really much of a hierarchy. We have no much bureaucracy. We're just there to create value, to teach the children. They say organizations organize, movements move. So we try to be a movement. And there's been tons of research about this
and how such movements create value. Basically, they have a website which allows people to do three things. Collect and distribute documented knowledge, like coursework, like experience papers, like how to talk to your local school board and so on. Connect those who know with those who don't. Like, for example, teachers who want to know,
they can be mentored. And we can connect and mobilize those who can solve or create something. For example, starting Code Clubs. So that's our aim. And we're getting there. We're getting a website which allows us to do that, which means that we don't need much staff. Like Computing High School in England,
they say no money, no staff. And I don't think we really need that in the long run. Sponsors, of course. Sponsors who can pay for robots, for Arduinos, to have in kids that can be lent and borrowed in the schools. Yes, of course, but not much for just administration. If you want to read more about this, I was allowed to write something in this paper here,
this column there, and it's also on their website. But I'd like to add something. I'm just a tech bloke from the industry. I'm just someone else. I mean, I just flipped the switch. It was an explosion waiting to happen. And the real heroes, they were the ones
who came before us. They were teachers, university lecturers, programmers, who did a lot on their own, but they didn't know about each other. So we gave them a home. And I think that our success is really attributed to their experience and they're willing to share
and have made us reach a long way just in a few months. So back to what you can do. I want you to think if you can take a friend in the business, a neighbor, if there's a school, if you're sitting on the school board for the parents, maybe you can suggest it. We're trying to make all this experience available
on the website, so you have as few obstacles as possible if you want to do this. And all you need to do is to have kids, the coursework, a venue, friends, and there you go. If you want to know more about this or join in,
I'm sure that, I mean, in England, you have computing at school, you have code clubs. In the US, you have code.org. I think you all have these things in all the countries. But here in Norway, we're on that website. Join in, volunteer, follow us on Twitter, follow Facebook, and form a code club.
Thank you.
We created a system for your safety. You must share your knowledge.
Sharing knowledge is very hard, at least I think so. Even if you're part of a very small team, having the overview of what's going on, especially in the code base perhaps, requires quite some effort from all the participants in the team.
And the problem with the lack of this overview is that it tends to make developers solve the same problems several times, right? You don't know what your colleague did two days ago. He might write some code that you actually need today,
so you didn't know, you write it as well. And the lack of knowledge also can make developers make errors. And you tend to repeat the history.
And that's not a good idea, right? We want to keep dry. So this clever man here said long before me, but he turned to agree that those who can't learn from the history are doomed to repeat it. So I think we all can agree that learning what our colleagues have been doing before us
is very important. If we don't do that, we'll end up in a maintenance hell. We have lots of duplicated code, lots of errors, and stuff like that. Now there are some practices. I'm pretty sure all of you are familiar with, like pair programming. Pair programming and code reviews
might help with this problem. But the thing is, you only share your knowledge with one other person. If you're five on that team, the rest of the guys don't know. So by now, you probably wonder, who am I?
My name is Thomas Peterson. I work at Miles in Bergen as a consultant. I do software development, architecture, and stuff like that at clients. And I'm here today to teach you a little thing that I started doing for, I don't know, maybe five, six years ago.
It's not a silver bullet, but it's something, and I can almost certainly promise you this, it's something that will improve the quality of what you as an individual, as well as your team, delivers. So with no further ado, it goes like this.
Drink a lot now. I do, and I think most of my colleagues do. You start the day with a cup of coffee. And then you go to your computer, you sit down, and you open some yesterday's news.
And you sit there for like 10, 15, 20 minutes, I don't know, an hour, perhaps, some of you, before you actually start working. And that's fine. You need something to get going in the morning. That's totally fine. But what I suggest you doing instead is replacing that old news from yesterday,
which isn't really that interesting either, with this. This is a commit log. For those who doesn't use Git, this is a commit log from Git. So you pull this up, and you can do it in, well, any way you like. If you use GitHub, it's excellent.
If you use TFS or whatever, you can do this. So you pull up this log, and you start reading what was done yesterday. So you go back to, well, not seven months ago, but at least back until yesterday, you read the commit messages,
and you also read the code that was checked in. When you find something good in here, you will learn something. When you find something not that good, you have an excellent opportunity to tell your coworker that maybe this wasn't a good idea. I can help you correct it.
And basically, it's just that. It's that simple. So if you start your morning routine with a cup of coffee and pull up this log and actually read through it, you will be amazed by how much stuff you actually will learn.
And you might experience, it can be a bit hard in the beginning, especially if you have a lot of crappy commits. Has anyone done that? Hands? No. Oh, there's one.
But guess what? You actually will get better at committing as well. So he has improved quite a lot already here. And you will learn a lot about committing and actually making the history in your source control useful. So you can actually use it
for more than just a backup of files. So not only will your commits be more focused, you probably end up with having one feature in there at a time. Code cleanups go into separate commits and stuff like that. And you probably also learn to make more specific commit messages.
It might take some time, a couple of months, years, who knows? But you will get better. And if you start doing this and encourage your team to do this, the source control, as I mentioned, will no longer just be a backup of your files.
You can actually use this history to both learn from it and also it's much more focused, so it's much more easier to go back and find errors and stuff like that. I want to mention one more thing.
And I briefly mentioned it in the beginning. Code reviews. Code reviews are also very hard. Not doing them necessarily, but getting people to do them. If you do this little exercise here,
which take you a couple of minutes each day, you also get code review for free. If you've spot an error in there, you can fix it or tell someone. And doing code review is a good practice, right? I hope everybody agrees. Who agrees? Okay, good.
It's not all me, then. So you get it for free. You might not even call it a code review. You can call it a check-in review or daily routine or whatever. And why does this work, in my opinion, or at least from my experience, is doing code review is another chore
or a task that are laid upon the developer. Developers want to code. They do not want to test. They do not want to do reviews and not write documentation. That's certainly. So it's another task that are laid upon them, and they really don't have the time for that.
Another thing is that a code review can, someone can feel like you're looking into their code trying to see if it's screwed up, right? With this approach, you eliminate all that. You do this for your sake, for learning, and then again, improving the entire team.
And actually, you'll be amazed how much you will find in code. Not only errors, but you'll learn a lot of stuff about domain you're working in, fix errors and stuff like that. So on Monday, most of you
will probably return to your office. And my hope or whatever is that you will actually start doing this, start learning from the history and stop repeating it, at least, and encourage also your teammates to do the same. As I said, I've done this for five, six years now.
The current team that I work in, I think almost everybody does it. And it's been an amazing help for us. And it's very easy. It takes basically no effort. You're going to drink that cup of coffee anyway, so. Thanks.
Yes, hello everyone. My name is Ulmar Temelk.
I work at BEC Consulting, and I'm here to talk about polyglot persistence. But first, I'm going to talk a little bit about architecture. What's the starting point of good architecture? My view, architecture often starts by figure out what needs and requirements
the customer has for the application you're going to make. And in order to create what the customer needs, we have to know the needs and we have to know their requirements. And we draw boxes and we draw arrows. And these systems talk with each other
on different protocols. Some on SOAP, some on REST, SQL, queues, et cetera. Some synchronous, some asynchronous. And we have different requirements for all these systems. And we keep on drawing and we create the system that meets the needs of the customer.
And at the end, we add a repository, one database. And the challenge we face by adding one database is that one database often won't be enough to meet the customer needs. Let's say you choose to use a relational database.
You get challenges with performance, with scaling, with object relational impedance. If you try to use a key value data store, you have problems with the relations between the data. You don't have relations in key value stores.
And if you use a graph database, you get challenges with scaling and object lookups. And whatever database you choose, you will meet some challenges. And we're aware of this. We know the strengths and weaknesses
of, let's say, a relational database. But most of the time, we don't do anything about it. And we have, it's so easy for us to always choose the same database technology. We don't, we don't inspect the requirements or needs of the customer when we choose database.
If we had done that, we would have seen that choosing the correct database technology isn't an easy task. So let's create an application. And this application so far has one requirement. It needs to read and write data.
That's the only requirement we have. And usually, if we want persistence in our architecture, it's because we need to read and write data. That's the primary need for persistence. And if we follow the, do the simplest thing
that could possibly work, and these are the needs, the best solution, in my mind, is a key value store. It has three operations. It has get, put, and delete. So we get the third one for free. Because given these needs, the most important thing
about the database is that it's easy to use, that it's fast, and it doesn't mess up your code. Your domain model will look the same in the database. But then the customer comes, and he has another need. He wants to search the data.
That's no surprise because so far we only have lookup, and that only gets us this far. Often you need to, let's say you have a personal database, you need to search by name, or you want to find all people living in Oslo.
And a key value store doesn't support those requirements. And a relational database does, but not very good. So we need search. And lots of systems have a search engine, like Microsoft figure that search is so important, so they're both fast in order to get
the search engine in SharePoint. And all content management system have both a database and a search engine. So in order to search, we add the search engine to our architecture. Not a big problem, architectural-wise. The application has to talk to both
the key value store and the search engine. And if you get some transactional problems, you can add some synchronization. So hopefully the data will stay the same. Then the customer comes along and has a third demand. He needs relational search.
And search engines doesn't do that very well. It can do it, but with a lot of duplication and heavy queries can be done, but it's not good at it. Let's say I'm creating an application from a consultant firm.
And we have an employee database. And the seller comes along and wants to find a consultant that has experience with F Sharp in two different projects, has a five-year experience as a consultant, has these specific types of certifications.
That's a query it's possible to run in a search engine. It's possible to run in a relational database, but that join is hard to write, hard to read, and heavy to execute. If you use a graph database,
that query is easy to write and read and really, really fast to execute. So we choose a graph database. The problem, though, is that this architecture doesn't look that very good.
The applications have to talk to three different databases and the databases has to know of each other. So let's remove those relations and let's remove the relations between the database. And reorganize, and we have this.
So we have three databases, but two of those databases doesn't do anything. They need some data. In order to give them data, we add a persistent event store. So every time there's a change to the data,
it's written to the key value store and the event store. And we hook up these other databases to that event store in order to those, so they can get some data. And it's a good thing if this event store is persistent. Because if we have to take down the search engine
and add a new hard drive or something, we can plug it in and retrieve all events since last time we were connected. Or if we want to add a completely new database, we can just play back all the old, all the events and fill up the new database.
That's what we have to do in order to meet these requirements. The customer has a need to generate reports. Suddenly the data warehouse people or statistics or economics comes along, and they have bought this wonderful reporting tool and they want statistics.
Problem is that we have a search engine, a key value store, and a graph database. And there's no reporting tools that talk to these data sources. And there's no problem. We just add a new listener to the event store. A relational database.
And there's several positive things about this, but first thing is that it's easy to add and it's easy to fill up with data because we have all the change events. And the second thing is that these folks can do whatever they want with this relational database because our application isn't dependent on it.
So they can run heavy reporting jobs in the middle of the day while our application is in production. We are not affected by it. So that is polyglot persistence. At least it's one way
of implementing polyglot persistence. And we know there's no silver bullets in anything. Nothing programming language, not in operating systems, not in libraries, and not in databases. Relational database may seem like a silver bullet, but it's not. It's not the best solution for every problem.
You pick the database that satisfies the needs you have there and then, and you build an architecture that allows you to expand and accept new requirements from the customer because those requirements will come.
Thank you. The structure needs for my talk today.
And during our project, we asked ourselves several times, is Microsoft moving into hardware business because we saw that the hardware requirements they have to implement enterprise search is huge. I'm going into that.
To simplify our task in the project, we needed to index 40 million documents and to persist the 10 queries per second. That is, I use a query from our middleware into the index is one query.
And that comes from a user texting in a search box and hitting the search button. But before I continue, how many of you work with SharePoint today? And how many of you work with search?
Yeah, some of you, good. All right, let's continue. So to implement search or create the infrastructure that supports these needs, we need our web front end. That is where we present our search results, where we present the search box.
We need some indexes. That's where the system persists the index or create the binary index of the content that we want searchable. We actually need four of these. There's our maximum limit that SharePoint defines. It needs to, or the maximum limit
is 10 million documents per index. We also need to get the documents from the source system. Our source system can be either SharePoint itself. It can be file systems, it can be databases, and it can be any custom content sources that it creates.
And of course, also web pages. There's a new component in SharePoint 2013 that is analytics, and that is a process that can give you information on how people are using the system, and especially search. We need some admin components.
We need our query processing. That's where the logistics or administration of the query, each query has to ask all the indexes, and it has to get the answer from all the indexes, and it manipulates it and sends it to the server. That's the query processing unit, and what it does.
One more admin component, it's a search admin. For documents to come into the index, it needs to be handled in some way, and it needs to be processed, converted from a binary data to a text form that the index can store, and that's done in the document processing units
or components, and they have a lot of them. That makes us be able to process from 50 to 100 documents per second when we do a full speed on the crawl. And we need a lot of databases. So now we have a setup that can handle 40 million documents.
In theory, it can handle 10 queries per second, but if one of these nodes disappears, the whole system is down. So we need redundancy, and we need another web front end, and we need a new set of indexes, and we need our crawling and analytics,
and of course the Docbook databases as well. So basically, to handle 40 million documents, we need 12 nodes. In our project, we used
a virtual system, virtual machines, which were hosted on two physical servers. What we also did is we split the two parts of the farm into two separate data centers, which means that our whole data center can have their power cut off,
and the search system will still be up and running and working fine, both with updating to new content and also giving results from queries. We also added our own business middleware, which is called Compare-a-Front,
which adds some additional functionality to search. We have done some content enrichment, where you can do some custom changes to the document coming into the index. Basically, so the infrastructure and investments,
we have 12 VMs, for each VM we have eight cores, basically results in 96 cores, a lot of memory. We started off with 16 gigabytes of memory, but we had to double that
to get the performance that we needed. We need a lot of disk, system disk and data disk. The data disk contains the log files and binary index. That has actually improved dramatically from the last version of FAST, so that's quite good.
And our search system is also very IO dependent. So, per indexer, the recommendation is 200 IOPS. So that basically means we need a system that can handle at least 1,600 IOPS. That's quite much. But this is also a big improvement
from the last version of enterprise search in SharePoint. In addition to this, we need fiscal servers. We have two. We need database servers, load balancers. We need a good sound system and someone that can set that up properly.
Domain controllers and other networking. You need a lot of licenses. SharePoint, SQL, Windows, Visual Studio. And in our project, you also needed to compare front licenses. And to properly work with developing on SharePoint, on this SharePoint search solution.
We also needed a development environment. That is a single VM with four cores and 16 gigabytes of memory. We had a QA system, which was two nodes.
The same size as the production. We had a test system. It's a single node, the same specs as the production. And we had a user acceptance test environment, which was exactly the same as production. So a lot of hardware. We have an extra two fiscal servers
to handle the UAT system and the rest of the virtual server was placed on a separate development environment. So key points that you have learned. What to work with search in SharePoint 2013.
And doing more than just setting it up and turning on search. That means that you have to customize how search works for a customer's needs. We need a lot of funding. We see that.
And at the same time, a lot of time. In this project, it was quite special because we had a tap project. We worked on a system, which was in beta mode and no documentation whatsoever. And we saw that you really need documentation
working with SharePoint. We didn't have that. But we had a good network. As I work at Compario, we have a good relationship with the old fast, which was bought by Microsoft and who still develop the search for SharePoint.
And we really needed to pick the ones that we knew worked with us and to get the information that we needed. We saw that automated deployment was really effectful and it helped us a lot. That made us be able to just deploy
the functionality that we have now and continue working on development new stuff. Saved us a lot of time. You can improve performance dramatically by adding more CPU to the search servers.
Adding memory, we saw that by doubling memory. We saw the search query performance quite dramatically reduced for the search time. Optimized SQL and have load balancing network, good.
We had an issue with distributed cache where we had a lot of timeouts. So we had to tune the cache to be bigger than the default is. That might be because you have a lot of uses on the system. And the last one, you have to know your antivirus.
Turn it off completely on the servers or at least exclude the index folders. That will hit hard on query performance. That's it. Any questions? We are at the compare-a-stand. Talk to us.
And, bear time. Thanks.