Web Scraping Best Practises

Video thumbnail (Frame 0) Video thumbnail (Frame 611) Video thumbnail (Frame 1424) Video thumbnail (Frame 2364) Video thumbnail (Frame 4229) Video thumbnail (Frame 5497) Video thumbnail (Frame 8505) Video thumbnail (Frame 11052) Video thumbnail (Frame 14833) Video thumbnail (Frame 18671) Video thumbnail (Frame 23532) Video thumbnail (Frame 30834) Video thumbnail (Frame 31496) Video thumbnail (Frame 32196) Video thumbnail (Frame 33416) Video thumbnail (Frame 39482) Video thumbnail (Frame 40503) Video thumbnail (Frame 41206)
Video in TIB AV-Portal: Web Scraping Best Practises

Formal Metadata

Web Scraping Best Practises
Title of Series
Part Number
Number of Parts
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date
Production Place
Bilbao, Euskadi, Spain

Content Metadata

Subject Area
Shane Evans - Web Scraping Best Practises Python is a fantastic language for writing web scrapers. There is a large ecosystem of useful projects and a great developer community. However, it can be confusing once you go beyond the simpler scrapers typically covered in tutorials. In this talk, we will explore some common real-world scraping tasks. You will learn best practises and get a deeper understanding of what tools and techniques can be used and how to deal with the most challenging of web scraping projects! We will cover crawling and extracting data at different scales - from small websites to large focussed crawls. This will include an overview of automated extraction techniques. We'll touch on common difficulties like rendering pages in browsers, proxy management, and crawl architecture.
Keywords EuroPython Conference EP 2015 EuroPython 2015
Scripting language Metropolitan area network Inclusion map Slide rule World Wide Web Consortium Computer animation Maxima and minima Binary file World Wide Web Consortium
Scripting language Metropolitan area network World Wide Web Consortium Escape character Computer animation Multiplication sign Internet forum Curve fitting Hand fan Physical system World Wide Web Consortium
Metropolitan area network Markup language Executive information system Multiplication sign Projective plane Source code Maxima and minima Grand Unified Theory Binary file Semantics (computer science) Value-added network Inclusion map Goodness of fit Computer animation Internetworking Universe (mathematics) Software framework Form (programming) World Wide Web Consortium
Statistics Multiplication sign Execution unit Source code Mereology Distance Discrete element method Mathematics Goodness of fit Internetworking Googol Forest Representation (politics) Software testing Data structure Website World Wide Web Consortium Scripting language Metropolitan area network World Wide Web Consortium Stapeldatei NP-hard Content (media) Sampling (statistics) Limit (category theory) Band matrix Software development kit Arithmetic mean Computer animation Internet forum Website Library (computing)
Metropolitan area network Regulärer Ausdruck <Textverarbeitung> Haar measure Source code Content (media) Division (mathematics) Maxima and minima Euler angles Process (computing) Film editing Computer animation Googol Boom (sailing) Website Arc (geometry) World Wide Web Consortium
Complex (psychology) Web crawler Link (knot theory) Markup language Code Multiplication sign Semantics (computer science) Arm Computer icon Twitter Mach's principle Term (mathematics) Googol Data structure Website Error message Task (computing) Scripting language Metropolitan area network Demo (music) Projective plane Ext functor Bit Connected space Type theory Computer animation Personal digital assistant Logic Website Pattern language HTTP cookie Library (computing)
Web crawler Mainframe computer State of matter Modal logic Multiplication sign Source code Combinational logic Set (mathematics) Permutation Uniform resource locator Order (biology) Mathematics Hypermedia Single-precision floating-point format Arc (geometry) Physical system Scripting language Metropolitan area network Software bug Link (knot theory) Theory of relativity Mapping Gradient Bit Web crawler 10 (number) Intranet Order (biology) Website Writing Web page Stapeldatei Link (knot theory) Limit (category theory) Event horizon Goodness of fit Googol World Wide Web Consortium Stapeldatei Matching (graph theory) Scaling (geometry) Uniqueness quantification System call Length of stay Computer animation Search engine (computing) Personal digital assistant Internet forum Boom (sailing) Vertex (graph theory)
Web page Point (geometry) Stapeldatei Standard deviation Web crawler Server (computing) State of matter Logarithm Maxima and minima Control flow Regular graph Formal language Uniform resource locator Internetworking Googol Single-precision floating-point format String (computer science) Metropolitan area network Scaling (geometry) Information Smoothing Projective plane State of matter Content (media) Instance (computer science) Software maintenance System call Open set Computer animation Personal digital assistant Boom (sailing) Website Pattern language Right angle Pressure Surjective function Library (computing)
Web page Standard deviation Parsing Correspondence (mathematics) Multiplication sign Syntaxbaum Set (mathematics) Mereology Rule of inference Arm Number Revision control Uniformer Raum Different (Kate Ryan album) Computer configuration Googol Aerodynamics Data structure Arc (geometry) Condition number Area Metropolitan area network Dependent and independent variables Constructor (object-oriented programming) Expression Content (media) Binary file Port scanner Regulärer Ausdruck <Textverarbeitung> System call Parsing Computer animation Personal digital assistant Boom (sailing) Right angle Musical ensemble Object (grammar) Table (information) Library (computing)
Discrete group Complex (psychology) Building Context awareness Multiplication sign Visual system Arm Medical imaging Heegaard splitting Machine learning Metropolitan area network World Wide Web Consortium Pattern recognition Algorithm File format Structural load Web page Moment (mathematics) Token ring Sequence 19 (number) Element (mathematics) Parsing 10 (number) Category of being Type theory Website Self-organization Online chat Web page Point (geometry) Link (knot theory) Token ring Codierung <Programmierung> Virtual machine Maxima and minima Plastikkarte Rule of inference Field (computer science) Value-added network Element (mathematics) Tabu search String (computer science) Data structure Gamma function Form (programming) Newton's law of universal gravitation Module (mathematics) Information Demo (music) Interface (computing) Projective plane Cartesian coordinate system CAN bus Word Computer animation Personal digital assistant Logic Normed vector space Library (computing)
Web page Metropolitan area network Scaling (geometry) Parsing Electronic program guide Projective plane Virtual machine Electronic program guide Content (media) Sampling (statistics) Website Website Chi-squared distribution
Web page Metropolitan area network World Wide Web Consortium Web crawler Pattern recognition Computer virus Link (knot theory) Virtual machine Content (media) Information privacy Value-added network Medical imaging Web service Computer animation Googol Boom (sailing) Website
Multiplication sign Bit rate Open set Mereology Programmer (hardware) Mathematics Web service Invariant (mathematics) Different (Kate Ryan album) Cuboid Data conversion Extension (kinesiology) Information security Vulnerability (computing) Chi-squared distribution Area Scripting language Metropolitan area network World Wide Web Consortium Software bug Arm Software developer Demoscene Type theory Category of being Data management Vector space Textsystem Internet service provider Artistic rendering Website Spacetime Web page Slide rule Identifiability Proxy server Link (knot theory) Web browser Black box Template (C++) Product (business) Regular graph Robotics Natural number Profil (magazine) Googol Software testing Data structure Proxy server Scaling (geometry) Projective plane Expression Content (media) Plane (geometry) Particle system Computer animation Doubling the cube Thermal radiation Musical ensemble Communications protocol
Metropolitan area network Computer animation Googol Maxima and minima Quicksort
Web crawler Variety (linguistics) Multiplication sign Client (computing) Power (physics) Direct numerical simulation Goodness of fit Mathematics Computer configuration Software framework Data structure Office suite Error message World Wide Web Consortium Metropolitan area network Default (computer science) World Wide Web Consortium Scaling (geometry) Information Software developer Content (media) Planning Basis <Mathematik> Bit Fehlererkennung Category of being Arithmetic mean Process (computing) Computer animation Personal digital assistant Website Heuristic Quicksort Library (computing)
that they have everyone thanks for coming and I'm very sorry but the technical difficulties and we that it would have to set of and prepare and and I really don't please try not to look ahead to fire in the slides and who's going to be difficult and
but there you OK so I'm gonna talk about web
script and best practices and
I really call this advanced web scraping and because of going to touch on a lot of advanced topics and but it's not advanced in the sense that you need to be passed to begin elaborating to understand and so I changed the best practices and I hope that it everybody can follow this stock and understand what's going on in the capitalist system shows no
and so that about me let's see how an 8 years ago about that I started scraping had anger and those around the time when we did this script this greatly with our
framework and then since that time we've been involved in a couple of projects as great including portion and frontier and if you don't know what they are don't worry I'll get to that later and so why would you want to greater well thus
a good sources of data on the internet and actually come across a lot of companies and universities and research labs and of all different sizes were using scraping and but you know getting data from where this is difficult and you can't align API you can't rely on and semantic markup so that's where which really
means the distance that's a baby can read them very well because it's more but basically and with greater has been on the increase recently and we've seen that ourselves but this has been also something we've seen from other companies reporting the stats from company called in capsular and that provide antibody scraping technology and and it's a sample of their customers so is probably not to the representative of the internet as a whole and but still it is very interesting to see and another thing that it that I can see from this as well as that of smaller websites have a larger percentage of all traffic probably because they have less users but it's something to keep in mind and especially if you had bad thoughts and you know they cause more trouble for smaller websites as moderate might bandwidth limit for example and and many HTTP libraries they don't compress content and so you need to go over and and their units and also cost you know being a batch of means you're web very hard to maintain and this is in the forest problem cost is websites change and so when I think about scraping I like to think of it as in 2 parts and the 1st is actually getting the content so it's
finding good sources of content and and downloading of and the 2nd is the extraction and actually extracting structured data from that time content and kind of structure this talk into parts as well that policy so as an example of web scraping and adjusted that's creeping up it's script all the time and is not just people testing of great years or something like that or tools
and but actually a couple weeks ago we posted a job ad on a website and the next day it was up on a job to for jobless support and and none was posted there so we thought well how that happened and and I think we're greater and so the question for the audience would be to think about how would you write paper and I I would break it down into you had was fine with
sources of content and headway extracted data it turns out that we tweeted about the job so hashtag remote working and so maybe somebody picked up into the cut retweeted and that we need source content and and we did you
semantic markup so perhaps the extracted from that and that's relatively to write such as reproductive do this is relatively easy task is a big dude and they may be and but then if you wanted to do say handle cases where people didn't use semantic markup I wanted to find people who didn't posts to a tweet about it so posted some other web sites then it becomes a much bigger and much more complex task and anything that kind of highlights the scope of what pre webscraping from the kind of very easy and to on accident occurring on to very ambitious and very difficult projects that happened and so getting on with onto downloading and yeah like mentioned Python requests library and probably many people know as the great library French distinctly and and doing simple things simple as it should be but when you start scraping and and is more skilled and you really want to worry a bit more about a few of the things that like for example retrying requests that failed but certainly when when we started out you know you could run on a scraping it it might take days to finish and and then about 3 quarters of the way through you get a network error you get and you know the the website itself grating but suddenly returned 500 internal server rare for for 10 minutes and so if you don't have some policy to handle this and it's it's a huge pain in the ass so did you wanna think about that and I also this in this example you can see I'm using in a session and but I don't know if you consider not because it's small but consider using sessions with Python requests and sessions Britain and the cookies they also use a connection keep alive so you don't end up repeatedly opening and closing connections at the but I would say as
soon as you start calling you really want to think about using scrapie right away and this this little example here is is not much code uses best babies crawl spider which is a common pattern for scraping for crawling and you know just defining 1 rule structural and that's enough to go from the ah your Python websites for this conference to use actually follow all the links to speakers and then you just need to fill in some code to parse the speaker details so it's really not much gold it's and it solves all the problems like highlighting our solves all problems like me trying etc. and you can catch the data locally which is good for like most of the time and yeah so a single as Singapore like that often turned into a calling multiple websites and had icon US in 2014 we did a demo and ended it's up and spreading of get up and it's called Picon speakers and where we actually great uh data from a whole lot of tech conferences and this is a really good example of that because it shows you can this shows a way to manage and how script project looks we've got lot of spiders and provides a lot facilities for managing not like you can list of of all the spiders that are there and the spider is the logic that we write for a given websites and and it also shows best practices in terms of you know it's easy which may be due to put Common Logic in common places and shared across some of the websites when they're calling the same type of thing and there's a lot of scope for reason and so it's 1184 skating what the sites of ViaScribe is no brain and
some tips for calling and find good source of things and some people maybe might not think about using site match against rejection as a site map spider that that makes this a pretty easy and transparent and but often you know it to be much more efficient way to get to content and that also means of course don't follow on necessary links and yeah this is said you can reason of resources for everybody following stuff that doesn't follow and consider call order um yes so if you're discovering links and web sites it made might make sense to crawl and to crawl breath 1st um and limit the depth to go to this this can help you avoid quota were maybe you're and all repeated scraping calendar for example and just going through the system is common example that I used to working in a company before that had had a search engine and you can cause every now and then would enter policy-making to it and follow all the search facets and turn every permutation combination search and and this generated future course and so on as you decide to scroll scale of and so let where I was talking here about maybe single websites grades which is I guess the most common use case at least special prescriptive and you know single scripts can be big right writing we frequently do maybe hundreds of millions of pages and by that scale say for example you're running a vertical search folks crawler and then we're talking maybe maybe tens of billions or even hundreds of billions of discovered your house and so on so you you might call a a certain set of pages but the amount of your relatives government-paid on those pages so your entire states that you need to keep in New York frontiers what can be can much much larger and still maintaining all of that is is a bit a headache it's a lot of data and 1 common way to do it is people just write letter somewhere um and then perform a big batch computation to maybe to figure out the next set of unique euros to crawl and typically using Hadoop a MapReduce it's a very common thing and it's and maybe not just good example that an and then incremental crawling uh would be where you are continues crying actually would be where your continuously feeding your last year your crawlers and this has the advantage that you can respond much more quickly to changes that you don't to stop the call resume and but it also
nowadays maybe you want to repeatedly hit some websites media following you know as social media or something like that there are good sources of events and so it's it's much more useful and blood is much more complex at the same time it's it's a harder problem to solve
maintain politeness is very little point on the bottom and but it's something really wanna consider when you're doing it on any scale I think almost anybody can fire up a lot of instances nowadays and easy to and are your favorite platform and and just done a lot of things 10 of those pages really quickly and without putting much thought into what those pages are and are particularly the the impact is going to have the web sites across uh in a larger call where you're calling from multiple servers for you would typically was you typically and going to call a single web sites from a single server and that's everything and maintain politeness the state whenever you common policies are you don't you don't break it so our frontier I thought I briefly mention is and Alexander is very complicated token it yesterday and this is the python project that we worked on and were working on the fact that that implements this profiteer here and so it maintains all the data about visited your and tells you what you should call next and there's a few different configurable back installation so you can use an embedded in your scrapie crawler you can just use it in API with their own thing and and you know it implements some more sophisticated revisit policies so if you say want to go back to some pages more often than others so then maybe you know keep keep constant pressure and you can do that and and I think the Alexander particularly talk about doing it scale and so he to call of Spanish Internet and ideas that we talking about that in the poster session is also discomfort so just to summarise quickly what we talked about downloading and requesters and also and for simple cases but once you start going out it's better to be descriptor quickly maybe you would even want to stop there and if you need to do anything really complicated as superscalar skills considered from 10 smoothing onto extraction extraction is the 2nd and part of that that I want to talk about and of course patterns of a language for extracting for extracting content information with strings are missing the data discovery talks at this conference about by managing
data with and but even just a simple right you know about that and just languages and to the standard library make it very easy to the text content and regular
expressions of course is is this 1 thing that built into the library and probably uh yeah which I should mention something about it and it's red expressions of Brentford textual content and the yeah works great with things like telephone numbers they're post and but if you find yourself ever matching against the HTML tags annotation of content you probably made a mistake and it's probably going to be a better way to do it uh license cold all the time of of regular expressions and yet works fine but it's hard to understand and then modify and an alternate exon doesn't work fine and so other techniques music HTML parsers and so that we have we have some great options uh yeah so if you want this is when you want to extract based on the content of based on the hate the structure of it's not pages and so often you OK this area here surrounded by this underneath the table that is engaged in a parser is actually the way to go yeah so just a brief example Elliott and right inside a had some examples of his team parsers and an XML hitched enough 5 live Beautiful Soup grumble land correspondences so that conditional parser talk about them in the morning minutes something more you can't see that and so just as as a brief example of what they do is take some time take some role hits to now here that that looks like text and an entry a parse tree and my favorite way of dealing and then you know you some technique usually these part of these parsers provides a method to navigate this parse tree and extract the it's a interesting and I don't you can see that all skip this quickly but there are a quite like XPath and as a way to do this that it's very powerful you can and in this case just select all both sides a bowl tag under Dave you know the the text from the 2nd of time uh it lets you as specified rules if we were learning and if you're going to do doing a lot of this uh the uh yeah here's a here's an example from scrapie or you don't really need to read that um but basically just lets you scripted provides a nice way for you to call x paths are CSS selectors and on responses and so this is probably the very definitely the the most common way to scrape content from a small set of known websites and right I just want to mention beautiful super as well and this is a very popular Python library and maybe in the early days it was a bit slow and but the more recent version you can use different parser ends and so you can even use use because went up by smile and the main difference with uh with the example I should previously is that it it is isn't pure Python API so you can navigate content using using Python constructs and
and Python objects verses XPath expressions and give
that the other thing is of course but you might not need to do this at all maybe somebody has already written something to extract what you're looking for and so definitely maybe this stuff you would
need to think of and some examples of things that we've done and it is the a lot in form of the module for scraping that automatically fields not because informs 19th websites and we have a deep parser module that takes textual strings and can can the data object Thomas and and web pages another another and and another project of which looks at the HTML page and will pull out links that perform pagination and the 2 which is often used and I was going to like them with this but I think we're probably short on time and and you know maybe it's not tendencies we enough technical problems already and what's that portion is a visual a visual way to build web scrapers and it's applicable in many of the cases where we had previously mentioned where we would use XPath you soup and what its advantages is that a nice you i where you can visually say 0 I want to select this element this is the title this is that as image this is the text and I was gonna demo this about scraping and then you're fighting websites and maybe we if something was probably later I can do this with an but it's really good at it is a lot of time however and it's not as applicable you know if you really want to be if you if you have some kind of complex rules and complex extraction logic and you can you can you might not always work with this and at all costs you want to use any of the previously mentioned stuff like automatically and extracting dates and things there might not be built into Portuguese and so scaling up extraction using portions great it's much quicker to write extracting extraction for a websites but at some point it becomes pointless again now you might be scraping 20 website that's fine 100 people have used a discrete thousands and by tens of thousands maybe even hundreds of thousands at this point you want to look for different techniques and there's some libraries that can extract articles from any page there is easy to use and I want to focus on quickly on a library called web structure that we worked on and that helps with automatically extracting data from from his pages and an example of the use of named entity recognition and so in this case we want to find elements in the text and assignment categories and so we started annotating Web pages of that type of stuff at the top of the label webpages basically with what we want to extract them as examples and we're going to use a tool called WebAnnotator and but there are others and here's an example of labeling and in this case we we want to find organization names and so the whole tea cafés is 1 of his organization in and we would label is within a sentence within a page and that format is not so useful for machine learning and and for the kind of uh 2 of you want to use so we would and of course that text split into tokens each token in this case is the word and then and we label every single token in the whole page being either outside of the what we has been as being at the beginning of an organization are inside an organisation and given that encoding then we can apply more standard machine learning algorithms and is yet in in our case we find conditional random fields and uh as as it is a good way to go about it but a important point is that need to take into account the sequence of tokens and are the sequencing of of information some features so we've needed basically not just the tokens itself but actual features and the features can be things like about the token itself and but they can also take into account the surrounding context and this is a very important point and we we take we can take into account the surrounding text or even you know it's 2 moments that embedded in and so it is the you can be quite so 1 way to do it and this is uh what we've been doing recently and is to use our web struck projects and and this helps you know load the annotations that were done previously in WebAnnotator and called you might call back to your Python modules that you write yourself to do the feature extraction and then at interfaces with with something like Python serous serous we and to actually from destruction so just this just
briefly to summarize and you know we use a different technologies depending on the scale of extraction and the HTML parsing and Portia are very good for a single pages in the web sites are from what the web sites if we don't have too many and the machine learning approaches are very good if we have a lot of data we compromise appeared on maybe the accuracy and but that's the nature and that I just
want to briefly mention sample of a project we will some recently actually were still working on it and you might know this actually artery and it's a it's a gallery of contemporary art in London and we did a project with them for that to create content for their global gallery guide now this is a ambitious project to showcase artworks and artistic exhibitions from around the world so it's fun project and it's nice to look at art works of and so
course we use greatly for the crawling we deployed it to 3 which is a scraping of as a service for running across and and we we use web page or 1 of the tools I mentioned earlier to and to actually pagination and so the crawl we we prioritize the links to follow and so we do so using machine learning so we don't and find ways to many resources on each website with great and once we hit the target Web pages we then use web pages to to paginate and so that's called inside and the extraction side we use Web stopped very much like privacy described and 1 interesting thing that came up I thought was that when we were extracting and images dropped after artists it we often that and wrong and then
we had to use it classification to be actually classify them and based on the image content using face recognition to see which 1 where artists versus artworks and so it was working pretty well this is a scripting 11 thousand websites hopefully to continue and increase so 1
important thing of course is to measure accuracy and the test everything improve incrementally and and it's also that's not treat these things that too much like a black box to understand what's going on don't make random changes that it tends to not work so well so we feel we've covered downloading content we covered extracting it seems like we have everything to go and great at large scale but there's still plenty of problems and describe such and you in the last 5 minutes and of course web pages have an regular structure and this can break crawled pretty badly it happens all the time and from people using superficially some websites look like structure but it turns out somebody was using the template word processor or something and then there's the tools of radiation that that can and other times I don't know maybe the developers have to much time there and then they write a million different kinds templates and you can discover halfway through that websites the multivariate testing and and it looks different next time you run across and I wish there was a silver bullet some solution I got the for there's not and and and the problem that will come up this is sites requiring JavaScript MR browser rendering and we tend to have uh we we have a service called splash which is a scriptable browser presented at http so this is very useful to integrate scrapie am and some other services and you can write your scrapers in Python and and this has had the browser you know that have plastic in the browser rendering an and weakness script extension space and in love and selenium is another project and if you're start thinking like OK follow this link type this year and so he was a great way to go we have finally of course we can look at my that's where these vectors something see what's happening this is maybe the most common thing for scraping programmers and this is quite efficient you can just you know is often there's an API behind the scenes you can actually use them and proxy management is is another thing that you might want consider and because some websites will give you different constant depending on where you are and we we called 1 website that actually did currency conversions and so of what was being very clever by selecting currency at the start but it turns out the website that double conversion and some products were center to different and so did discover that won't work for the ages so odd the band hosting centers often where they've had wanted to abuse of particle be some of that this is just part of the nature of scraping and current and yeah so for reliability sometimes for speed you might want to consider properties please don't use open proxies um they sometimes modify content is just justice is not a good idea and toward the I generally don't like it for a large content scraping and it's not really what is intended for what and we we don't some things with maybe government agencies there's security In the security area where we really don't want any blowback from scraping and and and you know it needs to be released in arms and otherwise there plenty of private providers but varying quality and finally the last slide is just briefly want to mention about ethics of web scraping and I think the most important question to ask yourself is what harm is your scraping doing and you don't technical side are with the content that scrape use using it you hurting vector-sensor getting it from the yeah so Cromwell unidentified following reason the race it's best practice to use and user it to identifiers of these agent that respect robots that next expression protocols that's when you visit the website and and that is that you have some questions thanks
be with what the text wonderful talk with questions emissions have again into some websites and so he she is a tool that will generate some fake credentials suffers local become a profile of the programmer or profile of farmer or for experiments thanks OK so about logging into websites wanted the to the mentioned just that finds the log box and lets you configure your you use ID and that you want to use so he doesn't
handle managing multiple contacts and I have seen people do that but it's not something done myself and in society and that sort thing that we really and any
any questions of 1st of all
thanks for the PostScript library and most and we're using it as the basis for the variety of these guys used to thinking in the audience and the idea is that the ball rolling border and then the basement of somebody's the contributors um you few
this is more 1 them out there but I don't know whether they're being shy Faria I'd probably have a few questions little only has a couple of years to 1st is that like to mention by querying was that was an awesome development of the change for us from XPath yeah and have you may be tried and this is 1 thing we use regularly and pros and yeah I heard about the properties of the rejected and I think the microscope for including other approaches to to 2 extraction of and the 1 is the you maybe you think about a master spiders sort of spiders that can you said that API is are brittle and yeah but you could still think of lower web frameworks and some behave in similar ways and maybe you could get away to extract certain information from certain kinds of websites yeah absolutely and like we have a collection inspired her the foreign engines for example and this is not individual websites but as the underlying engine powering and that works really well they have little ability collections of those kind of things and my example but API really meant to this API as as as in general and there are often quite useful but some cases they don't have a constant after and and in some cases the content is maybe like behind the it's a bit less fragile than what's the website and that that's been my experience and what definitely if there is a website and it's available you should you should check it out it works fine distributed of that and just last question a little bit more technical but do have a the plans for anything of 2 I don't know what to read to handle throttling or to handle role what's the next or to reschedule 500 hours or something that you know there's a oral throttle blogging yet but that doesn't mean it slows down significantly on a good website yeah don't doesn't work for that of so websites things that and you welcome and the option is an interesting 1 and often internally what we do is we we deployed all the frosted by default and then overwrite it when we know the website can do better MR differently so it is case especially appalling addressing the website there's small websites it's worth tuning them yourself it's hard to find good heuristics and and and that the it's something we do all the time when you write individual scrapers and and the interest in your thoughts about how we could and come up with some better heuristics by default and is definitely very interesting topic and and again we try again and again it's great because we try stuff and that by default but you can configure their for example the HTTP error codes that signify an error that you want to be retried they're not always consistent and across websites thank you that's always slight fall to the we try things you mention this briefly under the top then do you actually like to things like the back office and engineers and stuff because from my job we have some very interesting situations with synchronized clients another fun that is good to avoid yeah yeah yeah that that the actually and I glossed over the details and I mean I said we run great betrayed and but that takes care of a lot of the kind structure that we took the lead and and Alexander gave a talk on the Prof frontier which is crawling scale and is a lot more that goes into that that that happens outside creepy itself and the very 1st thing of course that we noticed this easy status calling from easy-to-use DNS there's all over the place and what the they there's several technical hurdles and the common thing to do it do a logical qualities thank thank you Shane instruments