Ten Unicode Characters You Should Know About as a Programmer

Video in TIB AV-Portal: Ten Unicode Characters You Should Know About as a Programmer

Formal Metadata

Ten Unicode Characters You Should Know About as a Programmer
Title of Series
Number of Parts
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
There are a lot of things that can go wrong when working with Unicode data. Some examples of unmeant behavior: You try to downcase "I" to "i", but your Turkish friends want it to be a dotless "ı" Your UI is broken, because people use empty usernames, despite the String#blank? check You think "C" is the same letter as "С", but your system does not think so and crashes Using ten characters as representatives, I will highlight some Unicode characteristics which require a programmer's attention and demonstrate how Ruby's solid Unicode support can be of useful assistance!
Point (geometry) Web page Laptop Standard deviation Cluster sampling Code Execution unit Virtual machine Maxima and minima Coma Berenices XML Unicode Theory Number Revision control Programmer (hardware) Operator (mathematics) Codierung <Programmierung> Social class Execution unit Multiplication Point (geometry) Code Regulärer Ausdruck <Textverarbeitung> Formal language Type theory Word Personal digital assistant Computing platform Ranking Writing Relief
Point (geometry) Standard deviation Context awareness Presentation of a group Dot product Texture mapping Range (statistics) Division (mathematics) Mathematical analysis Parameter (computer programming) Unicode Rule of inference Formal language Number Revision control Database normalization Different (Kate Ryan album) Canonical ensemble Single-precision floating-point format String (computer science) Operator (mathematics) Personal digital assistant Representation (politics) Software testing Information security Library (computing) Social class Curve Execution unit Meta element Matching (graph theory) Information Physical law Latin square Code Mereology Measurement Hand fan Personal digital assistant Revision control Normal (geometry) Right angle Text editor Pressure Form (programming) Local ring Row (database)
Group action Code Texture mapping Sheaf (mathematics) Combinational logic Set (mathematics) Water vapor Parameter (computer programming) Protein folding Mechanism design Roundness (object) Computer configuration Personal digital assistant Physical system Exception handling Parameter (computer programming) Menu (computing) Control flow Regulärer Ausdruck <Textverarbeitung> Formal language Demoscene Category of being Radical (chemistry) Process (computing) Vector space Hill differential equation Reading (process) Row (database) Protein folding Web page Game controller Observational study Algorithm Line (geometry) Characteristic polynomial Virtual machine Control flow Similarity (geometry) Revision control Term (mathematics) String (computer science) Operator (mathematics) Operating system Representation (politics) Spacetime ASCII Context awareness Characteristic polynomial Uniqueness quantification Line (geometry) Cursor (computers) System call Word Personal digital assistant String (computer science) Table (information)
Email Group action Hoax Distribution (mathematics) Range (statistics) Water vapor Unicode Area Medical imaging Roundness (object) Computer configuration Different (Kate Ryan album) Computer network Office suite Bounded variation Data compression God Physical system Metropolitan area network Hecke operator Bit Sequence Formal language Twitter Category of being Curve fitting Reading (process) Writing Spacetime Point (geometry) Connectivity (graph theory) Characteristic polynomial Maxima and minima Control flow Computer Template (C++) Number Goodness of fit Bridging (networking) Term (mathematics) Operating system Representation (politics) Energy level Codierung <Programmierung> Computing platform Standard deviation Focus (optics) Uniqueness quantification Content (media) Code Reflexive space Basis <Mathematik> Symbol table Integrated development environment Network topology Free variables and bound variables Library (computing) Flag Standard deviation Code Visual system 1 (number) Sheaf (mathematics) Set (mathematics) Peg solitaire Function (mathematics) Food energy Formal language Facebook Bit rate Oval Pressure volume diagram Pattern language Electronic visual display ARPANET Flag Data conversion Endliche Modelltheorie Category of being Area Presentation of a group Clique-width Venn diagram Point (geometry) Range (statistics) Price index Regulärer Ausdruck <Textverarbeitung> Flow separation Disk read-and-write head Exergie Microprocessor Type theory Vector space Species Bounded variation Functional (mathematics) Sweep line algorithm Virtual machine 2 (number) Local Group Sequence Revision control Band matrix String (computer science) Reduction of order Musical ensemble Spacetime Software testing Installable File System Dialect Multiplication ASCII Characteristic polynomial Poisson-Klammer Forcing (mathematics) Plane (geometry) Symbol table Single-precision floating-point format Voting
Point (geometry) Game controller Code Texture mapping Execution unit 8 (number) Mereology Unicode Food energy Twitter Database normalization Term (mathematics) Computer configuration Analogy Personal digital assistant Flag Multiplication Graph (mathematics) Clique-width Characteristic polynomial Latin square Content (media) Electronic mailing list Control flow Regulärer Ausdruck <Textverarbeitung> Sequence Type theory Subject indexing Radical (chemistry) Computer configuration Personal digital assistant Electronic visual display Spacetime Flag
Row (database) Coma Berenices
the the the the the the
the the the the the the I am I nearly this and I'm
here to tell you about like funny unique odd characters you should know about as a programmer and also I'm writing this review what we had a terrible funny things and relief which you may be no or concerto so the 1st character and I have to tell me about this is on as all that idea what this could be home cooked me so this 1 character even though it's made up of a sweet different strokes and that's the thing so it's a type character and its com combined out of multiples of characters and to understand this we have to understand what UT Code is about so when they're upset they say unique of words a unique number for a character and this means that it's all about then coding that how the actual bytes smoke but it's about now what bitch character is which number to refer to them all at once not by changing them coding but to have 1 unique encoding that all right so the number for every character so when it's this is just a smidgen of BDP damages like the page rank select all the encodings the that I'm not unique old and unique doesn't replace every every 1 of them but a lot of them so often you talking about characters of you are actually saying is called points and called points this this unit number every company new got got so a character can also be made of multiple points and that's what we've seen at the beginning and is all socially called class like the use perceived character and and read we have a lot to get that we can work with of those because carry just because there the backslash X units and this whole thing you read those of unstung Karekin Kaneria do get can retention come maybe the these known the new theory for it so benefits example which is all of what we what we all learned so that we can use the dot operator in regular expressions match for any character it actually legislation and told point so so the 2nd version of what we want to use in most of the cases because it leaves this units as long as it this 1 character and because it's so important in Rabin took 5 we are getting at each perfume class and a curfew classes method which will return the same falls alright what it is I'm not doing this on my laptop but by machine this is why this looks like the way it should but this illustrates in the point also like the 2nd character it's an aid with 2 adults evolved and this is like a character
which is pretty common in the German language so I have it on like he wrote but when I press this button it bolts of okolje like the combined class that you see again it's again combined of to cope points but it will create a single co point you
which represents the same character soul how
can we Myrtice's planning in at 1st example you it's more easily visible but but you can see is that in the 1st example which is to combine curve what is needed to cope indeed law 1 is just a cynical point which is a different company but the possess the same character and I'm using the useful tool you which also analyzing co point 8 so but what if we don't care about this different or presentations representations that's well the characters the nest and this is rare but I will often called normalization comes into play and this rule transform the to quote the work coke Callpoint version into the cynical of which and it's included in the recessionary sensor 3 giving automatically required which is unusual for the standard every it's like 3 salaries also and this 1 is and you can use it but to this calling string not unique about normalized and as you can see here that the 1st one honest there is still a long double talk comma decimal 1 and the 2nd 1 is cynical women and men analyzed in the same church so this is where the character it's the little litany small character and you might think what's so special about to their own fans actually the special thing as this not to the old and that's a different matter this Surobi after 0 3 B F and this is to create on that at all and can this setting the thing as and this is the real they're all well as you differences but not the real ones and that that among which we have more news and every look at it again we see we see that this this a similar in character and not at all a record holder tests like 75 characters that look really really summary and the unique of conventional they catalysts where they Reddy can look up those securities some more examples there's a character true christian article point which means to pressure Martin and of course it's really confusable of a single but of choosing a letter special also you don't have to go to higher Unical ranges you can sort of ASCII at the than that meta l lost your taste and and a number 1 there also Greek confusable so it's it's a problem that Boultbee ever totally sold itself is always ritual thing also and I put the true sees here match shouldn't she looked totally the same because the Cyrillic and him see they're the same character so it's it's by purpose developing and after bottles different points this can lead to a what kinds of security issues this but if I had a good right up on and what do we do not want that we know all are there because we're characters and White saying that you can use of this tiny microtron so yeah confusable and there but it uses the dialog thing for right but that by the antigen to 1 to check if 2 strings articles is slow character looks a little like 2nd I but actually it's it's a so catalyzed science which you which has another don't worry about it we during my visit measurements I by genital Texas ex-top not about so the thing is we think that the Khot of for a cult cult of a set up casing operation of 9 editors just just a stroke without without a doubt but what about like Turkic languages like Turkish they they put it never got above so the outpacing operation on pacing is language-dependent operations and you need to context of the of managed to do it correctly how so when the when there are a wait we have got this method and before the we to look for it was an unique odor at all so again the letter that my favorite 2 thoughts that Bolar piece of all sensory 2 dogs for over we can properly up cases condensed we can even do that we can also get all behavior by passing the ASCII she has their we can even put in some local It's of information for example tragic through gets the court that we want but there are cases that the only our method that supports this is argument you can also
has a useful kcal capitalists all this is pretty good that this made it into that into really are and the be around that there's another method of don't using a string its called case folding is also supported as the 4th argument if you just 1 we suggest used to fold you fold humor but it uses a different article was so in this example and it's the German job matter it's the German at the shop and the real using the normal don't casing it will return the lowercase version of it but if using the case folding mechanism which is meant for comparing strings we get another representation of this vector which is in this case just let has there's more to be careful about so I've told you the the thing an option buy this possible to to passengers down as an artist but it's called a than but it's not working at all so daunting this yet we don't have covered the whole word for example in the in the Netherlands so they haven't really while interesting way to to to to upper case and watching to listen read and also there's other strings case compared methods man's they looked really really similar but in the case cf. question mark 1 uses the case falling that had just told you about and the other 1 it will only use plain ASCII so be careful there no for some more from characters this is a control character called next line was and is called new land and so I don't know how do you know how you doing neurons that plenty of race already depending on your operating system sold will you would the use of maybe just a little 98 character or the carriage return character or a combination of both learned but because of this confusion but there was another character introduce you to to give you line like which is called the next line but is more often that wasn't so successful because the adoption was not so good rounds whatever on on my machine at least it in Burke's if I just went out the bolt give me in another they did the demand likely but it's check on the systems you're wrong or don't use this character I mean won't want us us to do this but you should know that it exists and they're checking for line breaks you should know this could be an option to traveling and the next thing characters interesting also for another reason where which is not visible but this is an ASCII table and in the for the 1st 2 rows which means like the 1st 30 true characters there but they're special characters called control characters on and because they they they don't run the normal cu but there like of doing line breaks or similar stuff and they're like introduced the nasty was foods that's is only the 1st 32 characters so later people wanted to have more like control of control of you want to do more a study needed more control characters so they introduced by summer the call for characters are this next page each variable soul of the origin of set of other characters is called scene C C 0 terms of the new 1 is called C 1 and It's not supported anyway in code uses some them because of so compatible with them what they I essentially do do nothing except for the next line operator that I've shown you they don't do anything so if you need them then in new data that you're passing be really careful about book for their own purposes so how can they work with the summary the next line operator is of it's it's not much by of the normal were regular expression which matches for spaces but by this 1 using a unique properties in text lands also control characters can also be matched this what the C as the property and you could also use of their characteristics channel for further research if it's C. C. Waters you want so the next section this character is worry White so this the 3 and action and GM the thing is if you are using it's found there is a chance that you might quick suffer so for example if I as I have using it and from the terminal so you can see that the cursor is somehow misplaced and that it's not detected that that's all also if you change your name are conceived but it totally breaks the melody maybe you can see it in the next I want to change
it back but it was impossible because my changing heck section was all thank and that's not the only character there's a more which are also like to tell you look different on my machine because it's also is also a really depending on the test on the on the is represented so you gee jets as not well-defined but it's more like it's not defined at all which is of course computing for fixed the environment selected from minerals rounds that's especially true for a lot of Asian characters so all the use of a convergence Bauer started to assign some myths if it's like 1 can space which would for a space to a lot of Asian characters but even there there's a category of force characters which is called ambiguous and they can be either single or double so the user after suffer library often has served God in an option to to 2 due to display them as more traditions stick judgments so you can use so you display with glycogen and if you want to have or proper checks and the 1st character you see that's about it so woman as it's of character which only like 1 space and the seconds to spaces but where the 3rd 1 is 1 of these ambiguous characters the start it so you have to pass and actually doctrine we want to have it displayed as 1 or 2 species right now that we know about all Knuth Nobel's this let's go to this character images just a placeholder for so our sorrow and included character that is are forbidden to use so there are 2 kinds of was called points that are not that old and the new code and they're and coding related so as a set you itself is independent from the actual encoding but they're like sweep of encoding sutures ages of 16 and you have 32 and you could can use them in the you can use whatever encoding and you want to and the you just 16 and going is a needs a special area of coal points through to to the function so the section of template till F of F it's it's plopped because because you have 16 needs it to represent 1 more unique characters and you could but the the thing is and you give aid and of 32 you can actually I have this or company and it but it's just not allowed the 2nd of the 2nd version of the inlet little poisons that to large ones which is again related to you just 16 soul but that the highest ColorPoint point you represent it can represent a future 16 as well 10 as which just about 1 million and you could represent much article points with the different encodings but some of the unique of contention that no while this is the ARPANET and I will read out to you so this is just an example that is low and you just 16 while only need 2 Mbytes used to the purpose and 1 character and if it goes to forward bites its needs to needs both special surveyed where points so because they're not allowed real also be you to trade them with other this living string literals and as you are running but a related read or really if if you really need them you can use by different techniques to to greater such data and we also gives you some some of the methods that you can use work refer with strengthen encodings and and 1 of them has developed encoding method and if your data contains those forbidden code points well this method mood returned force and also have the scrapbook method strings which will replace all of these characters with we but this replacement character that you've seen at the beginning and besides all this illegal characters you can you and there's also a section of about both characters per legal which are allowed to represent but which are not assigned to anything for example there that the well 10 ff adjust the highest hope Honourable the whole sentence it's not assigned up to anything and it will never be assigned to do anything because of because 3 kinds of conquest and not move not ever be assigned anything and 1 of the office is called the vote and not so good name unknown characters so I get 66 characters in unique revision of assignment just global never be assigned also a huge section of private use points break and coat with fantasy languages or follow or at the levels of your of science yeah what you you cannot see any but like the the the Naur 1 S 8 S S this is a battle Apple ovals on Apple compressible be little to spread an apple and the the above this they want to lower world song of 2 machines you see that went along with it but they're not defined by the Convention so you can as all there they will be displaying display correctly on different platforms from them there's also well you portion of cocones which hasn't been assigned yet which is called research with see if this Kraftwerk's no it doesn't so much what it shows is that's and it's that's a lot of reserve components so that's almost altering cold ones are not assigned yet so there's plenty of space for future future assignments the private on their abroad as much of standardized go points that's their bridges the outputs look back to Republican work with them in really again using the slower property of unique of properties in text and regular expressions you can match for 1 characters it's a little bit cumbersome you we have to write code point there we can so much cheaper reduce Coplans's Bourdieu's and the reserve funds if you don't care about demand characters you can just use the the unassigned property and if you do care you have to do this negative look at positive they need and and the regular expression work so in then see in this section this what about the the brackets it's about the space in between which look like usual space my space but is all the usual at space it's simple break space but again if no way to see it's no break space and that's not the only space you have and if you make a bad tons of them and only some matched and considered as white
space made a convincing but there are also a lot of characters rich 1 of which and also wasn't as our space so all this can be true to all of problems suffered some will just recently there was a fake multiple nets work and it looked like that looked like it was published but it wasn't that bad but the what the harmony but they put a space at the end so it just looked like that most public published by the water and also in the education is very good well maybe see it did would just use drinks strings warranty this is probably related to like what about checking for but then coupons correctly if a together some more so the energy of these decimal going rose separator this is the writer wants to the space this this era with China will needed later was this so the middle class and on my machine is actually invisible this is a the braille patent languages Braille selected writing system for blind people and and it's little dots in in in in the in the picture of the character and of course you also need a Rh represented no doubts and this is my this 1 is playing can also then this is an example of a character which was not matchless well as my sister the this 1 which is the space for Asian characters and this was also space which is if I understand correctly musical note that the OneNote it's really you that's where it's a cylinders but as though perceptual for and these acts of all these 5 pieces by the best of the classic characters called ignore both which just tryin' nothing or but more and more correctly they if you display system doesn't it doesn't know how to displace in these characters they Egypt should should just random nothing and it's not only a few characters the whole of Copan range while from East 0 to be 0 FFF is this I considered the normal terms some examples of 4 characters that you can find there the so-called variations actors that their purpose is to to them to have a Michelin variation on the preceding character which is especially useful for example in the southern Chinese letters there but it's not important of focus in that off cormorants true strokes were 3 and this variation vector can then tell those display and and to render correct but to carry surfaced totally invisible more famous is variations electoral and number 16 which based on some text based emoji on some platforms during to image-based ones and and actors 16 is the other out but again it really depends on the operating system for example and also 1 of the you know so I'm slightly so are are always shown on 1 for example microprocessor as a picture on 1 of the systems they they might be always the shoulders of text based in Moultrie and there's no way to change it's only a few characters that you can while the want and use a reagent that on and then another ignorable kind of characters that tech character but which was introduced to create a name which tech sequences but you shouldn't do it they're just deprecated so don't greater than chicks against its invisible sequence in your text which which describe something so it's the whole of ASCII ECoG encoded again but with different characters but don't use them irritancy so what have you learned of invisible characters so show all bytes spaces some matched by the normal regular expression of the just by such acts somewhat watch matched by by using the unique properties and takes too much for rates basis and you can also magic curable characters using this again rather cumbersome and so on that could point couple called point property and some even more much by anything 1 where you can use the characteristics jam again which has a somewhat support to that detecting if if a character is of ignorable or Planck this is my last section that's the Maltese action but apparently on this or in Gerda while the emoji of the Copan sequence that should Hochschule a cook at a meal the cook of the light skin tone is working correctly thought so if it would be working correctly but it tends to look like this this of look center the stands yet is constructed all of 4 Copeland's and go to introduce here too much about the emoji as offenses because it's really I I could do it or about emotion census itself vote it's like 7 ways of creating emoji and so on that and also can be enormous the model that uses this Europe China trees it's not about in the previous section It's or its end as a character that it combines true the military and test this thing better yet mended emoji so there are a lot of emoji and theoretical you can just makes any of them there's this X KCD rich had this is a fine idea of like combining combining what kind of Okmulgee and activists who was who's from the uniform conversions at all actually if you're using those Europe joy they all benefits of emojis the problem though it will be displayed at all in in in books at or Facebook or whatever because none of the vendors knows about them and this is what the emoji senators about its recommends some see cramps us which then the vendors bubble wonderment while the other way around and another type of emoji our country flexed soul the deleted can't reflexive said there rather easy you just use true with regional indicators symbols like the flag of Porterville it's not encoded directly assisting local point you make it up by using the P entities original indicators symbol however recently there was also introduced to have suffered regions and you would guess that they just use the return to Canada is again but no remember that I said text sequences adjective emitted from they got on deprecated and no you can use take characters in some kind of you it in another manner to true to create a sub flexed which
leads to a strange behavior for example there's such a loser was wondering about my a week ago was sold to Twitter allowed to have like 50 characters in your user name and while they want to choose to use and the use of like Belshe flexed but could only based like 3 so what's 1 was working at this where complicated to construct such a such as region fact this is normal Maltese for Plex flat then you do your text of these invisible recently and deprecated tech feathers and then you have to pay to cancel tech through to end the sequence and there you have you Scotland flag and this is also the reason why at the end doesn't like flak probably with just pace of some coal points to spend about rare was enough to display this that fact but not enough text and the children work with this summary you can use another might wooden called unit code of analogy and it contains a regular expression just like huge and the ball much like all types of characters is 1 of the major characters even if they are not all of them usable on your attention that and also contains the the list of our recommended emoji with it and all this was a lot of content so let's have a show every we learn about atrophy instead a character can that with which is just that term for characters and if you're not to match for any characters you probably want to use the backslash X syntax and not the dot in regular expressions and the for me to the fights we get always strange graphene it's extraneous graph impressive energy then to normalize all characters in this things into the chest of only 1 representation of off of a character that multiple it's a part of Tucson Library and called unique of normalized confusable I can be done with the unit of this is gen if case he mappings starting with would be to look for and it's all in the only Jiamin works as a trucking option but does more to come according to yesterday we have control characters that's what you should know about and where a lot of them are not specified in in any way but displaying characters on a on a fixed with terminals complicated so if you using that were broke up then you are probably already using unique adjusted this already so is free to use it and other stuff as well and to detect it so and colored coal points you can use the medical method to just replace that you can use to describe method and there are a lot of characters that are allowed to to be part of your data about which are not specified 1 and you can match the use the unscented index and Bourdieu syntax now a lot of them was my characters like the only white spaces but also it horrible once but she doesn't even runners space of granite just nothing and and a multi and a multi hour on a difficult to to that because there's so many ways to to to represent of represent what you want to represent that the rooms look little thank
fewer and and so


  398 ms - page object


AV-Portal 3.20.2 (36f6df173ce4850b467c9cb7af359cf1cdaed247)