1 00:00:01,720 --> 00:00:04,640 Hello, and welcome to yet another Worked 2 00:00:04,640 --> 00:00:08,096 Exercise for Python for Informatics: Exploring Information. 3 00:00:08,096 --> 00:00:14,460 I'm Charles Severance, I am your tour guide to this particular worked example. 4 00:00:14,460 --> 00:00:18,090 And this material is Creative Commons Attribution, and I hope to 5 00:00:18,090 --> 00:00:21,240 see the clever and cool things that you can do with it. 6 00:00:21,240 --> 00:00:25,610 This is part of my www.pythonlearn.com website that 7 00:00:25,610 --> 00:00:28,610 helps you to get, learn Python sort of 8 00:00:28,610 --> 00:00:30,870 from the very beginning with videos and other 9 00:00:30,870 --> 00:00:33,640 materials to help you go along with the book. 10 00:00:33,640 --> 00:00:34,590 Of course, the book is Creative 11 00:00:34,590 --> 00:00:37,770 Commons Attribution, so pretty much everything here 12 00:00:37,770 --> 00:00:41,719 is free for you to use and, and adapt as you see fit. 13 00:00:43,410 --> 00:00:48,546 So we're taking a sort of a working example from the tuples 14 00:00:48,546 --> 00:00:53,468 chapter, and the, the task that we're given 15 00:00:53,468 --> 00:00:58,640 is find the ten most common words in a file. 16 00:00:58,640 --> 00:01:00,890 And we will, will do that, and so if 17 00:01:00,890 --> 00:01:04,290 we take a look at the basic overall structure. 18 00:01:04,290 --> 00:01:06,080 We're going to read a file. 19 00:01:06,080 --> 00:01:09,230 We're going to make a dictionary, an empty dictionary, and we're going to 20 00:01:09,230 --> 00:01:13,090 read the line, we're going to read the file line by line. 21 00:01:13,090 --> 00:01:15,690 We're going to split each line, then we're going to loop through 22 00:01:15,690 --> 00:01:20,670 the line, using the dictionary pattern from the dictionaries chapter, 23 00:01:20,670 --> 00:01:25,150 using get so that we can both access and initialize 24 00:01:25,150 --> 00:01:28,070 new words that we haven't seen at the same time. 25 00:01:29,450 --> 00:01:32,140 And then we are going to do, we need to sort, right? 26 00:01:32,140 --> 00:01:35,350 But we need to sort by the value instead of the key. 27 00:01:35,350 --> 00:01:36,710 So we're going to make a list. 28 00:01:38,270 --> 00:01:42,010 And then we're going to loop through the results of the dictionary. 29 00:01:42,010 --> 00:01:45,530 And we're going to make a new tuple. 30 00:01:45,530 --> 00:01:47,380 But instead of being key, value, which is what 31 00:01:47,380 --> 00:01:50,510 you get from items, we're going to do value, key. 32 00:01:50,510 --> 00:01:54,350 And then we'll sort that in reverse order, and then we'll 33 00:01:54,350 --> 00:01:59,990 simply print the first ten elements of that sorted in reverse order. 34 00:01:59,990 --> 00:02:03,330 And because we're constructing carefully this new tuple 35 00:02:03,330 --> 00:02:07,610 value, key in the right order, value will 36 00:02:07,610 --> 00:02:09,560 be what's sorted on first, and then in 37 00:02:09,560 --> 00:02:11,740 cases where value matches, it will sort on key. 38 00:02:11,740 --> 00:02:15,210 Whereas if we just took items, which is key, value tuples, 39 00:02:15,210 --> 00:02:17,920 then it would sort on the key, and that's quite easy. 40 00:02:17,920 --> 00:02:23,590 So, that's the basic idea so let's go ahead and and get started. 41 00:02:28,570 --> 00:02:31,690 So I'm going to grab some of this 42 00:02:31,690 --> 00:02:33,020 and save myself a little bit of time, 43 00:02:35,070 --> 00:02:36,200 and paste that in. 44 00:02:37,540 --> 00:02:41,920 I'm going to use actually romeo-short, because I don't want to mess with it. 45 00:02:41,920 --> 00:02:47,590 romeo has punctuation in it, so I'm going to go and I'll call this 46 00:02:49,940 --> 00:02:50,420 under tuples 47 00:02:50,420 --> 00:02:56,160 I'm going to call this guy topten.py. 48 00:02:57,160 --> 00:03:01,920 I'll see if it's in my tuples folder, yep, just always sanity checking. 49 00:03:01,920 --> 00:03:07,805 So let's, let's go ahead to pythonlearn and grab that romeo-short file. 50 00:03:07,805 --> 00:03:18,491 [BLANK_AUDIO] 51 00:03:18,491 --> 00:03:20,320 So I'm going to grab romeo-short. 52 00:03:20,320 --> 00:03:26,680 If I look at romeo, romeo-short has conveniently has the, or just romeo 53 00:03:26,680 --> 00:03:33,165 conveniently has the punctuation removed and it's small, so we can do debugging. 54 00:03:33,165 --> 00:03:35,810 romeo-full is a little harder, and I cover that in the 55 00:03:35,810 --> 00:03:38,990 book in some more detail to get rid of punctuation and stuff. 56 00:03:38,990 --> 00:03:42,270 Not too bad, not too bad, but we won't talk about it here. 57 00:03:42,270 --> 00:03:45,380 So I'm going to grab romeo.txt, and I'll save 58 00:03:45,380 --> 00:03:51,740 this to my dictionaries, my tuples folder as romeo.txt. 59 00:03:52,740 --> 00:03:54,160 And I should be done with that now. 60 00:03:54,160 --> 00:03:55,500 And I'm done with that. 61 00:03:55,500 --> 00:03:56,680 So if I take a look at tuples again 62 00:03:56,680 --> 00:04:00,760 And I've got romeo.txt in this file topten.py. 63 00:04:00,760 --> 00:04:04,510 And it's not romeo, it's romeo.txt. 64 00:04:04,510 --> 00:04:06,480 And I'm just going to, split. 65 00:04:06,480 --> 00:04:09,420 I mean, I think, I hope by now you can read a text 66 00:04:09,420 --> 00:04:15,490 file [LAUGH] and split it into words, and count those words in a dictionary. 67 00:04:15,490 --> 00:04:17,460 So I'm just going to print out the dictionary, 68 00:04:17,460 --> 00:04:20,080 the counts dictionary that we've got at the end. 69 00:04:20,080 --> 00:04:21,200 And let's see how that works. 70 00:04:21,200 --> 00:04:24,600 So I'll go onto my desktop and this is under tuples. 71 00:04:25,725 --> 00:04:31,260 [SOUND] And so there's romeo.txt and topten.py. 72 00:04:31,260 --> 00:04:36,140 So let's go ahead and run topten.py. 73 00:04:36,140 --> 00:04:37,410 And so there we go. 74 00:04:37,410 --> 00:04:43,050 We have just sort roared through, broke it into words, broke it into lines. 75 00:04:43,050 --> 00:04:48,230 And we have a nice convenient little thing here. 76 00:04:48,230 --> 00:04:52,340 Now, you know what, I'm going to do one thing that's not in the example. 77 00:04:52,340 --> 00:04:54,960 This, the fact that these are upper case, I don't know 78 00:04:54,960 --> 00:04:57,250 if it's going to cause a problem or not, but I want to, 79 00:04:57,250 --> 00:04:59,670 this is, this bothers me. 80 00:04:59,670 --> 00:05:04,950 And so, so I'm going to, I'm going to force all the words to be lower case. 81 00:05:04,950 --> 00:05:12,920 And so I'm going to call, I'm going to make a thing wrd equals word.lower. 82 00:05:12,920 --> 00:05:13,830 I think it's lower. 83 00:05:13,830 --> 00:05:15,100 We'll find out fast enough. 84 00:05:15,100 --> 00:05:17,860 So I'm just going to grab a lower-case version of word and make 85 00:05:17,860 --> 00:05:23,210 it be w-r-d, and that's what I'm going to put into my dictionary. 86 00:05:23,210 --> 00:05:26,070 So let me run this again and see if I got it right, yeah. 87 00:05:26,070 --> 00:05:29,560 So now we see things like 'who' is lower case. 88 00:05:29,560 --> 00:05:32,140 I don't think it changed any of the numbers; it may have. 89 00:05:32,140 --> 00:05:37,030 But, I just, I think we shouldn't treat upper case or we 90 00:05:37,030 --> 00:05:40,160 should make this case insensitive, so we'll map everything to lower case. 91 00:05:40,160 --> 00:05:42,850 So I just did that by mapping it to lower, and then 92 00:05:42,850 --> 00:05:49,010 using w-r-d instead of word after I made a copy. 93 00:05:49,010 --> 00:05:50,820 So, so, here we go. 94 00:05:50,820 --> 00:05:52,380 So then what I'm going to do is I've got my 95 00:05:52,380 --> 00:05:55,850 counts, and I'm going to print the results of the items. 96 00:05:57,380 --> 00:06:03,600 So that gives us a list, instead of just showing it, it gives us a list of tuples. 97 00:06:03,600 --> 00:06:04,880 So there's a tuple. 98 00:06:04,880 --> 00:06:06,450 Its a two-item tuple. 99 00:06:06,450 --> 00:06:09,800 Tuple, tuple, and it's key-value pairs. 100 00:06:09,800 --> 00:06:10,450 Okay? 101 00:06:10,450 --> 00:06:15,000 And we can, this is useful now, in particular because we can construct a for 102 00:06:15,000 --> 00:06:22,850 loop that goes through for kie, vaal. 103 00:06:22,850 --> 00:06:23,790 Key and value. 104 00:06:27,660 --> 00:06:33,800 Kie, vaal equals, I'm sorry, in count.items. 105 00:06:33,800 --> 00:06:39,260 I get all confused trying to create really non-mnemonic variables. 106 00:06:39,260 --> 00:06:46,850 So we got kie and vaal and, you know, it's, it's two iteration variables. 107 00:06:46,850 --> 00:06:47,350 And 108 00:06:52,040 --> 00:06:54,540 the way it works is, you know, 109 00:06:54,550 --> 00:06:59,622 these will iterate through each tuple, tuple, tuple. 110 00:06:59,622 --> 00:07:02,990 And then kie will be the first thing in the tuple, which is 111 00:07:02,990 --> 00:07:05,730 the key, and the value will be the second thing in the tuple. 112 00:07:05,730 --> 00:07:07,070 key, val; key, val. 113 00:07:07,070 --> 00:07:10,970 And so, kie will bounce through these at the same 114 00:07:10,970 --> 00:07:14,180 time as vaal is going to bounce through the corresponding values. 115 00:07:14,180 --> 00:07:15,436 So that's quite nice. 116 00:07:15,436 --> 00:07:20,960 So just, and I always love sanity checks, so I'm going 117 00:07:20,960 --> 00:07:26,090 to print out kie and vaal, just print them out. 118 00:07:29,020 --> 00:07:30,180 Yeah, something's wrong. 119 00:07:31,750 --> 00:07:35,185 Kie, vaal, and, oh, counts. 120 00:07:35,185 --> 00:07:37,260 So if I, so here we go. 121 00:07:37,260 --> 00:07:39,600 Traceback line 11. 122 00:07:39,600 --> 00:07:41,380 The name count is not defined. 123 00:07:41,380 --> 00:07:44,030 I was so quick, I didn't even read what it told me. 124 00:07:44,030 --> 00:07:47,084 So look at line 11, going on line 11, and then I forgot my s. 125 00:07:47,084 --> 00:07:48,240 So it's pretty much. 126 00:07:48,240 --> 00:07:49,450 you know, once you get used to this, it's 127 00:07:49,450 --> 00:07:51,980 pretty much telling me what it wants to do. 128 00:07:51,980 --> 00:07:53,590 So now I'm going through my tuples. 129 00:07:53,590 --> 00:07:54,300 Here's my tuples. 130 00:07:54,300 --> 00:07:55,920 Now I'm going through the tuples one at a time. 131 00:07:55,920 --> 00:07:59,080 Just dumping them out in a loop, and away we go. 132 00:07:59,080 --> 00:08:03,870 Now, the whole problem is we want to make a new tuple. 133 00:08:05,890 --> 00:08:07,390 Okay? 134 00:08:08,550 --> 00:08:13,200 And I'm going to call this newtup, newtup. 135 00:08:13,200 --> 00:08:17,110 And I'm going to create a tuple, and I can do this by just putting these two things 136 00:08:17,110 --> 00:08:23,670 in parentheses, except that I want vaal first, and kie second, okay? 137 00:08:23,670 --> 00:08:27,860 So I want vaal first and kie second, and so now I'm going to print out newtup. 138 00:08:32,900 --> 00:08:37,630 Now you'll see that as I loop through each of the key-value pairs, I'm just 139 00:08:37,630 --> 00:08:39,700 making a new tuple where they're reversed, 140 00:08:39,700 --> 00:08:41,720 where the value is first and the key is second. 141 00:08:41,720 --> 00:08:43,350 Right? 142 00:08:43,350 --> 00:08:45,810 So value, key; value, key; value, key. 143 00:08:45,810 --> 00:08:48,510 And now I'm going to stick this in a list. 144 00:08:50,730 --> 00:08:54,890 And I'm going to make, I'm going to call my list flipped. 145 00:08:54,890 --> 00:08:58,850 And I'm going to say, give me an empty list. 146 00:09:00,240 --> 00:09:06,820 So, so I mean, I'm starting an empty list called flipped, and 147 00:09:06,820 --> 00:09:13,340 I am going to tell flipped to append to itself. 148 00:09:13,340 --> 00:09:16,200 So I'm now extending this list each time through the loop. 149 00:09:16,200 --> 00:09:18,050 Take that newtup that I just made. 150 00:09:18,050 --> 00:09:19,830 So we've got newtup. 151 00:09:19,830 --> 00:09:22,320 And I'll take my print statement out, because that'll be obvious. 152 00:09:22,320 --> 00:09:26,260 So I'm going to append newtup to the end of my flipped list. 153 00:09:26,260 --> 00:09:28,060 And then I'll print, at the end of this, 154 00:09:30,320 --> 00:09:32,610 print flipped. 155 00:09:35,045 --> 00:09:37,530 Okay? So I make a tuple. 156 00:09:37,530 --> 00:09:39,230 Instead of printing it I just append it to this list, 157 00:09:39,230 --> 00:09:41,190 then I print the whole list out afterwards. 158 00:09:43,190 --> 00:09:46,445 Okay, so this is a little messy, but here is the first list. 159 00:09:46,445 --> 00:09:49,705 It's got key, value, and here's the second list. 160 00:09:49,705 --> 00:09:54,520 It's the exact same list as before except it's value, key. 161 00:09:54,520 --> 00:09:57,230 Now, it's exact same list, we didn't take anything out, we 162 00:09:57,230 --> 00:10:01,040 didn't add anything, we didn't sum anything, we didn't do anything. 163 00:10:01,040 --> 00:10:06,410 So then, what is it that we're going to do with this list? 164 00:10:06,410 --> 00:10:09,915 Well, we can use the totally, well, we 165 00:10:09,915 --> 00:10:12,380 could use sorted but we're not going to do that. 166 00:10:12,380 --> 00:10:13,460 I want do this. 167 00:10:13,460 --> 00:10:20,360 I'm going to say, I want to take that flipped list and I want to sort it. 168 00:10:21,540 --> 00:10:25,572 Sort and then print flipped. 169 00:10:25,572 --> 00:10:27,850 Just sort it. 170 00:10:27,850 --> 00:10:28,950 Now the key of sorting, 171 00:10:28,950 --> 00:10:33,890 the key of awesomeness of sorting, is that sorting is done by repeated comparison, 172 00:10:33,890 --> 00:10:39,630 and it's comparing this tuple to this tuple. 173 00:10:39,630 --> 00:10:41,940 But the rules of tuple comparison is it first checks 174 00:10:41,940 --> 00:10:44,550 the first pair, and then it checks the second pair. 175 00:10:44,550 --> 00:10:48,660 It only checks the second pair if the first pair doesn't match. 176 00:10:48,660 --> 00:10:52,490 So 3, 1 doesn't match and so it sorts it 177 00:10:52,490 --> 00:10:54,890 based on the 3 and the 1 without even looking. 178 00:10:54,890 --> 00:10:59,374 But then when there is two 3's that match, it actually does check the second two words. 179 00:10:59,374 --> 00:11:07,556 So 3, and will sort. The and and the the will decide 180 00:11:05,556 --> 00:11:10,450 where the 3, and and the 3, the tuples end up sorting. 181 00:11:10,450 --> 00:11:12,070 If that makes any sense. 182 00:11:12,070 --> 00:11:12,740 Okay? 183 00:11:12,740 --> 00:11:14,100 So let's just print this out. 184 00:11:16,500 --> 00:11:22,850 Actually, let's print, print flipped, not clipped. 185 00:11:22,850 --> 00:11:24,970 We can make a clipped one too later. 186 00:11:24,970 --> 00:11:26,210 And I'm actually going to get rid of this 187 00:11:26,210 --> 00:11:29,970 print, so it's going to print flipped before and after. 188 00:11:29,970 --> 00:11:31,390 So watch this. 189 00:11:31,390 --> 00:11:34,760 So here's flipped before it's been sorted, in whatever 190 00:11:34,760 --> 00:11:37,220 order it was in the dictionary in the first place. 191 00:11:37,220 --> 00:11:40,280 And now, here's flipped in the final order. 192 00:11:40,280 --> 00:11:43,190 And you see that you know, it's in, it's 193 00:11:43,190 --> 00:11:46,260 in sort of reverse order, where, you know, 194 00:11:47,550 --> 00:11:50,980 3 and, is, and not are here at the end, the 1's 195 00:11:50,980 --> 00:11:54,970 are in alphabetical order within the 1's, because tuples, check. 196 00:11:54,970 --> 00:11:56,270 Oops, don't do that, don't do that. 197 00:11:56,270 --> 00:11:56,960 That was nasty. 198 00:11:58,360 --> 00:12:00,330 And that was really nasty. 199 00:12:00,330 --> 00:12:02,970 It's because I was copying and pasting. 200 00:12:02,970 --> 00:12:08,500 So within all the 1's it actually sorted based on the alphabetic sort of the second 201 00:12:08,500 --> 00:12:11,460 parameter, or the second item in the tuple 202 00:12:11,460 --> 00:12:15,780 whereas in among the 2's all sorted later. 203 00:12:15,780 --> 00:12:19,190 So this actually isn't quite what I wanted. 204 00:12:19,190 --> 00:12:23,980 And so I'm going to say, reverse equals True. 205 00:12:23,980 --> 00:12:27,790 Because I actually want to sort it so the big numbers are at the front. 206 00:12:27,790 --> 00:12:29,930 So let's run that little bit. 207 00:12:29,930 --> 00:12:31,570 So check that, right? 208 00:12:31,570 --> 00:12:33,670 So now I got my same sort. 209 00:12:33,670 --> 00:12:36,010 It's got the unsorted, no it's got the 210 00:12:36,010 --> 00:12:39,340 sorted, but the big numbers are at the beginning. 211 00:12:39,340 --> 00:12:40,610 And you'll notice it's kind of in 212 00:12:40,610 --> 00:12:43,010 reverse alphabetical order because the big words are 213 00:12:43,010 --> 00:12:44,620 at the beginning among the 3's, but 214 00:12:44,620 --> 00:12:47,400 eh, whatever, we're close enough for government work. 215 00:12:47,400 --> 00:12:50,560 So now I want to print out the top ten, okay? 216 00:12:50,560 --> 00:12:52,380 Oops, don't, I did that again. 217 00:12:54,150 --> 00:12:56,520 I want to print out the top ten. 218 00:12:56,520 --> 00:12:59,280 I'll just do like the top five. 219 00:12:59,280 --> 00:12:59,980 Do the top five. 220 00:13:01,000 --> 00:13:03,120 So, I could do something simple. 221 00:13:03,120 --> 00:13:04,470 We'll get rid of this. 222 00:13:04,470 --> 00:13:07,460 We trust that it's now properly sorted. 223 00:13:07,460 --> 00:13:09,210 I could say, 224 00:13:12,170 --> 00:13:17,650 print this list up to but not including 5, by using list slicing. 225 00:13:19,080 --> 00:13:21,970 So there we go, there's the top five. 226 00:13:21,970 --> 00:13:25,000 But that looks kind of ugly, maybe it does, maybe it doesn't. 227 00:13:25,008 --> 00:13:28,600 So I'm going to write a super cool for loop, but again simple, right? 228 00:13:28,600 --> 00:13:32,180 Just for me to check it I could say, you know, print the first five flipped. 229 00:13:32,180 --> 00:13:35,610 This little slice list slicing is a great way if you're using a 230 00:13:35,610 --> 00:13:39,080 lot of data to kind of keep your data from being totally insane. 231 00:13:39,080 --> 00:13:40,140 Okay? 232 00:13:40,340 --> 00:13:42,340 So I'm going to write a for loop, 233 00:13:43,340 --> 00:13:55,208 for key and value in flipped, and now I'm going to say 234 00:13:57,208 --> 00:14:01,330 So I'm going to say, only go through the first five. 235 00:14:01,330 --> 00:14:03,540 So this loop will only run for the first 236 00:14:03,540 --> 00:14:05,870 five. Just like this only prints the first five, 237 00:14:05,870 --> 00:14:09,610 this is going to iterate only through the first five. 238 00:14:09,610 --> 00:14:11,520 And key and value will go through the keys and the values 239 00:14:11,520 --> 00:14:16,360 values of these first five. 3, the. key, value. 3, is. 240 00:14:16,360 --> 00:14:18,835 And now I'm just going to print a sweet cool message out. 241 00:14:27,835 --> 00:14:30,349 [SOUND] And I'm just printing completely 242 00:14:30,349 --> 00:14:33,770 insane, partially mnemonic things. 243 00:14:33,770 --> 00:14:36,850 So I print out my top five keys and values. 244 00:14:36,850 --> 00:14:37,910 So let's take a look at that. 245 00:14:37,910 --> 00:14:39,130 Let's go ahead and run that baby. 246 00:14:40,970 --> 00:14:46,410 Winner 3 is the; 3, is; 3, and; so we saw five of these things. 247 00:14:47,540 --> 00:14:49,610 Five of them, and we're seeing the key and the value. 248 00:14:49,610 --> 00:14:54,200 It will probably be more easily read if I just did that. 249 00:14:55,300 --> 00:14:55,790 Right? 250 00:14:55,790 --> 00:15:02,580 So 3, the; 3, is; those are the top five words in my little program. 251 00:15:02,580 --> 00:15:04,360 So let's kind of take a look at this and review. 252 00:15:06,140 --> 00:15:06,640 Right? 253 00:15:08,680 --> 00:15:10,420 We open the file. 254 00:15:10,420 --> 00:15:12,090 We make an empty dictionary. 255 00:15:12,090 --> 00:15:17,220 We do our standard file loop, splitting each line. 256 00:15:17,220 --> 00:15:19,010 Newlines are kind of chomped up in the split, 257 00:15:19,010 --> 00:15:20,290 so we don't have to worry too much about it. 258 00:15:20,290 --> 00:15:22,370 We loop through the words. 259 00:15:22,370 --> 00:15:25,110 We change it to make it go lower case for each word. 260 00:15:25,110 --> 00:15:29,410 We use the get pattern to both initialize new words to 1 261 00:15:29,410 --> 00:15:32,890 and add 1 to existing words, while using this awesome getness. 262 00:15:34,050 --> 00:15:36,000 Now we want to flip. 263 00:15:38,240 --> 00:15:42,250 We want to flip the, the order in each tuple. 264 00:15:42,250 --> 00:15:45,185 We want to make the tuples from key, value tuples, 265 00:15:45,185 --> 00:15:48,230 which we get from items, to value, key tuples. 266 00:15:48,230 --> 00:15:51,570 We append each of those tuples to a list, then with 267 00:15:51,570 --> 00:15:56,620 the tuples that have the, the vaal and the kie in them. 268 00:15:57,910 --> 00:15:59,400 I can't write with this very well. 269 00:16:00,610 --> 00:16:03,285 We sort them in reverse order, but the vaal 270 00:16:03,285 --> 00:16:05,250 is going to be the most important thing it gets sorted on 271 00:16:05,250 --> 00:16:07,970 because it's the left-most thing in the tuple. 272 00:16:07,970 --> 00:16:10,791 And then we simply print out the last five. 273 00:16:10,791 --> 00:16:14,817 The top five of this tuples after the sort. 274 00:16:14,817 --> 00:16:21,258 And so that, pretty simple, print out the top five words in this particular file. 275 00:16:21,258 --> 00:16:22,262 Okay? 276 00:16:22,262 --> 00:16:25,270 Simple enough. 277 00:16:25,270 --> 00:16:25,930 Thanks a lot. 278 00:16:25,930 --> 00:16:27,210 And see you see on the net.