Hello, and welcome to yet another Worked
Exercise for Python for Informatics: Exploring Information.
I'm Charles Severance, I am your tour guide to this particular worked example.
And this material is Creative Commons Attribution, and I hope to
see the clever and cool things that you can do with it.
This is part of my www.pythonlearn.com website that
helps you to get, learn Python sort of
from the very beginning with videos and other
materials to help you go along with the book.
Of course, the book is Creative
Commons Attribution, so pretty much everything here
is free for you to use and, and adapt as you see fit.
So we're taking a sort of a working example from the tuples
chapter, and the, the task that we're given
is find the ten most common words in a file.
And we will, will do that, and so if
we take a look at the basic overall structure.
We're going to read a file.
We're going to make a dictionary, an empty dictionary, and we're going to
read the line, we're going to read the file line by line.
We're going to split each line, then we're going to loop through
the line, using the dictionary pattern from the dictionaries chapter,
using get so that we can both access and initialize
new words that we haven't seen at the same time.
And then we are going to do, we need to sort, right?
But we need to sort by the value instead of the key.
So we're going to make a list.
And then we're going to loop through the results of the dictionary.
And we're going to make a new tuple.
But instead of being key, value, which is what
you get from items, we're going to do value, key.
And then we'll sort that in reverse order, and then we'll
simply print the first ten elements of that sorted in reverse order.
And because we're constructing carefully this new tuple
value, key in the right order, value will
be what's sorted on first, and then in
cases where value matches, it will sort on key.
Whereas if we just took items, which is key, value tuples,
then it would sort on the key, and that's quite easy.
So, that's the basic idea so let's go ahead and and get started.
So I'm going to grab some of this
and save myself a little bit of time,
and paste that in.
I'm going to use actually romeo-short, because I don't want to mess with it.
romeo has punctuation in it, so I'm going to go and I'll call this
under tuples
I'm going to call this guy topten.py.
I'll see if it's in my tuples folder, yep, just always sanity checking.
So let's, let's go ahead to pythonlearn and grab that romeo-short file.
[BLANK_AUDIO]
So I'm going to grab romeo-short.
If I look at romeo, romeo-short has conveniently has the, or just romeo
conveniently has the punctuation removed and it's small, so we can do debugging.
romeo-full is a little harder, and I cover that in the
book in some more detail to get rid of punctuation and stuff.
Not too bad, not too bad, but we won't talk about it here.
So I'm going to grab romeo.txt, and I'll save
this to my dictionaries, my tuples folder as romeo.txt.
And I should be done with that now.
And I'm done with that.
So if I take a look at tuples again
And I've got romeo.txt in this file topten.py.
And it's not romeo, it's romeo.txt.
And I'm just going to, split.
I mean, I think, I hope by now you can read a text
file [LAUGH] and split it into words, and count those words in a dictionary.
So I'm just going to print out the dictionary,
the counts dictionary that we've got at the end.
And let's see how that works.
So I'll go onto my desktop and this is under tuples.
[SOUND] And so there's romeo.txt and topten.py.
So let's go ahead and run topten.py.
And so there we go.
We have just sort roared through, broke it into words, broke it into lines.
And we have a nice convenient little thing here.
Now, you know what, I'm going to do one thing that's not in the example.
This, the fact that these are upper case, I don't know
if it's going to cause a problem or not, but I want to,
this is, this bothers me.
And so, so I'm going to, I'm going to force all the words to be lower case.
And so I'm going to call, I'm going to make a thing wrd equals word.lower.
I think it's lower.
We'll find out fast enough.
So I'm just going to grab a lower-case version of word and make
it be w-r-d, and that's what I'm going to put into my dictionary.
So let me run this again and see if I got it right, yeah.
So now we see things like 'who' is lower case.
I don't think it changed any of the numbers; it may have.
But, I just, I think we shouldn't treat upper case or we
should make this case insensitive, so we'll map everything to lower case.
So I just did that by mapping it to lower, and then
using w-r-d instead of word after I made a copy.
So, so, here we go.
So then what I'm going to do is I've got my
counts, and I'm going to print the results of the items.
So that gives us a list, instead of just showing it, it gives us a list of tuples.
So there's a tuple.
Its a two-item tuple.
Tuple, tuple, and it's key-value pairs.
Okay?
And we can, this is useful now, in particular because we can construct a for
loop that goes through for kie, vaal.
Key and value.
Kie, vaal equals, I'm sorry, in count.items.
I get all confused trying to create really non-mnemonic variables.
So we got kie and vaal and, you know, it's, it's two iteration variables.
And
the way it works is, you know,
these will iterate through each tuple, tuple, tuple.
And then kie will be the first thing in the tuple, which is
the key, and the value will be the second thing in the tuple.
key, val; key, val.
And so, kie will bounce through these at the same
time as vaal is going to bounce through the corresponding values.
So that's quite nice.
So just, and I always love sanity checks, so I'm going
to print out kie and vaal, just print them out.
Yeah, something's wrong.
Kie, vaal, and, oh, counts.
So if I, so here we go.
Traceback line 11.
The name count is not defined.
I was so quick, I didn't even read what it told me.
So look at line 11, going on line 11, and then I forgot my s.
So it's pretty much.
you know, once you get used to this, it's
pretty much telling me what it wants to do.
So now I'm going through my tuples.
Here's my tuples.
Now I'm going through the tuples one at a time.
Just dumping them out in a loop, and away we go.
Now, the whole problem is we want to make a new tuple.
Okay?
And I'm going to call this newtup, newtup.
And I'm going to create a tuple, and I can do this by just putting these two things
in parentheses, except that I want vaal first, and kie second, okay?
So I want vaal first and kie second, and so now I'm going to print out newtup.
Now you'll see that as I loop through each of the key-value pairs, I'm just
making a new tuple where they're reversed,
where the value is first and the key is second.
Right?
So value, key; value, key; value, key.
And now I'm going to stick this in a list.
And I'm going to make, I'm going to call my list flipped.
And I'm going to say, give me an empty list.
So, so I mean, I'm starting an empty list called flipped, and
I am going to tell flipped to append to itself.
So I'm now extending this list each time through the loop.
Take that newtup that I just made.
So we've got newtup.
And I'll take my print statement out, because that'll be obvious.
So I'm going to append newtup to the end of my flipped list.
And then I'll print, at the end of this,
print flipped.
Okay? So I make a tuple.
Instead of printing it I just append it to this list,
then I print the whole list out afterwards.
Okay, so this is a little messy, but here is the first list.
It's got key, value, and here's the second list.
It's the exact same list as before except it's value, key.
Now, it's exact same list, we didn't take anything out, we
didn't add anything, we didn't sum anything, we didn't do anything.
So then, what is it that we're going to do with this list?
Well, we can use the totally, well, we
could use sorted but we're not going to do that.
I want do this.
I'm going to say, I want to take that flipped list and I want to sort it.
Sort and then print flipped.
Just sort it.
Now the key of sorting,
the key of awesomeness of sorting, is that sorting is done by repeated comparison,
and it's comparing this tuple to this tuple.
But the rules of tuple comparison is it first checks
the first pair, and then it checks the second pair.
It only checks the second pair if the first pair doesn't match.
So 3, 1 doesn't match and so it sorts it
based on the 3 and the 1 without even looking.
But then when there is two 3's that match, it actually does check the second two words.
So 3, and will sort. The and and the the will decide
where the 3, and and the 3, the tuples end up sorting.
If that makes any sense.
Okay?
So let's just print this out.
Actually, let's print, print flipped, not clipped.
We can make a clipped one too later.
And I'm actually going to get rid of this
print, so it's going to print flipped before and after.
So watch this.
So here's flipped before it's been sorted, in whatever
order it was in the dictionary in the first place.
And now, here's flipped in the final order.
And you see that you know, it's in, it's
in sort of reverse order, where, you know,
3 and, is, and not are here at the end, the 1's
are in alphabetical order within the 1's, because tuples, check.
Oops, don't do that, don't do that.
That was nasty.
And that was really nasty.
It's because I was copying and pasting.
So within all the 1's it actually sorted based on the alphabetic sort of the second
parameter, or the second item in the tuple
whereas in among the 2's all sorted later.
So this actually isn't quite what I wanted.
And so I'm going to say, reverse equals True.
Because I actually want to sort it so the big numbers are at the front.
So let's run that little bit.
So check that, right?
So now I got my same sort.
It's got the unsorted, no it's got the
sorted, but the big numbers are at the beginning.
And you'll notice it's kind of in
reverse alphabetical order because the big words are
at the beginning among the 3's, but
eh, whatever, we're close enough for government work.
So now I want to print out the top ten, okay?
Oops, don't, I did that again.
I want to print out the top ten.
I'll just do like the top five.
Do the top five.
So, I could do something simple.
We'll get rid of this.
We trust that it's now properly sorted.
I could say,
print this list up to but not including 5, by using list slicing.
So there we go, there's the top five.
But that looks kind of ugly, maybe it does, maybe it doesn't.
So I'm going to write a super cool for loop, but again simple, right?
Just for me to check it I could say, you know, print the first five flipped.
This little slice list slicing is a great way if you're using a
lot of data to kind of keep your data from being totally insane.
Okay?
So I'm going to write a for loop,
for key and value in flipped, and now I'm going to say
So I'm going to say, only go through the first five.
So this loop will only run for the first
five. Just like this only prints the first five,
this is going to iterate only through the first five.
And key and value will go through the keys and the values
values of these first five. 3, the. key, value. 3, is.
And now I'm just going to print a sweet cool message out.
[SOUND] And I'm just printing completely
insane, partially mnemonic things.
So I print out my top five keys and values.
So let's take a look at that.
Let's go ahead and run that baby.
Winner 3 is the; 3, is; 3, and; so we saw five of these things.
Five of them, and we're seeing the key and the value.
It will probably be more easily read if I just did that.
Right?
So 3, the; 3, is; those are the top five words in my little program.
So let's kind of take a look at this and review.
Right?
We open the file.
We make an empty dictionary.
We do our standard file loop, splitting each line.
Newlines are kind of chomped up in the split,
so we don't have to worry too much about it.
We loop through the words.
We change it to make it go lower case for each word.
We use the get pattern to both initialize new words to 1
and add 1 to existing words, while using this awesome getness.
Now we want to flip.
We want to flip the, the order in each tuple.
We want to make the tuples from key, value tuples,
which we get from items, to value, key tuples.
We append each of those tuples to a list, then with
the tuples that have the, the vaal and the kie in them.
I can't write with this very well.
We sort them in reverse order, but the vaal
is going to be the most important thing it gets sorted on
because it's the left-most thing in the tuple.
And then we simply print out the last five.
The top five of this tuples after the sort.
And so that, pretty simple, print out the top five words in this particular file.
Okay?
Simple enough.
Thanks a lot.
And see you see on the net.