The following content is
provided under a Creative
Commons license.
Your support will help MIT
OpenCourseWare continue to
offer high quality educatiohal
resources for free.
To make a donation or to view
additional materials from
hundreds of MIT courses, visit
MIT OpenCourseWare at
ocw.mit.edu.
PROFESSOR: As this says, and
all the handouts say
that this is 6.450.
It's a first year graduate
course in the principles of
digital communication.
It's sort of the major first
course that you take as a
graduate student in the
communication area.
The information theory course
uses this as a prerequisite,
uses it in a rather
strong way.
6.432, the stochastic process
course uses it as a
prerequisite.
6.451, which is the companion
course which follows after
this uses it as a
prerequisite.
It's sort of material that you
have to know if you're going
to go into the communication
area.
It deals with this huge
industry, which is the
communication industry.
You look at the faculty here in
the department and you're a
little surprised to see that
there are so few faculty
members in digital
communication, so many faculty
members in the computer area.
You wonder why this is.
It's sort of this historical
accident, because somehow or
other the department didn't
realize as these two fields
were growing, the one that
looked more glamorous for a
long time was the
computer field.
Nobody in retrospect understands
why that is.
But at the same time
that's how it is.
You will not see as much
excitement here.
You also will see that because
of the big bust in the year
2000, there's much less
commercial activity in the
communications field now.
As students, you ought
to relish this.
You ought to really be happy
about this if you want to go
into the communication field,
because at some point in the
future, all of the failure to
start new investments, and all
of the large communication
companies means a very
explosive growth is
about to start.
So I don't think this is a
dead field by any means.
Looking back historically, every
time the communication
field has seemed dead, and I
can think of three or four
such times in history, those
have been the exact times
where it was great to
get into the field.
There was a time in the early
'70s when all the
theoreticians who worked in
communication theory all were
going around with long faces
saying everything that we can
do has been done, the field is
dead, nothing more to do,
might as well give up and
go into another field.
This was really exactly the time
that a number of small
communication companies were
really starting to grow very
fast because they were using
some of the modern ideas of
communication theory and they
were using these modern ideas
to do new things.
You now see companies like
Qualcomm that were started
then, Motorola of whom
major divisions were
started back then.
An enormous amount of activity
was starting just then, partly
because the field seemed to be
moribund and the theoreticians
were turning and deciding well,
I guess we have to do
something practical now.
If you want to find a good
practical engineer, if you
wind up in industry, if you wind
up as an entrepreneur or
if you wind up climbing the
ladder and being a chief
executive, if you want to find
somebody who will really do
important things, find somebody
who understands
theory who has decided that they
suddenly want to start to
do practical things.
Because if you point to the
people who have really done
exciting things in this field,
those are the ones who have
done most of it.
OK, so anyway, the big field,
it's an important field.
What we want to do is to study
the aspects of communication
systems that are unique to
communication systems.
In other words, we don't want
to study hardware here, we
don't want to study software
here, because hardware is
pretty much the same for any
problem you want to look at.
It's not that highly
specialized, and software is
not either.
So we're not going to study
either of those things, we're
going to study much more the
architectural principles of
communication.
We're going to study a lot of
the theory of communication,
and I want to explain why it is
that we're going to do that
as we move on.
Because it's not at all clear at
first why we want to study
the silly things that we'll be
studying in the course, and I
want to give you some idea today
of why we're pushed into
doing that.
As we start doing these
things, I'll give
you more of an idea.
I know when I was a student I
couldn't figure out why I was
learning the things that
I was learning.
I just found that some things
were much more interesting
than others, and the things that
tended to be interesting
were the more theoretical
things, so I started looking
at them harder and thought
gee, this is neat stuff.
But I had no idea that it was
ever going to be important.
So I'm going to try to explain
to you why this is important,
and something today about what
the proper inter-relationship
is of engineering and theory.
As another example of my own
life, for a long time I
thought -- this is rather
embarrassing to say --
but I was a theoretician and not
much of an engineer for a
very long time.
And I kept worrying about this
and saying well the stuff I'm
doing is fun and I enjoy doing
it, but why should
anybody pay me for it?
Of course, I was being paid by
MIT, I was being paid as a
consultant, but I felt I was
robbing all these people.
And finally, I decided a
rationale for all of this.
Most people felt that theory was
silly, and therefore, they
would keep asking me well,
if you're so smart
why aren't you rich?
I didn't have any very good
answer to that because I felt
I was smart because I understood
all this theory,
but I wasn't rich.
So I thought gee, what I've
gotta do is do enough
engineering to get rich, and
then when people ask me that
question I can say I am.
I would suggest to all you, if
you have theoretical leadings,
to focus on some engineering
also so you can become rich,
not because there's anything to
do with the money, but just
because it lets you hold your
head up high in a society that
values money a lot more
than values intellect.
So it's important.
OK.
This is a part of this
two-term sequence.
One comment here is I'm not sure
that whether 6.451, which
is the second part of this
sequence, is going to be
taught in the spring or not.
It might not be taught until
spring of the following year.
If a large number of you decide
that you really want to
do this and do it now, make
yourselves heard a little bit
because the idea of a lot of
people want this course will
have something to do on whether
somebody manages to
teach it or not.
As I was starting to say, theory
has had a very large
impact on almost every field
we can think of.
I mean you think of
electromagnetism, you think of
all these other fields
and theory is very
important in them.
But in communication systems it
is particularly important.
It's partly important because
back in 1948, Claude Shannon
who was the inventor of
information theory -- you
usually don't invent theories,
but he really did.
This came whole cloth out
of this one mind.
He had these ideas turning
around in his head for about
eight years while the second
World War was going on.
He was working on other things,
which were, in fact,
very important things, and he
was doing those things for the
government.
He was working on aircraft
control and things like that.
He was working on
cryptography.
But he was just fascinated by
these communication problems
and he couldn't get
his mind off them.
He took up cryptography because
it happened to use all
the ideas that he was really
interested in.
He wrote something about
cryptography during the middle
of the war, which made many
people believe that he had
done his cryptography work
before he did his
communication work.
But in fact, it was the
other way around.
This was purely the fact that
he could write about
cryptography, and he never liked
to write, so he didn't
want to write about
communication because it was a
more complicated field
and he didn't
understand the whole thing.
So for 25 years it was a
theory floating around.
You will learn a good deal
of what this theory is.
Most graduate courses on
communication don't spend much
time on information theory.
This was not a mistake many,
many years ago, but it is a
mistake now, because
communication theory at its
core is information theory.
Most people realize this.
Most kids recognize it,
most people in the
street recognize it.
When they start to use a system
to communicate with the
internet, what do
they talk about?
Do they talk about
the bandwidth?
No.
They might call it bandwidth but
what they're talking about
is the number of bits per second
they can communicate.
This is a distinctly information
theoretic idea,
because communication used to
be involved with studying
different kinds of wave forms.
Suddenly, at this point, people
are studying all these
other things and are based on
Claude Shannon's ideas,
therefore, we will teach some
of those ideas as we go.
As a matter of fact, I was
talking a little bit about the
fact that there's been a sort of
a love-hate relationship by
communication engineers for
information theory.
Back in 1960, a long, long time
ago when I got my PhD
degree, the people at MIT, the
people who started this field,
many of them told me that
information theory was dead at
that point.
There'd been an enormous amount
of research done on it
in the ten years before 1960.
They told me I should go into
vacuum tubes, which was the
coming field at that time.
Two points to that story.
If people tell you that
information theory isn't
important, please remember that
story or else tell them
to start to study
vacuum tubes.
The other point of it -- well,
I forgot what the other point
of it is so let's go on.
Anyway, information theory is
very much based on a bunch of
very abstract ideas.
I will start to say what that
means as we go on, but
basically it means that
it's based on very
simple-minded models.
It's based on ignoring all
the complexities of the
communication systems and
starting to play with toy
models and trying to
understand those.
That's part of what we have to
understand to understand why
we're doing what we're doing.
A very complex relationship
between modeling, theory,
exercises and engineering
design.
Maybe I shouldn't be talking
about that now and maybe the
notes shouldn't talk about it,
but I want to open the subject
because I want you to be
thinking about that
throughout the term.
Because it's something that
most people really don't
understand.
Something that most teachers
don't understand.
It's something that most
students don't understand.
When you're a student, you do
the exercises you do because
you know you have to do them.
There was a sort of apocryphal
story awhile ago that somebody
told me about a graduate
student, not at MIT, thank
God, who went into see the
person teaching a course on
wireless and saying, I don't
want to know all that theory,
I don't want to have to think
about these things, just tell
me how to do the problems.
Now, we will explain the
enormous stupidity of that as
we move on.
Part of the stupidity is that
the problems that you work on
as a graduate student, even in
your thesis, the problems that
you work on are not
real problems.
The problems that we work on
everywhere in engineering are
toy problems.
We work on them because we want
to understand some aspect
of the real problem.
We can never understand the
whole problem, so we study one
aspect, then we study
another aspect.
The reason for all these
equations that we have, it's
not a way to understand
reality.
It really isn't.
It's a way to understand little
pieces of reality.
You take all of those and after
you have all of these in
your mind, you then start to
study the real engineering
problem and you say I put
together this and this and
this and this and this -- this
is important, this is not so
important, here, this
is more important.
And then finally, if you're a
really good engineer, instead
of writing down an equation, you
take an envelope, as the
story goes, and you scribble
down a few numbers, and you
say the way this system ought to
be built is the following.
That's the way all important
systems get built, I claim.
Important systems are not
designed by the kinds of
things you do in homework
exercises.
The things you do in homework
exercises are designed to help
you understand small
pieces of problems.
You have to understand those
small pieces of problems in
order to understand the whole
engineering problems, but you
don't do simulations to build
a communications system.
You understand what the whole
system is, you can see it in
your mind after you think about
it long enough, and
that's where good system
design comes from.
Simulations are a big part of
all of this, writing down
equations are a big
part of it.
But when you're all done, if
you design an engineering
system and you don't see in your
mind why the whole thing
works, you will wind up with
something like Microsoft Word.
That's the truth.
So the exercises that we're
going to be doing are really
aimed at understanding
these simple models.
The point of the exercises is
not to get the right answer.
The answer is totally irrelevant
to everything.
What's important is you start to
understand what assumptions
in the problem lead to
what conclusions.
How if you change something
it changes something else.
That's when you're using system
design, because in
system design you can never
understand the whole thing.
You have to look at what parts
of it are important, what
parts of it aren't important,
and the only way you can do
that is having a catalog of the
simple-minded things in
the back of your mind.
Practical engineers get that by
dealing with real systems
day-to-day all their lives.
They see catastrophes occur.
Those things tell them what to
avoid and what not to avoid.
Students can do this in a
much more efficient way.
Obviously, in the practical
experience also, but the
experience that you get from
looking at these exercises and
looking at this analytical
material in the right way,
mainly looking at it is how do
you analyze these simple toy
models, how do you make
conclusions from them is what
let's you start being
a good engineer.
I'm stressing this because at
some point this term, all of
you, all of you at different
times are going to start
saying why the hell
am I doing this?
Why am I dealing with this
stupid little problem?
Stupid little problems can get
incredibly frustrating at
times, because you see something
that's so simple and
you can't understand it.
Then you say well, this
simple thing can't
be important anyway.
What I really want to do is
understand what this overall
system is, and you give
up at that point.
Don't do that.
Sometimes there's a very
difficult question about what
you mean by something
being simple.
I will sometimes say that
something is very simple, and
I will offend many you to
whom it's not simple.
The point is something
becomes simple after
you understand it.
Nothing is simple before
you understand it.
So there's that point at which
the light goes off in your
head at which it
becomes simple.
When I say that something is
simple, what I mean is that if
you think about it long enough
a light will go off in your
head and it will
become simple.
It's not simple to start with.
Other things are just messy.
You've all heard of mathematical
problems which
are just ugly.
There are these things of
extraordinary complexity.
There are things where there
just isn't any structure.
You see a lot of these things
in computer science.
I talked about Microsoft Word.
Part of the reason it's such a
lousy language is because the
problem is incredibly
difficult.
It's incredibly unstructured.
So people come up with things
that don't make any sense,
because that inherently
is not simple.
OK, these other things that
we'll be studying here are
inherently simple and the only
problem is how do you get to
the point of seeing these
simple ideas.
OK, so that's enough
philosophy.
No, not quite enough, a
little more of that.
All the everyday communication
systems that we deal with are
incredibly complex in terms of
the amount of hardware, the
amount of software in them.
If you look at the whole
system and you try to
understand it as a whole,
you don't have prayer.
There's no way to do it.
These things work and they can
be understood because they're
very highly structured.
They have simple architectural
principles.
What do I mean by architectural
principles?
It's a word that engineers
use more and more.
It started off with computer
systems because it became
essential there to start
thinking in terms of
architecture.
It's exactly the same thing
that you mean when you're
talking about a house
or a building.
Mainly the architecture
is the overall design.
It's the way the different
pieces fit together.
In engineering, architecture
is the same thing.
It's the thing that happens
when your eyes fuzz over a
little bit and you can't see
any of the details anymore,
and suddenly what you're doing
is you're looking at the whole
thing as an entity
and saying how do
the pieces fit together.
Well, what we're going to
be focusing on here.
One of the major keys to making
these things simpler is
to have standardized
interfaces and to have layering.
And we'll talk a little bit
about each of these because
our entire study of this subject
is based on studying
the different layers that
we'll wind up with.
OK, the most important and most
critical interface in a
communication system
is between the
source and the channel.
Today, that standard interface
is almost always
a binary data stream.
You all know this.
When you talk about
your modab, how do
you talk about it?
48 kilobits per second
if you're an
old-fashioned kind of guy.
1.2 megabits if you're a medium
technology person, or
several gigabits if you're one
of these persons who likes to
gobble up huge amounts
of stuff.
That's the way you describe
channels -- how many bits per
second can you send?
The way you describe sources is
how many bits per second do
you need as a way of viewing
that source.
When you store a picture, how
do you talk about it?
You don't talk about in terms of
the colors or any of these
other things, picture
is a bunch of bits.
That, at the fundamental
level, is
what we're doing here.
When you deal with source
coding, you're talking
fundamentally about the problem
of taking whatever the
source is, and it can be any
sort of thing at all -- it can
be voice, it can be text, it can
be emails, the emails with
viruses in it, whatever -- and
the problem is you want to
turn it into a bit stream.
Then this bit stream gets
presented to the channel.
Channels and the channel
encoding equipment don't have
any idea of what those
bits mean.
They don't want to have any idea
of what those bits mean.
That's what an interface
means.
It means the problem of
interpreting the bits, the
problem of going from something
which you view as
having intelligence down to
a sequence of bits is the
problem of the source coder.
The problem of taking those bits
and moving them from one
place to another is the function
of the channel
designer, the network designer
and all of those things.
When you talk about information
theory, it's a
total misnomer from
the beginning.
Information theory does not deal
with information at all,
it deals with data.
We will understand what that
distinction is as we go on.
When you talk about a bit
stream, what you're talking
about is a sequence of data.
It doesn't mean anything.
It's just that a one is
different from a zero, and you
don't care how it's different,
it just is.
So the channel input
is a binary stream.
It doesn't have any
meaning as far as
the channel is concerned.
The bit stream does have a
meaning as far as the source
is concerned.
So the problem is the source
encoder takes these bits,
takes the source, whatever it
is, with its meaning and
everything else, turns it
into a stream of bits --
you would like to turn it into
as few bits as possible.
Then you take those bits, you
transmit them on the channel,
the channel designer understands
what the physical
medium is all about, understands
what the network
is all about, and doesn't
have a clue as to what
all those bits mean.
So the picture is
the following.
That's the major layering of
all communication systems.
Here's the input, which is
whatever it happens to be.
Here's the source encoder.
The source encoder has to
know a great deal about
what that input is.
You have to know the structure
of the input.
You have a source encoder for
voice and you use it to try to
encode a video stream it's not
going to work, obviously.
If you try to use a source
encoder for English and try to
use it on Chinese, it probably
won't work very well either.
It won't work for anything
other than what
it's intended for.
So the source encoder has to
understand the source.
When I say understand the
source, what do I mean?
It means to understand
the probabilistic
structure of the source.
This is another of the things
that Shannon recognized
clearly that hadn't been
recognized at all before this.
You have to somehow understand
how to view what's coming out
of the source, this picture,
say, as one of a possible set
of pictures.
If you know ahead of time that
a communication system is
going to be sending the
Gettysburg Address or one of
50 other highly-inspiring
texts, what do you do?
Do you try to encode
these things, all
these different texts?
Of course not.
You just assign number one to
the Gettysburg Address, number
two to the second thing that
you might be interested in.
You send a number and then at
the output out comes the
Gettysburg Address because
you've stored that there.
So that's the idea
of source coding.
What is important is not the
complexity of the individual
messages, it's the probabilistic
structure that
tells you what are the different
possibilities.
That's what happens when
you try to turn
things into binary digits.
You have one sequence of binary
digits for each of the
possible things that you
want to represent.
Then in the channel you have
the same sort of thing.
You have noise, you have all
sorts of other crazy things on
the channel.
You have a sequence of bits
coming in, you have a sequence
of bits coming out.
The fundamental problem of the
channel is very, very simple.
You want to spit out the
same bits that came in.
You can take a certain amount
of delay doing that, but
that's what you have to do.
In order to do that, you have to
understand something about
the probabilistic structure
of the noise.
So both ways you have
probabilistic structure.
It's why you have to understand
probability at the
level of 6.041, which is the
undergraduate course here
studying probability.
If you don't have that
background, please don't take
the course because you're
going to be crucified.
If you think you can learn
it on the fly, don't.
You can't learn it on the fly.
As a matter of fact, if you've
taken the course, you still
don't understand it, and you
will need to scramble a little
bit to understand it at a
deeper level in order to
understand what's
going on here.
6.041 is a particularly good
undergraduate course.
I think it's probably taught
here better than it's taught
at most places.
But you still don't understand
it the first time through.
It's a tricky, subtle subject
you really need to understand
enough of it, If you have
no exposure to it.
If you've taken a course
called statistics and
probability, which first teaches
you statistics and
then tucks in a little bit of
probability at the end, go
learn probability first because
you don't have enough
of it to take this subject.
The other part of dealing with
these channels is that
suddenly we have to deal with
4AA analysis and all of these
things, if you don't have
some kind of subject
in signals and systems.
Some computer scientists
don't learn that
kind of material anymore.
Again, you can't learn it on
the fly because you need a
little bit of background
for it.
Those two prerequisites
are really essential.
Nothing else is essential.
The more mathematics you know
the better off you are, but we
will develop what we
need as we go.
Now, we talked about this
fundamental layer in all
communication systems, which is
between source coding and
channel coding.
You first turn the source into
bits, and then you turn the
bits into something you can
transmit on the channel.
Source coding breaks down into
three pieces itself.
It doesn't have to break down
into those three pieces, but
it usually does.
Most source coding you start out
with a wave form, such as
with voice, or with pictures you
start out with what's more
complicated than a wave form,
some kind of mapping from
x-coordinate and y-coordinate
and time, and you map all of
those things into something.
Our problem here is we want to
take these analog wave forms
or generalizations of.
It turns out that all we have
to deal with here is the
analog wave forms because
everything else
follows easily from that.
So there's one question.
How do you turn an analog
wave form into
a sequence of numbers?
This is the common way of
doing source coding.
Many of you who have been
exposed to the sampling
theorem, you think that's the
way to do it, this is
one way to do it.
We will talk a great
deal about this.
You will find out that the
sampling theorem really isn't
the way to do it.
Although again, it's a simple
model, it's the way to start
dealing with it.
Then you learn the various
other parts of
that as you go along.
After you wind up with
sequences, sequences of
numbers, we worry about how
to quantize those numbers,
because fundamentally we're
trying to get from
wave forms into bits.
So the three stages in that
process are first go to
sequences, go from sequences to
symbols -- namely, you have
a finite number of symbols,
which are quantized versions
of those analog numbers
which can be anything.
Then finally, you encode
the symbols into bits.
The coding goes in the
opposite order.
Here's a picture which shows it
better than the words do.
From the input wave form, you
sample it or something more
generalized than sampling it.
That gives you a sequence
of numbers.
You quantize those numbers,
that gives
you a symbol sequence.
You have a discrete coder, which
turns those symbols into
a sequence of bits
in some nice way.
Then you have this reliable
binary channel.
Notice what I've done here.
This thing is that entire
system we were
talking about before.
This has a channel encoder in
it, a channel, a channel
decoder and all of that.
But what we've been saying in
this layering idea is when
you're dealing with source
coding you ignore all of that.
You say OK, it's Tom's job who
was designing the channel
encoder to make sure that
my bits come out
here as the same bits.
If the bits are wrong
it's his fault.
If the bits are right, and this
output wave form doesn't
look like the input wave
form, it's my fault.
So part of the layering idea is
we just recreate this whole
thing here as the same bits and
we don't worry about what
happens when the bits
are different.
The other part of this is
that all of this breaks
down in the same way.
Namely, at this interface
between here and here, the
idea is the same.
Namely, there's an input wave
form that comes in, there's a
sequence of numbers here.
The job of this analog filter,
as we call it, and you'll see
why we call it that later, is
to take numbers here, turn
them back into wave forms.
These numbers are supposed to
resemble these numbers in some
way or other.
I can deal with this part of the
problem totally separately
from dealing with the other
parts of the problem.
Namely, the only problem here is
how do I take numbers, turn
them into wave forms.
There are a bunch of other
problems hidden here, like
these numbers are not going to
be exactly the same as these
wave forms because I have
quantization involved here.
Since the numbers are not
exactly the same, we have to
deal with questions
of approximation.
How to approximations in numbers
come out in terms of
approximations in terms
of wave forms?
That's where you need things
like the sampling theorem and
the generalizations of it that
we're going to spend
a lot of time with.
Then you go down
to this point.
At this point the quantizer
produces a string of symbols.
Whatever those symbols happen
to be, but those symbols
might, in fact, just be integers
whereas the things
here were real valued numbers,
and in quantizing we turn real
numbers into intergers by saying
-- well, it's the same
way you approximate things
all the time.
You round off the pennies
in your checkbook.
It's the same idea.
So the problem here is the
quantizer produces these
symbols here.
The job of all of this stuff
is to recreate those same
symbols here.
So the symbols now go into
something that does the
reverse of a quantizer --
we'll call it a table look-up
because that's
sort of what it is.
So we can study this part of the
problem separately, also.
You don't have to understand
anything about this problem to
understand this problem.
Finally, we have symbols here
going into a discrete encoder.
We have an interesting problem
which says how do we take
these symbols which have some
kind of probabilistic
structure to them and turn them
into binary digits in an
efficient way.
If the symbols here, for
example, are binary symbols
and the symbols happen to be 1
with probability 999 out of
1,000, and zero with probability
1 in 1,000, you
don't really just want to take
these symbols and map them in
a straightforward way, 1 into
1 and zero into zero.
You want to look at whole
sequences of them.
You want to do run length
coding, for example.
You want to count how long it
is between these zero's, and
then what you do is send those
counts and you encode them.
So there are all kinds of tricks
that you can use here.
But the point is, this problem
is separate, and this problem
is separate from this problem.
Now what we're going to do in
this course is start out with
this problem, because this is
the cleanest and the neatest
of the three problems.
It's the one you can
say the most about.
It's the one which has the
nicest results about it.
In fact, a lot of source coding
problems just deal with
this one part of it.
Whenever you're encoding text,
you're starting out with
symbols which are the letters
of whatever language you're
dealing with and perhaps a bunch
of other things -- ask a
code, you've generated 256
different things, including
all the letters, all the capital
letters, all of the
junk that used to be on
typewriters, and some of the
junk that's now in word
processing languages, and you
have a way of mapping those
into binary digits.
We want to look at better
ways of doing that.
So this, in fact, covers the
whole problem of source coding
when you're dealing with
text rather than
dealing with wave forms.
This problem here combined with
this problem deals with
all of these many problems where
what you're interested
in is taking a sequence of
numbers or a sequence of
whatjamacallits and
quantitizing
them and then coding.
Finally, when you get to the
input wave forms and output
wave forms, you've solved both
of these problems and you can
then focus on what's
important here.
This is a good case study of why
the information theoretic
approach works and
why theory works.
Because if you started out with
the problem of saying I
want to build something to
encode voice, and you try to
do all of these together, and
you look up all the things you
can find about voice encoding on
the web and you read all of
them, what you will wind up
with, I can guarantee you, is
you will know an enormous amount
of jargon, you will
know an enormous number of
different systems, which each
half work, you will not have the
slightest clue as to how
to build a better system.
So you have to take the
structured viewpoint, which is
really the only way to do things
to find new and better
ways of doing things.
Now there are some
extra things.
I have over-simplified it.
I want to over-simplify
things in this course.
What I will always do is I'll
try to cheat you into thinking
the problem is simpler than it
is, and then I will talk about
all of the added little nasties
that come in as soon
as you try to use any
of this stuff.
There are always nasties.
What I hope you will all do by
the end of the term is find
those little nasties on your
own before I start to talk
about them.
Namely, when we start to talk
about a simple model, be
interested in the simple model,
study it, find out what
it's all about, but at the same
time, for Pete's sake,
ask yourself the question.
What does this have to do with
the price of rice in China or
with anything else?
And ask those questions.
Don't let those questions
interfere with understanding
the simple model, but you've got
to always be focusing on
how that simple model relates
to what you visualize as a
communication system problem.
It usually will have
some relation but
not a complete relation.
Here the problems are in
this binary interface.
The source might produce packets
or the source might
produce a stream of data.
In other words, if what you're
dealing with is the kind of
situation where you're an
amateur photographer, you go
around taking all sorts of
pictures, and then you encode
these pictures --
what do I mean by encoding
the pictures?
You turn them into
binary digits.
Then you want to send your
pictures to somebody else, but
what you're really doing is
sending these packets to
someone else.
You're not sending a stream of
data -- you have 10 pictures,
you want to send those
10 pictures.
At the output of the whole
system, you hope you see 10
separate pictures.
You hope there's something in
there which can recognize that
out of the stream of binary
digits that come across the
channel, there's some packet
structure there.
Well we're not going to talk
about that really.
Whenever you start dealing with
this problem of source
encoding, you also need to worry
a little bit about the
packet structure.
What are the protocols for
knowing when something new
starts, when something
new ends.
Those kinds of problems are
dealt with mostly in a network
course here, and you can find
out all sorts of things about
them there.
But there are these two general
types of things --
packets and streams of data.
In terms of understanding voice
encoding, you can study
them both together.
Why can you study them
both together?
Because the packets are long
and because the packets are
long because they include a lot
of data, you have a little
bit of end effect about how to
start it, a little bit of end
effect about how to end it,
but the main structural
problem you'll be dealing
with is what to do
with the whole thing.
It's an added piece that comes
at the end to worry about how
do you denote the beginning
and the end.
That's a problem you have with
stream data also, because
stream data does not start
at time minus infinity.
Whatever the stream data is
coming from, what it's coming
from did not at time,
t equals infinity.
You just think of it that way
because you recognize you can
postpone the problem of how do
you start it and how do you
end it, and hopefully you can
postpone it until somebody
else has taken over the job.
What the channel accepts is
either of these binary streams
or packets, but then
queueing exists.
In other words, you have stuff
coming into a channel --
sometimes I'm sitting at home
and I want to send long files,
I want to send this
whole textbook
I'm writing to somebody.
It queues up, it takes a long
time to get it out of my
computer and into this other
person's computer.
It would be nice if I could
send it at optical speeds.
But I don't care much.
I don't care much whether it
takes a second or a minute or
an hour to get to this
other person.
He won't read it for a week
anyway, if he reads it at all,
so what difference
does it make?
But anyway, we have these
queueing problems, which are,
again, separable.
They're dealt with mostly
in the network course.
What we're mostly interested in
is how to reduce bit rate
for sources.
How do you encode things
more efficiently?
Data compression is the word
usually given to this.
How do you compress things into
a smaller number of bits
using the statistical
structure of it?
Then how do you increase the
number of bits you can send
over a channel?
That's the problem
with channels.
If you have a channel that'll
send 4,800 bits per second,
which is what telephone
lines used to do.
If you build a new product
that sends 9,600 bits per
second, which was a major
technological achievement back
in the '70s and '80s, suddenly
your company becomes a very
important communication
player.
If you then take the same
channel and learn how to
communicate at megabits over it,
that's a major thing also.
How did people learn
to do that?
Mostly by using all the ideas
that the people used in going
from 4,800 bits per second
to 9,600 bits per second.
It was the people who did that
first job who were the primary
people involved in
the second job.
The point of that is it doesn't
really make any
difference as an engineer
whether you're going from
4,800 to 9,600 or from 9,600 to
19.2 or from 19.9 to 38.4
or whatever it is,
or so forth up.
Each one of these is just
putting in a few new goodies,
recognizing a few new things,
making the system
a little bit better.
Let's talk about the
channel now.
The channel is given --
in other words, it's not under
the control of the designer.
This is something we haven't
talked about yet.
In all of these communication
system design problems, one of
the things that gets straight is
when you're worrying about
the engineering of something,
what parts of the system can
you control and what parts
of it can't you control.
Even more when you get into
layering -- what parts can you
control and what parts can
other people control.
Namely, if I have some source
and I'm trying to transmit it
over some physical channel and I
have two engineers -- one of
them is a data compressor, and
the other is a channel guy.
What the channel guy is trying
to do is to find a way of
sending more bits over
this channel than
could be done before.
What the data compressor is
trying to do is to find a way
to encode the source
into fewer bits.
If the two come together,
the whole thing works.
If the two don't come together,
then you have to
fire the engineers, hire some
new engineers or do something
else or you get fired.
So that's the story.
The things you can't change
here, you can't change the
structure of the source usually,
and you can't change
the structure of the channel.
There's some very interesting
new problems coming into
existence these days by
information theorists who are
trying to study biological
systems.
One of the peculiar things in
trying to understand how
biological organisms, which have
remarkable systems for
communicating information from
one place to another, how they
manage to do it.
One of the things you find is
that this separation that we
use in studying electronic
communication does not exist
in the body at all.
In other words, what we
call a source gets
blended with the channel.
What we call the statistics of
the source get very much
blended in with what this
organism is trying to do.
If the organism cannot send such
refined data, it figures
out ways to get the data that
it needs which is essential
for its survival or it
dies and some other
organism takes over.
So you find the system which has
no constraints in it, and
it evolves with all
of these things
changing at the same time.
The channels are changing,
the source statistics are
changing, and the way the source
data is encoded is
changing, and the way the data
is transmitted is changing.
So it's a whole new
generalization of this whole
problem of how do
you communicate.
Here though, the problem we're
dealing with is these fixed
channels, which you can think
of, depending on what your
interests are, you can view the
canonical channel as being
a piece of wire in a telephone
network, or even view it as
being a cable in a cable TV
system, or you can view it as
being a wireless channel.
We will talk about all of
these in the course.
The last three weeks of the
course is primarily talking
about wireless because that's
where the problems are most
interesting.
But in any situation we look
at, the channel is fixed.
It's given to us and
we can't change it.
All we can do fiddle around with
the channel encoding and
the channel decoding.
The channels that we're most
interested in usually send
wave forms.
That brings up another
interesting question.
We call this digital
communication, and what I've
been telling you is that the
sources that we're interested
in are most often wave
form sources.
The channels that we're
interested in are most often
channels that send wave forms,
receive wave forms and add
noise, which is wave forms.
Why do we call it digital
communication?
Anybody have a clue as to why
we might call all of this
digital communication?
Yeah?
AUDIENCE: [INAUDIBLE]
PROFESSOR: Because the analog
gets represented in binary
digits, because we have chosen
essentially, because of having
studied Shannon at some point,
to turn all the analog sources
into binary bit streams.
Namely, when you talk about
digital communication, what
you're really doing is saying I
have decided there will be a
digital interface between
source and channel.
That's what digital
communication is.
That's what most communication
today is.
There's very little
communication today that
starts out with an analog wave
form and finds a way to
transmit it without first going
into a binary sequence.
So here's a picture of one
of the noises that
we're going to study.
We have an input which
is now a wave form.
We have noise, which
is a wave form.
And we have an output,
which is a wave form.
This input here is going to be
created somehow in a process
of modulation from this binary
stream that's coming into the
channel encoder.
So somehow or other we're
taking a binary stream,
turning it into a wave form.
Now, think a little bit about
what might be essential in
that process.
If you know something about
practical engineering systems
for communication, you know that
there's an organization
in the U.S. called the FCC, and
every other country has
its companion set of initials,
which says what part of the
radio spectrum you can use and
what part of the radio
spectrum you can't use.
You suddenly realize that
somehow it's going to be
important to turn these binary
streams into wave forms, which
are more or less compressed
into some frequency bands.
That's one of the problems
we're going to
have to worry about.
But at some level if I take
a whole bunch of different
binary sequences, one thing I
can do, for example, I could
take a binary sequence
of length 100.
How many such binary sequences
are there?
Has length 100, the first bit
can be 1 or zero, second bit
can be 1 or zero -- that's
four combinations.
Third bit can be 1 or zero
also -- that makes eight
combinations of three bits.
There are 2 to the 100th
combinations
of 100 binary digits.
So a theoretician says fine,
I'm at those 2 to the 100th
different combinations of binary
digits into evenly
spaced numbers between
zero and 1.
Then I will take those evenly
spaced numbers between zero
and 1 and I'll modulate some
wave form, whatever wave form
I happen to choose, and I will
send that wave form.
In the absence of noise, I pick
up that wave form and
turn it back into my binary
sequence again.
What's the conclusion
from this?
The conclusion is if you don't
have noise there's no
constraint on how much
data you can send.
I can send as much data as I
want to, and there's nothing
to stop me from doing it,
if I don't have noise.
Yeah.
AUDIENCE: So, is that
where something that
[UNINTELLIGIBLE] basically
comes in.
Like a difference between one
signal and the next signal.
PROFESSOR: Somehow I have to
keep these things seperate.
I don't necessarily have to
separate one binary digit from
the next binary digit in time.
I could, in fact, do something
like creating 2 to the 100th
different wave forms or I could
do anything in between.
I could take each sequence of
eight bits, turn them into 1
of 256 wave forms, transmit one
wave form, then a little
bit later transmit another
wave form and so forth.
So I can split up the pie in any
way I want to, and we'll
talk about that a
lot as we go on.
Now, I split them up into
different wave forms, which I
send at spaced times.
I somehow have to worry about
the fact that if I send them
over a finite bandwidth, there
is no way in hell that you can
take finite bandwidth
wave forms and send
them in finite time.
Anything which lasts for a
finite amount of time spreads
out in bandwidth.
Anything which is in a finite
amount of bandwidth spreads
out in time.
That's what you ought
to know from 6.003.
So we're dealing with an
unsoluble problem here.
Well, Nyquist back in 1928
solved that problem.
He said well no, you can't make
the wave form separate,
but you can make the
sample separate.
People earlier had figured out
they could do that with the
sampling theorem, but the
sampling theorem wasn't very
practical, but Nyquist's way
of doing it was practical.
So, in fact, you can separate
these wave forms.
But you still have the
problem how do you
deal with the noise.
Well, one of the favorite ways
of dealing with noise is to
assume that it's something
called white Gaussian noise.
What is white Gaussian noise?
White Gaussian noise is noise
which no matter where you look
it's sitting there.
You can't get away from it.
You move around to different
frequencies, the noise is
still there.
You move around to different
times, it's still there.
It is somehow uniformed
throughout time
and throughout frequency.
Kind of an awkward thing because
if I developed a
receiver which didn't have any
bandwidth constraint on it,
and I looked at this noise
coming in, which was spread
out over all frequencies,
the noise would
burn out the receiver.
In other words, this white
Gaussian noise
has infinite power.
That shouldn't bother
you too much.
You deal with unit impulses
all the time.
A unit impulse has infinite
energy also.
You probably never thought
about it because most
undergraduate courses try very
hard to conceal that fact from
you because they like to use
impulses for everything, but
impulses have infinite energy
associated with them, and
that's kind of a nasty thing.
But anyway, we will deal with
the fact that impulses have
infinite energy, and therefore,
not use them to
transmit data.
And we will deal with the fact
that this noise has infinite
power, and find out how
to deal with that.
And we will learn how to
understand the statistical
structure of that noise.
So we'll deal with all of these
things when we start
studying these channels.
As we do that we will find out
that no matter what you do on
a channel like this, there are
some fundamental constraints
on how much data can
be transmitted.
I can take you through a
little bit of a thought
experiment which will sort
of explain that, I think.
It's partly what we were just
starting to say over here.
A simple-minded way to look at
this problem is I have binary
digits coming in.
I take these binary digits and
I separate them into shorts
sequences of m binary
digits each.
So, I take the first m binary
digits, then I take the next m
binary digits, and the next m
binary digits and so forth.
A sequence of m binary digits,
there are 2 to the m
combinations of this.
So I map these 2 to the m
combinations into one of 2 to
the m different numbers.
Now, you look at this noise
and you say I have to keep
some separation between
these numbers.
So, with a separation between
the numbers, say the
separation is 1, let this
define the number 1.
That's something we
theoreticians do all the time.
So I have 2 to the m
different levels.
I have to send this in
a certain bandwidth.
The sampling theorem says I
can't send things that are
more than twice the bandwidth
of this system, so I have a
bandwidth of 2w.
I can send m binary digits
in this bandwidth, w.
Therefore, I can send bits
at a certain rate.
This is all very crude but it's
going to lead us to an
interesting trade-off.
I can increase m and by
increasing m I can get by with
a smaller bandwidth, because I
have these symbols, these m
bit symbols coming along
at a slower rate.
So by increasing m, I can
decrease the bandwidth, by
increasing m, I can reduce the
bandwidth, but by reducing the
bandwidth, the power
is going up.
You can see that as I change
m, the power is going up
exponentially with m.
In other words, the trade-off
between bandwidth and power is
kind of nasty in this
simple-minded picture.
Let me jump ahead.
For this added of white
Gaussian noise with a
bandwidth constraint, Shannon
showed that the capacity was w
times log of 1 plus the power in
the system times the noise
power times w.
Now, I don't expect you to
understand this formula now,
and there's a lot of very tricky
and somewhat confusing
things about it.
One thing is that the power and
the noise is going up with
the bandwidth that you're
looking at.
Because I said the bandwidth is
everywhere, you can't avoid
it, and therefore, if you use a
wider bandwidth system, you
get more noise coming into it.
If you have a certain amount of
power that you're willing
to transmit and a certain
amount of noise, and the
number of bits per second
you can transmit is
going up with w.
But look at this affect with p
here, the effect with p is
logarithmic, which says as I
increase this parameter m and
I put more and more bits into
one symbol, this quantity is
going up exponentially, which
means the logarithm of this is
going up just linearly with m.
This gives you, if you look
at it carefully, the same
trade-off as the simple-minded
example I was talking about.
That simple-minded example
does not explain
this formula at all.
This formula's rather deep.
We probably won't
even completely
have proven this term.
What Shannon's results says is
that no matter what you do, no
matter how clever you are with
this noise that exists
everywhere, the fastest you
can transmit with the most
sophisticated coding schemes you
can think of is this rate
right here, which is what you
would think it might be just
from the scaling in terms of w
and m, where when you increase
m the power goes up
exponentially.
It's the same kind of answer
you get with that simple
thought experiment.
The fact that you have a 1 in
here is a little bit strange,
and we'll find out where
that comes from.
The idea here is that
this noise is
something you can't beat.
The noise is fundamental.
It's a fundamental part of every
communication system.
As I always like to say, noise
is like death and taxes, you
can't avoid them, they're there,
and there's nothing you
can do about them.
Like death and taxes, you can
take care of yourself and live
longer, and with taxes, well,
you can become wealthy and
support the government and make
all sorts of payments and
reduce your taxes that way.
Or you can hire a
good accountant.
So you can do things
there also.
But those things are
still always there.
So you're always stuck
with this.
One of the major parts of the
course will be to understand
what this noise is in a
much deeper context
than we would otherwise.
So, I'll save that and come
back to it later.
We have the same kind of
layering in channel encoding.
When I talk about channeling
encoding, I'm not talking
about any kind of complicated
coding technique.
6.451 talks about
all of those.
What I'm talking about is simply
the question of how do
you take a sequence of bits,
turn it into a sequence of
wave forms in such a way that
at the receiver we can take
that sequence of wave forms and
match them reliabily back
into the original bits.
There's a standard way of doing
this, which is to take
the sequence of bits,
first send them
through a discrete decoder.
What the discrete encoder
code will do is
something like this.
And match bits at one rate into
bits at a higher rate,
and this let's you correct
channel errors.
A simple example of this is
you match zero into zero,
zero, zero.
1 into 1, 1, 1.
If the rest of your system,
namely, if this part of the
system makes a single error at
some point, this part of the
system escapes from it.
I sort of put these on one
slide but I couldn't.
This takes a single zero, turns
it into three zeros.
These three zeros go into this
modulator, which maps this
into some wave from responsive
to these three zeros.
Wave form goes through here,
noise gets added, the
modulator, the text, those
binary digits, comes out with
some approximation to
these three zeros.
Every once in awhile because
of the noise, one of these
bits comes out wrong, and
because they come out wrong
this discrete decoder says
ah-ha, I know that what was
sent/put in was either zero,
zero, zero or 1, 1, 1.
It's more likely to have one
error than to have two errors,
and therefore, any time
one error occurs I
can decode it correctly.
Now, that is a miserable code.
A guy by the name of Hamming
became very famous for
generalizing this just a little
bit into another single
error correcting code, which
came through at a slightly
higher rate but still corrected
single errors.
He became inordinately famous
because he was a tireless
self-promoter and people think
that he was the person who
invented error correction
coding.
He really wasn't.
And his heirs will probably --
I don't know.
Anyway, coding of this type,
namely mapping binary digits
into longer strings of binary
digits in such a way that you
can correct binary errors, has
always been a major part of
communication system research.
A large number of communication
systems work in
exactly this way.
In fact, the older systems which
use coding tended to
work this way.
This was a very popular way of
trying to design system.
You would put a total layering
across here.
Namely, this part of the system
didn't know that there
was any coding going on here.
At this point where you're doing
discrete decoding, you
don't know anything about what
the detector is doing here.
It doesn't take a lot of
imagination to say if you have
a detector here and there's this
symbol that comes through
here, noise gets added to it,
you're trying to decode this
symbol, and there's this
Gaussian noise, which is just
sort of spread around,
it's concentrated.
Usually when errors occur,
errors occur just barely.
What you see at this point is
you almost flip a coin to
decide whether a zero or a 1 was
transmitted at this point.
If this poor guy had that
information, this poor guy
could work much, much better.
Namely, this guy could correct
double errors instead of
single errors, because if one
bit came through with a clear
indication that that's what it
was, and the other two bits
came through saying well, I
can't really tell what these
are, then this guy could
correct two errors
instead of one error.
So this is a good example where
this layering in here is
really a rotten idea.
Most modern systems don't
do that strict layering.
But a whole lot of modern
systems do, in fact, do this
binary encoding here and they
use the principles that came
out of the binary encoding
from a long time ago.
The only extra thing they do
here is they use this extra
information from the
demodulator.
This is something called
soft decoding
instead of hard decoding.
In other words, what comes out
of the demodulator is soft
decisions, and soft decisions
say well, I think it was this
but I'm not sure or I am
sure and so forth.
There's a whole range
of graduations.
We will study detection theory
and we'll understand how that
works and we'll understand how
you use those soft decisions
and all of that stuff and
it's kind of neat.
The modulation part of this is
what maps bit sequences into
wave forms.
When we map bit sequences into
wave form, there's this very
simple idea.
If I take a single bit that's
either zero or 1, I map a zero
into one wave form, I map a 1
into another wave form, I send
either wave form a
or wave form b.
I somehow want to make those
wave forms as different from
each other as I can.
Now, what do you mean by making
wave forms different?
I know what it means to make
numbers different.
I know that zero and 3
are more different
than zero and 2 are.
Namely, I have a good measure
of distance for
a sequence of numbers.
One of the things that we have
to deal with is how do you
find an appropriate measure of
distance for wave forms.
When I find the measure of
distance for wave forms, what
do I want to make it
responsive to?
What I'm trying to do is to
overcome this noise, and
therefore, I have to find the
measure of distance which is
appropriate for what the
noise is doing to me.
Well that's why we have to study
wave forms as much as we
do in this course.
What we will find is that you
can treat wave forms in the
same way as you can treat
sequences of numbers.
We will find that there's a
vector space associated with
wave forms, which is exactly
the same as a vector space
associated with infinite
sequences of numbers.
It's almost the same as the
vector space associated with
finite sequences of wave form,
and that vector space
associated with a finite
sequence of numbers is exactly
the vector space that you've
been looking at all of your
lives, and which you use
primarily as a notational
convenience to talk about
a sequence of numbers.
What's the distance that
you use there?
Well, you take the difference
between each number in the
sequence, you square it, you
add it up and you take the
square root of it.
Namely, you look at the
energy difference
between these sequences.
What we will find remarkably
is when we do things right,
the appropriate distance to talk
about on wave forms, is
exactly what comes from the
appropriate looking at sequences.
In fact, we'll take wave forms
and break them into sequences.
That's a major part of this
modulation process.
You think of that in terms
of the sampling theorem.
Namely, you take a bunch of
numbers, you take each number
in the sampling theorem and you
put a little sin x over x
hat around it and you transmit
that, and then you add up all
of these different samples,
which is a sequence of
numbers, each of them with these
little sin x over x caps
around them.
The neat thing about the sin x
over x caps around them is
they all go to zero
with these sample
points and it all works.
Therefore, the sequences
look almost the
same as a wave form.
We'll find out that that
works in general, so we
will do most of that.
Modern practice often combines
all these layers into what's
called coding modulation.
In other words, they go further
than just this idea of
soft decisions, and actually
treat the problem in general
of what do you do with bits
coming into a sequence, into a
system, how do you turn those
into wave forms that can be
dealt with an appropriate way.
And we'll talk just a little
bit about that and mostly
6.451 talks about that.
I want to talk a little bit
right at the end, I'm almost
finished, about computational
complexity in these
communication systems that we're
trying to worry about.
One of the things that you will
become curious about as
we move along is that we are
sometimes suggesting doing
things in ways that
look very complex.
The word complex is an
extraordinarily complex word.
None of us know what it means
because it means so many
different things.
I was saying that these
telephone systems are
inordinately complex.
In a sense they aren't because
they have an incredible number
of pieces, all of which
are the same.
So you can understand a
telephone system in terms of
understanding a relatively small
number of principles.
You can understand these
Gaussian channels in terms of
understanding a small number
of principles.
So they're complex if you don't
know how to look at
them, they're simple if you do
know how to look at them.
Here what I'm talking about is
how many chips do you need to
build something?
How complicated are
those chips?
Fundamentally, how much does it
cost to build the system?
Well, one of the curious things
and one of the things
that has made information theory
so popular in designing
communication systems is that
we're almost at the point
where chips are free.
Now, what does that mean?
It means if you want to do
something more complicated it
hardly costs anything.
This is sort of Moore's law.
Moore's law says every year
you can put more and more
stuff on a single chip.
You can do this cheaply and
the chips work faster and
faster so you can do more and
more complicated things very,
very easily.
There are some caveats.
Low cost with high complexity
requires large volume.
In other words, it takes a very
long time to design a
chip, it takes an enormous
amount of lead time --
I mean we're not going to study
these details in the
course, but I want you to be
aware of why it is that in a
sense complexity doesn't
matter anymore.
If you spend a long enough time
designing it and you can
produce enough of them, you can
do extraordinarily complex
things very, very cheaply.
I mean you see this with
all the cellular
phones that you buy.
I mean cellular phones now are
actually give-away devices.
You look at them to see what's
in them and they're
inordinately complicated.
You buy a personal computer
today and it's 1,000 times
more powerful than the biggest
computers in the
world 30 years ago.
You can do everything
more cheaply now
than you could before.
What's the hitch?
Well, you see the hitch when
you program a computer,
namely, if you look at word
processing languages, yes,
they become bigger and bigger
every year, they become more
and more complex, they will
do more and more things.
As far as their function of word
processing, some of us
think there have been advances
in the last 30 years.
Others, like myself, feel that
there have not been advances
and, in fact, we're
going backwards.
The problem is we have so much
complexity we don't know how
to deal with it anymore.
Most of us have become
blinking 12's.
You know what a blinking
12 is.
It's all of the electronic
equipment you have in your
home, whenever you don't program
it right you see a 12
blinking on and off, which is
where the clock is and the
clock hasn't been set,
and therefore,
it's blinking at you.
Most people who have complicated
devices, whether
they're audio devices or what
have you, are using less than
one percent of the fancy
features in them.
You spend hours and hours trying
to bring that from one
percent up to two percent.
So that the cost of complexity
is not in what you can build,
but it's in this conceptual
complexity.
If you design an algorithm and
you understand the algorithm,
it hardly makes any difference
how complicated it is, unless
the complexity is going up
exponentially with something.
That's the first caveat.
It's actually the second
caveat, too.
Complex systems are often not
well thought through.
They often don't work
or are not robust.
The non-robustness is the
worst part of it.
The third caveat is when you
have special applications.
Since they involve small
numbers, you're not going to
build a lot of them, it takes
a long time to design them.
The only way you can build
special applications is to
make them minor modifications of
other things that have been
done before.
So this question of how
complicated the systems we can
build are, if you can
make enough of
them they become cheap.
If you have enough lead time
they become cheap.
Which says that this layering
of communication system that
we're talking about really
ultimately makes sense.
Starting next time, we're going
to start talking about
this first part of source
coding, which is the discrete
part of source coding.
We will have those notes
on the web shortly.
You can look ahead at
them, if you would
like to, before Monday.
Please review the probability
that you are supposed to know
because you will need
it very shortly.
Thanks.
