Least squares approximation

WEBVTT
Kind: captions
Language: en

00:00:00.430 --> 00:00:02.385
Let's say I have
some matrix A.

00:00:02.385 --> 00:00:05.500
Let's say it's an n-by-k
matrix, and I have the

00:00:05.500 --> 00:00:10.450
equation Ax is equal to b.

00:00:10.450 --> 00:00:17.540
So in this case, x would have to
be a member of Rk, because

00:00:17.540 --> 00:00:22.960
we have k columns here, and
b is a member of Rn.

00:00:22.960 --> 00:00:25.680
Now, let's say that it just so
happens that there is no

00:00:25.680 --> 00:00:35.460
solution to Ax is equal to b.

00:00:35.460 --> 00:00:37.100
What does that mean?

00:00:37.100 --> 00:00:39.730
Let's just expand out A.

00:00:39.730 --> 00:00:41.200
I think you already know
what that means.

00:00:41.200 --> 00:00:45.210
If I write a like this, a1, a2,
if I just write it as its

00:00:45.210 --> 00:00:49.250
columns vectors right there,
all the way through ak, and

00:00:49.250 --> 00:00:55.750
then I multiply it times x1,
x2, all the way through xk,

00:00:55.750 --> 00:00:57.690
this is the same thing as
that equation there.

00:00:57.690 --> 00:00:59.760
I just kind of wrote out
the two matrices.

00:00:59.760 --> 00:01:06.650
Now, this is the same thing as
x1 times a1 plus x2 times a2,

00:01:06.650 --> 00:01:13.870
all the way to plus xk times ak
is equal to the vector b.

00:01:13.870 --> 00:01:17.330
Now, if this has no solution,
then that means that there's

00:01:17.330 --> 00:01:23.430
no set of weights here on the
column vectors of a, where we

00:01:23.430 --> 00:01:25.060
can get to b.

00:01:25.060 --> 00:01:27.560
Or another way to say it is, no
linear combinations of the

00:01:27.560 --> 00:01:30.500
column vectors of a will
be equal to b.

00:01:30.500 --> 00:01:39.290
Or an even further way of saying
it is that b is not in

00:01:39.290 --> 00:01:41.880
the column space of a.

00:01:41.880 --> 00:01:45.720
No linear combination of these
guys can equal to that.

00:01:45.720 --> 00:01:48.000
So let's see if we can
visualize it a bit.

00:01:48.000 --> 00:01:51.370
So let me draw the column
space of a.

00:01:51.370 --> 00:01:54.460
So maybe the column space of
a looks something like this

00:01:54.460 --> 00:01:56.360
right here.

00:01:56.360 --> 00:02:00.392
I'll just assume it's
a plane in Rn.

00:02:00.392 --> 00:02:02.300
It doesn't have to be a plane.

00:02:02.300 --> 00:02:04.340
Things can be very general, but
let's say that this is the

00:02:04.340 --> 00:02:05.615
column space.

00:02:05.615 --> 00:02:09.560
This is the column space of a.

00:02:09.560 --> 00:02:12.300
Now, if that's the column space
and b is not in the

00:02:12.300 --> 00:02:14.800
column space, maybe we
can draw b like this.

00:02:14.800 --> 00:02:18.180
Maybe b, let's say this is the
origin right there, and b just

00:02:18.180 --> 00:02:19.250
pops out right there.

00:02:19.250 --> 00:02:20.560
So this is the 0 vector.

00:02:20.560 --> 00:02:23.990
This is my vector b, clearly
not in my column spaces,

00:02:23.990 --> 00:02:26.070
clearly not in this plane.

00:02:26.070 --> 00:02:29.540
Now, up until now, we would
get an equation like that.

00:02:29.540 --> 00:02:31.900
We would make an augmented
matrix, put in reduced row

00:02:31.900 --> 00:02:34.990
echelon form, and get a line
that said 0 equals 1, and we'd

00:02:34.990 --> 00:02:37.430
say, no solution, nothing
we can do here.

00:02:37.430 --> 00:02:39.870
But what if we can do better?

00:02:39.870 --> 00:02:41.885
You know, we clearly can't
find a solution to this.

00:02:41.885 --> 00:02:43.610
But what if we can find
a solution that

00:02:43.610 --> 00:02:45.570
gets us close to this?

00:02:45.570 --> 00:02:51.030
So what if I want to find some
x, I'll call it x-star for

00:02:51.030 --> 00:02:58.090
now, where-- so I want to find
some x-star, where A times

00:02:58.090 --> 00:03:07.340
x-star is-- and this is
a vector-- as close as

00:03:07.340 --> 00:03:16.560
possible-- let me write this--
as close to b as possible.

00:03:16.560 --> 00:03:19.450
Or another way to view it, when
I say close, I'm talking

00:03:19.450 --> 00:03:25.830
about length, so I want to
minimize the length of-- let

00:03:25.830 --> 00:03:26.670
me write this down.

00:03:26.670 --> 00:03:38.260
I want to minimize the length
of b minus A times x-star.

00:03:41.940 --> 00:03:43.990
Now, some of you all
might already know

00:03:43.990 --> 00:03:44.840
where this is going.

00:03:44.840 --> 00:03:48.130
But when you take the difference
between 2 and then

00:03:48.130 --> 00:03:51.250
take its length, what
does that look like?

00:03:51.250 --> 00:03:53.172
Let me just call Ax.

00:03:53.172 --> 00:03:55.820
Ax is going to be a member
of my column space.

00:03:55.820 --> 00:03:57.415
Let me just call that v.

00:03:57.415 --> 00:04:00.610
Ax is equal to v.

00:04:00.610 --> 00:04:06.660
You multiply any vector in Rk
times your matrix A, you're

00:04:06.660 --> 00:04:09.040
going to get a member of
your column space.

00:04:09.040 --> 00:04:16.019
So any Ax is going to be
in your column space.

00:04:16.019 --> 00:04:20.850
And maybe that is the vector v
is equal to A times x-star.

00:04:20.850 --> 00:04:24.860
And we want this vector to get
as close as possible to this

00:04:24.860 --> 00:04:26.890
as long as it stays--
I mean, it has to be

00:04:26.890 --> 00:04:28.160
in my column space.

00:04:28.160 --> 00:04:31.020
But we want the distance between
this vector and this

00:04:31.020 --> 00:04:33.040
vector to be minimized.

00:04:33.040 --> 00:04:36.470
Now, I just want to show you
where the terminology for this

00:04:36.470 --> 00:04:37.370
will come from.

00:04:37.370 --> 00:04:40.510
I haven't given it its
proper title yet.

00:04:40.510 --> 00:04:43.840
If you were to take this
vector-- let just call this

00:04:43.840 --> 00:04:47.980
vector v for simplicity-- that
this is equivalent to the

00:04:47.980 --> 00:04:51.240
length of the vector.

00:04:51.240 --> 00:04:53.350
You take the difference between
each of the elements.

00:04:53.350 --> 00:05:02.980
So b1 minus v1, b2 minus v2,
all the way to bn minus vn.

00:05:02.980 --> 00:05:05.650
And if you take the length of
this vector, this is the same

00:05:05.650 --> 00:05:06.150
thing as this.

00:05:06.150 --> 00:05:09.100
This is going to be equal
to the square root.

00:05:09.100 --> 00:05:10.600
Let me take the length
squared, actually.

00:05:10.600 --> 00:05:14.690
The length squared of this is
just going to be b1 minus v1

00:05:14.690 --> 00:05:21.900
squared plus b2 minus v2 squared
plus all the way to bn

00:05:21.900 --> 00:05:23.870
minus vn squared.

00:05:23.870 --> 00:05:25.500
And I want to minimize this.

00:05:25.500 --> 00:05:29.180
So I want to make this value the
least value that it can be

00:05:29.180 --> 00:05:32.990
possible, or I want to get the
least squares estimate here.

00:05:32.990 --> 00:05:36.550
And that's why, this last minute
or two when I was just

00:05:36.550 --> 00:05:39.720
explaining this, that was just
to give you the motivation for

00:05:39.720 --> 00:05:52.810
why this right here is called
the least squares estimate, or

00:05:52.810 --> 00:05:56.180
the least squares solution,
or the least squares

00:05:56.180 --> 00:06:00.160
approximation for the equation
Ax equals b.

00:06:00.160 --> 00:06:04.200
There is no solution to this,
but maybe we can find some

00:06:04.200 --> 00:06:09.140
x-star, where if I multiply A
times x-star, this is clearly

00:06:09.140 --> 00:06:11.900
going to be in my column space
and I want to get this vector

00:06:11.900 --> 00:06:14.010
to be as close to
b as possible.

00:06:14.010 --> 00:06:17.180
Now, we've already seen in
several videos, what is the

00:06:17.180 --> 00:06:21.090
closest vector in any
subspace to a vector

00:06:21.090 --> 00:06:23.200
that's not in my subspace?

00:06:23.200 --> 00:06:26.940
Well, the closest vector to
it is the projection.

00:06:26.940 --> 00:06:31.120
The closest vector to b, that's
in my subspace, is

00:06:31.120 --> 00:06:37.540
going to be the projection of
b onto my column space.

00:06:37.540 --> 00:06:39.940
That is the closest
vector there.

00:06:39.940 --> 00:06:47.530
So if I want to minimize this,
I want to figure out my

00:06:47.530 --> 00:06:57.380
x-star, where Ax-star is equal
to the projection of my vector

00:06:57.380 --> 00:07:02.460
b onto my subspace or onto
the column space of A.

00:07:02.460 --> 00:07:03.780
Remember what we're
doing here.

00:07:03.780 --> 00:07:08.010
We said Axb has no solution, but
maybe we can find some x

00:07:08.010 --> 00:07:10.310
that gets us as close
as possible.

00:07:10.310 --> 00:07:13.920
So I'm calling that my least
squares solution or my least

00:07:13.920 --> 00:07:15.940
squares approximation.

00:07:15.940 --> 00:07:19.340
And this guy right here is
clearly going to be in my

00:07:19.340 --> 00:07:23.210
column space, because you take
some vector x times A, that's

00:07:23.210 --> 00:07:25.520
going to be a linear combination
of these column

00:07:25.520 --> 00:07:27.420
vectors, so it's going to
be in the column space.

00:07:27.420 --> 00:07:33.880
And I want this guy to be as
close as possible to this guy.

00:07:33.880 --> 00:07:36.900
Well, the closest vector in my
column space to that guy is

00:07:36.900 --> 00:07:38.720
the projection.

00:07:38.720 --> 00:07:41.840
So Ax needs to be equal
to the projection of b

00:07:41.840 --> 00:07:44.440
on my column space.

00:07:44.440 --> 00:07:46.890
It needs to be equal to that.

00:07:46.890 --> 00:07:50.090
But this is still pretty
hard to find.

00:07:50.090 --> 00:07:52.620
You saw how, you know, you took
A times the inverse of A

00:07:52.620 --> 00:07:54.160
transpose A times A transpose.

00:07:54.160 --> 00:07:56.990
That's hard to find that
transformation matrix.

00:07:56.990 --> 00:08:01.110
So let's see if we can find an
easier way to figure out the

00:08:01.110 --> 00:08:04.870
least squares solution, or kind
of our best solution.

00:08:04.870 --> 00:08:06.470
It's not THE solution.

00:08:06.470 --> 00:08:08.900
It's our BEST solution
to this right here.

00:08:08.900 --> 00:08:11.290
That's why we call it the least
squares solution or

00:08:11.290 --> 00:08:14.070
approximation.

00:08:14.070 --> 00:08:17.130
Let's just subtract b from
both sides of this and we

00:08:17.130 --> 00:08:19.230
might get something
interesting.

00:08:19.230 --> 00:08:24.390
So what happens if we take Ax
minus the vector b on both

00:08:24.390 --> 00:08:25.360
sides of this equation?

00:08:25.360 --> 00:08:27.690
I'll do it up here
on the right.

00:08:27.690 --> 00:08:30.925
On the left-hand side we
get A times x-star.

00:08:30.925 --> 00:08:32.720
It's hard write the x and
then the star because

00:08:32.720 --> 00:08:33.970
they're very similar.

00:08:33.970 --> 00:08:36.460
And we subtract b from it.

00:08:36.460 --> 00:08:38.610
We subtract our vector b.

00:08:38.610 --> 00:08:46.020
That's going to be equal to the
projection of b onto our

00:08:46.020 --> 00:08:48.980
column space minus b.

00:08:48.980 --> 00:08:51.210
All I did is I subtracted
b from both

00:08:51.210 --> 00:08:52.800
sides of this equation.

00:08:52.800 --> 00:08:58.585
Now, what is the projection
of b minus our vector b?

00:08:58.585 --> 00:09:01.490
If we draw it right here, it's
going to be this vector

00:09:01.490 --> 00:09:05.480
right-- let me do it in
this orange color.

00:09:05.480 --> 00:09:06.860
It's going to be this
right here.

00:09:06.860 --> 00:09:09.410
It's going to be that vector
right there, right?

00:09:09.410 --> 00:09:13.910
If I take the projection of b,
which is that, minus b, I'm

00:09:13.910 --> 00:09:16.490
going to get this vector. you
we could say b plus this

00:09:16.490 --> 00:09:19.350
vector is equal to
my projection

00:09:19.350 --> 00:09:21.490
of b onto my subspace.

00:09:21.490 --> 00:09:24.880
So this vector right
here is orthogonal.

00:09:24.880 --> 00:09:28.580
It's actually part of the
definition of a projection

00:09:28.580 --> 00:09:34.460
that this guy is going to be
orthogonal to my subspace or

00:09:34.460 --> 00:09:35.360
to my column space.

00:09:35.360 --> 00:09:39.390
And so this guy is orthogonal
to my column space.

00:09:39.390 --> 00:09:47.930
So I can write Ax-star minus
b, it's orthogonal to my

00:09:47.930 --> 00:09:51.260
column space, or we could
say it's a member of the

00:09:51.260 --> 00:09:56.070
orthogonal complement
of my column space.

00:09:56.070 --> 00:10:00.660
The orthogonal complement is
just the set of everything,

00:10:00.660 --> 00:10:03.240
all of the vectors that are
orthogonal to everything in

00:10:03.240 --> 00:10:05.050
your subspace, in your column
space right here.

00:10:05.050 --> 00:10:07.990
So this vector right here
that's kind of pointing

00:10:07.990 --> 00:10:11.280
straight down onto my plane
is clearly a member of the

00:10:11.280 --> 00:10:14.210
orthogonal complement
of my column space.

00:10:14.210 --> 00:10:17.730
Now, this might look familiar
to you already.

00:10:17.730 --> 00:10:20.670
What is the orthogonal
complement of my column space?

00:10:20.670 --> 00:10:25.320
The orthogonal complement of
my column space is equal to

00:10:25.320 --> 00:10:31.920
the null space of a transpose,
or the left null space of A.

00:10:31.920 --> 00:10:33.330
We've done this in many,
many videos.

00:10:33.330 --> 00:10:40.240
So we can say that A times my
least squares estimate of the

00:10:40.240 --> 00:10:43.860
equation Ax is equal to
b-- I wrote that.

00:10:43.860 --> 00:10:49.010
So x-star is my least squares
solution to Ax is equal to b.

00:10:49.010 --> 00:10:55.010
So A times that minus
b is a member of the

00:10:55.010 --> 00:10:57.770
null space of A transpose.

00:10:57.770 --> 00:10:59.370
Now, what does that mean?

00:10:59.370 --> 00:11:05.020
Well, that means that if I
multiply A transpose times

00:11:05.020 --> 00:11:12.470
this guy right here, times
Ax-star-- and let me, no I

00:11:12.470 --> 00:11:14.740
don't want to lose the vector
signs there on the x.

00:11:14.740 --> 00:11:15.880
This is a vector.

00:11:15.880 --> 00:11:17.420
I don't want to forget that.

00:11:17.420 --> 00:11:20.710
Ax-star minus b.

00:11:20.710 --> 00:11:24.210
So if I multiply A transpose
times this right there, that

00:11:24.210 --> 00:11:26.630
is the same thing is that,
what am I going to get?

00:11:26.630 --> 00:11:29.400
Well, this is a member of the
null space of A transpose, so

00:11:29.400 --> 00:11:32.400
this times A transpose has
got to be equal to 0.

00:11:32.400 --> 00:11:35.350
It is a solution to A transpose
times something is

00:11:35.350 --> 00:11:40.120
equal to the 0 vector.

00:11:40.120 --> 00:11:40.440
Now.

00:11:40.440 --> 00:11:42.770
Let's see if we can simplify
this a little bit.

00:11:42.770 --> 00:11:57.750
We get A transpose A times
x-star minus A transpose b is

00:11:57.750 --> 00:12:02.900
equal to 0, and then if we add
this term to both sides of the

00:12:02.900 --> 00:12:11.180
equation, we are left with A
transpose A times the least

00:12:11.180 --> 00:12:17.230
squares solution to Ax
equal to b is equal

00:12:17.230 --> 00:12:23.050
to A transpose b.

00:12:23.050 --> 00:12:24.500
That's what we get.

00:12:24.500 --> 00:12:26.220
Now, why did we do
all of this work?

00:12:26.220 --> 00:12:28.050
Remember what we started with.

00:12:28.050 --> 00:12:34.180
We said we're trying to find a
solution to Ax is equal to b,

00:12:34.180 --> 00:12:35.430
but there was no solution.

00:12:38.950 --> 00:12:46.080
So we said, well, let's find
at least an x-star that

00:12:46.080 --> 00:12:53.360
minimizes b, that minimizes
the distance

00:12:53.360 --> 00:12:56.000
between b and Ax-star.

00:12:56.000 --> 00:12:58.680
And we call this the least
squares solution.

00:12:58.680 --> 00:13:00.730
We call it the least squares
solution because, when you

00:13:00.730 --> 00:13:03.320
actually take the length, or
when you're minimizing the

00:13:03.320 --> 00:13:06.200
length, you're minimizing the
squares of the differences

00:13:06.200 --> 00:13:06.890
right there.

00:13:06.890 --> 00:13:09.240
So it's the least squares
solution.

00:13:09.240 --> 00:13:13.310
Now, to find this, we know
that this has to be the

00:13:13.310 --> 00:13:16.710
closest vector in our
subspace to b.

00:13:16.710 --> 00:13:19.840
And we know that the closest
vector in our subspace to b is

00:13:19.840 --> 00:13:25.040
the projection of b onto our
subspace, onto our column

00:13:25.040 --> 00:13:26.720
space of A.

00:13:26.720 --> 00:13:31.550
And so, we know that A--
let me switch colors.

00:13:31.550 --> 00:13:37.070
We know that A times our least
squares solution should be

00:13:37.070 --> 00:13:45.250
equal to the projection of b
onto the column space of A.

00:13:45.250 --> 00:13:49.030
If we can find some x in Rk that
satisfies this, that is

00:13:49.030 --> 00:13:50.665
our least squares solution.

00:13:50.665 --> 00:13:53.820
But we've seen before that
the projection b is

00:13:53.820 --> 00:13:55.090
easier said than done.

00:13:55.090 --> 00:13:56.660
You know, there's a
lot of work to it.

00:13:56.660 --> 00:13:58.390
So maybe we can do
it a simpler way.

00:13:58.390 --> 00:14:01.020
And this is our simpler way.

00:14:01.020 --> 00:14:04.270
If we're looking for this,
alternately, we can just find

00:14:04.270 --> 00:14:05.710
a solution to this equation.

00:14:05.710 --> 00:14:09.270
So you give me an Ax equal to
b, there is no solution.

00:14:09.270 --> 00:14:11.070
Well, what I'm going to do is
I'm just going to multiply

00:14:11.070 --> 00:14:14.230
both sides of this equation
times A transpose.

00:14:14.230 --> 00:14:17.440
If I multiply both sides of this
equation by A transpose,

00:14:17.440 --> 00:14:27.070
I get A transpose times Ax is
equal to A transpose-- and I

00:14:27.070 --> 00:14:30.790
want to do that in the same
blue-- A-- no, that's not the

00:14:30.790 --> 00:14:37.340
same blue-- A transpose b.

00:14:37.340 --> 00:14:39.330
All I did is I multiplied
both sides of this.

00:14:39.330 --> 00:14:45.470
Now, the solution to this
equation will not be the same

00:14:45.470 --> 00:14:47.750
as the solution to
this equation.

00:14:47.750 --> 00:14:51.380
This right here will always
have a solution, and this

00:14:51.380 --> 00:14:56.190
right here is our least
squares solution.

00:14:56.190 --> 00:14:59.880
So this right here is our
least squares solution.

00:14:59.880 --> 00:15:03.630
And notice, this is some matrix,
and then this right

00:15:03.630 --> 00:15:06.760
here is some vector.

00:15:06.760 --> 00:15:08.410
This right here is
some vector.

00:15:08.410 --> 00:15:11.220
So long as we can find a
solution here, we've given our

00:15:11.220 --> 00:15:15.450
best shot at finding a solution
to Ax equal to b.

00:15:15.450 --> 00:15:16.990
We've minimized the error.

00:15:16.990 --> 00:15:19.440
We're going to get Ax-star,
and the difference between

00:15:19.440 --> 00:15:21.420
Ax-star and b is going
to be minimized.

00:15:21.420 --> 00:15:23.590
It's going to be our least
squares solution.

00:15:23.590 --> 00:15:25.870
It's all a little bit abstract
right now in this video, but

00:15:25.870 --> 00:15:28.780
hopefully, in the next video,
we'll realize that it's

00:15:28.780 --> 00:15:31.690
actually a very, very
useful concept.


Projection is closest vector in subspace

WEBVTT
Kind: captions
Language: en

00:00:00.000 --> 00:00:03.360
Let's say I've got
some subspace, v,

00:00:03.360 --> 00:00:04.590
that's a plane in R3.

00:00:04.590 --> 00:00:08.050
So let me see if I can draw
it respectably well.

00:00:08.050 --> 00:00:11.130
Let me draw some plane in R3.

00:00:11.130 --> 00:00:14.950
Maybe it looks something
like that.

00:00:14.950 --> 00:00:17.256
That is my subspace.

00:00:17.256 --> 00:00:18.340
I think that's good enough.

00:00:18.340 --> 00:00:20.580
Let me see if I can draw it a
little bit better than that.

00:00:20.580 --> 00:00:21.710
There you go.

00:00:21.710 --> 00:00:24.240
So that is my plane in R3.

00:00:24.240 --> 00:00:26.280
This is v.

00:00:26.280 --> 00:00:27.530
This is a subspace.

00:00:30.230 --> 00:00:33.550
And let's say that I have
some other vector, x,

00:00:33.550 --> 00:00:34.845
any vector in R3.

00:00:34.845 --> 00:00:37.970
So my vector x looks
like this.

00:00:37.970 --> 00:00:40.570
That is my vector x.

00:00:40.570 --> 00:00:43.430
Now what I want to show you
in this video is that the

00:00:43.430 --> 00:00:46.770
projection of x onto our
subspace-- and let's say that

00:00:46.770 --> 00:00:49.050
this is our 0 vector
right there.

00:00:49.050 --> 00:00:51.880
I want to show you that the
projection of x onto our

00:00:51.880 --> 00:00:57.240
subspace is the closest vector
in our subspace to x.

00:00:57.240 --> 00:00:59.840
So let me draw that out,
and maybe it'll make

00:00:59.840 --> 00:01:00.790
a little more sense.

00:01:00.790 --> 00:01:03.150
So the projection of x on to
the subspace will look

00:01:03.150 --> 00:01:04.610
something like this.

00:01:04.610 --> 00:01:07.220
It will look something
like that.

00:01:07.220 --> 00:01:10.320
That right there, that green
vector right there, is the

00:01:10.320 --> 00:01:17.010
projection of the vector
x onto our subspace v.

00:01:17.010 --> 00:01:18.200
That's our vector x.

00:01:18.200 --> 00:01:23.600
Now, let's take some arbitrary
other vector in our subspace.

00:01:23.600 --> 00:01:26.800
Let's just take this one.

00:01:26.800 --> 00:01:28.690
This is just some
other arbitrary

00:01:28.690 --> 00:01:30.390
vector in our subspace.

00:01:30.390 --> 00:01:32.250
Let me draw a little
bit differently.

00:01:32.250 --> 00:01:35.280
We draw it like that.

00:01:35.280 --> 00:01:38.120
Let's call that vector v.

00:01:38.120 --> 00:01:40.400
That's clearly another vector
on our subspace.

00:01:40.400 --> 00:01:41.970
It lies on that plane.

00:01:41.970 --> 00:01:48.490
What I want to show you is that
the distance from x to

00:01:48.490 --> 00:01:51.875
our projection of x on to v is
shorter than the distance from

00:01:51.875 --> 00:01:54.270
x to any other vector.

00:01:54.270 --> 00:01:56.290
From the distance from x
to any other vector.

00:01:56.290 --> 00:01:58.480
And obviously, the way I've
drawn it, it looks pretty

00:01:58.480 --> 00:02:00.810
clear that this line is shorter
than that line.

00:02:00.810 --> 00:02:03.430
But that was just a particular
choice that I picked.

00:02:03.430 --> 00:02:04.830
Let's prove it that's general.

00:02:04.830 --> 00:02:08.139
So what I want to prove is that
the distance between x

00:02:08.139 --> 00:02:11.560
and its projection onto the
subspace-- and the way we can

00:02:11.560 --> 00:02:15.750
get that is essentially just
take the length of the vector

00:02:15.750 --> 00:02:23.920
of x minus the projection
of x onto my subspace.

00:02:23.920 --> 00:02:28.220
This length right here is this
length right here, is this

00:02:28.220 --> 00:02:31.260
length right here.

00:02:31.260 --> 00:02:37.180
So x minus the projection of x
onto v, that's going to be

00:02:37.180 --> 00:02:39.690
this vector right there.

00:02:39.690 --> 00:02:41.460
Let me do that in a
different color.

00:02:41.460 --> 00:02:44.230
Don't want to reuse
colors too often.

00:02:44.230 --> 00:02:46.430
That's going to be that
vector right there.

00:02:46.430 --> 00:02:50.200
We could call that vector a.

00:02:50.200 --> 00:02:53.190
It's clearly in the orthogonal
complement of v, because it's

00:02:53.190 --> 00:02:54.150
orthogonal to this guy.

00:02:54.150 --> 00:02:56.433
And that's the definition of a
projection, actually, so this

00:02:56.433 --> 00:02:59.760
is equal to a.

00:02:59.760 --> 00:03:04.020
My claim, what I want to show
you, is that this distance a

00:03:04.020 --> 00:03:13.610
is shorter than any distance
here, is less than or equal to

00:03:13.610 --> 00:03:17.800
the distance between x and
v, where v is any member.

00:03:17.800 --> 00:03:19.270
So that's this distance,
right here.

00:03:19.270 --> 00:03:22.960
This vector right here, that
distance right-- let me draw

00:03:22.960 --> 00:03:23.936
this vector.

00:03:23.936 --> 00:03:27.270
The vector x minus v
looks like this.

00:03:27.270 --> 00:03:29.350
It looks like that.

00:03:29.350 --> 00:03:32.850
That is the vector
x minus v, right?

00:03:32.850 --> 00:03:37.330
If you take v plus x minus v,
you're going to go to x.

00:03:37.330 --> 00:03:40.730
So what I want to show is that
this distance, the length of

00:03:40.730 --> 00:03:44.810
a, of the difference between x
and its projection, is always

00:03:44.810 --> 00:03:48.880
going to be less than the
distance between x and any

00:03:48.880 --> 00:03:51.000
other vector in the subspace.

00:03:51.000 --> 00:03:54.450
So that is x minus v.

00:03:54.450 --> 00:03:56.830
So let's see if we
can prove that.

00:03:56.830 --> 00:04:00.260
So let's take the square
of this distance.

00:04:00.260 --> 00:04:05.230
So the square of x-- actually,
let me do that.

00:04:05.230 --> 00:04:08.180
Yeah, let me write that way.

00:04:08.180 --> 00:04:10.420
So we want to concern ourselves
with the square of

00:04:10.420 --> 00:04:16.320
the distance of x minus v, where
x is some vector in R3,

00:04:16.320 --> 00:04:18.970
and v is some vector
in R3 that's also a

00:04:18.970 --> 00:04:19.959
member of our subspace.

00:04:19.959 --> 00:04:20.959
It sits on this plane.

00:04:20.959 --> 00:04:23.650
So what's the square of
this going to be?

00:04:23.650 --> 00:04:29.250
Well, x minus v is equal
to this vector.

00:04:29.250 --> 00:04:31.430
Let me draw a new vector here.

00:04:31.430 --> 00:04:34.780
It is equal to this--
wait, let me draw

00:04:34.780 --> 00:04:36.130
it in in this yellow.

00:04:36.130 --> 00:04:39.480
It's equal to this vector.

00:04:39.480 --> 00:04:43.600
It's equal to this yellow
vector plus a.

00:04:43.600 --> 00:04:47.200
Right? x minus v is-- this
magenta vector that starts

00:04:47.200 --> 00:04:50.680
here and goes there-- clearly
equal to this yellow vector

00:04:50.680 --> 00:04:52.040
plus this orange vector.

00:04:52.040 --> 00:04:55.570
So let me call that
yellow vector b.

00:04:55.570 --> 00:04:57.430
Now, what is b equal to?

00:04:57.430 --> 00:05:07.120
Well, b is going to be equal
to this vector, this green

00:05:07.120 --> 00:05:15.540
vector, which is the projection
of x onto v, minus

00:05:15.540 --> 00:05:16.620
this purple vector.

00:05:16.620 --> 00:05:18.480
Minus this mauve vector,
I guess.

00:05:18.480 --> 00:05:19.340
Minus v.

00:05:19.340 --> 00:05:20.840
That's what b is.

00:05:20.840 --> 00:05:25.630
So we could write x minus v as
being equal to the sum of the

00:05:25.630 --> 00:05:33.405
vector b plus the vector a.

00:05:36.356 --> 00:05:39.830
So x minus v is equal
to b plus a.

00:05:39.830 --> 00:05:43.450
And if we're taking the length
of x minus v squared, that's

00:05:43.450 --> 00:05:49.500
the same thing as the length
of b plus a squared.

00:05:49.500 --> 00:05:57.460
And that's just equal to b plus
a dotted with b plus a,

00:05:57.460 --> 00:06:03.660
which is the same thing
as b dot b.

00:06:03.660 --> 00:06:05.190
Let me write that a
little bit neater.

00:06:05.190 --> 00:06:07.540
Let me write it down here.

00:06:07.540 --> 00:06:13.980
This is going to be equal to,
I'll switch colors, b dot b.

00:06:13.980 --> 00:06:17.270
Right, b dot b plus b
dot a plus a dot b.

00:06:17.270 --> 00:06:22.580
So plus 2 times a dot
b, plus a dot a.

00:06:25.470 --> 00:06:28.110
Now, a and b are clearly
orthogonal.

00:06:28.110 --> 00:06:32.040
b is the difference between two
vectors in our subspace.

00:06:32.040 --> 00:06:35.070
The subspace is closed under
addition and subraction.

00:06:35.070 --> 00:06:39.360
So b is a member of
our subspace.

00:06:39.360 --> 00:06:43.790
a is orthogonal to everything
in our subspace, by

00:06:43.790 --> 00:06:45.740
definition.

00:06:45.740 --> 00:06:51.040
So since a is clearly orthogonal
to b, a is-- by

00:06:51.040 --> 00:06:54.380
definition-- going to be in the
orthogonal compliment of

00:06:54.380 --> 00:06:55.210
the subspace.

00:06:55.210 --> 00:06:57.760
This is going to be 0.

00:06:57.760 --> 00:07:01.600
And then this right here
will simplify to

00:07:01.600 --> 00:07:04.010
the length of b squared.

00:07:04.010 --> 00:07:06.420
And then this right here
is going to be plus

00:07:06.420 --> 00:07:09.950
the length of a squared.

00:07:09.950 --> 00:07:14.540
So we get the distance between x
and some arbitrary member of

00:07:14.540 --> 00:07:20.860
our subspace squared is equal
to the length of b, right

00:07:20.860 --> 00:07:23.580
here, plus the length
of a squared.

00:07:23.580 --> 00:07:28.800
Now, a was a distance between
our vector x and our

00:07:28.800 --> 00:07:30.050
projection, right?

00:07:30.050 --> 00:07:33.040
That's what the definition of
a was. a was the distance

00:07:33.040 --> 00:07:35.100
between our vector x
and our projection.

00:07:35.100 --> 00:07:38.060
Now, this number right
here is going to be

00:07:38.060 --> 00:07:40.410
at least 0 or positive.

00:07:40.410 --> 00:07:42.440
So this right here is definitely
going to be greater

00:07:42.440 --> 00:07:45.710
than or equal to a squared.

00:07:45.710 --> 00:07:51.200
Or another way to say it is that
the distance between x

00:07:51.200 --> 00:07:54.950
minus v squared is definitely
going to be greater than or

00:07:54.950 --> 00:08:00.050
equal to the distance
of a squared.

00:08:00.050 --> 00:08:05.570
Or the distance between x minus
v-- this is still going

00:08:05.570 --> 00:08:07.910
to be a positive quantity,
length is always going to be

00:08:07.910 --> 00:08:10.950
positive-- is greater
than or equal to the

00:08:10.950 --> 00:08:12.680
length of vector a.

00:08:12.680 --> 00:08:15.620
Or what's that length of vector
a? a is just this thing

00:08:15.620 --> 00:08:16.070
right here.

00:08:16.070 --> 00:08:18.280
So let's write our result.

00:08:18.280 --> 00:08:24.170
The length of the vector x
minus v, or the distance

00:08:24.170 --> 00:08:28.440
between x and some arbitrary
member of our subspace, is

00:08:28.440 --> 00:08:31.240
always going to be greater than
or equal to the length of

00:08:31.240 --> 00:08:37.289
a, which is just the distance
between x and the projection

00:08:37.289 --> 00:08:40.730
of x onto our subspace.

00:08:40.730 --> 00:08:41.890
So there you have it.

00:08:41.890 --> 00:08:46.740
We've shown, and the original
graph kind of hinted at it,

00:08:46.740 --> 00:08:53.530
that the projection of x onto v
is the closest vector in our

00:08:53.530 --> 00:08:55.850
subspace to x.

00:08:55.850 --> 00:09:00.310
It's closer than any other
vector in v to our arbitrary

00:09:00.310 --> 00:09:01.855
vector in R3, x.

00:09:01.855 --> 00:09:04.300
And we've proven
it right there.


Another example of a projection matrix

WEBVTT
Kind: captions
Language: en

00:00:00.580 --> 00:00:05.980
Let's say I have a subspace v
that is equal to all of the

00:00:05.980 --> 00:00:15.325
vectors-- let me write it this
way-- all of the x1, x2, x3's,

00:00:15.325 --> 00:00:24.980
so all the vectors like this
that satisfy x1 plus x2 plus

00:00:24.980 --> 00:00:28.050
x3 is equal to 0.

00:00:28.050 --> 00:00:31.030
So if you think about it, this
is just a plane in R3, so this

00:00:31.030 --> 00:00:33.180
subspace is a plane in R3.

00:00:33.180 --> 00:00:37.810
And I'm interested in finding
the transformation matrix for

00:00:37.810 --> 00:00:44.350
the projection of any vector
x in R3 onto v.

00:00:44.350 --> 00:00:45.840
So how could we do that?

00:00:45.840 --> 00:00:47.780
So we could do it like we
did in the last video.

00:00:47.780 --> 00:00:52.280
We could find the basis for
this subspace right there.

00:00:52.280 --> 00:00:54.580
And that's not too hard to do.

00:00:54.580 --> 00:00:59.290
We could say x1, if we assume
that, let's say, that x2 and

00:00:59.290 --> 00:01:02.080
x3 are kind of free variables,
then we could say that x1 is

00:01:02.080 --> 00:01:07.550
equal to minus x2, minus x3.

00:01:07.550 --> 00:01:10.940
And then let's just, just so we
can write it in, kind of,

00:01:10.940 --> 00:01:15.130
our parametric form, or if we
can write our solution set as

00:01:15.130 --> 00:01:19.650
the combination of basis
vectors, we can say x2 is

00:01:19.650 --> 00:01:21.520
equal to, let's say
it's equal to some

00:01:21.520 --> 00:01:23.930
arbitrary constant, C2.

00:01:23.930 --> 00:01:26.360
And let's say that x3
is equal to some

00:01:26.360 --> 00:01:28.900
arbitrary constant, C3.

00:01:28.900 --> 00:01:35.860
Then we can say that v, we can
rewrite v, we could say that v

00:01:35.860 --> 00:01:41.220
is-- I'll do it here-- v is
equal to the set of all x1's,

00:01:41.220 --> 00:01:51.590
x2's, and x3's that are equal
to C2 times-- so x1 is equal

00:01:51.590 --> 00:01:56.320
to minus-- let me rewrite
this with a C2--

00:01:56.320 --> 00:01:58.230
this is equal to C2.

00:01:58.230 --> 00:02:00.190
This is equal to C3.

00:02:00.190 --> 00:02:05.050
So x1 is equal to minus
C2, minus C3.

00:02:05.050 --> 00:02:12.470
So x1 is equal to minus 1 times
C2, plus C3 times what?

00:02:12.470 --> 00:02:16.560
Plus C3 times minus 1.

00:02:16.560 --> 00:02:19.550
And then what is x2 equal to?

00:02:19.550 --> 00:02:20.470
x2 is just equal to C2.

00:02:20.470 --> 00:02:24.420
So it's 1 times C2,
plus 0, times C3.

00:02:24.420 --> 00:02:28.450
x3 is just equal to C3,
so it's 0 times C2,

00:02:28.450 --> 00:02:30.700
plus 1, times C3.

00:02:30.700 --> 00:02:34.410
And so this is another way
of defining our subspace.

00:02:34.410 --> 00:02:39.810
All of the vectors that satisfy
this is equal to this

00:02:39.810 --> 00:02:40.590
definition here.

00:02:40.590 --> 00:02:43.660
It's all the vectors whose
components satisfy, or that

00:02:43.660 --> 00:02:47.450
lie in this plane, whose entries
lie in that plane.

00:02:47.450 --> 00:02:50.420
And that's for any real
numbers right there.

00:02:50.420 --> 00:02:55.590
Or another way of writing this,
is v is equal to the

00:02:55.590 --> 00:03:05.230
span of the vectors minus 1, 1,
and 0, and the vector minus

00:03:05.230 --> 00:03:08.430
1, 0, and 1.

00:03:08.430 --> 00:03:09.100
Just like that.

00:03:09.100 --> 00:03:12.000
And we know that these are
actually a basis for v because

00:03:12.000 --> 00:03:13.480
they're linearly independent.

00:03:13.480 --> 00:03:16.640
There's no way I can take linear
combinations of this

00:03:16.640 --> 00:03:19.390
guy and make the second
entry equal a 1 here.

00:03:19.390 --> 00:03:21.670
And likewise there's no way I
can take linear combinations

00:03:21.670 --> 00:03:24.480
of this guy and make this third
entry equal a 1 here.

00:03:24.480 --> 00:03:27.390
So these are also
a basis for v.

00:03:27.390 --> 00:03:30.560
So given, that just using the
technique we did before, we

00:03:30.560 --> 00:03:36.120
could set some vector, we could
set some matrix A equal

00:03:36.120 --> 00:03:45.930
to minus 1, 1, 0, and then
minus 1, 0, and 1.

00:03:45.930 --> 00:03:52.280
And then we can figure out that
the projection of any

00:03:52.280 --> 00:03:59.440
vector x in our 3 onto v is
going to be equal to, and we

00:03:59.440 --> 00:04:04.640
saw this, it's going to be equal
to A times the inverse

00:04:04.640 --> 00:04:07.210
of A transpose A.

00:04:07.210 --> 00:04:12.190
All of that times A transpose
and all of that times x.

00:04:12.190 --> 00:04:13.750
And you can do it.

00:04:13.750 --> 00:04:14.500
You have A here.

00:04:14.500 --> 00:04:16.860
You can figure out what the
transpose of A is very easy.

00:04:16.860 --> 00:04:19.615
You can take A transpose A,
then you can invert it.

00:04:19.615 --> 00:04:22.500
And it'll be very similar to
what we did in the last video.

00:04:22.500 --> 00:04:24.530
It'll be a little less work,
because this is a 3 by 2

00:04:24.530 --> 00:04:26.720
matrix, instead of
a 4 by 2 matrix.

00:04:26.720 --> 00:04:29.260
But you saw it is actually
a lot of work.

00:04:29.260 --> 00:04:31.800
It's very hairy and you might
make some careless mistakes.

00:04:31.800 --> 00:04:38.620
So let's figure out if there's
another way that we can come

00:04:38.620 --> 00:04:41.060
up with this matrix
right here.

00:04:41.060 --> 00:04:50.280
Now we know that if x is a
member of R3, that x can be

00:04:50.280 --> 00:04:58.550
represented as a combination of
some vector v, that is in

00:04:58.550 --> 00:05:05.720
our subspace, plus some vector
w, that is in the orthogonal

00:05:05.720 --> 00:05:11.820
complement of the subspace,
where v is a member of our

00:05:11.820 --> 00:05:19.760
subspace, and w is a member of
the orthogonal complement of

00:05:19.760 --> 00:05:20.950
our subspace.

00:05:20.950 --> 00:05:26.290
Now by definition, that right
there is the projection of x

00:05:26.290 --> 00:05:31.030
onto v, and this is the
projection of x onto the

00:05:31.030 --> 00:05:32.510
orthogonal complement of v.

00:05:32.510 --> 00:05:42.070
So we can write that x is equal
to the projection onto v

00:05:42.070 --> 00:05:49.650
of x, plus the projection onto
v's orthogonal complement, or

00:05:49.650 --> 00:05:53.190
the orthogonal complement
of v of x.

00:05:53.190 --> 00:05:57.190
So this is by definition, that
any member of R3 can be

00:05:57.190 --> 00:05:59.000
represented this way.

00:05:59.000 --> 00:06:02.220
Now if we want to write this as
matrix vector products, and

00:06:02.220 --> 00:06:04.620
two videos ago I showed you
that these are linear

00:06:04.620 --> 00:06:05.346
transformations.

00:06:05.346 --> 00:06:06.940
So let me write that here.

00:06:06.940 --> 00:06:08.230
So they're linear
transformations.

00:06:08.230 --> 00:06:10.780
So they can be written as
matrix vector products.

00:06:10.780 --> 00:06:12.190
You see that right there.

00:06:12.190 --> 00:06:19.130
Let me define this matrix, I
don't know, let me call this

00:06:19.130 --> 00:06:25.060
matrix T, let me
just call it T.

00:06:25.060 --> 00:06:29.250
And let me do another.

00:06:29.250 --> 00:06:32.180
Let me do a letter,
let me do B.

00:06:32.180 --> 00:06:40.200
And let's say that the
projection onto the orthogonal

00:06:40.200 --> 00:06:44.270
complement of v of x, let's say
that that's equal to some

00:06:44.270 --> 00:06:50.125
other matrix C, times x.

00:06:50.125 --> 00:06:52.650
And we know this is a linear
transformation, so it can be

00:06:52.650 --> 00:06:55.720
represented as some
matrix C times x.

00:06:55.720 --> 00:06:58.060
So what are these going
to be equal to?

00:06:58.060 --> 00:07:01.610
Well x, if I want to write it as
a linear transformation of

00:07:01.610 --> 00:07:06.730
x, I could just write it as the
3 by 3 identity matrix,

00:07:06.730 --> 00:07:08.370
times x, right?

00:07:08.370 --> 00:07:09.880
That's the same thing as x.

00:07:09.880 --> 00:07:12.710
That's going to be equal to the
projection of x onto v,

00:07:12.710 --> 00:07:15.800
well that's just the same
thing as B times x.

00:07:21.010 --> 00:07:23.870
And then plus the projection
of x onto v's orthogonal

00:07:23.870 --> 00:07:26.640
complement, well that's
just C times x.

00:07:26.640 --> 00:07:29.050
Plus C times x.

00:07:29.050 --> 00:07:32.390
And if you want to factor out
the x on this side, we know

00:07:32.390 --> 00:07:34.830
that the matrix vector
products exhibit the

00:07:34.830 --> 00:07:38.260
distributive property, so we
could write that the identity

00:07:38.260 --> 00:07:47.180
matrix times x is equal
to B plus C, times x.

00:07:47.180 --> 00:07:51.000
Or another way to view this
equation is that this matrix

00:07:51.000 --> 00:07:54.740
must be equal to these
two matrices.

00:07:54.740 --> 00:08:02.530
So we get that the identity
matrix in R3 is equal to the

00:08:02.530 --> 00:08:09.100
projection matrix onto v, plus
the projection matrix onto v's

00:08:09.100 --> 00:08:11.580
orthogonal complement.

00:08:11.580 --> 00:08:13.740
Remember, the whole point of
this problem is to figure out

00:08:13.740 --> 00:08:15.800
this thing right here,
is to solve or B.

00:08:15.800 --> 00:08:17.520
And we know a technique
for doing it.

00:08:17.520 --> 00:08:20.520
You take A transpose, you can do
this whole thing, but that

00:08:20.520 --> 00:08:21.760
might be pretty hairy.

00:08:21.760 --> 00:08:24.520
But maybe it's easy
to find this guy.

00:08:24.520 --> 00:08:25.400
Maybe, I don't know.

00:08:25.400 --> 00:08:27.780
It actually turns out in the
video, this one will be easy.

00:08:27.780 --> 00:08:31.050
So if it's easy to find this
guy, we can just solve for B.

00:08:31.050 --> 00:08:36.200
If we subtract C from both
sides, we get that B is equal

00:08:36.200 --> 00:08:41.760
to I, is equal to the identity
matrix, minus the

00:08:41.760 --> 00:08:45.960
transformation matrix for the
transformation onto v's

00:08:45.960 --> 00:08:48.300
orthogonal complement.

00:08:48.300 --> 00:08:50.730
So let's see what this is.

00:08:50.730 --> 00:08:54.460
Let's see if we can figure out
what C is right there.

00:08:54.460 --> 00:08:57.860
So let's go back to
our original.

00:08:57.860 --> 00:09:01.200
So remember-- let me rewrite
the problem actually--

00:09:01.200 --> 00:09:09.940
remember that v was equal to,
essentially it's equal to all

00:09:09.940 --> 00:09:24.140
of the x1's, x2's, x3's that
satisfy x1 plus x2, plus x3 is

00:09:24.140 --> 00:09:25.960
equal to 0.

00:09:25.960 --> 00:09:28.500
Or another way to say it is that
all the x1's, x2's, and

00:09:28.500 --> 00:09:39.800
x3's that satisfy the equation
1, 1, 1, times x1, x2, x3 is

00:09:39.800 --> 00:09:41.900
equal to the 0 vector.

00:09:41.900 --> 00:09:43.610
Or this case it'll just be 0.

00:09:43.610 --> 00:09:46.440
We could write the 0 vector
just like that.

00:09:46.440 --> 00:09:50.180
So 1 times x1, plus 1 times x2,
plus 1 times x3 is going

00:09:50.180 --> 00:09:51.350
to equal the 0 vector.

00:09:51.350 --> 00:09:53.350
This is another way
to write v.

00:09:53.350 --> 00:09:56.670
Now all of the x's that
satisfied this right here,

00:09:56.670 --> 00:09:58.110
what is that?

00:09:58.110 --> 00:10:02.170
This is saying that v is equal
to the null space of this

00:10:02.170 --> 00:10:03.920
matrix right there.

00:10:03.920 --> 00:10:06.550
The null space of this matrix
is all of the vectors that

00:10:06.550 --> 00:10:08.330
satisfy this equation.

00:10:08.330 --> 00:10:14.370
So v is equal to the null
space-- let me write it this

00:10:14.370 --> 00:10:19.440
way-- the null space of 1,
1, 1, just like that.

00:10:19.440 --> 00:10:22.050
Up here we, kind of, figured
out v in kind of the

00:10:22.050 --> 00:10:22.710
traditional way.

00:10:22.710 --> 00:10:25.730
We figured out that v is the
span of these things, but now

00:10:25.730 --> 00:10:30.600
we know that's the same thing is
the null space of 1, 1, 1.

00:10:30.600 --> 00:10:32.890
These two statements
are equivalent.

00:10:32.890 --> 00:10:37.080
Now we at least had a hunch that
maybe, you know, we could

00:10:37.080 --> 00:10:41.070
figure out, straight up, this B
here by doing all of this A

00:10:41.070 --> 00:10:43.120
transpose and, you know,
by doing all of

00:10:43.120 --> 00:10:43.970
this silliness here.

00:10:43.970 --> 00:10:47.040
But our hunch is maybe if
we could figure out the

00:10:47.040 --> 00:10:53.650
transformation matrix for the
orthogonal complement of v

00:10:53.650 --> 00:10:56.560
right there, that then we could
just apply this, kind

00:10:56.560 --> 00:10:59.600
of, that we can just solve for
B given that the identity

00:10:59.600 --> 00:11:02.840
matrix minus this guy is
going to be equal to B.

00:11:02.840 --> 00:11:08.590
So let's see if we can figure
out the projection matrix, if

00:11:08.590 --> 00:11:11.420
we can figure out the
transformation matrix for the

00:11:11.420 --> 00:11:14.260
orthogonal projection, for
x onto the orthogonal

00:11:14.260 --> 00:11:15.820
projection of v.

00:11:15.820 --> 00:11:17.380
So this is v.

00:11:17.380 --> 00:11:21.550
What is v compliment?

00:11:21.550 --> 00:11:26.650
v compliment is going to be
equal to the orthogonal

00:11:26.650 --> 00:11:29.680
complement, or v perp is going
to be equal to the orthogonal

00:11:29.680 --> 00:11:35.340
complement of the null space
of this matrix right here.

00:11:35.340 --> 00:11:36.210
Which is equal to what?

00:11:36.210 --> 00:11:41.180
Remember, the null space, its
orthogonal complement-- a null

00:11:41.180 --> 00:11:49.220
space's orthogonal complement is
equivalent to the row space

00:11:49.220 --> 00:11:52.620
or the column space
of A transpose.

00:11:52.620 --> 00:11:54.650
We saw that multiple times.

00:11:54.650 --> 00:11:58.690
Or you could say the orthogonal
complement of the

00:11:58.690 --> 00:12:00.990
row space is the null space.

00:12:00.990 --> 00:12:03.120
We've seen this many,
many times before.

00:12:03.120 --> 00:12:07.790
So the orthogonal complement of
this guy is going to be the

00:12:07.790 --> 00:12:10.880
column space of his transpose.

00:12:10.880 --> 00:12:15.305
So the column space of the
transpose of this guy.

00:12:15.305 --> 00:12:21.560
So it's 1, 1, 1,
just like that.

00:12:21.560 --> 00:12:27.570
Or we can write that v's
orthogonal complement is equal

00:12:27.570 --> 00:12:32.490
to the span of 1, 1, 1.

00:12:32.490 --> 00:12:35.320
The column space of this matrix,
we only have one

00:12:35.320 --> 00:12:38.540
column in it, so its column
space is going to be the span

00:12:38.540 --> 00:12:39.860
of that one column.

00:12:39.860 --> 00:12:43.340
So just to visualize what
we're doing here, that

00:12:43.340 --> 00:12:46.780
original equation for v, that
satisfies that, that's just

00:12:46.780 --> 00:12:48.030
going to be some plane in R3.

00:12:51.660 --> 00:12:53.570
That is v right there.

00:12:53.570 --> 00:12:55.100
And now we just figured
out what v's

00:12:55.100 --> 00:12:56.610
orthogonal complement is.

00:12:56.610 --> 00:12:58.050
It's going to be a line in R3.

00:12:58.050 --> 00:12:59.270
It's going to be all
of the linear

00:12:59.270 --> 00:13:01.880
combinations of this guy.

00:13:01.880 --> 00:13:03.640
So it's going to be
some line in R3.

00:13:03.640 --> 00:13:04.900
I haven't drawn it.

00:13:04.900 --> 00:13:06.960
You know this is going to be
tilted more, and so is this,

00:13:06.960 --> 00:13:08.000
but it's going to
be some line.

00:13:08.000 --> 00:13:10.840
So this is the orthogonal
complement of v.

00:13:10.840 --> 00:13:12.840
So let's see if we
can figure out.

00:13:12.840 --> 00:13:18.820
So remember, the projection--
let me do it this way.

00:13:18.820 --> 00:13:22.920
So this is the basis for v's
orthogonal complement.

00:13:22.920 --> 00:13:27.200
So let's construct
some matrix.

00:13:27.200 --> 00:13:28.760
I don't know, let me
use a letter that I

00:13:28.760 --> 00:13:30.020
haven't used before.

00:13:30.020 --> 00:13:34.250
Let me construct some matrix D,
whose columns are the basis

00:13:34.250 --> 00:13:37.710
vectors for the orthogonal
complement of v.

00:13:37.710 --> 00:13:40.580
Well, there's only one
basis vector, so

00:13:40.580 --> 00:13:42.730
it's going to be that.

00:13:42.730 --> 00:13:45.930
And we learned, in the last
video and the video before

00:13:45.930 --> 00:13:53.550
that, that the projection of
any vector in R3 onto v's

00:13:53.550 --> 00:14:00.580
orthogonal complement is going
to be equal to D times D

00:14:00.580 --> 00:14:09.520
transpose D inverse, times
D transpose, times x.

00:14:09.520 --> 00:14:13.230
Or another way to view it is
that this thing right here,

00:14:13.230 --> 00:14:17.590
that thing right there is the
transformation matrix for this

00:14:17.590 --> 00:14:18.470
projection.

00:14:18.470 --> 00:14:20.455
That is the transformation
matrix.

00:14:23.940 --> 00:14:25.020
matrix

00:14:25.020 --> 00:14:29.620
So let's see if this is easier
to solve this thing than this

00:14:29.620 --> 00:14:32.190
business up here, where we
had a 3 by 2 matrix.

00:14:32.190 --> 00:14:34.490
That was the whole motivation
for doing this problem.

00:14:34.490 --> 00:14:38.180
To figure out the projection
matrix for v's subspace, we'd

00:14:38.180 --> 00:14:40.150
have to do this with
the 3 by 2 matrix.

00:14:40.150 --> 00:14:41.100
It seems pretty difficult.

00:14:41.100 --> 00:14:44.720
Instead, let's find the
projection matrix to get to

00:14:44.720 --> 00:14:46.780
the production onto
v's orthogonal

00:14:46.780 --> 00:14:48.770
complement, which is this.

00:14:48.770 --> 00:14:51.050
So what is D transpose?

00:14:51.050 --> 00:14:55.910
So D transpose is just going
to be equal to 1, 1, 1.

00:14:55.910 --> 00:14:58.230
What is D transpose times D?

00:14:58.230 --> 00:15:00.770
Well, that's D transpose.

00:15:00.770 --> 00:15:03.220
This is D, just like that.

00:15:03.220 --> 00:15:04.540
So what is this going
to be equal to?

00:15:04.540 --> 00:15:06.460
This is just the dot product
of that and that.

00:15:06.460 --> 00:15:13.210
1 times 1, plus 1 times 1, plus
1 times 1, it equals 3.

00:15:13.210 --> 00:15:17.690
So this thing right here is
equal to a 1 by 1 matrix 3.

00:15:17.690 --> 00:15:18.540
So let's write it down.

00:15:18.540 --> 00:15:26.700
So this is equal to D-- which
is this matrix, 1, 1, 1--

00:15:26.700 --> 00:15:29.770
times D transpose D inverse.

00:15:29.770 --> 00:15:32.670
So D transpose D is just
a 1 by 1 matrix.

00:15:32.670 --> 00:15:34.010
And we're going to have
to invert it.

00:15:34.010 --> 00:15:36.750
Actually, I've never defined the
inverse of a 1 by 1 matrix

00:15:36.750 --> 00:15:39.510
for you just now, so it's
just mildly exciting.

00:15:39.510 --> 00:15:41.420
Times D transpose.

00:15:41.420 --> 00:15:45.280
So D transpose looks
like this, 1, 1, 1.

00:15:45.280 --> 00:15:46.940
And then all of that's
times x.

00:15:46.940 --> 00:15:49.920
But this is the transformation
matrix right there.

00:15:49.920 --> 00:15:53.490
Now what is the inverse
of a 1 by 1 matrix?

00:15:53.490 --> 00:15:58.930
Now you just have to remember
that A inverse times A is

00:15:58.930 --> 00:16:00.730
equal to the identity matrix.

00:16:00.730 --> 00:16:04.850
If we're dealing with a 1 by 1
matrix, then I'm just trying

00:16:04.850 --> 00:16:11.030
to figure out what, let's say,
what matrix times 3 is going

00:16:11.030 --> 00:16:13.880
to be equal to the 1 by
1 identity matrix.

00:16:13.880 --> 00:16:26.500
So let's say that 3 inverse
times 3 has to be equal to the

00:16:26.500 --> 00:16:27.600
identity matrix.

00:16:27.600 --> 00:16:29.710
1 by 1 identity matrix.

00:16:29.710 --> 00:16:32.600
Well, the only matrix that's
going to make this work out,

00:16:32.600 --> 00:16:35.040
to get this entry I'll just take
this guy's entry times

00:16:35.040 --> 00:16:37.880
that guy's entry, is going to
be this guy right here.

00:16:37.880 --> 00:16:40.500
The inverse of this 1
by 1 matrix has to

00:16:40.500 --> 00:16:42.920
be the matrix 1/3.

00:16:42.920 --> 00:16:46.200
1/3 times 3 is equal to 1.

00:16:46.200 --> 00:16:48.700
This is almost trivially
simple, but this is the

00:16:48.700 --> 00:16:53.270
inverse, that right there is the
inverse matrix, for the 1

00:16:53.270 --> 00:16:55.020
by 1 matrix 3.

00:16:55.020 --> 00:16:57.790
So this right here
is just 1/3.

00:16:57.790 --> 00:16:59.360
And we could actually
just take that out.

00:16:59.360 --> 00:17:01.660
It's a 1 by 1 matrix,
which is essentially

00:17:01.660 --> 00:17:03.430
equivalent to a scalar.

00:17:03.430 --> 00:17:08.599
So this is going to be equal--
let me just draw a line here--

00:17:08.599 --> 00:17:13.547
this thing is equal to 1/3--
actually, I don't want to

00:17:13.547 --> 00:17:13.930
confuse you.

00:17:13.930 --> 00:17:15.050
Let me rewrite it.

00:17:15.050 --> 00:17:21.410
So we get the projection of
any vector in R3 onto the

00:17:21.410 --> 00:17:28.510
orthogonal complement of v, is
equal to 1/3, that's 1/3,

00:17:28.510 --> 00:17:33.330
times the vector 1, 1, 1,
times-- sorry, or wait, that

00:17:33.330 --> 00:17:36.120
is a vector or the matrix 1
on 1-- times that matrix

00:17:36.120 --> 00:17:38.880
transposed, 1, 1, 1.

00:17:38.880 --> 00:17:40.860
And then all of that times x.

00:17:40.860 --> 00:17:44.580
And you can see, this is a lot
simpler than if we have to do

00:17:44.580 --> 00:17:48.040
all of this business
with this matrix.

00:17:48.040 --> 00:17:50.160
That's a harder matrix
to deal with.

00:17:50.160 --> 00:17:51.910
This 1, 1, 1 matrix
is very easy.

00:17:51.910 --> 00:17:55.840
Now what is this going
to be equal to?

00:17:55.840 --> 00:18:00.930
This is going to be equal to
1/3 times, we have a 3 by 1

00:18:00.930 --> 00:18:03.510
times a 1 by 3 matrix,
so it's going to

00:18:03.510 --> 00:18:05.195
result in a 3 by 3 matrix.

00:18:07.880 --> 00:18:08.560
And what do we get?

00:18:08.560 --> 00:18:12.980
So this first entry is going to
be 1 times 1, which is 1.

00:18:12.980 --> 00:18:14.970
The second entry is going to
be 1 times 1, which is 1.

00:18:14.970 --> 00:18:16.840
The third entry is going to
be 1 times 1, which is 1.

00:18:16.840 --> 00:18:19.450
I think you see the pattern.

00:18:19.450 --> 00:18:22.430
The second grow, first column,
1 times 1, which is 1.

00:18:22.430 --> 00:18:26.480
So this is going to be a
3 by 3 matrix of 1's.

00:18:26.480 --> 00:18:29.060
So just like that we were able
to get-- that was a pretty

00:18:29.060 --> 00:18:32.780
straightforward situation--
we were able to get the

00:18:32.780 --> 00:18:38.170
projection matrix for any
vector in R3 onto v's

00:18:38.170 --> 00:18:39.820
orthogonal complement.

00:18:39.820 --> 00:18:49.270
Now, we know that this thing
right here is our original C

00:18:49.270 --> 00:18:50.510
that we said.

00:18:50.510 --> 00:18:55.760
And we said that the identity
matrix-- we wrote it up here.

00:18:55.760 --> 00:18:58.760
Let me refer back to what
I wrote way up here.

00:18:58.760 --> 00:19:02.410
We said, look, the identity
matrix is equal to the

00:19:02.410 --> 00:19:07.590
transformation matrix for the
projection onto v, plus the

00:19:07.590 --> 00:19:10.790
transformation matrix for the
projection onto v's orthogonal

00:19:10.790 --> 00:19:11.780
complement.

00:19:11.780 --> 00:19:15.490
Or we can write that the
transformation matrix for the

00:19:15.490 --> 00:19:20.540
projection onto v is equal to
the identity matrix minus the

00:19:20.540 --> 00:19:23.990
transformation matrix for the
projection onto v's orthogonal

00:19:23.990 --> 00:19:25.240
complement.

00:19:30.050 --> 00:19:35.430
So if we say that the projection
onto v of x is

00:19:35.430 --> 00:19:40.780
equal to B times x, we know that
B is equal to the 3 by 3

00:19:40.780 --> 00:19:44.980
identity matrix, minus C, and
this is C right there.

00:19:44.980 --> 00:19:52.560
So B is equal to the identity
matrix-- so that's just 1, 0,

00:19:52.560 --> 00:20:04.160
0, 0, 1, 0, 0, 0, 1-- minus C,
minus 1/3, times 1, 1, 1, 1,

00:20:04.160 --> 00:20:06.850
1, 1, 1, 1, 1, just like that.

00:20:06.850 --> 00:20:08.405
And what is this going
to be equal to?

00:20:11.650 --> 00:20:15.720
Let's see, let's, in our heads,
multiply this out.

00:20:15.720 --> 00:20:18.040
All of these entries are going
to be 1/3 essentially, if we

00:20:18.040 --> 00:20:20.240
multiply this out like that.

00:20:20.240 --> 00:20:24.200
So if we have 1 minus 1/3.

00:20:24.200 --> 00:20:25.155
I could write it
out like that.

00:20:25.155 --> 00:20:28.440
It's 1/3, 1/3, 1/3.

00:20:28.440 --> 00:20:29.530
Everything is 1/3.

00:20:29.530 --> 00:20:36.740
1/3, 1/3, 1/3, 1/3, 1/3, 1/3,
and this just becomes a 1.

00:20:36.740 --> 00:20:40.810
So 1 minus 1/3 is 2/3.

00:20:40.810 --> 00:20:43.290
And all of the 1's minus 1/3
are going to be 2/3, so we

00:20:43.290 --> 00:20:45.910
could just go down
the diagonal.

00:20:45.910 --> 00:20:49.880
And then the 0's minus 1/3 are
going to be minus 1/3.

00:20:49.880 --> 00:20:53.750
Minus 1/3, minus
1/3, minus 1/3.

00:20:53.750 --> 00:20:58.090
You have minus 1/3, minus
1/3, and minus 1/3.

00:20:58.090 --> 00:21:01.740
And just like that, we've been
able to figure out our

00:21:01.740 --> 00:21:05.450
projection, our transformation
matrix, for the projection of

00:21:05.450 --> 00:21:09.390
any vector x onto v, by
essentially finding this guy

00:21:09.390 --> 00:21:12.010
first, for finding the
transformation matrix for the

00:21:12.010 --> 00:21:15.760
projection of any x onto v's
orthogonal complement.

00:21:15.760 --> 00:21:17.870
Anyway, I thought that
was pretty neat.

00:21:17.870 --> 00:21:25.850
And you could rewrite this as v
equal to 1/3 times 2, 2, 2,

00:21:25.850 --> 00:21:28.470
2's along back the diagonals
and then you have minus 1's

00:21:28.470 --> 00:21:29.720
everywhere else.

00:21:33.040 --> 00:21:35.600
Anyway, see you in
the next video.


Subspace projection matrix example

WEBVTT
Kind: captions
Language: en

00:00:00.750 --> 00:00:04.090
Let's say I've got some subspace
V, which tends to be

00:00:04.090 --> 00:00:07.750
our favorite letter for
subspaces, and it's equal to

00:00:07.750 --> 00:00:11.170
the span of two vectors in R4.

00:00:11.170 --> 00:00:17.840
Let's say that the first vector
is 1 0 0 1, and the

00:00:17.840 --> 00:00:26.350
second vector is 0 1 0 1.

00:00:26.350 --> 00:00:28.210
That is my subspace V.

00:00:28.210 --> 00:00:30.220
And you can see that these
are going to be a basis.

00:00:30.220 --> 00:00:32.810
That these are linearly
independent.

00:00:32.810 --> 00:00:36.150
Two vectors that are linear-- or
any set of vectors that are

00:00:36.150 --> 00:00:39.380
linearly independent and that
span a subspace are a basis

00:00:39.380 --> 00:00:39.960
for that subspace.

00:00:39.960 --> 00:00:42.020
You can see they are linearly
independent.

00:00:42.020 --> 00:00:43.040
This guy's has a 1 here.

00:00:43.040 --> 00:00:45.250
There's no way you can take some
combination of this guy

00:00:45.250 --> 00:00:46.820
to somehow get a 1 there.

00:00:46.820 --> 00:00:47.960
And this guy has a 1 here.

00:00:47.960 --> 00:00:49.730
There's no way you can get some
linear combination of

00:00:49.730 --> 00:00:51.440
these zeroes here
to a 1 there, so

00:00:51.440 --> 00:00:54.050
they're linearly dependent.

00:00:54.050 --> 00:01:00.500
You can also call this
a basis for V.

00:01:00.500 --> 00:01:03.920
Now, given that, let's see
if we can find out the

00:01:03.920 --> 00:01:08.730
transformation matrix for the
projection of any arbitrary

00:01:08.730 --> 00:01:11.360
vector onto this subspace.

00:01:11.360 --> 00:01:14.440
So let's say that X-- we're
dealing in R4 here, right?

00:01:14.440 --> 00:01:21.030
Let's say that x is a member
of R4, and I want to figure

00:01:21.030 --> 00:01:24.170
out a transformation
matrix for the

00:01:24.170 --> 00:01:29.320
projection onto V of x.

00:01:29.320 --> 00:01:34.040
Now, in the last video, we came
up with a general way to

00:01:34.040 --> 00:01:34.850
figure this out.

00:01:34.850 --> 00:01:38.400
We said if A is a transformation
matrix-- sorry.

00:01:38.400 --> 00:01:43.900
If A is a matrix who's columns
are the basis for the

00:01:43.900 --> 00:01:52.350
subspace, so let's say A is
equal to 1 0 0 1, 0 1 0 1.

00:01:52.350 --> 00:01:55.197
So A is a matrix whose columns
are the basis for our

00:01:55.197 --> 00:02:00.130
subspace, then the projection
of x onto V would be equal

00:02:00.130 --> 00:02:02.350
to-- and this is kind of hard.

00:02:02.350 --> 00:02:04.600
The first time you look at, it
gives you a headache, but

00:02:04.600 --> 00:02:08.030
there's a certain pattern or
symmetry or a way of-- you

00:02:08.030 --> 00:02:11.140
could say it's A times, you're
gonna have something in the

00:02:11.140 --> 00:02:17.790
middle, and then you have A
transpose times your vector x.

00:02:17.790 --> 00:02:20.100
And the way I remember it is in
the middle, you have these

00:02:20.100 --> 00:02:21.760
two guys switched around.

00:02:21.760 --> 00:02:25.790
So then you have A transpose
A, and you take

00:02:25.790 --> 00:02:27.040
the inverse of it.

00:02:29.820 --> 00:02:31.940
You probably won't be using
this in your everyday life

00:02:31.940 --> 00:02:34.500
five or ten years from now, so
it's OK if you don't memorize

00:02:34.500 --> 00:02:37.390
it, but temporarily, put this
in your medium-term memory

00:02:37.390 --> 00:02:40.110
because it's a good thing
to know for doing these

00:02:40.110 --> 00:02:41.490
projection problems.

00:02:41.490 --> 00:02:45.640
So if we want to find the
general matrix for this

00:02:45.640 --> 00:02:48.790
transformation, we just have to
determine what this matrix

00:02:48.790 --> 00:02:53.000
is equal to, and that's just a
bunch of matrix operations.

00:02:53.000 --> 00:02:53.980
So that's A.

00:02:53.980 --> 00:02:55.840
What is A transpose?

00:02:55.840 --> 00:03:05.450
A transpose is going to be equal
to just all the rows

00:03:05.450 --> 00:03:06.020
turn into columns.

00:03:06.020 --> 00:03:08.960
So the first column becomes
the first row.

00:03:08.960 --> 00:03:12.165
So it becomes 1 0 0 1.

00:03:12.165 --> 00:03:17.740
The second column becomes
the second row 0 1 0 1.

00:03:17.740 --> 00:03:19.290
That's what A transpose is.

00:03:19.290 --> 00:03:21.270
Now, what's A transpose A?

00:03:21.270 --> 00:03:25.100
To figure out that, I want to
figure out what A transpose

00:03:25.100 --> 00:03:26.680
times A is.

00:03:26.680 --> 00:03:29.210
So let me multiply A
transpose times A.

00:03:29.210 --> 00:03:31.370
So I'll rewrite A right here.

00:03:31.370 --> 00:03:36.120
1 0 0 1, 0 1 0 1.

00:03:36.120 --> 00:03:38.290
This is giving us some
good practice on

00:03:38.290 --> 00:03:39.890
matrix-matrix products.

00:03:39.890 --> 00:03:41.590
This is going to be
equal to what?

00:03:41.590 --> 00:03:45.660
Well, first of all, this is
a 2-by-4 matrix, and I'm

00:03:45.660 --> 00:03:49.400
multiplying it by a 4-by-2
matrix, so it's going to be a

00:03:49.400 --> 00:03:52.150
2-by-2 matrix.

00:03:52.150 --> 00:03:55.410
So the first entry is
essentially the dot product of

00:03:55.410 --> 00:03:57.870
that row with that column.

00:03:57.870 --> 00:04:06.100
So it's 1 times 1 plus 0
times 0 plus 0 times 0

00:04:06.100 --> 00:04:07.840
plus 1 times 1.

00:04:07.840 --> 00:04:11.250
So it's just going to be 2 for
that first entry right there.

00:04:11.250 --> 00:04:13.710
And then you take the dot
product of this guy with this

00:04:13.710 --> 00:04:14.760
guy right here.

00:04:14.760 --> 00:04:20.860
So it's 1 times 0, which is 0,
plus 0 times 1, which is 0,

00:04:20.860 --> 00:04:26.090
plus 0 time 0, which is 0, plus
1 times 1, which is 1.

00:04:26.090 --> 00:04:32.970
Now, we do this guy dotted with
this column right there.

00:04:32.970 --> 00:04:40.170
0 times 1 is 0 plus 1 time 0 is
0 plus 0 times 0 is 0 plus

00:04:40.170 --> 00:04:41.525
1 times 1 is 1.

00:04:44.200 --> 00:04:47.740
And then finally,
this row dotted

00:04:47.740 --> 00:04:49.640
with this second column.

00:04:49.640 --> 00:04:51.430
Second row, second column.

00:04:51.430 --> 00:04:55.660
0 times 0 is 0, 1 times 1
is 1, 0 times 0 is 0,

00:04:55.660 --> 00:04:56.580
1 times 1 is 1.

00:04:56.580 --> 00:04:59.530
So we have 1 times
1 plus 1 times 1.

00:04:59.530 --> 00:05:01.450
It's going to be 2.

00:05:01.450 --> 00:05:02.800
It's going to be equal to 2.

00:05:02.800 --> 00:05:07.590
So this right here
is A transpose A.

00:05:07.590 --> 00:05:08.520
But that's not good enough.

00:05:08.520 --> 00:05:12.680
We need to figure out what the
inverse of A transpose A is.

00:05:12.680 --> 00:05:13.710
This is A transpose A.

00:05:13.710 --> 00:05:16.720
But we need to figure out
A transpose A inverse.

00:05:16.720 --> 00:05:19.300
So what's the inverse of this?

00:05:19.300 --> 00:05:20.420
So let me write it here.

00:05:20.420 --> 00:05:26.050
The inverse A transpose
A inverse is going

00:05:26.050 --> 00:05:26.970
to be equal to what?

00:05:26.970 --> 00:05:29.060
It's 1 over the determinant
of this guy.

00:05:29.060 --> 00:05:30.185
What's the determinant here?

00:05:30.185 --> 00:05:32.920
It's going to be 1 over the
determinant of this.

00:05:32.920 --> 00:05:37.510
The determinant is 2 times 2,
which is 4, minus 1 times 1.

00:05:37.510 --> 00:05:40.910
So it's 4 minus 1, which is 3.

00:05:40.910 --> 00:05:47.090
So 1 over the determinant times
this guy, where if I

00:05:47.090 --> 00:05:53.170
swap these two, so I swap the
1's-- sorry, I swap the 2's.

00:05:53.170 --> 00:06:01.210
So this 2 goes here, and then
this orange 2 goes over here.

00:06:01.210 --> 00:06:04.570
And then I make these
1's negative.

00:06:04.570 --> 00:06:09.730
This becomes a minus 1 and
this becomes a minus 1.

00:06:09.730 --> 00:06:12.010
We learned that this is a
general solution for the

00:06:12.010 --> 00:06:13.510
inverse of a 2-by-2 matrix.

00:06:13.510 --> 00:06:17.350
I think it was 10 or 11 videos
ago, and you probably learned

00:06:17.350 --> 00:06:19.420
this in your Algebra II class,
frankly, but there you go.

00:06:19.420 --> 00:06:22.420
We have A transpose A inverse.

00:06:22.420 --> 00:06:23.620
So we have this guy.

00:06:23.620 --> 00:06:26.020
We have this whole guy here
is just this matrix.

00:06:26.020 --> 00:06:28.850
I could multiply the 1/3 into
it, but I don't have to do

00:06:28.850 --> 00:06:29.930
that just yet.

00:06:29.930 --> 00:06:31.670
But let's figure out the
whole matrix now.

00:06:31.670 --> 00:06:34.700
The whole A times this
guy, A transpose A

00:06:34.700 --> 00:06:37.340
inverse times A transpose.

00:06:37.340 --> 00:06:39.350
Let me write it this way.

00:06:39.350 --> 00:06:49.630
So the projection onto the
subspace V of x is going to be

00:06:49.630 --> 00:06:50.880
equal to A.

00:06:54.020 --> 00:07:02.270
1 0 0 1-- let me write a little
bit bigger like this.

00:07:02.270 --> 00:07:11.680
So 1 0 0 1, 0 1 0 1 times A
transpose A inverse, right?

00:07:11.680 --> 00:07:16.810
A times A transpose A inverse,
which is this guy right here.

00:07:16.810 --> 00:07:19.370
Let's just put the 1/3 out
front just because

00:07:19.370 --> 00:07:20.780
that's just a scalar.

00:07:20.780 --> 00:07:24.570
I'll put the 1/3 out front
times this guy.

00:07:24.570 --> 00:07:28.700
This A transpose A inverse
is 1/3 times 2

00:07:28.700 --> 00:07:32.690
minus 1, minus 1, 2.

00:07:32.690 --> 00:07:34.860
And then I'm going to multiply
it times A transpose.

00:07:38.200 --> 00:07:40.180
And all that times
our vector x.

00:07:40.180 --> 00:07:41.880
So A transpose is right there.

00:07:45.440 --> 00:07:52.340
It is 1 0 0 1, 0 1 0 1.

00:07:52.340 --> 00:07:56.440
And then all of that's going
to be times your vector x.

00:07:56.440 --> 00:07:59.070
So we still have some
nice matrix-matrix

00:07:59.070 --> 00:08:00.070
products ahead of us.

00:08:00.070 --> 00:08:04.290
Let see if we can do these.

00:08:04.290 --> 00:08:07.160
So the first one, let's just
multiply these two guys.

00:08:09.680 --> 00:08:12.210
I don't think there's any
simple way to do it.

00:08:12.210 --> 00:08:16.370
This is a 2-by-2 matrix and this
is a 2-by-4 matrix, so

00:08:16.370 --> 00:08:17.740
when I multiply them,
I'm going to end up

00:08:17.740 --> 00:08:23.230
with a 2-by-4 matrix.

00:08:23.230 --> 00:08:27.430
Let me write that 2-by-4
matrix right here.

00:08:27.430 --> 00:08:30.080
And then I can write this
guy right here.

00:08:30.080 --> 00:08:35.450
1 0 0 1, 0 1 0 1.

00:08:35.450 --> 00:08:38.070
And then I have the 1/3 that
was from A transpose A

00:08:38.070 --> 00:08:40.760
inverse, but I put the scaling
factor out there.

00:08:40.760 --> 00:08:43.039
And all of this is equal to the
projection of x onto V.

00:08:43.039 --> 00:08:45.040
So let's do this product.

00:08:45.040 --> 00:08:48.690
So this first entry is going to
be 2 times 1 plus minus 1

00:08:48.690 --> 00:08:51.930
times 0, so that is just 2.

00:08:51.930 --> 00:08:57.250
Then you're going to have 2
times 0 plus minus 1 times 1.

00:08:57.250 --> 00:08:58.895
Well, that's minus 1.

00:08:58.895 --> 00:09:02.040
Then you have 2 times 0
plus minus 1 times 0.

00:09:02.040 --> 00:09:04.110
Well, that's just 0.

00:09:04.110 --> 00:09:08.010
And then you're going to
have 2 times 1 plus

00:09:08.010 --> 00:09:09.420
minus 1 times 1.

00:09:09.420 --> 00:09:10.350
That's 2 minus 1.

00:09:10.350 --> 00:09:11.640
That's just 1, right?

00:09:11.640 --> 00:09:14.790
2 times 1 plus minus
1 times 1.

00:09:14.790 --> 00:09:15.430
Fair enough.

00:09:15.430 --> 00:09:16.930
Now, let's do the second row.

00:09:16.930 --> 00:09:21.730
Minus 1 times 1 plus 2 times
0, so that's just minus 1.

00:09:21.730 --> 00:09:24.460
Minus 1 times 0 plus
2 times 1.

00:09:24.460 --> 00:09:26.170
Well, that's just 2.

00:09:26.170 --> 00:09:28.150
Minus 1 times 0 plus
2 times 0.

00:09:28.150 --> 00:09:29.750
That's just 0.

00:09:29.750 --> 00:09:33.640
Minus 1 times 1 plus
2 times 1.

00:09:33.640 --> 00:09:39.020
Well, that's minus 1 plus
2, so that is 1.

00:09:39.020 --> 00:09:41.560
Almost there, and, of course, we
have to multiply it times x

00:09:41.560 --> 00:09:42.950
at the end.

00:09:42.950 --> 00:09:44.350
That's what the transformation
is.

00:09:44.350 --> 00:09:47.050
But this right here is our
transformation matrix.

00:09:47.050 --> 00:09:48.390
One more left to do.

00:09:48.390 --> 00:09:50.620
Let's hope I haven't made any
careless mistakes and that I

00:09:50.620 --> 00:09:53.685
won't make any when doing
this product.

00:09:53.685 --> 00:09:55.480
This is going to be a little
more complicated because this

00:09:55.480 --> 00:09:59.350
is a 4 by 2 times a 2 by 4.

00:09:59.350 --> 00:10:03.550
I'm going to end up with
a 4-by-4 matrix.

00:10:03.550 --> 00:10:06.890
Let me give myself some
breathing room here because

00:10:06.890 --> 00:10:11.260
I'm going to generate a 4-by-4
matrix right there.

00:10:11.260 --> 00:10:12.820
And so what am I going to get?

00:10:12.820 --> 00:10:17.690
So this first entry is going
to be 1 times 2 plus

00:10:17.690 --> 00:10:20.060
0 times minus 1.

00:10:20.060 --> 00:10:22.840
So it's just going
to be equal to 2.

00:10:22.840 --> 00:10:29.100
The next entry: 1 times-- this
row times any column here is

00:10:29.100 --> 00:10:31.560
just going to be the first entry
in the column because it

00:10:31.560 --> 00:10:32.090
gets zeroed out.

00:10:32.090 --> 00:10:35.600
So 1 times 2 plus 0 times
minus 1 is just 2.

00:10:35.600 --> 00:10:39.500
1 times minus 1 plus 0 times
2 is just minus 1.

00:10:39.500 --> 00:10:42.570
1 times 0 plus 0 times 0 is 0.

00:10:42.570 --> 00:10:46.690
1 times 1 plus 0 times
1 is just 1.

00:10:46.690 --> 00:10:48.920
When you take this row and you
multiply it times these

00:10:48.920 --> 00:10:51.710
columns, you literally just
got your first row there.

00:10:51.710 --> 00:10:58.100
Now, let's do this row
times these columns.

00:10:58.100 --> 00:11:00.340
Now, you've got a 0 here, so
you're going to have a 0 times

00:11:00.340 --> 00:11:02.010
the first entry of all
of these and a 1

00:11:02.010 --> 00:11:02.820
times the second one.

00:11:02.820 --> 00:11:06.800
So 0 times 2 plus 1 times
minus 1 is minus 1.

00:11:06.800 --> 00:11:09.450
0 times minus 1 plus
1 times 2 is 2.

00:11:09.450 --> 00:11:11.450
You're just going to get
the second row here.

00:11:11.450 --> 00:11:14.110
2 0 1.

00:11:14.110 --> 00:11:16.140
That actually makes sense,
because if you just look at

00:11:16.140 --> 00:11:18.990
this part of the matrix, it's
the 2-by-2 identity matrix.

00:11:18.990 --> 00:11:22.500
So, anyway, that's a little hint
why this looks very much

00:11:22.500 --> 00:11:24.060
like that, but we're just
going to go through this

00:11:24.060 --> 00:11:25.340
matrix product.

00:11:25.340 --> 00:11:28.280
Now, you multiply this-- let me
do it in a different color.

00:11:28.280 --> 00:11:31.530
You multiply this guy times
each of these columns.

00:11:31.530 --> 00:11:34.440
That guy dotted with that is
just going to 0 because this

00:11:34.440 --> 00:11:37.475
guy's essentially the 0 row
vector, so you're just going

00:11:37.475 --> 00:11:39.330
to get a bunch of zeroes.

00:11:39.330 --> 00:11:44.540
And then, finally, this last
row, it's 1 times the first

00:11:44.540 --> 00:11:46.810
entry plus 1 times
the second entry.

00:11:46.810 --> 00:11:50.920
So this guy's going to be 2
plus minus 1, which is 1.

00:11:50.920 --> 00:11:53.000
Minus 1 plus 2, which is 1.

00:11:53.000 --> 00:11:55.160
0 plus 0, which is 0.

00:11:55.160 --> 00:11:57.240
And then 1 plus 1, which is 2.

00:11:57.240 --> 00:11:59.090
And all that times x.

00:11:59.090 --> 00:12:00.710
And there you have it.

00:12:00.710 --> 00:12:03.500
This is exciting!

00:12:03.500 --> 00:12:08.220
The projection onto V of x
is equal to this whole

00:12:08.220 --> 00:12:10.060
matrix times x.

00:12:10.060 --> 00:12:14.770
So this thing right here, I
could multiply the 1/3 into

00:12:14.770 --> 00:12:15.820
it, but we don't have
to do that.

00:12:15.820 --> 00:12:17.870
That'll just make it a little
bit more messy.

00:12:17.870 --> 00:12:20.720
This thing right here is the
transformation matrix.

00:12:25.270 --> 00:12:27.760
As you can see, since we're
transforming-- remember, this

00:12:27.760 --> 00:12:32.560
projection onto V,
this is a linear

00:12:32.560 --> 00:12:37.540
transformation from R4 to R4.

00:12:37.540 --> 00:12:40.610
You give me some member of R4,
and I'll give you another

00:12:40.610 --> 00:12:44.600
member of R4 that's in my
subspace that is the

00:12:44.600 --> 00:12:45.760
projection.

00:12:45.760 --> 00:12:48.740
So this is going to be
a 4-by-4 You can

00:12:48.740 --> 00:12:49.830
see it right there.

00:12:49.830 --> 00:12:51.610
Anyway, hopefully, you found
that useful to actually see a

00:12:51.610 --> 00:12:52.880
tangible result.

00:12:52.880 --> 00:12:55.430
R4 is very abstract, so this
would even be beyond our

00:12:55.430 --> 00:12:57.730
three-dimensional programming
example.

00:12:57.730 --> 00:12:59.900
We're dealing with a more
abstract data set where we're

00:12:59.900 --> 00:13:02.220
interested in finding
a projection.


A projection onto a subspace is a linear transformation

WEBVTT
Kind: captions
Language: en

00:00:00.670 --> 00:00:04.100
We've defined the notion of a
projection onto a subspace,

00:00:04.100 --> 00:00:06.870
but I haven't shown you yet that
it's definitely a linear

00:00:06.870 --> 00:00:07.960
transformation.

00:00:07.960 --> 00:00:10.965
Nor have I shown you that if
you know a basis for a

00:00:10.965 --> 00:00:13.990
subspace, how do you actually
find a projection onto it?

00:00:13.990 --> 00:00:17.080
So let's see if we can make
any progress on that here.

00:00:17.080 --> 00:00:19.220
So let's say I have
some subspace.

00:00:19.220 --> 00:00:28.750
Let's say v is a
subspace of Rn.

00:00:28.750 --> 00:00:31.720
And let's say I've got some
basis vectors for v.

00:00:31.720 --> 00:00:35.610
So let's say these are my basis
vectors, basis vector 1,

00:00:35.610 --> 00:00:39.160
2, and I have k of them.

00:00:39.160 --> 00:00:41.520
I don't know what v's
dimension is, but

00:00:41.520 --> 00:00:42.760
let's say it's k.

00:00:42.760 --> 00:00:52.230
It's got k basis vectors,
so it is a basis for v.

00:00:52.230 --> 00:00:56.680
And that means that any vector--
that's not v vector,

00:00:56.680 --> 00:01:02.840
that's v subspace-- now that
means that any vector-- let me

00:01:02.840 --> 00:01:07.100
call some vector, I don't know,
let's say, any vector a

00:01:07.100 --> 00:01:10.740
that is a member of my subspace
can be represented.

00:01:10.740 --> 00:01:13.170
That means that a can be
represented as a linear

00:01:13.170 --> 00:01:15.190
combination of these guys.

00:01:15.190 --> 00:01:17.040
So I'll make my linear
combination.

00:01:17.040 --> 00:01:25.130
Let's say it is, I don't know,
y1 times b1, plus y2 times b2,

00:01:25.130 --> 00:01:30.700
all the way to plus
yk times bk.

00:01:30.700 --> 00:01:32.370
That's what the definition
of a basis is.

00:01:32.370 --> 00:01:36.590
The span of these guys is your
subspace v, so any member of v

00:01:36.590 --> 00:01:37.830
can be represented as a linear

00:01:37.830 --> 00:01:40.220
combination of my basis vectors.

00:01:40.220 --> 00:01:47.340
Now, if I were to construct a
matrix-- let's make it an n by

00:01:47.340 --> 00:01:51.890
k matrix-- whose columns are
essentially the basis vectors

00:01:51.890 --> 00:01:53.930
of my subspace.

00:01:53.930 --> 00:01:56.430
So A looks like this,
the first column is

00:01:56.430 --> 00:01:58.310
my first basis vector.

00:01:58.310 --> 00:02:01.800
My second column is my second
basis vector, and I go all the

00:02:01.800 --> 00:02:05.280
way to my k column, and I have
k columns, is going to be my

00:02:05.280 --> 00:02:07.610
k'th basis vector.

00:02:07.610 --> 00:02:10.449
If I'm going to have my k'th
basis vector-- let me make the

00:02:10.449 --> 00:02:14.140
closing bracket the same color
as my opening bracket, just

00:02:14.140 --> 00:02:16.500
like that-- it's going to have
n rows, because each of these

00:02:16.500 --> 00:02:18.360
basis vectors are
members of Rn.

00:02:18.360 --> 00:02:21.140
Remember, v is a subspace of Rn,
so each of these guys are

00:02:21.140 --> 00:02:23.730
going to have n terms.
So this matrix is

00:02:23.730 --> 00:02:25.560
going to have n rows.

00:02:25.560 --> 00:02:29.620
Now, saying that any member
of the subspace v can be

00:02:29.620 --> 00:02:31.980
represented as a linear
combination of these basis

00:02:31.980 --> 00:02:36.360
vectors, is equivalent to saying
that any member, that

00:02:36.360 --> 00:02:43.170
a, that any member a of our
subspace v can be represented

00:02:43.170 --> 00:02:57.540
as the product of our matrix A,
times some vector y, where

00:02:57.540 --> 00:02:57.820
[INAUDIBLE]

00:02:57.820 --> 00:03:06.980
was equal to a, for some y,
that is a member of Rk.

00:03:06.980 --> 00:03:09.940
Now why is this statement and
that statement equivalent?

00:03:09.940 --> 00:03:12.600
Well you can imagine, if you
were to just multiply this

00:03:12.600 --> 00:03:20.110
times some vector y in Rk, so
it's y1, y2, all the way down

00:03:20.110 --> 00:03:27.090
to yk, this is going to be equal
to y1 times v1, plus y2,

00:03:27.090 --> 00:03:35.960
times b2, all the way to plus yk
times bk, which is the same

00:03:35.960 --> 00:03:36.280
thing as this.

00:03:36.280 --> 00:03:39.990
So you can always pick the
right linear combination.

00:03:39.990 --> 00:03:43.200
You can always pick the right
member, yk, so that you get

00:03:43.200 --> 00:03:45.720
the right linear combination of
your basis vectors to get

00:03:45.720 --> 00:03:48.500
any member of your subspace v.

00:03:48.500 --> 00:03:51.860
So any member of my subspace,
right there, can be

00:03:51.860 --> 00:03:56.930
represented as the product
of the matrix A with

00:03:56.930 --> 00:03:59.980
some vector in Rk.

00:03:59.980 --> 00:04:03.140
Now we don't know much about
this vector here in Rk.

00:04:03.140 --> 00:04:07.980
Now, the projection-- let's
say that x is just some

00:04:07.980 --> 00:04:17.620
arbitrary member of Rn-- the
projection of x onto our

00:04:17.620 --> 00:04:21.620
subspace v, that is by
definition going to be a

00:04:21.620 --> 00:04:23.890
member of your subspace.

00:04:23.890 --> 00:04:30.540
Or another way of saying it is
that this guy, the projection

00:04:30.540 --> 00:04:36.380
onto v of x is going to be equal
to my matrix A, is going

00:04:36.380 --> 00:04:39.320
to be equal to-- I'll do it in
blue-- is going to be equal to

00:04:39.320 --> 00:04:50.360
A times some vector y, or
some vector y in Rk.

00:04:50.360 --> 00:04:54.250
If we knew what that vector y
was, if we could always find

00:04:54.250 --> 00:04:58.060
it, then we would have a
formula, so to speak, for

00:04:58.060 --> 00:05:00.040
figuring out the projection
of x onto v.

00:05:00.040 --> 00:05:01.420
But we don't have that yet.

00:05:01.420 --> 00:05:07.160
All I've said is, any member of
v can be represented as a

00:05:07.160 --> 00:05:12.760
product of our matrix A, which
has the basis for v as

00:05:12.760 --> 00:05:15.260
columns, and some
member of Rk.

00:05:15.260 --> 00:05:18.670
That just comes out of the fact
that these guys span v,

00:05:18.670 --> 00:05:19.960
that any member of
v is a linear

00:05:19.960 --> 00:05:21.580
combination of those guys.

00:05:21.580 --> 00:05:25.620
We know that the projection of
x onto v is a member of our

00:05:25.620 --> 00:05:28.760
subspace v, it has to
be inside of v.

00:05:28.760 --> 00:05:32.550
So it can also be represented
this way.

00:05:32.550 --> 00:05:35.740
Now what was our definition
of our projection?

00:05:35.740 --> 00:05:38.420
Our definition of our
projection, we say--

00:05:38.420 --> 00:05:39.330
Well, let me write
it this way.

00:05:39.330 --> 00:05:46.850
We know that x can be
represented as the sum of the

00:05:46.850 --> 00:05:55.910
projection onto v of x, plus
some member of v complement.

00:05:55.910 --> 00:06:04.140
Or maybe I could even write,
plus the projection onto the

00:06:04.140 --> 00:06:06.410
orthogonal complement of v.

00:06:11.710 --> 00:06:12.620
You could write this way.

00:06:12.620 --> 00:06:16.810
I could have also written this
as w, where w is a member of v

00:06:16.810 --> 00:06:17.170
complement.

00:06:17.170 --> 00:06:18.160
Actually, let me write
it that way.

00:06:18.160 --> 00:06:19.140
That might make it simpler.

00:06:19.140 --> 00:06:23.720
I don't want to get too many
projections here-- plus w,

00:06:23.720 --> 00:06:37.800
where w is a unique member
of the orthogonal

00:06:37.800 --> 00:06:39.420
complement of v.

00:06:39.420 --> 00:06:42.610
Or you could say it this way,
if you subtract a projection

00:06:42.610 --> 00:06:48.900
of x onto v from both sides, you
get x minus the projection

00:06:48.900 --> 00:06:52.285
of x onto v, is equal to w.

00:06:54.790 --> 00:06:57.510
Or another way to say it is that
this guy right here is

00:06:57.510 --> 00:07:01.280
going to be a member of the
orthogonal complement of v,

00:07:01.280 --> 00:07:03.700
right, because this is
the same thing as w.

00:07:03.700 --> 00:07:06.840
Now what's the orthogonal
complement of v?

00:07:06.840 --> 00:07:09.070
We go back to this
matrix here.

00:07:09.070 --> 00:07:11.500
I have these basis vectors.

00:07:11.500 --> 00:07:13.410
Right here is the columns.

00:07:13.410 --> 00:07:21.310
So the column space of A is
going to be equal to v, right?

00:07:21.310 --> 00:07:23.720
The column space of
A is just the span

00:07:23.720 --> 00:07:25.100
of these basis vectors.

00:07:25.100 --> 00:07:28.420
And by definition, that is
going to be equal to my

00:07:28.420 --> 00:07:29.630
subspace v.

00:07:29.630 --> 00:07:32.760
Now what is the orthogonal
complement of v?

00:07:32.760 --> 00:07:36.200
The orthogonal complement of v
is going to be the orthogonal

00:07:36.200 --> 00:07:37.920
complement of my column space.

00:07:42.420 --> 00:07:46.160
And what's the orthogonal
complement of a column space?

00:07:46.160 --> 00:07:50.840
That's equal to the null space
of A transpose, or you could

00:07:50.840 --> 00:07:53.750
also call that the left
null space of A.

00:07:53.750 --> 00:07:56.600
But we've seen that many,
many, videos ago.

00:07:56.600 --> 00:08:00.980
So we could say that x minus the
projection of x onto v as

00:08:00.980 --> 00:08:07.240
a member of-- let me write
it this way-- x minus the

00:08:07.240 --> 00:08:14.230
projection onto v of x is a
member of the orthogonal

00:08:14.230 --> 00:08:18.050
complement of my column space
of my matrix, which is equal

00:08:18.050 --> 00:08:22.870
to the null space
of A transpose.

00:08:22.870 --> 00:08:25.210
That's the orthogonal
complement of v.

00:08:25.210 --> 00:08:28.810
This is the same thing as the
orthogonal complement of v.

00:08:28.810 --> 00:08:30.540
But what does this mean?

00:08:30.540 --> 00:08:31.910
What does this mean
right here?

00:08:31.910 --> 00:08:36.640
This means that if I take A
transpose, and I multiply it

00:08:36.640 --> 00:08:39.200
times this vector, because it's
a member of A transpose's

00:08:39.200 --> 00:08:40.580
null space.

00:08:40.580 --> 00:08:44.540
So if I multiply it times that
vector right there-- so,

00:08:44.540 --> 00:08:48.470
projection of x onto v-- then
I'm going to get 0.

00:08:48.470 --> 00:08:49.840
I'm going to get the 0 vector.

00:08:49.840 --> 00:08:52.160
That's the definition
of a null space.

00:08:52.160 --> 00:08:53.730
So let's write this out
a little bit more.

00:08:53.730 --> 00:08:54.990
Let's see if we can
algebraically

00:08:54.990 --> 00:08:56.240
manipulate it a bit.

00:08:56.240 --> 00:08:59.020
So if we distribute this matrix
vector product, we get

00:08:59.020 --> 00:09:06.150
A transpose times the vector
x, minus A transpose, times

00:09:06.150 --> 00:09:13.520
the projection-- actually let
me write this this way.

00:09:13.520 --> 00:09:16.400
Instead of keep writing the
projection onto v of x, what

00:09:16.400 --> 00:09:17.820
did we say earlier
in this video?

00:09:17.820 --> 00:09:22.900
We said the projection of v onto
x can be represented as

00:09:22.900 --> 00:09:26.750
the matrix product of
the matrix A, times

00:09:26.750 --> 00:09:29.330
some vector y in Rk.

00:09:29.330 --> 00:09:30.690
That's where we started
off the video.

00:09:30.690 --> 00:09:32.240
So let me write it that way,
because that's what going to

00:09:32.240 --> 00:09:34.050
simplify our work
a little bit.

00:09:34.050 --> 00:09:37.460
So I'm going to distribute
the A transpose, A

00:09:37.460 --> 00:09:38.560
transpose times x.

00:09:38.560 --> 00:09:41.300
And then A transpose, minus A
transpose, times this thing.

00:09:41.300 --> 00:09:49.300
This thing I can write as A
times some vector y, and this

00:09:49.300 --> 00:09:52.460
is just a byproduct of the
notion that the projection is

00:09:52.460 --> 00:09:54.220
a member of our subspace.

00:09:54.220 --> 00:09:56.140
Because it's a member of our
subspace, it's going to be

00:09:56.140 --> 00:09:59.480
some linear combination of
the column vectors of A.

00:09:59.480 --> 00:10:04.800
We saw that up here, so it can
be represented in this way.

00:10:04.800 --> 00:10:12.400
So instead of projection onto
v of x, I can just write Ay.

00:10:12.400 --> 00:10:16.500
This thing and this thing are
equivalent, because this thing

00:10:16.500 --> 00:10:17.940
is a member of v.

00:10:17.940 --> 00:10:21.250
And then all of that is going
to be equal to 0.

00:10:21.250 --> 00:10:24.980
And then if we add this to both
sides of this equation,

00:10:24.980 --> 00:10:38.500
we get that A transpose x is
equal to A transpose A of y.

00:10:38.500 --> 00:10:39.580
Now this is interesting.

00:10:39.580 --> 00:10:41.590
Remember where we started
off here.

00:10:41.590 --> 00:10:50.010
We said that the projection onto
v of x is equal to Ay for

00:10:50.010 --> 00:10:56.040
some y that is a member of Rk.

00:10:56.040 --> 00:10:59.160
If we knew what that y was, if
we could always solve for that

00:10:59.160 --> 00:11:02.450
y, then the projection of
x would be well defined.

00:11:02.450 --> 00:11:05.230
And we could always just
figure it out.

00:11:05.230 --> 00:11:07.840
Now can we solve for y here?

00:11:07.840 --> 00:11:11.070
Well, we'll be able to solve
for y if we can take the

00:11:11.070 --> 00:11:13.250
inverse of that matrix.

00:11:13.250 --> 00:11:16.360
If this major is always
invertible, then we're always

00:11:16.360 --> 00:11:17.870
going to be able to
solve for y here.

00:11:17.870 --> 00:11:20.370
Because we just take the inverse
of this matrix and

00:11:20.370 --> 00:11:22.120
multiply it times the
left side of both

00:11:22.120 --> 00:11:23.660
sides in this equation.

00:11:23.660 --> 00:11:28.450
Now if you remember, three
videos ago, I think it was

00:11:28.450 --> 00:11:34.180
three videos ago, I showed you
that if I have a matrix A

00:11:34.180 --> 00:11:43.220
whose columns are linearly
independent, then A transpose

00:11:43.220 --> 00:11:48.890
A is always invertible.

00:11:48.890 --> 00:11:51.580
The whole reason why I did that
video was for this moment

00:11:51.580 --> 00:11:52.930
right here.

00:11:52.930 --> 00:11:56.430
Now, what about our matrix A?

00:11:56.430 --> 00:11:59.650
Well, our matrix A has column
vectors that form the basis

00:11:59.650 --> 00:12:00.500
for a subspace.

00:12:00.500 --> 00:12:03.740
By definition, basis vectors
are linearly independent.

00:12:03.740 --> 00:12:07.510
So our matrix A has columns that
are linearly independent.

00:12:07.510 --> 00:12:09.850
And if you watched that video,
and if you believe what I told

00:12:09.850 --> 00:12:13.470
you, then you'll know that A
transpose A, in our case, is

00:12:13.470 --> 00:12:14.760
going to be invertible.

00:12:14.760 --> 00:12:16.570
It has to be inveritible.

00:12:16.570 --> 00:12:19.330
So let's take the inverse
of it and multiply

00:12:19.330 --> 00:12:20.350
it times both sides.

00:12:20.350 --> 00:12:24.940
If we take A transpose A
inverse-- we know that this

00:12:24.940 --> 00:12:29.030
exists, because A has linearly
independent columns-- and

00:12:29.030 --> 00:12:34.080
multiply it times this side
right here, A transpose x.

00:12:34.080 --> 00:12:37.430
And then on this side we get--
well, we're going to do the

00:12:37.430 --> 00:12:43.080
same thing-- A transpose A
inverse, times this thing

00:12:43.080 --> 00:12:47.060
right here, A transpose Ay.

00:12:47.060 --> 00:12:48.920
These two things when you
multiply them, when you

00:12:48.920 --> 00:12:51.340
multiply the inverse of a
matrix, times the matrix,

00:12:51.340 --> 00:12:53.470
you're just going to get
the identity matrix.

00:12:53.470 --> 00:12:56.910
So that's just going to be equal
to the identity matrix.

00:12:56.910 --> 00:12:59.880
And the identity matrix times y
is just going to be y, so we

00:12:59.880 --> 00:13:02.790
get-- and this is a vector--
so if I flip them around, I

00:13:02.790 --> 00:13:09.080
get that y is equal to this
expression right here.

00:13:09.080 --> 00:13:14.720
A transpose A inverse, which'll
always exist, times A

00:13:14.720 --> 00:13:17.120
transpose, times x.

00:13:17.120 --> 00:13:20.970
Now we said the projection of x
onto v is going to be equal

00:13:20.970 --> 00:13:23.410
to A times y, for some y.

00:13:23.410 --> 00:13:26.660
Well we just solved for the y
using our definition of a

00:13:26.660 --> 00:13:27.180
projection.

00:13:27.180 --> 00:13:28.700
We just were able
to solve for y.

00:13:28.700 --> 00:13:34.710
So now, we can define our
projection of x onto v as a

00:13:34.710 --> 00:13:36.590
matrix vector product.

00:13:36.590 --> 00:13:44.020
So we can write the projection
onto v of our vector x is

00:13:44.020 --> 00:13:51.030
equal to A, times y, and
y is just equal to

00:13:51.030 --> 00:13:52.280
that thing right there.

00:13:54.800 --> 00:14:01.630
So A times A transpose A
inverse-- which always exists

00:14:01.630 --> 00:14:05.600
because A has linearly
independent columns-- times A

00:14:05.600 --> 00:14:08.010
transpose, times x.

00:14:08.010 --> 00:14:10.480
And this thing right here, this
long convoluted thing,

00:14:10.480 --> 00:14:15.880
that's just some matrix, some
matrix which always exists for

00:14:15.880 --> 00:14:18.980
any subspace that
has some basis.

00:14:18.980 --> 00:14:22.200
So we've just been able to
express the projection of x

00:14:22.200 --> 00:14:27.340
onto a subspace as a matrix
vector product.

00:14:27.340 --> 00:14:29.920
So anything that can be any
matrix vector product

00:14:29.920 --> 00:14:31.670
transformation is a linear
transformation.

00:14:35.250 --> 00:14:37.710
And not only did we show that
it's a linear transformation,

00:14:37.710 --> 00:14:45.420
we showed that, look, if you can
give me the basis for v,

00:14:45.420 --> 00:14:48.810
I'm going to make those column
vectors equal to the column of

00:14:48.810 --> 00:14:50.080
some matrix A.

00:14:50.080 --> 00:14:53.250
And then if I take matrix A, if
I take its transpose, if I

00:14:53.250 --> 00:14:56.910
take A transpose times A, and
invert it, and if I multiply

00:14:56.910 --> 00:14:59.130
them all out in this way,
I'm going to get the

00:14:59.130 --> 00:15:02.000
transformation matrix
for the projection.

00:15:02.000 --> 00:15:04.440
Now this might seem really
complicated, and it is hard to

00:15:04.440 --> 00:15:07.720
do by hand for many, many
projections, but this is super

00:15:07.720 --> 00:15:10.340
useful if you're going to do
some three-dimensional

00:15:10.340 --> 00:15:11.590
graphical programming.

00:15:15.150 --> 00:15:25.370
Let's say you have some
three-dimensional object, and

00:15:25.370 --> 00:15:28.000
you want to know what it looks
like from the point of view of

00:15:28.000 --> 00:15:29.185
some observer.

00:15:29.185 --> 00:15:30.890
So let's say you have
some observer.

00:15:30.890 --> 00:15:34.270
Some observer's point of view
is essentially going to be

00:15:34.270 --> 00:15:36.020
some subspace.

00:15:36.020 --> 00:15:38.440
You want to see what the
projection of this cube onto

00:15:38.440 --> 00:15:41.970
the subspace, how would it
look to the person who's

00:15:41.970 --> 00:15:44.260
essentially on to this flat
screen right there.

00:15:44.260 --> 00:15:47.610
How would that cube look from
this point of view?

00:15:47.610 --> 00:15:51.890
Well if you know the basis for
this subspace, you can just

00:15:51.890 --> 00:15:54.260
apply this transformation.

00:15:54.260 --> 00:15:57.810
You can make a matrix whose
columns are these basis

00:15:57.810 --> 00:16:00.790
vectors for this observer's
point of view.

00:16:00.790 --> 00:16:04.860
And then you can apply this to
every vector in this cube in

00:16:04.860 --> 00:16:09.420
R3, and you'll know exactly how
this cube should look from

00:16:09.420 --> 00:16:11.030
this person's point of view.

00:16:11.030 --> 00:16:14.200
So this is actually a
super useful result.