# Linear Algebra Notes

Notes on some linear algebra topics. Sometimes I find things are just stated as being obvious, for example, the dot product of two orthogonal vectors is zero. Well... why is this? The result was a pretty simple but nonetheless it took a little while to think it through properly. Hence, these notes... they're my musings on the "why", even if the "why" might be obvious to the rest of the world!

## Matricies And Simultaneous Linear Equations

The set of linear equations: Can be represented in matrix form: Any vector , where , is a solution to the set of simultaneous linear equations. How can we solve this from the matrix. We create an augmented or patitioned matrix: Next this matrix must be converted to row echelon form. This is where each row has the same or less zeros to the right than the row below. For example, the following matrix is in row echelon form: But, this next one is not: We go from a matrix to the equivalent in row-echelon form using elementary row operations:

1. Row interchange:
2. Row scaling:

We can do this in python using the sympy package and the rref() function:

from sympy import Matrix, init_printing

init_printing(use_unicode=True)

system = Matrix(( (3, 6, -2, 11), (1, 2, 1, 32), (1, -1, 1, 1) ))
system
system.rref()[0]

Which outputs:

⎡1  0  0  -17/3⎤
⎢              ⎥
⎢0  1  0  31/3 ⎥
⎢              ⎥
⎣0  0  1   17  ⎦

To then apply this to our augmented matrix we could do something like the following. Lets say we have a set of 3 simultaneous equations with 3 unknowns: We'd make an augmented matrix like so: To solve this using Python we do:

from sympy import Matrix, solve_linear_system, init_printing
from sympy.abc import x, y, z

init_printing(use_unicode=True)

system = Matrix(( (3, 6, -2, 11), (1, 2, 1, 32), (1, -1, 1, 1) ))
system

solve_linear_system(system, x, y, z)

Which will give the following output:

⎡3  6   -2  11⎤
⎢             ⎥
⎢1  2   1   32⎥
⎢             ⎥
⎣1  -1  1   1 ⎦
{x: -17/3, y: 31/3, z: 17}

## Orthogonal Vectors and Linear Independence

Figure 1

Orthogonal vectors are vectors that are perpendicular to each other. In a standard 2D or 3D graph, this means that they are at right angles to each other and we can visualise them as seen in Figure 1: is at 90 degrees to both and . is at 90 degrees to and , and is at 90 degrees to and .

In any set of any vectors, , the vectors are said to be linearly linearly independent if every vector in the set is orthogonal to every other vector in the set.

So, how do we tell if one vector is orthogonal to another? The answer is the dot product which is defined as follows.

We know when two vectors are orthogonal when their dot product is zero: x and y are orthogonal.

But why is this the case? Lets imagine any two aribtrary vectors, each on the circumference of a unit circle (so we know that they have the same length and are therefore a proper rotation of a vector around the centre of the circle). This is shown in figure 2. From the figure, we know the following:

Figure 2

The vector is the vector rotated by degrees. We can use the following trig identities: Substitute these into the formauls above, we get the following. Which means that... For a 90 degree rotation we know that and . Substiuting these values into the above equations we can clearly see that... Therefore, any vector in 2D space that is will become and therefore the dot product becomes . And voilà, we know know why the dot product of two orthogonal 2D vectors is zero. I'm happy to take it on faith that this extrapolates into n dimensions :)

Another way of looking at this is to consider the law of cosines: :

Here and represent the magnitude of the vectors , , and , . represents the magnitude of the vector . Thus, we have this:

I.e., .

When the angle between the vectors is 90 degrees, and so that part of the equation dissapears to give us . Now... Recall that and that therefore .

Based, on this we can say that... Now, all of the and terms above will be cancelled out by the and terms, leaving us with... Which is just twice the dot product when the vectors are at 90 degrees - or just the dot product because we can divide through by 2.

## Orthogonal Matrices

So, onto orthogonal matrices. A matrix is orthogonal if . If we take a general matrix and multiply it by it's transpose we get...

The pattern looks pretty familiar right?! It looks a lot like , our formula for the dot product of two vectors. So, when we say , , and , we will get the following: a matrix of two row vectors , . If our vectors and are orthogonal, then the component must be zero. This would give us the first part of the identify matrix pattern we're looking for.

The other part of the identity matrix would imply that we would have to have and ... which are just the formulas for the square of the length of a vector. Therefore, if our two vectors are normal (i.e, have a length of 1), we have our identity matrix.

Does the same hold true for ? It doesn't if we use our original matrix A!... Oops, we can see that we didn't get the identity matrix!! But, perhaps we can see why. If was a matrix of row vectors then is a matrix of column vectors. So for we were multiplying a matrix of row vectors with a matrix of column vectors, which would, in part, give us the dot products as we saw. So if we want to do it would follow that for this to work now was to be a matrix of column vectors because we get back to our original ( would become a mtrix of row vectors):

So we can say that if we have matrix who's rows are orthogonal vectors and who's columns are also orthogonal vectors, then we have an orthogonal matrix!

Okay, thats great n' all, but why should we care? Why Are orthogonal matricies useful?. It turns our that orthogonal matricies preserve angles and lengths of vectors. This can be useful in graphics to rotate vectors but keep the shape they construct, or in numerical analysis because they do not amplify errors.

## Determinants

This is one of the better YouTube tutorials I've found that explains the concepts and not just the mechanics behind determinants, byt 3BlueBrown (awesome!).

## Eigenvectors and Eigenvalues

Eigenvectors and values exist in pairs: every eigenvector has a corresponding eigenvalue. An eigenvector is a direction, ... An eigenvalue is a number, telling you how much variance there is in the data in that direction ...