Vector Spaces in Linear Algebra

In the previous sections, we re-visited concepts about vectors and vector operations. You saw that some vectors can be added together to form other vectors. You also saw that vectors can be scaled in the positive or negative direction. These two properties are the starting point of this section.

Vector Spaces

They sound complex, but they are not. Intuitively, a vector space is a set of vectors such that if you add any two vectors or scale any vectors, the resultant vector is also present in this set. Formally, the definition is:

If 'v' and 'w' are in a vector space and c1 and c2 are scalars (constants),

then $L(v,w)=(c1\ast v+c2\ast w)$is also in the vector space.

This type of a function L is referred to as a linear combination of vectors.

Basis & Span

Basis

As we saw in the previous section, the basis of a certain vector space are vectors whose linear combinations can specify the entire vector space. For 2D real space, the typical choice of basis vectors are $\hat{i}$ and $\hat{j}$.

However, these are not the only basis of 2D space. Any two vectors in 2D real space can be the basis for the space. It is worth noting that the actual values of the scalars that multiply the vectors will be different for different choices of basis.

Span

The span of two vectors is the set of all of their linear combinations. Calculating span is a way of asking the question, what are all possible vectors you can create by scaling and adding the two given vectors? For $(\overrightarrow{v1},\overrightarrow{v2})$ in 2 dimensions, the span is all the values of $c_1\ast v_1+c_2\ast v2$ . There are 3 possible cases:

• If the vectors are not collinear, span is all of 2D space, which is the plane defined by these two vectors
• If the vectors are collinear, their linear combination will only exist on that line, hence the span is the line defined by these two vectors
• If both the vectors are 0, then the span is the origin.

When we scale this up to 2 vectors in 3 dimensions, any 2 vectors will still have a plane, a line and a point as its possible spans

$span(v_1,v_2)\in \{2Dspace,1Dspace,0Dspace\}$

Although that notation isn't mathematically precise, it describes the set in which the span of two vectors lies. The actual spans depends on the relationships of these vectors with each other.

What happens when we move to 3 vectors in 3 dimensions? The span of vectors $(\overrightarrow{v1},\overrightarrow{v2},\overrightarrow{v3})$ is all the possible values of  $c_1\ast v_1+c_2\ast v2+c_3\ast v_3$ .

Now we have 3D space, a plane, a line and a point as its possible spans. 

$span(v_1,v_2,v_3)\in \{3Dspace,2Dspace,1Dspace,0Dspace\}$

The actual span depends on the relationships of these vectors with each other.

This brings us to the most important consequence of expressing vectors algebraically - moving further up dimensions is mathematically trivial! Now that we have a framework of working with linear combinations in 2 and 3 dimensions, we can use the same framework to work with 4 dimensions or more, even though we are incapable of visualising these dimensions.

This property is why it is easy to express hundreds or even thousands of columns (as are found in large datasets today) as independent axes in space (dimensions), and work with vectors that are present in that n-dimensional space.

Many data science and machine learning problems boil down to searching for the right vector in an immensely large vector space. Once you have a good intuition for how vector spaces work, you will be more equipped to deal with these search problems.

Let's take an application. In some text models, all the possible words in some documents are modeled as different axes. Let's find out how.

Additional Reading

1. Here is a video further explaining linear combinations and spans.
2. Here is a link to Mikolov’s original paper describing word embeddings, a vector space used in text analytics
3. A brief video lecture by Prof. Chris Manning from Stanford, explaining very briefly how vectors are used for representing words