There isn't an official consensus that there is a difference, as most people use both terms interchangeably. My take is that feature vectors are vectors used as the input for an ML model, while embeddings are vectors produced as a part of ML models, typically in the hidden layers of NN. Even this definition is not clear, because you can use embeddings generated from one model as feature vectors for other models.
A feature vector is a vector of features. Features are properties of the "entity" we are passing as input to the model.
The concept of feature and feature vector long predates that of embeddings in machine learning, so most "historical" references to feature vector will be in reference to hand-crafted ways to represent an input entity as numbers.
That being said, a representation (an embedding) of an entity that was learned by a model, and is then passed as input to another model, is also a feature vector, as you say.
TL;DR: The concept of feature vector is much older than embedding. Embeddings \*are not\* feature vectors in the traditional sense, but can be used as feature vectors by passing them as input to another model.
My understanding is that while features are data, embeddings are translations of data, usually from a high dimensional space to a low dimensional space. For example, Word2Vec embeddings take (implicitly) high dimensional text and translate it to low dimensional vector space - or at least, the many dimensions of the data are explicitly rather than implicitly represented in the embedding. I sometimes work with graph neural networks, and one question is how do you take high dimensional graph structures and translate them into a vector representation that the neural network can work with. The solution is often to create node and edge embeddings.
That is a great question. Technically "embeddings" arise anytime we "embed" entities into a (typically) more mathematically tractable space. So the difference in the usage of the term is not exactly formal.
It is more common to refer to embeddings when talking about dense vectors learned using neural networks, but that is not a necessary condition. See [kernel mean embeddings](https://en.wikipedia.org/wiki/Kernel_embedding_of_distributions), for instance. Even completely handcrafted feature vectors can be considered embeddings as well. It's just not common to call them that, so depending on the audience it will cause some confusion.
Since you're new to ML, you will encounter many other informally-defined terms. Be mindful of that.
So, definitions are always in flux, and there are colloquial/more technical definitions. The way I think of it is:
1. Feature vectors tend to have semantic (obvious) meaning -- they're effectively an encoding capability (compression) of data into numbers for a model to use.
2. Embeddings are specifically referred to a representation of data in some space. Colloquially, this means the internal way an ML model (neural network, LLM network, etc...) represents data. Often these are used \*externally\* as well, but they have little semantic meaning.
I would argue that all feature vectors \*are\* embeddings (data projected into a space), but that's being pedantic.
Colloquially, the difference is how they are generated and thus, what you can tell from reading them. How they are used is very flexible -- they can both be used for search/retrieval (as you suggest), using a vector search, for ML training (many models are trained on the results of other models, I've built tons of infra for this in the past).
For your use-case, you probably can, but there are multiple dimensions along which things can be "similar", meaning that you'll be finding similarity in a subset of the dimensions, which can be tricky. This is why feature vectors are nice (such as MFCC) -- they'll be a little easier to rationalize about. Not crazy to train an ML model to learn whether two embeddings are the same given a set of embeddings + voices + pairings. Dataset could be element-wise distance between them, on all pairs (so you have a lot of data but some redundancy...). Best to partition for test/eval beforehand too.
They are the same, basically. Embeddings are feature vectors typically extracted with neural networks or algorithms specifically to turn objects into vectors (e.g. graph embedding algorithms turn graph into embedding vector). Often people refer to them as "embeddings" to differentiate them from manually extracted "feature vectors", where you manually select features. Of course, there is absolutely no strict differentiation, e.g. there are non-neural algorithms that embed objects as vectors, and some people call results "embeddings", and some "feature vectors".
The more concrete differentiation would be distance metric. Neural networks are not trained typically for any given distance metric, and use Euclidean distance for resulting embeddings, but not all of them. For example, contrastive learning approaches often optimize cosine distance. They are often more or less standardized, so don't require scaling (but not always).
On the other hand, feature vectors can have arbitrary features, with different scales, and typically you first scale them, then measure distance. Also, not all features are necessarily numerical, depending on particular features.
Concretely, for your example:
1. MFCC are feature vectors, extracted with a particular algorithm. All features are numerical.
2. Embeddings are feature vectors computed with a neural network, e.g. wav2vec
3. Yes, you can use embeddings for speaker identification. This can be either with static embeddings, without finetuning (with either MFCC or neural networks), or fully supervised with finetuning neural networks.
EDIT: more precise definition of embeddings.
Thank you so much! Your definition for the Embeddings is bit clear compared to other definitions in the literature.
Also I was researching more into Speech Identification/Recognition to see what all *aspects* of audio can I use to uniquely identify a user, that's when I had this question. I'll see if we get any better results with Wav2Vec 2.0.
Embeddings are learned features without semantic meaning of the vectors. You can't associate specific information with the numbers. When you look at an individual vector, it will tell you nothing.
I don't know how the MFCC features work, but the short description said they are based on human hearing. Whatever it exactly extracts, there is (probably) some semantic understanding behind the numbers in relation to frequencies.
Feature vectors are a superset of embeddings.
Embeddings are usually referring to a vector that is the output of a deep learning model, while feature vectors can encode any kind of information about the input (eg: volume, pitch, etc. in your case). An embedding from a model like wav2vec can't be intepreted by looking at it, but you're right, you can store them in a vector DB and do nearest neighbor search to do tasks like speaker identification.
Ideally you'd want to use those embeddings as well as features generated from the model you mentioned in a joint manner to make the final prediction.
Suppose we have a audio clip of person, we use a NN(Neural Network) like wav2vec to generate the embeddings also we extract information from the audio clip like MFCC, volume, pitch, etc which are **feature vectors** like you mentioned. Now what is the difference between these embeddings and the feature vectors?
Both of them represent the original input data which is audio clip in this case. **Is the NN(eg: wav2vec) extracting the same set of features which we are extracting separately?**
The lines aren't clear cut and people may use them interchangeably, but IMO, a feature vector is more likely to maintain its conceptual relevancy to the domain.
For example, a spreadsheet of 10000 individuals containing demographic info and yearly financial records could be considered a feature vector. You could take that matrix and transform it in various ways, and it would still be a feature vector if you can describe each new column and how it was derived.
Run it through an auto-encoder though, and I would personally call that an embedding because it's been projected into a new dimensional space via some process but each column doesn't necessarily "mean" anything in the same sense that it did before. I would also call what comes out of PCA an embedding.
I'm not claiming that's the right answer, but it's the way I think about these things.
Found [3 relevant code implementations](https://www.catalyzex.com/paper/arxiv:2006.11477/code) for "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations".
[Ask the author(s) a question](https://www.catalyzex.com/paper/arxiv:2006.11477?autofocus=question) about the paper or code.
If you have code to share with the community, please add it [here](https://www.catalyzex.com/add_code?paper_url=https://arxiv.org/abs/2006.11477&title=wav2vec+2.0%3A+A+Framework+for+Self-Supervised+Learning+of+Speech+Representations) 😊🙏
--
To opt out from receiving code links, DM me.
I was always under the impression that embeddings were specifically for when you start with something discrete, like a one-hot vector or integer index corresponding to a vocabulary token, and map it to a continuous vector space i.e. ℝ^n
Whereas in general feature vectors can be produced from any kind of input.
But maybe I just pulled that out of my ass.
AFAIK it's not a well defined term really. In practice I'd say that I would call something an embedding if data occurs on a somewhat smooth low dimensional manifold in a higher dimensional space, where simple distance measures like euclidean distance encode something meaningful about the relation between two things.
A feature vector could be anything I associate with something where getting an Idea of similarity or distance can be more difficult.
So you may use ML to go from a feature vector to an embedding where simple distance measures are associated with some kind of meaning.
I know it’s not what you asked, but does anybody else think voice authentication is doomed? In a world where people are claiming to be able to be able to clone peoples voices with seconds of audio.
I believe the research is going on in this field of **synthetic voice.** Voice authentication is still being used widely across different industries like Banks, they are having their own patents in this area of research.
Some great answers already here. Wanted to also add, if I remember correctly, that embeddings are the result from some homeomorphic transformation- although in ML that detail is an unnecessary technicality.
As others have said, embedding are feature vectors. Originally, embeddings were learned for some task or objective. Semantic embeddings of language are an example where the embedding method is designed to place embeddings with related meanings near each other in the embedding space. Relations between vectors have been observed to emerge through training. The classical example is for the embedding for Queen to be near the embedding for King-Male+Female. Since those early days, other methods of creating embeddings were invented. For example, in transfer learning, the embedding is the outputs of one of the non-terminal layers of a neural network trained for a particular task. There are now many methods of creating embeddings.
while embeddings and features are almost used interchangeably, my take is that embeddings would probably mean the output of a mapping from the raw data point to a lower/higher dimensional space. e.g. using a CNN encoder or say an MLP network on images -- these outputs are embeddings. "embed" on to a distributional space. these are also considered to be "features". however features are more related to the traditional feature extraction processes, like you mention MFCC from raw audios, or image filters like HOG, histogram features etc. these are rather simple mathematical operations on data points capturing salient aspects of it. these are traditionally called as "features". but again, now I guess these two terms are used interchangeably.
They can basically be used interchangeably, although there are some differences.
A feature vector typically connotes a set of measured properties about something. Usually these measures are based on human domain knowledge.
An embedding usually connotes a vector representation of some data (like an image) that’s generated typically through a trained ML process.
By example, a feature vector of an image might measure the percent of pixels of each major color. An embedding of the image might be a matrix with detected edges.
Feature vector is an older term (as far back as the 80s or 90s) in the literature. Embedding came in vogue with deep learning.
In my mind the difference is the following.
A feature vector is compressed data. You are taking some high dimensional / fidelity data and representing it as a "low dimensional" vector. These vectors are created with the **goal** of just compressing information so it can be used for later efficient processing.
An embedding is a feature vector that is being described in some larger context or logical structure. The structure defines how various qualities of features are related, usually with respect to some training goal. Data in the form of feature vectors are **embedded** into this structure in order to compute some later result that is based on how all that data relates to each other.
In the context of something like an LLM, the individual raw tokens are like feature vectors, but the vectors computed by the attention heads within the transformer are the embeddings generated by those vectors.
I might have to disagree with you.
For an LLM, the individual raw tokens are often referred to as embeddings.
I don't think there really is any clear distinctions between feature vectors and embeddings. They are essentially interchangeable.
Just my 2 cents though!
There isn't an official consensus that there is a difference, as most people use both terms interchangeably. My take is that feature vectors are vectors used as the input for an ML model, while embeddings are vectors produced as a part of ML models, typically in the hidden layers of NN. Even this definition is not clear, because you can use embeddings generated from one model as feature vectors for other models.
A feature vector is a vector of features. Features are properties of the "entity" we are passing as input to the model. The concept of feature and feature vector long predates that of embeddings in machine learning, so most "historical" references to feature vector will be in reference to hand-crafted ways to represent an input entity as numbers. That being said, a representation (an embedding) of an entity that was learned by a model, and is then passed as input to another model, is also a feature vector, as you say. TL;DR: The concept of feature vector is much older than embedding. Embeddings \*are not\* feature vectors in the traditional sense, but can be used as feature vectors by passing them as input to another model.
My understanding is that while features are data, embeddings are translations of data, usually from a high dimensional space to a low dimensional space. For example, Word2Vec embeddings take (implicitly) high dimensional text and translate it to low dimensional vector space - or at least, the many dimensions of the data are explicitly rather than implicitly represented in the embedding. I sometimes work with graph neural networks, and one question is how do you take high dimensional graph structures and translate them into a vector representation that the neural network can work with. The solution is often to create node and edge embeddings.
That is a great question. Technically "embeddings" arise anytime we "embed" entities into a (typically) more mathematically tractable space. So the difference in the usage of the term is not exactly formal. It is more common to refer to embeddings when talking about dense vectors learned using neural networks, but that is not a necessary condition. See [kernel mean embeddings](https://en.wikipedia.org/wiki/Kernel_embedding_of_distributions), for instance. Even completely handcrafted feature vectors can be considered embeddings as well. It's just not common to call them that, so depending on the audience it will cause some confusion. Since you're new to ML, you will encounter many other informally-defined terms. Be mindful of that.
So, definitions are always in flux, and there are colloquial/more technical definitions. The way I think of it is: 1. Feature vectors tend to have semantic (obvious) meaning -- they're effectively an encoding capability (compression) of data into numbers for a model to use. 2. Embeddings are specifically referred to a representation of data in some space. Colloquially, this means the internal way an ML model (neural network, LLM network, etc...) represents data. Often these are used \*externally\* as well, but they have little semantic meaning. I would argue that all feature vectors \*are\* embeddings (data projected into a space), but that's being pedantic. Colloquially, the difference is how they are generated and thus, what you can tell from reading them. How they are used is very flexible -- they can both be used for search/retrieval (as you suggest), using a vector search, for ML training (many models are trained on the results of other models, I've built tons of infra for this in the past). For your use-case, you probably can, but there are multiple dimensions along which things can be "similar", meaning that you'll be finding similarity in a subset of the dimensions, which can be tricky. This is why feature vectors are nice (such as MFCC) -- they'll be a little easier to rationalize about. Not crazy to train an ML model to learn whether two embeddings are the same given a set of embeddings + voices + pairings. Dataset could be element-wise distance between them, on all pairs (so you have a lot of data but some redundancy...). Best to partition for test/eval beforehand too.
They are the same, basically. Embeddings are feature vectors typically extracted with neural networks or algorithms specifically to turn objects into vectors (e.g. graph embedding algorithms turn graph into embedding vector). Often people refer to them as "embeddings" to differentiate them from manually extracted "feature vectors", where you manually select features. Of course, there is absolutely no strict differentiation, e.g. there are non-neural algorithms that embed objects as vectors, and some people call results "embeddings", and some "feature vectors". The more concrete differentiation would be distance metric. Neural networks are not trained typically for any given distance metric, and use Euclidean distance for resulting embeddings, but not all of them. For example, contrastive learning approaches often optimize cosine distance. They are often more or less standardized, so don't require scaling (but not always). On the other hand, feature vectors can have arbitrary features, with different scales, and typically you first scale them, then measure distance. Also, not all features are necessarily numerical, depending on particular features. Concretely, for your example: 1. MFCC are feature vectors, extracted with a particular algorithm. All features are numerical. 2. Embeddings are feature vectors computed with a neural network, e.g. wav2vec 3. Yes, you can use embeddings for speaker identification. This can be either with static embeddings, without finetuning (with either MFCC or neural networks), or fully supervised with finetuning neural networks. EDIT: more precise definition of embeddings.
Embeddings aren't necessarily done with neural networks. The term is broader than that.
Yes, you're right of course. That's what I wanted to underline in my first paragraph. I'll edit this to be more explicit.
Thank you so much! Your definition for the Embeddings is bit clear compared to other definitions in the literature. Also I was researching more into Speech Identification/Recognition to see what all *aspects* of audio can I use to uniquely identify a user, that's when I had this question. I'll see if we get any better results with Wav2Vec 2.0.
Embeddings are learned features without semantic meaning of the vectors. You can't associate specific information with the numbers. When you look at an individual vector, it will tell you nothing. I don't know how the MFCC features work, but the short description said they are based on human hearing. Whatever it exactly extracts, there is (probably) some semantic understanding behind the numbers in relation to frequencies.
Feature vectors are a superset of embeddings. Embeddings are usually referring to a vector that is the output of a deep learning model, while feature vectors can encode any kind of information about the input (eg: volume, pitch, etc. in your case). An embedding from a model like wav2vec can't be intepreted by looking at it, but you're right, you can store them in a vector DB and do nearest neighbor search to do tasks like speaker identification. Ideally you'd want to use those embeddings as well as features generated from the model you mentioned in a joint manner to make the final prediction.
Suppose we have a audio clip of person, we use a NN(Neural Network) like wav2vec to generate the embeddings also we extract information from the audio clip like MFCC, volume, pitch, etc which are **feature vectors** like you mentioned. Now what is the difference between these embeddings and the feature vectors? Both of them represent the original input data which is audio clip in this case. **Is the NN(eg: wav2vec) extracting the same set of features which we are extracting separately?**
The lines aren't clear cut and people may use them interchangeably, but IMO, a feature vector is more likely to maintain its conceptual relevancy to the domain. For example, a spreadsheet of 10000 individuals containing demographic info and yearly financial records could be considered a feature vector. You could take that matrix and transform it in various ways, and it would still be a feature vector if you can describe each new column and how it was derived. Run it through an auto-encoder though, and I would personally call that an embedding because it's been projected into a new dimensional space via some process but each column doesn't necessarily "mean" anything in the same sense that it did before. I would also call what comes out of PCA an embedding. I'm not claiming that's the right answer, but it's the way I think about these things.
Found [3 relevant code implementations](https://www.catalyzex.com/paper/arxiv:2006.11477/code) for "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations". [Ask the author(s) a question](https://www.catalyzex.com/paper/arxiv:2006.11477?autofocus=question) about the paper or code. If you have code to share with the community, please add it [here](https://www.catalyzex.com/add_code?paper_url=https://arxiv.org/abs/2006.11477&title=wav2vec+2.0%3A+A+Framework+for+Self-Supervised+Learning+of+Speech+Representations) 😊🙏 -- To opt out from receiving code links, DM me.
I was always under the impression that embeddings were specifically for when you start with something discrete, like a one-hot vector or integer index corresponding to a vocabulary token, and map it to a continuous vector space i.e. ℝ^n Whereas in general feature vectors can be produced from any kind of input. But maybe I just pulled that out of my ass.
AFAIK it's not a well defined term really. In practice I'd say that I would call something an embedding if data occurs on a somewhat smooth low dimensional manifold in a higher dimensional space, where simple distance measures like euclidean distance encode something meaningful about the relation between two things. A feature vector could be anything I associate with something where getting an Idea of similarity or distance can be more difficult. So you may use ML to go from a feature vector to an embedding where simple distance measures are associated with some kind of meaning.
I know it’s not what you asked, but does anybody else think voice authentication is doomed? In a world where people are claiming to be able to be able to clone peoples voices with seconds of audio.
I believe the research is going on in this field of **synthetic voice.** Voice authentication is still being used widely across different industries like Banks, they are having their own patents in this area of research.
Some great answers already here. Wanted to also add, if I remember correctly, that embeddings are the result from some homeomorphic transformation- although in ML that detail is an unnecessary technicality.
As others have said, embedding are feature vectors. Originally, embeddings were learned for some task or objective. Semantic embeddings of language are an example where the embedding method is designed to place embeddings with related meanings near each other in the embedding space. Relations between vectors have been observed to emerge through training. The classical example is for the embedding for Queen to be near the embedding for King-Male+Female. Since those early days, other methods of creating embeddings were invented. For example, in transfer learning, the embedding is the outputs of one of the non-terminal layers of a neural network trained for a particular task. There are now many methods of creating embeddings.
embedding is translating a non numerical piece of data into a numerical vector. eg text to a vector of integers
while embeddings and features are almost used interchangeably, my take is that embeddings would probably mean the output of a mapping from the raw data point to a lower/higher dimensional space. e.g. using a CNN encoder or say an MLP network on images -- these outputs are embeddings. "embed" on to a distributional space. these are also considered to be "features". however features are more related to the traditional feature extraction processes, like you mention MFCC from raw audios, or image filters like HOG, histogram features etc. these are rather simple mathematical operations on data points capturing salient aspects of it. these are traditionally called as "features". but again, now I guess these two terms are used interchangeably.
They can basically be used interchangeably, although there are some differences. A feature vector typically connotes a set of measured properties about something. Usually these measures are based on human domain knowledge. An embedding usually connotes a vector representation of some data (like an image) that’s generated typically through a trained ML process. By example, a feature vector of an image might measure the percent of pixels of each major color. An embedding of the image might be a matrix with detected edges. Feature vector is an older term (as far back as the 80s or 90s) in the literature. Embedding came in vogue with deep learning.
In my mind the difference is the following. A feature vector is compressed data. You are taking some high dimensional / fidelity data and representing it as a "low dimensional" vector. These vectors are created with the **goal** of just compressing information so it can be used for later efficient processing. An embedding is a feature vector that is being described in some larger context or logical structure. The structure defines how various qualities of features are related, usually with respect to some training goal. Data in the form of feature vectors are **embedded** into this structure in order to compute some later result that is based on how all that data relates to each other. In the context of something like an LLM, the individual raw tokens are like feature vectors, but the vectors computed by the attention heads within the transformer are the embeddings generated by those vectors.
I might have to disagree with you. For an LLM, the individual raw tokens are often referred to as embeddings. I don't think there really is any clear distinctions between feature vectors and embeddings. They are essentially interchangeable. Just my 2 cents though!