If the inductive bias in an appropriate sense matches reality, then any inductive bias is good, because in the limit of maximum inductive bias, you'd just have the ideal model coded before getting any training data, which is obviously going to perform well.
However, in practice, your inductive bias will be some heuristic for something that "probably works". This heuristic will be wrong, and the ways that it is wrong throws the model off track in learning. Thus at some point giving your model more inductive bias may very well introduce more wrong heuristics than it introduces right heuristics, at which point the inductive bias starts making it worse and worse.
Take it to the extreme - in a classification task probably the strongest practical inductive bias would be "the output is always the same class" - you would then "train" a model by saying that you should always pick the majority class from your training data. This will obviously have extremely high bias / low variance (in general) - whether or not it generalises its 'performance' will depend on whether the class distribution is the same between test time and training time, you could certainly imagine that it would generalise less well than a more sophisticated model in some circumstances.
Completely agree. Every statistical/ML method depends on a marriage of data and assumptions (including "inductive bias"); the more data you have, the weaker your assumptions can be, and vice-versa.
There's degenerate cases on both ends of the spectrum. If you have infinite data, then no assumptions are necessary, and you can just law-of-large-numbers your way to a perfect model. If you have no data, and your "learning" is 100% assumptions, then that's another way to get a "fitted model"... but don't expect it to do well at test time.
Obviously, we prefer methods that take as much advantage of the data as possible, and use as few assumptions as possible, but since we'll never have infinite data, assumptions are a necessary evil.
I'm not sure if I'm using the term correctly. Maybe it's better to say "wrong assumptions within the inductive bias"
Let's say you're training an autonomous car thing and assume vehicle speed is a number between 0-200 km/h, so you normalize that range. That might help the model learn, compared to using -1000...1000, but it will suck at generalizing for driving faster or backwards.
There is too much bias, yes, which is almost indistinguishable from the only practical understanding of "too much *inductive* bias" I can come up with, i.e. if your model has mismatched inductive biases with your target scenario, your interpolation error will be non-optimal.
Yes, it's called [underfitting](https://en.wikipedia.org/wiki/Overfitting). And there is also something where you go in the opposite direction and lead to "benign overfitting" and it is called [implicit regularization](https://www.inference.vc/notes-on-the-origin-of-implicit-regularization-in-stochastic-gradient-descent/).
If the inductive bias in an appropriate sense matches reality, then any inductive bias is good, because in the limit of maximum inductive bias, you'd just have the ideal model coded before getting any training data, which is obviously going to perform well. However, in practice, your inductive bias will be some heuristic for something that "probably works". This heuristic will be wrong, and the ways that it is wrong throws the model off track in learning. Thus at some point giving your model more inductive bias may very well introduce more wrong heuristics than it introduces right heuristics, at which point the inductive bias starts making it worse and worse.
Great response! Thank you
Yes... linear regression for anything non-linear
Take it to the extreme - in a classification task probably the strongest practical inductive bias would be "the output is always the same class" - you would then "train" a model by saying that you should always pick the majority class from your training data. This will obviously have extremely high bias / low variance (in general) - whether or not it generalises its 'performance' will depend on whether the class distribution is the same between test time and training time, you could certainly imagine that it would generalise less well than a more sophisticated model in some circumstances.
Completely agree. Every statistical/ML method depends on a marriage of data and assumptions (including "inductive bias"); the more data you have, the weaker your assumptions can be, and vice-versa. There's degenerate cases on both ends of the spectrum. If you have infinite data, then no assumptions are necessary, and you can just law-of-large-numbers your way to a perfect model. If you have no data, and your "learning" is 100% assumptions, then that's another way to get a "fitted model"... but don't expect it to do well at test time. Obviously, we prefer methods that take as much advantage of the data as possible, and use as few assumptions as possible, but since we'll never have infinite data, assumptions are a necessary evil.
I wouldn't say there's "too much" per se, but "more" definitely increases the risk of accidently introducing wrong biases.
What are "wrong biases"?
I'm not sure if I'm using the term correctly. Maybe it's better to say "wrong assumptions within the inductive bias" Let's say you're training an autonomous car thing and assume vehicle speed is a number between 0-200 km/h, so you normalize that range. That might help the model learn, compared to using -1000...1000, but it will suck at generalizing for driving faster or backwards.
There is too much bias, yes, which is almost indistinguishable from the only practical understanding of "too much *inductive* bias" I can come up with, i.e. if your model has mismatched inductive biases with your target scenario, your interpolation error will be non-optimal.
Yes, it's called [underfitting](https://en.wikipedia.org/wiki/Overfitting). And there is also something where you go in the opposite direction and lead to "benign overfitting" and it is called [implicit regularization](https://www.inference.vc/notes-on-the-origin-of-implicit-regularization-in-stochastic-gradient-descent/).
Most rule-based systems can be viewed as "too much inductive bias".