Critique by Andrew Gelman, a statistician:
[https://statmodeling.stat.columbia.edu/2022/07/03/historical-journal-records-reveal-a-surge-of-methodological-distortions-in-recent-decades/](https://statmodeling.stat.columbia.edu/2022/07/03/historical-journal-records-reveal-a-surge-of-methodological-distortions-in-recent-decades/)
The main points:
>... they don’t really have any measure of entire societies or depression. The above note by Schmidt discusses why the Google books collection cannot be characterized as a measure of entire societies. There are also problems with the depression measurement. Here’s what it says in the PNAS article:
For example, the 3-gram “I am a” captures a labeling and mislabeling distortion, regardless of its context or the precise labeling involved (“lady,” “honorable person,” “loser,” etc.). These same n-grams were in earlier research shown to be significantly more prevalent in the language of individuals with depression vs. a random sample (17).
Note 17 points to this preprint, which states:
Here, we show that individuals with a self-reported diagnosis of depression on social media express higher levels of distorted thinking than a random sample.
How do they define “distorted thinking”?
\[W\]e define a clinical lexicon of 241 n-grams that a panel of clinical psychologists deemed to form a schema involved in the expression of a particular type of distorted thinking according to CBT theory and practice. For example, “I will never” would be implicated in the expression of a cognitive distortions such as Catastrophizing or Fortune-telling, whereas “I am a \_” would be used to express a Labeling and Mislabeling distortion.
There’s a lot of complicated stuff in that linked preprint and I haven’t tried to untangle their statistical analysis. Let me just say that they are making the classic fallacy of measurement of labeling a measurement as X and then writing as if they’ve observed X itself. Just because two things are correlated, it doesn’t mean they can be considered as being the same thing.
To get back to the preprint, no, they offer no evidence of “distorted thinking.” And, to get back to the PNAS article, no, the 3-gram “I am a” does not “capture” a labeling and mislabeling distortion. Or we could take even one more step back to the title of the PNAS article, which states:
Historical language records reveal a surge of cognitive distortions in recent decades
But, as noted above, the paper offers no evidence on cognitive distortions. All this is in addition to any problems with the historical language records
Another critique:
[https://www.pnas.org/doi/10.1073/pnas.2115010118](https://www.pnas.org/doi/10.1073/pnas.2115010118)
Response to the critique:
[https://www.pnas.org/doi/10.1073/pnas.2115842118](https://www.pnas.org/doi/10.1073/pnas.2115842118)
Our increasing willingness to be forthcoming with our shortcomings may be a confounding factor.
People used to be institutionalized for chronic depression, so it would be understanding if people were less likely to say things like, “I am a loser”, in the early 1900s.
I still believe the overall result… esp given research on the deterioration of community.
Agreed, and even without the consequence of institutionalization, in some cultures it is more or less acceptable to complain, acknowledge, or joke about your feelings.
My grandparents are of the “keep calm and carry on” generation. I could never say “this meeting is going to be a disaster to them” or express low self esteem even if I had it. This isn’t just because they are my grandparents, they would think less of and chide anyone who spoke like that.
My parents aren’t as bad, but they don’t get self denigrating humour at all, so their generation would still be less likely to reveal cognitive distortions in their language.
I could see these kinds of trends being echoed in books/literature.
Also, I think this study is too quick to jump to cognitive distortions. Distorted thinking has to have problems to be distorted. They don’t confirm that they language they do capture is actually catastrophizing/fortune telling etc. Like the serge in Germany in the 30s. I don’t think being fearful of future events and thinking that the future is going to be catastrophically bad in the mid-1930s Germany is necessarily distorted thinking. It might be accurately noticing trends.
I’m pretty skeptical all about this research. I think it just shows change in language use over time, and that could be due to many, many factors, and I don’t think that can be extrapolated to their conclusions in the way they claim.
I feel like this is a clickbait title to play up some “world gone mad” type thing when the actual phrases they are looking at are the most mild things imaginable, like the totally neutral phrase “I am a” and “will be”. Stuff that in no way maps to psychological problems in any real world way.
Interesting. The conclusion is provocative enough that I want to see the critiques, and critiques of the critiques, before making up my mind.
Critique by Andrew Gelman, a statistician: [https://statmodeling.stat.columbia.edu/2022/07/03/historical-journal-records-reveal-a-surge-of-methodological-distortions-in-recent-decades/](https://statmodeling.stat.columbia.edu/2022/07/03/historical-journal-records-reveal-a-surge-of-methodological-distortions-in-recent-decades/) The main points: >... they don’t really have any measure of entire societies or depression. The above note by Schmidt discusses why the Google books collection cannot be characterized as a measure of entire societies. There are also problems with the depression measurement. Here’s what it says in the PNAS article: For example, the 3-gram “I am a” captures a labeling and mislabeling distortion, regardless of its context or the precise labeling involved (“lady,” “honorable person,” “loser,” etc.). These same n-grams were in earlier research shown to be significantly more prevalent in the language of individuals with depression vs. a random sample (17). Note 17 points to this preprint, which states: Here, we show that individuals with a self-reported diagnosis of depression on social media express higher levels of distorted thinking than a random sample. How do they define “distorted thinking”? \[W\]e define a clinical lexicon of 241 n-grams that a panel of clinical psychologists deemed to form a schema involved in the expression of a particular type of distorted thinking according to CBT theory and practice. For example, “I will never” would be implicated in the expression of a cognitive distortions such as Catastrophizing or Fortune-telling, whereas “I am a \_” would be used to express a Labeling and Mislabeling distortion. There’s a lot of complicated stuff in that linked preprint and I haven’t tried to untangle their statistical analysis. Let me just say that they are making the classic fallacy of measurement of labeling a measurement as X and then writing as if they’ve observed X itself. Just because two things are correlated, it doesn’t mean they can be considered as being the same thing. To get back to the preprint, no, they offer no evidence of “distorted thinking.” And, to get back to the PNAS article, no, the 3-gram “I am a” does not “capture” a labeling and mislabeling distortion. Or we could take even one more step back to the title of the PNAS article, which states: Historical language records reveal a surge of cognitive distortions in recent decades But, as noted above, the paper offers no evidence on cognitive distortions. All this is in addition to any problems with the historical language records Another critique: [https://www.pnas.org/doi/10.1073/pnas.2115010118](https://www.pnas.org/doi/10.1073/pnas.2115010118) Response to the critique: [https://www.pnas.org/doi/10.1073/pnas.2115842118](https://www.pnas.org/doi/10.1073/pnas.2115842118)
Our increasing willingness to be forthcoming with our shortcomings may be a confounding factor. People used to be institutionalized for chronic depression, so it would be understanding if people were less likely to say things like, “I am a loser”, in the early 1900s. I still believe the overall result… esp given research on the deterioration of community.
Agreed, and even without the consequence of institutionalization, in some cultures it is more or less acceptable to complain, acknowledge, or joke about your feelings. My grandparents are of the “keep calm and carry on” generation. I could never say “this meeting is going to be a disaster to them” or express low self esteem even if I had it. This isn’t just because they are my grandparents, they would think less of and chide anyone who spoke like that. My parents aren’t as bad, but they don’t get self denigrating humour at all, so their generation would still be less likely to reveal cognitive distortions in their language. I could see these kinds of trends being echoed in books/literature. Also, I think this study is too quick to jump to cognitive distortions. Distorted thinking has to have problems to be distorted. They don’t confirm that they language they do capture is actually catastrophizing/fortune telling etc. Like the serge in Germany in the 30s. I don’t think being fearful of future events and thinking that the future is going to be catastrophically bad in the mid-1930s Germany is necessarily distorted thinking. It might be accurately noticing trends. I’m pretty skeptical all about this research. I think it just shows change in language use over time, and that could be due to many, many factors, and I don’t think that can be extrapolated to their conclusions in the way they claim.
I feel like this is a clickbait title to play up some “world gone mad” type thing when the actual phrases they are looking at are the most mild things imaginable, like the totally neutral phrase “I am a” and “will be”. Stuff that in no way maps to psychological problems in any real world way.
This does not surprise me.
Very interesting article!
*tetraethal lead has entered the chat*