More sensationalism, playing fast and loose with numbers, incorrect interpretations, etc. in "Why We Sleep"

One of the more egregious errors in the book, which immediately caused me to raise my eyebrows, is in Chapter 8 in the section titled “Weight Gain and Obesity”.

Walker presents a plot (Figure 13) overlaying average sleep amount and percentage of obese individuals in the United States since ~1950 (without any citations of course), and remarks:

“people who sleep less are the same individuals who are more likely to be overweight or obese. Indeed, if you simply plot the reduction in sleep time over the past fifty years on the same graph as the rise in obesity rates across the same time period, shown in Figure 13, the data infer this relationship clearly.”

Overlaying two trendlines and “inferring” some correlation like this is completely unscientific nonsense, and is highly misleading. This is nothing more than the common correlation-causation fallacy, which any grad student would be ashamed to make, let alone an academic.

Upon my own reading I found that the book as a whole commits a number of cardinal sins of academic writing, which I would severely mark any student down for:
Insufficient Citations:
All throughout university we strongly emphasise to students the importance of correct citations and referencing. However, apparently this all goes out the window when you come to write a popular science book. I understand that this is sometimes due to publishers not wanting to include footnotes or citations for aesthetic or typesetting reasons, however the fact that there are some citations spinkled through the work suggests that this is more down to laziness or apathy.

Inappropriate Attribution:
In the book credit is often given to the research lead or project director rather than the lead author(s) of a mentioned study. For example, in Chapter 8 in the section “Sleep Loss, Genes, and DNA” he refers to “Dr. Derk-Jan Dikj, who directs the Surrey Sleep Research Center” and later states that “Dijk and his prolific team examined gene expression…” before going on to describe the results of a study (without citation again).

The study in question is “Effects of insufficient sleep on circadian rhythmicity and expression amplitude of the human blood transcriptome” (Moller-Levet, Archer, Bucca, et al.) [2013] {10.1073/pnas.1217154110}. However, Dijk is the last of eleven authors on the paper, which is lead-authored by Carla S. Möller-Levet. Indeed, looking at the author contributions, Dijk was not involved in performing the research or analyzing the data - and yet only his name appears in Walker’s book. Unfortunately, this speaks to deeper systemic problems in academia that occur all over the place. There is also a cruel irony in the fact that the grad students doing a lot of the (uncredited) hard work in this area are probably criminally sleep-deprived.

Playing Fast and Loose with Numbers:
Like other popular science writers Walker prioritises numerical aesthetics over numerical accuracy. For example, referring to the study cited above he states that:

“the activity of a hefty 711 genes was distorted […] about half of those 711 genes had been abnormally revved up […] while the other half had been diminished […]”.

Referring to the original paper on page 2 under the section “Effects of Sleep Restriction on the Blood Transcriptome” the 711 genes (which represented “3.1% of the genes determined as present in the arrays”), “444 were down-regulated and 267 were up-regulated following sleep restriction” - so 62.4% were up-regulated and 37.6% were down-regulated. So let’s just call it 50/50 I guess? While this doesn’t really alter the point he is trying to make, and is ultimately a bit pedantic on my part, it does speak to a willingness to massage numbers.

Incorrect Interpretations:
More troubling instances are where the numerical slopiness has a direct impact on the conclusions drawn. For example, in Chapter 8 in the section “Sleep Loss and the Immune System” he refers to work by Dr. Aric Prather regarding sleep and colds. He describes the results of the study (without citation) “Behaviorally Assessed Sleep and Susceptibility to the Common Cold” (Prather, Janicki-Deverts, Hall, Cohen) [2015] {10.5665/sleep.4968}. Walker states that:

“In those sleeping five hours on average, the infection rate was almost 50 percent”.

However, looking at the paper, of the 164 participants (Walker refers to this as “more than 150”) “124 (75.6%) were infected and 48 (29.3%) developed a biologically verified cold”. From page 4, Figure 1 of the paper, ~45% of those who slept LESS than 5 hours developed an objective cold (~16 out of 36 people), while ~30% of those who slept 5-6 hours (16 out of 54 people) developed a cold. So somehow 45% sleeping LESS than 5 hours transforms into 50% sleeping 5 hours ON AVERAGE. Speaking anecdotally, less than 5 hours of sleep is completely different than an average 5 hours of sleep

This also raises the issue of many of these studies using categorical ranges for fitting data, in which a person sleeping 5 hours 10 minutes is lumped in with a person sleeping 5 hours and 55 minutes, while a person sleeping 5 hours 55 minutes is in a completely different category from a person sleeping 6 hours and 5 minutes. Due to the small sample sizes used, if only a couple of people are on the boundary of two categories then how they are counted could completely change the results. For example, in the study above it would only take 4 people shifting from the <5 category to the 5-6 category for the infection rates to change from ~45% and ~30% respectively to ~33.33…% and ~37% respectively - destroying the “clear, linear relationship” referred to by Walker.

Of course, all of this does not mean that the results are incorrect, or that the conclusions of the original works are in any way erroneous, but when discussing the findings of any scientific work there is often a lot of nuance and important statistical qualification (particularly regarding sample sizes and error bounds) that simply gets lost in translation. I think it is incumbent upon popular science communicators to try to communicate some of this nuance, and improve the overall numerical literacy of the public, rather than simply resorting to a set of easily digestible, often sensationalised, “facts” that are stripped of all caveats.

Sensationalism:
For example, take Chapter 8 again in the section titled “Sleep Loss and the Reproductive System”. In this section Walker implies that getting 5 hours of sleep for a week

“effectively ‘ages’ a man by ten to fifteen years in terms of testosterone virility”.

He has made this bold claim in a number of his talks as well. He gives no citation for this fact, except for mentioning that it was done at the University of Chicago. Upon researching this, it appears he is referencing a Research Letter in JAMA: “Effect of 1 Week of Sleep Restriction on Testosterone Levels in Young Healthy Men” (Leproult, Van Cauter) [2011] {10.1001/jama.2011.710}. This uses an extremely small sample of only 10 men who spent “1 week of 8-hour bedtimes” at home, followed by 3 nights of 10 hours in the lab, followed by 8 nights of 5 hours in the lab (more than a week). In their letter they state that

“Daytime testosterone levels were decreased by 10% to 15% in this small convenience sample of young healthy men who underwent 1 week of sleep restriction to 5 hours per night”.

So how does Walker get to ‘aging’ by ten to fifteen years? Well, in the comment it also states that “By comparison, normal aging is associated with a decrease of testosterone levels by 1% to 2% per year.” So, by simple division, 10 to 15% becomes 10 to 15 years. Of course, it could just as easily be 6.7 to 10 years, or 5 to 7.5 years, or whatever number you want frankly given the incredibly small sample size that, unsurprisingly, is never mentioned in the book.

4 Likes

Chapter 8 also contains another erroneous description of an existing study. In the section “Sleep Loss and the Reproductive System” Walker refers to work by Tina Sundelin on “how attractive you look when sleep-deprived”. This appears* to refer to the study “Beauty sleep: experimental study on the perceived health and attractiveness of sleep deprived people” (Axelsson, Sundelin, Ingre, Van Someren) [2010] {10.1136/bmj.c6614}.

Walker states that:

“In one of the sessions, the participants were given just five hours of sleep before being put in front of the camera, while in the other session, these same individuals got a full eight hours of sleep.”

He goes on to say:

“The faces pictured after one night of short sleep were rated as looking more fatigued, less healthy, and significantly less attractive.”

However, in the cited paper it says that the researchers

“photographed the faces of 23 adults (mean age 23, range 18-31 years, 11 women) between 14.00 and 15.00 under two conditions in a balanced design: after a normal night’s sleep (at least eight hours of sleep between 23.00-07.00 and seven hours of wakefulness) and after sleep deprivation (sleep 02.00-07.00 and 31 hours of wakefulness)”

Walker completely fails to mention that the individuals who received five hours of sleep were also sleep deprived for 31 hours. This fact is even directly stated in the abstract of the paper:

“Participants were photographed after a normal night’s sleep (eight hours) and after sleep deprivation (31 hours of wakefulness after a night of reduced sleep)”

This completely changes the interpretation of the paper. Being able to tell visually if someone has had 5 hours of sleep is considerably different from being able to tell if they have had five hours of sleep AND have been sleep deprived for 31 hours.

*Note that there are a number of other works carried out by Sundelin and her colleagues that involve photographs of sleep deprived individuals. However, only the “Beauty sleep” paper uses specifically the three features mentioned as part of the study by Walker (“perceived health, tiredness, and attractiveness”). Furthermore, all of the other works by Sundelin also involve some form of sleep deprivation, and none of them involve just 5 hours of sleep without any sleep deprivation, so it is not simply a case of two different studies being mixed up or confused. In addition, Walker writes that Sundelin “ratified the long-held concept of ‘beauty sleep’”, so it seems extremely likely that the “Beauty sleep” paper is the correct paper despite the erroneous description by Walker.

The other relevant works by Sundelin involving sleep and photographs:
"Cues of Fatigue: Effects of sleep deprivation on facial appearance" (Sundelin, Lekander, Kecklund, Van Someren, Olsson, Axelsson) [2013] {10.5665/sleep.2964}
This study “consisted of 10 individuals (five women) photographed at 14:30 after normal sleep and after 31 h of sleep deprivation following a night with 5 h of sleep.” This is the same sleep regiment of sleep deprivation as “Beauty sleep”, but it involves different assessment criteria from those mentioned by Walker, namely precise physical descriptors such as “hanging eyelids, redder eyes, more swollen eyes, darker circles under the eyes, paler skin, more wrinkles/fine lines, and more droopy corners of the mouth”

"Negative effects of restricted sleep on facial appearance and social appeal" (Sundelin, Lekander, Sorjonen, Axelsson1) [2017] {10.1098/rsos.160918}
This study involved the participants “being instructed to sleep for 8 h for two consecutive nights and once after only being allowed 4 h in bed for two consecutive nights”. This involved different criteria including sociability, trustworthiness, attractiveness, health. It was also published in 2017.

"The effect of sleep deprivation on objective and subjective measures of facial appearance" (Holding, Sundelin, Cairns, Perrett, Axelsson) [2019] {10.1111/jsr.12860}
This involved “either one night of total sleep deprivation (n = 91) or a night of normal sleep at home (n = 90)”. This was also published after Why We Sleep.

2 Likes

Slightly less important, but something I had spotted in my reading of the book:

In some species, many of the birds in a flock will sleep with both halves of the brain at the same time. How do they remain safe from threat? The answer is truly ingenious. The flock will first line up in a row. With the exception of the birds at each end of the line, the rest of the group will allow both halves of the brain to indulge in sleep. Those at the far left and right ends of the row aren’t so lucky. They will enter deep sleep with just one half of the brain (opposing in each), leaving the corresponding left and right eye of each bird wide open. In doing so, they provide full panoramic threat detection for the entire group, maximizing the total number of brain halves that can sleep within the flock. At some point, the two end-guards will stand up, rotate 180 degrees, and sit back down, allowing the other side of their respective brains to enter deep sleep.

I have had difficulty tracking this down. Everything seems to lead back to this study, which put the ducks in a row. I’m not sure if they do this in the wild? It seems like they would stand in groups in the wild, not in rows. https://www.nature.com/articles/17037

Would love to learn more about this if anyone is aware of other sources.

From the study:

We established four groups of mallard ducks (Anas platyrhynchos), each containing four individuals. Video recordings of sleep behaviour were made with the birds arranged in a row.

I didn’t spend too long looking into it, but all the issues spotted so far suggest to me that it is possible the paper was misremembered by the author, and that ducks do not in fact do this in the wild! (although maybe they do! would be fascinating if they did)

I don’t suppose anyone knows if ducks do in fact do this in the wild? I wonder if it is just common knowledge amongst ornithologists. The whole ‘stand up and turn around’ part is completely absent from the study, so maybe there is a different one that I failed to find!

1 Like

It appears that this is the correct study. There was a Radiolab episode (https://www.wnycstudios.org/podcasts/radiolab/episodes/91528-sleep) back in 2007 about sleep that interviewed one of the authors (Charles Amlaner), and the description matches up closely with that given by Walker. From the transcript of the interview (square brackets are used to demarcate interruptions from the host):

We put four birds in a row. [Four mallard ducks, this time in the lab and they watched them sleep.] The two birds in the center of this row slept with both eyes closed. The birds on the outer edges, both left and right, slept with one eye closed and one eye opened.

And later on (emphasis my own):

"What happened is that the birds that were on the outer edge, then would rotate [stand up, turn around] 180 degrees. [And then sit back down.] And the other eye would then get some sleep and consequently the opposite hemisphere would get some sleep

The statement about the ducks standing up and turning around 180 degrees is not in the original research paper, or the Nature communication, and only seems to appear in interviews such as this, or in blog posts or books that refer to it, so it is likely that Walker read about it from a second hand source (or he could always have spoken with the authors directly).

Regarding whether they do this in the wild, earlier in the podcast Amlaner states that they performed the mallard experiment after:

We observed that ducks sometimes will get together into groups. [Like, on a log. Four ducks will get together and snooze in a neat little line.]

More details about the experiment can be found in the original research paper “Facultative control of avian unihemispheric sleep under the risk of predation” (Rattenborg, Lima, Amlaner) [1999] {10.1016/S0166-4328(99)00070-4}. In this paper the authors also state that:

Video recordings of sleep behavior were performed with groups of four birds arranged in a row, an orientation frequently observed in the wild when mallards sleep on floating logs (pers. observ.)

So the statement that “the flock will first line up in a row” before going to sleep is a little bit misleading, as it appears to be due to the environment.

An earlier pair of studies “Sleeping and vigilance in birds I. Field observations of the mallard (Anas platyrhynchos)” (Lendrem) [1983] {10.1016/S0003-3472(83)80076-1} and “Sleeping and vigilance in birds II. An experimental study of the Barbary dove” (Streptopella risoria) (Lendrem) [1984]{10.1016/S0003-3472(84)80343-7} which were about peeking rather than unihemispheric slow-wave sleep, found that vigilance in the birds, when observed in the wild, was correlated with how at risk the birds were, such as their distance from the shore, or the size of the flock and their position within it.

Interestingly, this paper cites “Pre-dusk and nocturnal behaviour of Goldeneye, with notes on population composition” (Linsell) [1969], which describes how they observed sea ducks in the wild “patrolling” the flock at regular intervals while the others slept (ostensibly to protect them from threats):

A little later the most remarkable form of behaviour was observed. Two ‘brown-heads’ were seen to leave the oval end of the raft and swim its entire length, one on either side. On reaching the last bird they turned about and swam back to the oval end. The ‘patrolling’ was carried out, always by two ‘brown-heads’ at regular intervals of about a quarter of an hour throughout the night.

A fun little observation to find in the literature.

I added excerpts from your comments here: https://guzey.com/books/why-we-sleep/#appendix-serious-problems-in-chapter-8-found-by-a-reader

@ish wonderful, thank you. The transcript does closely match the claims in the book. I’m happy enough with that!

The fact that they would be in a line makes sense when they are perched on a log, also. I hadn’t thought of that possibility.

Unrelated, Bill Gates has recommended the book in his end of year book suggestion list: https://www.gatesnotes.com/About-Bill-Gates/Holiday-Books-2019