New study examines how sense of smell affects mating and aggression in mice

Cover of December 2005 issue of Nature Neuroscience -----

New research by scientists at UCSF sheds light on how the odor detecting system in mice sends signals that affect their social behavior.

“Understanding how mice process cues from the olfactory system-which regulates the sense of smell—should provide insight into the fundamental principles that mammalian brains use to transform sensory information into behavior,” says lead investigator Nirao Shah, MD, PhD, UCSF assistant professor of anatomy.

“There are striking genetic and neuroanatomic similarities between mice and humans. We hope that such basic knowledge of how the brain functions will eventually be useful in understanding how the human brain generates behaviors in humans,” he adds.

The UCSF study focused on the sexual and aggressive behaviors triggered by the rodent’s olfactory system. The findings are published in the December issue of Nature Neuroscience. 

According to Shah, researchers traditionally have thought rodents detect pheromones through a specialized organ-the vomeronasal organ (VNO) in the nose-that is separate from the main olfactory system. Pheromones are olfactory cues that signal the social and sexual status of individuals of a species, and their detection is a key step in regulating behaviors such as mating and aggression.

The new study findings show, however, that male mice require intact functioning of the main olfactory epithelium (MOE) in order to detect pheromones that elicit sexual behavior and fighting. The MOE covers the olfactory region of the nasal cavity and contains sensory neurons that recognize odors and transmit this information to the brain.

The study is important, says Shah, because it establishes a novel and hitherto unsuspected role for the MOE in regulating mating and aggressive behavior in mice. He adds that while it appears humans do not have an intact VNO, they do possess a functioning MOE. 

In the study, researchers compared the behavior of mutant male mice that had been genetically modified to disrupt the functioning of their MOE with the behavior of normal mice, also known as wild-type mice. Wild-type female and male mice were placed at separate times in cages with the male mutants.

When comparing the mutant males with wild-type males, study results showed that all wild-type males mated with test females, while none of the mutants engaged in sexual behavior with the females. “Typically, when you take a female and put her in a cage, the male will initially sniff her, a process known as chemoinvestigation, and then mate with her,” Shah says. “But the mutants without the MOE even showed a profound defect in initiating chemoinvestigation of the test females.”

When a wild-type male encounters another male, he will typically sniff the male and then initiate a fight. However, the mutant males were defective in sniffing the test wild-type males, and they did not attack the wild-type males, Shah notes. “This suggests a broad and essential role for the MOE in regulating mating and aggression.”

More research is needed to understand the role of the olfactory system and human behavior, but clearly this is an important connection for people, according to Shah.  “The booming perfume industry attests to our belief that odors can attract potential suitors, and there are some studies in humans that suggest pheromone-type signals might have subtle effects on the regulation of certain physiological functions, such as the menstrual cycle.”

Stimulation of the other senses also has a direct effect on behavior, he adds. “When you touch a hot pan, you immediately draw your hand away. Or, if you place your finger in an infant’s palm, the baby will grasp it instinctively.”

The current research findings might also be relevant for controlling pest populations, which utilize olfactory cues in a manner similar to mice, according to Shah. 

The study was co-authored by Vidya S. Mandiyan, BA, UCSF staff research associate and Jennifer K. Coats, BS, UCSF graduate student researcher. 

The research was supported by the National Institutes of Health, the Burroughs Wellcome Fund, the Sandler Family Supporting Foundation, and the Research Evaluation and Allocation Committee at UCSF.

UCSF is a leading university that consistently defines health care worldwide by conducting advanced biomedical research, educating graduate students in the life sciences, and providing complex patient care.