Testing the Tests: COVID-19 Antibody Assays Scrutinized for Accuracy by UCSF, UC Berkeley Researchers

As the United States and much of the world move toward relaxing shelter-in-place restrictions to let people move about more freely, public health experts hope to rely on antibody tests to determine who has been infected with the COVID-19 virus and may be immune – at least temporarily – and who is still susceptible.

A project launched by UC San Francisco and UC Berkeley scientists to evaluate some of the more than 120 available antibody test kits – only a handful of which have received Emergency Use Authorization from the Food and Drug Administration – should provide the test performance data these doctors and public health officials need to decide which tests to employ and to understand how reliable the results are. The work is supported by the Chan Zuckerberg Biohub (CZ Biohub), Anthem Blue Cross Blue Shield and other local donors.

In head-to-head comparisons of a dozen tests, the researchers have already found that many of the tests performed reasonably well, especially two weeks or more after infection, when levels of antibodies in the blood begin to peak. But many of the test kits have false positive rates that may exceed the proportion of people who have been infected in some communities. That means that a large proportion of those testing positive on an antibody test may not actually have had COVID-19.

Some government officials have suggested providing those who test positive with a get-out-of-jail-free card – an “immunity passport” to a normal life. However, additional critical information is still required before assuming that antibody tests can safely predict protection from future infections, the researchers caution.

“It's the Wild West right now. These tests are widely available and many people are buying and deploying them, but I realized that they had not been systematically validated and we needed to figure out which ones would really work,” said Patrick Hsu, PhD, an assistant professor of bioengineering at UC Berkeley and investigator at the Innovative Genomics Institute, a joint research collaboration between the two campuses focused on CRISPR. “This is a huge unmet need for public health.”

Hsu is leading the effort with Alex Marson, MD, PhD, an associate professor of microbiology and immunology at UCSF and scientific director for biomedicine at the IGI, Caryn Bern, MD, MPH, a professor of epidemiology and biostatistics at UCSF, and Jeffrey Whitman, MD, a clinical fellow in pathology at UCSF and a resident in laboratory medicine. Marson is also a CZ Biohub investigator.

They have posted their first results online in a paper they will submit to a journal, and will continue to update their website so that state and federal policymakers have the information they need before purchasing serology tests. The team cautions, however, that the paper is a preliminary report of work that has not been certified by peer review, and should not be relied upon to guide clinical practice or health-related behavior or reported in news media as established information.

Antibody Tests Complement PCR Diagnostics

Current diagnostic tests, such as the standard RT-PCR (reverse transcription-polymerase chain reaction) test conducted on samples obtained from nasopharyngeal swabs, can tell doctors if someone is currently infected, but antibody tests might be able identify people who have been exposed to the virus even weeks after an initial infection. Antibody tests could be particularly useful for identifying those who have been infected but never showed symptoms. Some controversial studies have suggested that the proportion of such cases could be as high as one in four.

When infected by a virus like SARS-CoV-2, the cause of COVID-19, the body initially produces antibodies known as IgM (immunoglobulin-M), in an attempt to neutralize the virus. Later, as the body’s adaptive immune system revs up, IgM levels go down and the body ramps up production of IgG, which more specifically targets the viral invader.

Antibody tests, also called serology tests because they are conducted on blood samples, such as from a finger prick, can assess levels of both IgM and IgG, and the relative levels could indicate whether a person is in the early or late stages of infection.

As such, antibody tests can complement the information from PCR tests, since even these relatively accurate tests can give false negatives. PCR tests for coronavirus generally exhibit lower sensitivity if performed several days to a week after symptom onset, probably because of decreasing levels of virus in the upper respiratory tract where samples for testing are commonly taken.

Antibody tests may eventually provide clues to how long immunity lasts, and what levels of antibodies are truly protective from subsequent SARS-CoV-2 infections. It remains unclear whether infection with SARS-CoV-2 produces long-lasting immunity.

The UCSF/UC Berkeley team has so far evaluated 10 point-of-care tests – tests much like home pregnancy or HIV tests, which are called, in general, lateral flow assays – and two different set of tests based on a common laboratory antibody detection method called ELISA (enzyme-linked immunosorbent assay).

Each of the dozen was tested against roughly 300 blood samples. Of these, 108 samples were obtained before July 2018, so presumably came from people who could not have contracted COVID-19. Most of the remainder came from COVID-19 patients seen at Zuckerberg San Francisco General Hospital (ZSFG) or UCSF Medical Center. About 130 samples were from individuals who had tested positive for COVID-19 by PCR testing, and around 50 were from people had been tested for other viruses.

“We have carefully curated the specimens so that we can systematically study how these different antibody tests perform at different times since symptom onset, and across many samples, making our study one of the most comprehensive to date,” Hsu said.

Nevertheless, the team is hampered by the lack of a definitive antibody test with which to compare the many new test kits on the market.

“One of the cornerstones of lab medicine is that a new test is compared to a definitive reference or gold standard,” Marson said. “We do not have a gold standard yet for COVID-19 serology testing, so we are amassing data on a standardized set of blood samples and really looking at how each of these tests performs in relationship to all the others.”

Sensitivity Versus Specificity

The COVID-19 patient samples represented blood taken at various stages of illness starting about three days after initial onset of symptoms. Patients ranged in age from 22 to over 90, and were primarily (69 percent) of Hispanic/Latinx ethnicity – the demographic largely served by ZSFG, and one of the segments of San Francisco’s population hit hardest by COVID-19.

Each test was assessed for sensitivity – how likely it is to detect antibodies in the blood of coronavirus-positive patients – and for specificity – how good they are at distinguishing people who are infected from those who are not. A high specificity means a low false positive rate.

Of the 10 point-of-care tests, “there are multiple tests that have specificities greater than 95 percent. So there is some reason for guarded optimism,” Marson said. “Although it is important to point out that if these infections are rare in a population, a false positive rate of 5 percent could cloud the picture of the information coming in.”

“Several of our tests had specificities over 98 percent, which is critical for reopening society,” Hsu added.

The extent to which positive results by serology may reflect a protective immune response will require further study, Hsu emphasized. “More research is needed to understand if antibody assays can be used as predictors of protection against reinfection and to prioritize return to work.”

Hsu and Marson noted that, while point-of-care tests are meant to be yes/no tests – either you have antibodies or you don’t – they actually display positive results within a range that can be helpful in judging how certain the results may be. Doctors can set a high bar – a darker band on the dipstick-like tests – to be more certain of positive results, but at the expense of losing sensitivity to small antibody levels. “Many of the false positives were associated with fainter bands,” Marson noted.

“Accurate use of these tests will depend on adequate training of test readers,” Hsu said. The researchers are now working on methods to standardize rapid serology test interpretation, which could eventually be used with a cellphone camera.

The UCSF/Berkeley team also worked with researchers at Massachusetts General Hospital (MGH) who had independently assessed three antibody kits. One of those overlapped with the Bay Area team’s set of tests, and researchers at MGH confirmed their assessment of it.

In one of the weekly Zoom meetings within IGI, Hsu learned of Marson’s interest in the accuracy of antibody tests, and they teamed up to acquire and evaluate test kits. They recruited Whitman and Bern, who have experience evaluating antibody test kits for other diseases, and Whitman was able to acquire patient blood samples from ZSFG and UCSF, and access to lab equipment in that hospital’s Department of Experimental Medicine.

Along with a group of graduate students and postdoctoral researchers from UCSF and UC Berkeley, the team has been working around the clock to acquire new test kits, run the assays, and evaluate the results. They continue to expand the number of tests they are evaluating, and are obtaining new blood samples from patients who have recovered from COVID-19 to look at antibody levels long after recovery. With this information, they can determine how long antibodies persist and how that correlates with both the severity of the illness and subsequent immunity.

“This is a huge, huge community effort,” Hsu said. “A lot of people really came together. One of the things I think is cool about this study is how many people repurposed themselves from what we normally do to respond to this pandemic. Personally, I find it very inspiring.”

About UCSF: The University of California, San Francisco (UCSF) is exclusively focused on the health sciences and is dedicated to promoting health worldwide through advanced biomedical research, graduate-level education in the life sciences and health professions, and excellence in patient care. UCSF Health, which serves as UCSF's primary academic medical center, includes top-ranked specialty hospitals and other clinical programs, and has affiliations throughout the Bay Area.

About UC Berkeley: Founded in 1868, the University of California, Berkeley, is an iconic top-ranking public research university offering some 350 undergraduate and graduate degree programs ranging from the arts and humanities to the sciences and technology. It is committed to innovation, social justice and the greater good.