International Journal of Infectious Diseases 131 (2023) 111–114 Contents lists available at ScienceDirect International Journal of Infectious Diseases journal homepage: www.elsevier.com/locate/ijid Perspective Observed negative vaccine effectiveness could be the canary in the coal mine for biases in observational COVID-19 studies Korryn Bodner 1 , ∗, Michael A. Irvine 2 , 3 , Jeffrey C. Kwong 4 , 5 , 6 , 7 , 8 , 9 , 10 , Sharmistha Mishra 1 , 5 , 6 , 11 , 12 , 13 1 MAP Centre for Urban Health Solutions, St. Michael’s Hospital, Unity Health Toronto, Toronto, Canada 2 British Columbia Centre for Disease Control, Vancouver, Canada 3 Health Sciences, Simon Fraser University, Burnaby, Canada 4 Public Health Ontario, Toronto, Canada 5 ICES, Toronto, Canada 6 Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, Toronto, Canada 7 Clinical Public Health Division, Dalla Lana School of Public Health, University of Toronto, Toronto, Canada 8 Centre for Vaccine Preventable Diseases, University of Toronto, Toronto, Canada 9 Department of Family and Community Medicine, University of Toronto, Toronto, Canada 10 University Health Network, Toronto, Canada 11 Institute of Medical Science, University of Toronto, Toronto, Canada 12 Institute of Health Policy, Management and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, Canada 13 Department of Medicine, University of Toronto, Toronto, Canada a r t i c l e i n f o Article history: Received 25 November 2022 Revised 23 February 2023 Accepted 14 March 2023 Keywords: Bias Immunity Test-negative-design Cohort-studies SARS-CoV-2 a b s t r a c t Since the emergence of the SARS-CoV-2 Omicron variant, multiple observational studies have reported negative vaccine effectiveness (VE) against infection, symptomatic infection, and even severity (hospi- talization), potentially leading to an interpretation that vaccines were facilitating infection and disease. However, current observations of negative VE likely stem from the presence of various biases (e.g., ex- posure differences, testing differences). Although negative VE is more likely to arise when true biological efficacy is generally low and biases are large, positive VE measurements can also be subject to the same mechanisms of bias. In this perspective, we first outline the different mechanisms of bias that could lead to false-negative VE measurements and then discuss their ability to potentially influence other protec- tion measurements. We conclude by discussing the use of suspected false-negative VE measurements as a signal to interrogate the estimates (quantitative bias analysis) and to discuss potential biases when communicating real-world immunity research. © 2023 The Author(s). Published by Elsevier Ltd on behalf of International Society for Infectious Diseases. This is an open access article under the CC BY-NC-ND license ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ) v t c O d b t p h f c t e n i S p h a h 1 l Observational studies are essential for measuring the effects of accination in real-world settings [1] . At the end of 2021, observa- ional studies measuring vaccine effectiveness found negative vac- ine effectiveness (VE) against infection [2 , 3] for the SARS-CoV-2 micron variant. These negative VE measurements attracted me- ia attention and generated widespread concern about the possi- le harmful effects of COVID-19 vaccines [4 , 5] . Although these ini- ial studies occurred during Omicron’s early emergence (when re- orted cases may be nonrepresentative), subsequent studies also ave found negative VE measurements against symptomatic in- ∗ Corresponding author: Tel: + 1-416-864-5746; fax: + 1-416-864-5310. E-mail address: k.bodner@mail.utoronto.ca (K. Bodner) . m s p ttps://doi.org/10.1016/j.ijid.2023.03.022 201-9712/© 2023 The Author(s). Published by Elsevier Ltd on behalf of International Soc icense ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ) ection [6–10] and against hospital admission [11] . Negative vac- ine efficacy is biologically plausible [12] ; however, before nega- ive VE measurements can be interpreted as negative biological fficacy, it is important to first consider, examine, and commu- icate the plausibility and likelihood that negative measurements nstead stem from biases, such as confounding and selection bias. uch biases are known to affect VE estimates from retrospective, opulation-based observational studies that largely rely on linked ealth-administrative data [13 , 14] . The key aims of this perspective re to (i) summarize patterns of negative VE and their potential eaning for immunity research (paragraphs 2 and 6); (ii) discuss ources (or mechanisms) of bias related to negative VE using a pro- osed bias classification framework (paragraphs 3-5; Figure 1 ); and iety for Infectious Diseases. This is an open access article under the CC BY-NC-ND https://doi.org/10.1016/j.ijid.2023.03.022 http://www.ScienceDirect.com http://www.elsevier.com/locate/ijid http://crossmark.crossref.org/dialog/?doi=10.1016/j.ijid.2023.03.022&domain=pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ mailto:k.bodner@mail.utoronto.ca https://doi.org/10.1016/j.ijid.2023.03.022 http://creativecommons.org/licenses/by-nc-nd/4.0/ K. Bodner, M.A. Irvine, J.C. Kwong et al. International Journal of Infectious Diseases 131 (2023) 111–114 Figure 1. Bias classification framework for observed negative VE against infection. Sources (or mechanisms) of bias capable of producing false-negative VE against infection either affect (1) true levels of infection by vaccination status or (2) observed levels of infection by vaccination status. An example of a mechanism of bias for (1) is differences in exposure (arrows), where higher exposure for vaccinated individuals compared to unvaccinated individuals produces higher infection levels among vaccinated. This bias persists when all infected and uninfected vaccinated and unvaccinated are observed (dark blue and red). In contrast, an example of a mechanism of bias for (2) is differences in testing by vaccination status (due to testing behavior or testing access) where testing differences only result in the perception of higher infection levels. This bias occurs because the unobserved uninfected and infected (light blue and light orange) are excluded from VE measurements. VE, vaccine effectiveness. ( s s g o c o s t F t m e a e V e n a c a a l v t v s s a d t ( b t o ( t a s t n t i n a i c c u t i iii) highlight methods, study designs, and reporting guidelines de- igned to help communicate and address biases in observational tudies (paragraphs 8-10). There are general patterns across current VE studies that sug- est biases are likely the cause of negative VE measurements. First, bserved negative VE has not been consistently found for Omi- ron across VE studies with some studies reporting measurements f negative VE [2 , 3 , 6–10] and others reporting only positive mea- urements [15 , 16] . Second, even within studies that reported nega- ive VE, the observed negative VE only occurred in some instances. or example, in one study, researchers observed a negative VE for hose individuals aged 18-59 years but found positive VE measure- ents for those aged 60 years and older [10] . This pattern of co- xisting negative and positive estimates within a single study has lso occurred for different vaccination dosages [2 , 6] and for differ- nt times since vaccination [3 , 9] . Finally, observations of negative E typically have been found in scenarios where biases could more asily cause an estimate to appear falsely negative. For example, egative VE was often reported for those groups with fewer doses nd with less recent vaccinations [3 , 7 , 8] when true biological effi- acy is likely lower. Here, the same degree of bias that could cause VE measurement to appear negative may only have resulted in positive but underestimated VE measurement when true bio- ogical efficacy was higher (e.g., with three doses or more recent accinations). Different sources of bias produce negative VE by influencing ei- her (1) the true levels of infection or symptomatic infection by 112 accination status (e.g., by differences in exposure) or (2) the ob- erved levels of infection or symptomatic infection by vaccination tatus (e.g., by differences in testing) ( Figure 1 ). Although the bi- ses related to (1) cause true underlying (symptomatic) infection ifferences that exist even given perfect sampling, biases related o (2) cause a perception of differences that do not reflect the real symptomatic) infection levels present in the population. Although oth categories of bias are important to discuss in the broader con- ext of VE studies, their relevance for a specific study will depend n the type of VE being measured. The biases related to (1) and 2) can both influence VE measurements of infection and symp- omatic infection; the biases related to (2) are less likely to play role when measuring VE against severe outcomes (e.g., inten- ive care unit admission, intubation, and death) because testing for hese outcomes is unlikely to differ across vaccinated and unvacci- ated individuals [14] . The mechanisms of bias that influence the true levels of infec- ion can produce false-negative VE when they lead to vaccinated ndividuals becoming infected at higher rates than their unvacci- ated counterparts. These biases are related to the uncontrolled nd often unknown differences in contact, exposure, susceptibil- ty, and immunity. For example, higher contact rates among vac- inated compared with unvaccinated populations (e.g., due to vac- ine mandates) can cause higher infection levels in vaccinated pop- lations and therefore can produce observed negative VE when he true biological efficacy is low but still positive [17] . Similarly, f vaccinated individuals experience higher network-level exposure K. Bodner, M.A. Irvine, J.C. Kwong et al. International Journal of Infectious Diseases 131 (2023) 111–114 r w ( t b p d c m w v p a m b a t i w e a s u w ( t [ s c s t c i a c i b d a [ V l f s b n A O t v V a w f t F t t t i a w v m h p f m n m s t t m A s c k b t s a a m b s w m m i o s t s V i r s a c [ a t b m V i i b a d p t I c t d p a t r D isk (e.g., essential workers), higher susceptibility (e.g., individuals ith compromised immune systems), or lower previous infection e.g., high VE for previous variants), the ratio of infected vaccinated o infected unvaccinated is increased. A larger ratio means VE can e underestimated if the analyses are unable to account for the reviously mentioned relationships [14] . Finally, for test-negative esigns, the factors that influence infection levels of the control ases also have the potential to lead to false-negative VE measure- ents. For example, the test-negative controls may include persons ith other vaccine preventable infections. If the probabilities of accination for COVID-19 and vaccination for other infections are ositively correlated, the vaccination of the other infection can act s a confounder, potentially causing COVID-19 VE to be underesti- ated [1] . The mechanisms of bias related to differential observations y vaccination (and sometimes infection) status can also cre- te the perception of negative VE. The ability to observe infec- ions may be influenced by top-down testing policies (e.g., clin- cal or employment-based criteria for who can access a test), hich can vary across jurisdictions and institutions (e.g., differ- nt hospitals may institute different testing policies). They can lso be influenced by individual testing-behaviors, which can be haped by experiences, such as living in households with individ- als at greater risk of severity (e.g., older adults and/or persons ith compromised immune systems). VE can be underestimated and negative VE observed) when vaccinated individuals have ei- her more access to testing or are more likely to seek testing 18] . Although a test-negative design can help to correct for some ources of bias (e.g., selection bias due to differences in health are seeking behavior [19 , 20] ), this correction depends on having trict criteria for both enrollment and testing [21] . Similarly, al- hough measuring VE against symptomatic infection may be per- eived as being less prone to bias than VE against infection, test- ng biases can still persist, for example, when testing-behaviors lso vary with symptoms [14] . Overall, multiple sources of bias an converge to influence VE estimates with variations in test- ng, further shaping the direction and magnitude of these other iases [22] . False-negative VE measurements occur when the biases that rive measurements downward become large enough to overcome vaccine’s true positive benefit ( i.e ., its true biological efficacy) 17] . This reason is likely why, as outlined previously, negative E estimates have been often observed in scenarios when bio- ogical efficacy is expected to be lower [3 , 7 , 8] . Omicron has been ound to confer an overall lower biological efficacy in reducing usceptibility with vaccination [23 , 24] , which means there have een more opportunities for the previously mentioned mecha- isms of bias to potentially cause false-negative VE measurements. lthough observed negative VE has been mostly limited to the micron period (with some examples from pre-Omicron [25,32] ), he same mechanisms of bias could have been present for any ariant and time period but did not lead to an observed negative E. Observational studies related to COVID-19 immunity include ex- mining the roles of previous infection and/or hybrid-immunity ith vaccination [26,27] . The measurements of the protective ef- ect of previous infection and hybrid-immunity are also susceptible o the same and similar mechanisms of bias as VE measurements. or example, studies estimating the effectiveness of previous infec- ion may be subject to a selection bias related to a previous infec- ion (e.g., those without a previous infection being less likely to est than those with a history of previous infections [27] ); stud- es estimating effectiveness of hybrid-immunity may also addition- lly be subject to a selection bias related to vaccination (e.g., those ho are unvaccinated being less likely to test than those who are accinated). These immunity estimates can also be influenced by 113 isclassification bias or by unknown differences in risk-averse be- aviors if they are unaccounted for in the analysis [26] . Therefore, revious infection and hybrid-immunity studies also require care- ul assessments of potential biases that could affect the protection easurements. We posit that the findings of the negative VE can act like a ca- ary in a coal mine, signaling that mechanisms of biases likely re- ain at play in the study design or analytic approaches. An ob- ervational study design subject to fewer biases is the prospec- ive cohort design [29] (e.g., SARS-CoV-2 Immunity and Reinfec- ion Evaluation study), which can include systematic and repeated easurements of exposure, potential confounders, and outcomes. lthough more resource-intensive than a retrospective cohort de- ign, systematic SARS-CoV-2 testing, infection, and symptom data an reduce the bias related to both differential testing and un- nown previous infections [14] . Beyond signaling the potential of ias, observing negative VE further signals the importance of quan- itative bias analyses as part of retrospective (and prospective) ob- ervational VE studies. Quantitative bias analysis (QBA) involves a process of system- tically examining and testing for the potential impact of system- tic errors. The goal is to help estimate the potential direction and agnitude of biases and to quantify the uncertainties around these iases [28] . QBA generally involves visualizing the causal relation- hips between variables using causal directed acyclic graphs [28] , hich can be used to conceptualize a priori the various possible echanisms of bias that could create false-negative VE measure- ents. When feasible, the collection or use of external data could nform QBA by helping determine the influence of each mechanism f bias on the VE measurements. For example, contact and testing urveys [18 , 30] could help elucidate whether contact, exposure, or esting vary by vaccination status and if so, provide an estimated trength of the relationships. The communication surrounding the interpretation of negative E measurements is also important. The Strengthening the Report- ng of Observational Studies in Epidemiology guidelines provide ecommendations about detailing which potential biases were con- idered and controlled for, which biases could remain (e.g., as visu- lized with directed acyclic graphs), and of those remaining, which ould have influenced estimates (assessed via sensitivity analysis) 31] . A recent VE study also explicitly highlighted the presence of suspected bias in their summary/abstract before outlining its po- ential causes in their discussion [32] . Future communication could enefit from interpreting the finding of one or more negative VE easurements, including what it means when interpreting other E estimates in the same study. The success of a vaccine campaign is influenced by the qual- ty of and trust in real-world evaluations by observational stud- es [33] . The arrival of negative VE created a media sensation and rought up valid concerns about the potential use of vaccination s our primary control measure for COVID-19 epidemics. Although iscussing the plausibility of true negative biological efficacy is im- ortant, we posit that emphasis should first be placed on the exis- ence, causes, and implications of false-negative VE measurements. gnoring the existence of false-negative VE measurements and their auses could inadvertently undermine strategies of future vaccina- ion programs and increase vaccine mistrust. By investing in un- erstanding and addressing false-negative VE, we will not only im- rove our abilities to interpret existing and future VE studies but lso create opportunities to develop new frameworks and methods hat can generally advance how we conduct real-world immunity esearch. eclarations of competing interest The authors have no competing interests to declare. K. Bodner, M.A. Irvine, J.C. Kwong et al. International Journal of Infectious Diseases 131 (2023) 111–114 F C G R E A i w R [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ unding This work was funded by a Public Health Agency of Canada OVID-19 Immunology Task Force COVID-19 Hot Spots Competition rant (grant 2021-HQ-0 0 0143 ). SM is supported by a Tier 2 Canada esearch Chair in Mathematical Modeling and Program Science . thical approval Approval was not required. cknowledgments The authors thank Jesse Knight and Mackenzie Hamilton for the nteresting discussions surrounding bias within VE studies and Lin- ei Wang for your helpful feedback on the manuscript draft. eferences [1] Doll MK, Pettigrew SM, Ma J, Verma A. Effects of confounding bias in coro- navirus disease 2019 (COVID-19) and influenza vaccine effectiveness test- negative designs due to correlated influenza and COVID-19 vaccination behav- iors. Clin Infect Dis 2022; 75 :e564–71. doi: 10.1093/cid/ciac234 . [2] Buchan SA, Chung H, Brown KA, Austin PC, Fell DB, Gubbay JB, et al. Effective- ness of COVID-19 vaccines against Omicron or Delta infection. MedRxiv 2022 [accessed 10 January 2022]. doi: 10.1101/2021.12.30.21268565 . [3] Hansen CH, Schelde AB, Moustsen-Helm IR, Emborg H-D, Krause TG, Mølbak K, et al. Vaccine effectiveness against SARS-CoV-2 infection with the Omicron or Delta variants following a two-dose or booster BNT162b2 or mRNA-1273 vacci- nation series: a Danish cohort study. MedRxiv 2021 [accessed 10 January 2022]. doi: 10.1101/2021.12.20.21267966 . [4] Miller A. Canadian COVID-19 vaccine study seized on by anti-vaxxers — highlight- ing dangers of early research in pandemic . Ottawa: CBC News; 2022 . [5] Reuters fact Check Fact Check- Danish study did not conclude that COVID-19 vac- cines adversely impact immune systems or that COVID-19 vaccines are completely ineffective against the Omicron variant . London: Reuters; 2021 . [6] Altarawneh HN, Chemaitelly H, Ayoub HH, Tang P, Hasan MR, Yassine HM, et al. Effects of previous infection and vaccination on symptomatic omicron infections. N Engl J Med 2022; 387 :21–34. doi: 10.1056/NEJMoa2203965 . [7] Andrews N, Stowe J, Kirsebom F, Toffa S, Rickeard T, Gallagher E, et al. Covid- 19 vaccine effectiveness against the omicron (B.1.1.529) variant. N Engl J Med 2022; 386 :1532–46. doi: 10.1056/NEJMoa2119451 . [8] Chemaitelly H, Ayoub HH, AlMukdad S, Coyle P, Tang P, Yassine HM, et al. Duration of mRNA vaccine protection against SARS-CoV-2 Omicron BA.1 and BA.2 subvariants in Qatar. Nat Commun 2022; 13 :3082. doi: 10.1038/ s41467- 022- 30895- 3 . [9] Ranzani OT, Hitchings MDT, de Melo RL, de França GVA, Fernandes CFR de FR, Lind ML, et al. Effectiveness of an inactivated Covid-19 vaccine with ho- mologous and heterologous boosters against Omicron in Brazil. Nat Commun 2022; 13 :5536. doi: 10.1038/s41467- 022- 33169- 0 . [10] Cerqueira-Silva T, de Araujo Oliveira V, Paixão ES, Júnior JB, Penna GO, Werneck GL, et al. Duration of protection of CoronaVac plus heterologous BNT162b2 booster in the Omicron period in Brazil. Nat Commun 2022; 13 :4154. doi: 10.1038/s41467- 022- 31839- 7 . [11] Piernas C, Patone M, Astbury NM, Gao M, Sheikh A, Khunti K, et al. Associa- tions of BMI with COVID-19 vaccine uptake, vaccine effectiveness, and risk of severe COVID-19 outcomes after vaccination in England: a population-based cohort study. Lancet Diabetes Endocrinol 2022; 10 :571–80 Supplementary ap- pendix 1. doi: 10.1016/S2213-8587(22)00158-9 . 12] Huisman W, Martina BEE, Rimmelzwaan GF, Gruters RA, Osterhaus ADME. Vaccine-induced enhancement of viral infections. Vaccine 2009; 27 :505–12. doi: 10.1016/j.vaccine.2008.10.087 . [13] World Health Organization Evaluation of COVID-19 vaccine effectiveness: Interim Guidelines . Geneva: World Health Organization; 2021 . 114 [14] World Health Organization (WHO) Evaluation of COVID-19 vaccine effectiveness in a changing landscape of COVID-19 epidemiology and vaccination: second ad- dendum to the COVID-19 vaccine effectiveness methods interim guidance . Geneva: World Health Organization; 2022 . [15] Tseng HF, Ackerson BK, Luo Y, Sy LS, Talarico CA, Tian Y, et al. Effective- ness of mRNA-1273 against SARS-CoV-2 Omicron and Delta variants. Nat Med 2022; 28 :1063–71. doi: 10.1038/s41591- 022- 01753- y . [16] Gram MA, Emborg HD, Schelde AB, Friis NU, Nielsen KF, Moustsen-Helms IR, et al. Vaccine effectiveness against SARS-CoV-2 infection or COVID-19 hospi- talization with the Alpha, Delta, or Omicron SARS-CoV-2 variant: A nation- wide Danish cohort study. PLoS Med 2022; 19 :1–18. doi: 10.1371/journal.pmed. 1003992 . [17] Bodner K, Knight J, Hamilton MA, Mishra S. Testing if higher contact among vaccinated can be a mechanism for observed negative vaccine effectiveness. Am J Epidemiol 2023. doi: 10.1093/aje/kwad055 . [18] Glasziou P, McCaffery K, Cvejic E, Batcup C, Ayre J, Pickles K, et al. Testing behaviour may bias observational studies of vaccine effectiveness. J Assoc Med Microbiol Infect Dis Can 2022; 7 :242–6. doi: 10.3138/jammi- 2022- 0 0 02 . [19] Jackson ML, Nelson JC. The test-negative design for estimating influenza vac- cine effectiveness. Vaccine 2013; 31 :2165–8. doi: 10.1016/j.vaccine.2013.02.053 . 20] Sullivan SG, Tchetgen Tchetgen EJT, Cowling BJ. Theoretical basis of the test- negative study design for assessment of influenza vaccine effectiveness. Am J Epidemiol 2016; 184 :345–53. doi: 10.1093/aje/kww064 . 21] Lewnard JA, Cobey S. Immune history and influenza vaccine effectiveness. Vac- cines 2018; 6 :1–14. doi: 10.3390/vaccines6020028 . 22] Williams LR, Ferguson NM, Donnelly CA, Grassly NC. Measuring vaccine effi- cacy against infection and disease in clinical trials: sources and magnitudes of bias in Covid-19 vaccine efficacy estimates. Clin Infect Dis 2022; 75 :e764–73. doi: 10.1093/cid/ciab914 . 23] Zhou Y, Zhi H, Teng Y. The outbreak of SARS-CoV-2 Omicron lineages, immune escape, and vaccine effectivity. J Med Virol 2023; 95 :e28138. doi: 10.1002/jmv. 28138 . 24] Lyngse FP, Mortensen LH, Denwood MJ, Christiansen LE, Møller CH, Skov RL, et al. Household transmission of the SARS-CoV-2 Omicron variant in Denmark. Nat Commun 2022; 13 :5760. doi: 10.1038/s41467- 022- 33328-3 . 25] Chung H, He S, Nasreen S, Sundaram ME, Buchan SA, Wilson SE, et al. Ef- fectiveness of BNT162b2 and mRNA-1273 Covid-19 vaccines against symp- tomatic SARS-CoV-2 infection and severe Covid-19 outcomes in Ontario, Canada: test negative design study. BMJ 2021; 374 :n1943. doi: 10.1136/bmj. n1943 . 26] Goldberg Y, Mandel M, Bar-On YM, Bodenheimer O, Freedman LS, Ash N, et al. Protection and waning of natural and hybrid immunity to SARS-CoV-2. N Engl J Med 2022; 386 :2201–12. doi: 10.1056/NEJMoa2118946 . 27] Nordström P, Ballin M, Nordström A. Risk of SARS-CoV-2 reinfection and COVID-19 hospitalisation in individuals with natural and hybrid immunity: a retrospective, total population cohort study in Sweden. Lancet Infect Dis 2022; 22 :781–90. doi: 10.1016/S1473-3099(22)00143-8 . 28] Lash TL, Fox MP, Maclehose RF, Maldonado G, Mccandless LC, Greenland S. Good practices for quantitative bias analysis. Int J Epidemiol 2014; 43 :1969–85. doi: 10.1093/ije/dyu149 . 29] Gail MH, Altman DG, Cadarette SM, Collins G, Evans SJW, Sekula P, et al. Design choices for observational studies of the effect of exposure on disease incidence. BMJ Open 2019; 9 :e031031. doi: 10.1136/bmjopen-2019- 031031 . 30] Adu P, Binka M, Mahmood B, Jeong D, Buller-Tylor T, Damascene MJ, et al. Quantifying contact patterns: development and characteristics of the British Columbia COVID-19 population mixing patterns survey. Int J Infect Dis 2022; 116 :S30–1. doi: 10.1016/j.ijid.2021.12.073 . 31] Vandenbroucke JP, Von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Epidemiology 2007; 18 :805–35. doi: 10. 1097/EDE.0b013e3181577511 . 32] Hitchings MDT, Ranzani OT, Torres MSS, de Oliveira SB, Almiron M, Said R, et al. Effectiveness of CoronaVac among healthcare workers in the setting of high SARS-CoV-2 Gamma variant transmission in Manaus, Brazil: a test- negative case-control study. Lancet Reg Health Am 2021; 1 :10 0 025. doi: 10.1016/ j.lana.2021.10 0 025 . 33] Hungerford D, Cunliffe NA. Real world effectiveness of Covid-19 vaccines. BMJ 2021; 374 :n2034. doi: 10.1136/bmj.n2034 . https://doi.org/10.1093/cid/ciac234 https://doi.org/10.1101/2021.12.30.21268565 https://doi.org/10.1101/2021.12.20.21267966 http://refhub.elsevier.com/S1201-9712(23)00102-9/sbref0004 http://refhub.elsevier.com/S1201-9712(23)00102-9/sbref0005 https://doi.org/10.1056/NEJMoa2203965 https://doi.org/10.1056/NEJMoa2119451 https://doi.org/10.1038/s41467-022-30895-3 https://doi.org/10.1038/s41467-022-33169-0 https://doi.org/10.1038/s41467-022-31839-7 https://doi.org/10.1016/S2213-8587(22)00158-9 https://doi.org/10.1016/j.vaccine.2008.10.087 http://refhub.elsevier.com/S1201-9712(23)00102-9/sbref0013 http://refhub.elsevier.com/S1201-9712(23)00102-9/sbref0014 https://doi.org/10.1038/s41591-022-01753-y https://doi.org/10.1371/journal.pmed.1003992 https://doi.org/10.1093/aje/kwad055 https://doi.org/10.3138/jammi-2022-0002 https://doi.org/10.1016/j.vaccine.2013.02.053 https://doi.org/10.1093/aje/kww064 https://doi.org/10.3390/vaccines6020028 https://doi.org/10.1093/cid/ciab914 https://doi.org/10.1002/jmv.28138 https://doi.org/10.1038/s41467-022-33328-3 https://doi.org/10.1136/bmj.penalty -@M n1943 https://doi.org/10.1056/NEJMoa2118946 https://doi.org/10.1016/S1473-3099(22)00143-8 https://doi.org/10.1093/ije/dyu149 https://doi.org/10.1136/bmjopen-2019-penalty -@M 031031 https://doi.org/10.1016/j.ijid.2021.12.073 https://doi.org/10.1097/EDE.0b013e3181577511 https://doi.org/10.1016/j.lana.2021.100025 https://doi.org/10.1136/bmj.n2034 Observed negative vaccine effectiveness could be the canary in the coal mine for biases in observational COVID-19 studies Declarations of competing interest Funding Ethical approval Acknowledgments References