Objective The aim of this study was to investigate the impact of methodological choices in a meta-analysis of real-world evidence (RWE) comparing three non-vitamin K antagonist oral anticoagulants with vitamin K antagonists (VKAs) for the treatment of patients with non-valvular atrial fibrillation (NVAF). Methods The meta-analysis was based on a systematic review of RWE studies enrolling incident and prevalent patients aged >/=18 years with NVAF and receiving either rivaroxaban, dabigatran, apixaban, or a VKA. Five different scenarios were considered to explore the impact of the initial meta-analysis assumptions: (1) using studies that involved only incident patients; (2) excluding studies that only reported crude values and did not consider any adjustment; (3) including all studies independently of possible database overlap; (4) using studies with data on different dosages for rivaroxaban and dabigatran; and (5) assigning quality weights to studies to assess quality of reporting. These scenarios were run on three outcomes: ischemic stroke (IS), myocardial infarction (MI), and intracranial hemorrhage (ICH). Results Across all scenarios, rivaroxaban was associated with significantly lower risks of IS and ICH than VKAs. In most scenarios, dabigatran was associated with significantly lower risks of IS and ICH. In all scenarios, apixaban was associated with a significantly lower risk of ICH. Conclusions Sensitivity analyses showed the impact of similar assumptions was different depending on the outcome, and the drug considered. The development of recommendations and guidelines for the inclusion of RWE in meta-analyses could prove useful in evaluating the effectiveness of health care interventions.