As the application of computational methods in drug discovery pipelines becomes more widespread it is increasingly important to understand how reproducible their results are and how sensitive they are to choices made in simulation setup and analysis. Here we use ensemble simulation protocols, termed ESMACS (enhanced sampling of molecular dynamics with approximation of continuum solvent), to investigate the sensitivity of the popular molecular mechanics Poisson-Boltzmann surface area (MMPBSA) methodology. Using the bromodomain-containing protein 4 (BRD4) system bound to a diverse set of ligands as our target, we show that robust rankings can be produced only through combining ensemble sampling with multiple trajectories and enhanced solvation via an explicit ligand hydration shell. Simulation input and binding affinity calculation data available from: https://doi.org/10.5281/zenodo.1484050