I think the difference is negligible as they use the same backbone. I use Sky and I don't live far away from the cabinet but I've never experienced speeds drop during peak times.
Been with Sky Fibre for about 5 years now never had an issue with service or connection. I don't have access to Virgin here but if i did decide to change i would most likely go for BT.