To be absolutely fair, I don't think you'd notice a difference there. 105 and Ultegra are only one step apart in the Shimano family.
My Defy Advanced has complete 105 and it works brilliantly, but my alloy Defy 2 has Tiagra... which also works brilliantly and there's very little to choose between them. The 105 needs less lever travel to make a shift... that's about it. The biggest difference between 105 and Ultegra is a small weight saving, but not enough to make it a deal breaker IMHO.