I don't think it was taken out, just obscured so it wasn't the focus, I've been there enough times to have recognised bits of it.
It did make me smile that they made it from there to Sycamore Gap so quickly, but then he played around with geography in Days too. There are films when this kind of thing irritates me, but not with these.
The Bone Temple is due out in January - I'm really hoping for a bit of backstory as to how the various tribes got formed.
Mystery solved!
I just watched the opening scenes again and the castle is there, but they have moved it! In the overhead shot of the island at the start, and the next shot just of the castle it is there but its been placed in the North East corner not the South East one. Next theres another shot looking East where you can see most of the village with the Priory at the Southern end. But looking in the distance beyond the harbour the castle motte is there, but the castle is not!
I think they've just shrunk the island a fair bit, removing most of the Northern dune area and placing the castle up there instead.
And yeah geography changes in films can annoy me too, but in this case it makes sense so as to make the island seem more isolated/protected than it actually is.
Also (dont mean to be going on lol) there are some really good shots in the village later that if you watched them on their own, could be almost from a different film set on the island as it looks pretty much the same.
