So you've gone to Dorset and stood on the Chalk out line of a giant naked man.
I'm surprised they let people do that, being what it is.
A chalk outline in the ground.
I figure too many tourists standing on that may cause undue wear and taer, unless it's been mantained.
As for the Temple of the stars.... I pretty sure they have guided tours there. Seeing as your a local, you should check that out. It should be right on your doorstep, you right in the middle of it.
http://www.isleofavalon.co.uk/avalon-zodiac.html
Sorry to hear that Dinas Emyres isn't what seems cracked up to be.
I know of the lake or body of water and the main road beside it. As well as the the old railway line and bike trail.
As for it's size, perhaps it might be a knoll in amongst large mountain range that you were standing on. From what I have read in the source books and the comics. The fortress is really a small town or city. It's not there of course, but the area should be big enough. Could you imagine crenelated walls and towers resting on the flater panes rather than the mountain slopes.
Regarding your claim that there doesn't appear to be enough room for a small house rather than a castle city.
It may just be that your a giant.