East Ilsley lays a claim to be the site of the battle of Ashdown in AD871 when Alfred defeated the Danes. Chronicled by Bishop Asser "On the hill stood a solitary ash tree, revered for centuries by the druids before the Christians came," and as late as the Domesday Survey, Ilsley Hundred was known as"nachededorne" (Solitary Ash). Shortly after the battle a community must have originated bearing the name of "hildes laeg, or battlefield, this was then corrupted into" Hildesley" and then later Ilsley. In addition, there is a byway on the edge of the village known as Dennisford road. This is thought to be "Danesford" where the fleeing Danes were seen to be crossing the River Pang in disarray.
Geoffrey de Mandeville was granted lands in East Ilsley by The Conqueror: by the 13th century, the manor had passed to the De Bohuns, Earls of Hereford and Essex. In 1573 it passed to the House of Lancaster by the marriage of Mary de Bohun to Henry Bolingbroke, later Henry IV. The swan badge was the crest of the de Mandevilles inherited by the de Bohuns and thus it was used by Henry V as derived from his mother. It is a possibility, therefore that the hostelry may have been named for this connection to the sovereign and could even have been Crown property.