"Neutron star astronomy at ESO, from the NTT to the E-ELT"