Optimal modulation formats for 2 560 km optical transmission with low power penalty