Spin waves and phase transition on a magnetically frustrated square lattice with long-range interactions