Scalable and adaptive log manager in distributed systems