Adaptive sparse and dense hybrid representation with nonconvex optimization