Something in your life is withering away, becoming less important or less of an issue. May also suggest that something is withering away due to your neglect. What or whom have you not been giving enough attention to?