Wolverine's physical appearance in the comic books is late 40s to early 50s in all series. He can't be replaced with a 20-something or even a 30-something in the movies. Keep in mind he was born in 1886. Although he ages slowly, he still does actually age. He physically matured quickly but the maturing process slowed down to a crawl when he reached adulthood.
I don't know the comic X-Men series that well, but I agree that he cant be TOO much younger. However, they can take a younger actor and age him with makeup and allow him to age into the role naturally.