I teach almost 1000 students per academic year, and I can tell you: it’s absolutely stunning to see how many of my male students look so much older than they really are.
Fat guys, balding students, pale skeletons, wrinkles-wrinkles-wrinkles. Some of these students literally look older than me, while I could be their dad. (Which I’m not, as far as I know.)
More than anything else, the answer to your question depends on genes (not jeans — to be perfectly clear). Robert Redford in the 1986 movie Legal Eagles above, was 49 when the movie was shot, and sure, he had some wrinkles and a bit of a turkey neck, but his exceptional looks easily compensated to make him look as charismatic as ever.
Brad Pitt is another example: the man is in his early 60s but easily beats your average 38-year-old because his was blessed with his remarkable genes. Some of these guys start looking even better as they age (a phenomenon which is much more rare in women) — it’s not fair to many, if not most of us.
In general, I think there is so much variation among men that a specific “average age at which men begin to loose their looks” would give a deeply distorted perception on most individual cases — statistics would spectacularly fail at this particular one. That’s the long version —
The short version: it all depends on the looks.