There's an ambiguity when you say "academic reference". I can think of three meanings:
- Letter to help him get into graduate school or an academic position.
- Letter to help him get into undergraduate
- Letter to an employee about his academic competency.
I don't see any problem with writing him a letter for #2 based on your knowledge of him. For #1 and #3, it doesn't seem like you've been in the sort of relationship where you could write one.
Regardless of type, there's a second issue of whether you can write him a helpful letter. I've had several students ask me in the past (particularly for their entrance into medical school), and several of them were outstanding students. For the less than outstanding, what I do is, point out the sort of letter they would get from me and say I'm fine with sending that, e.g.,
You earned a C in my class which is not exceptionally good. If I wrote a letter, I would mention the degree to which I felt you performed in my class. It might not be the best letter you could get. IF you're still interested, I can send one for you.
(It should not be hard to adapt that to your purposes).
Presumably if he's asking you for a letter to get into undergraduate, part of the point could be to explain that he's a good student but had to quit for work, etc.