The reason that he refers to thénardier by name is because he has been on his own for so long, it is unlikely would think of him as a father. In fact, thénardier gave him away. Even if he remembers being his son, he wouldn't think of him as his father.