Well, my first question would be what is it exactly that you are trying to do. A line driver is something different than just a line output so I'm confused as to what you really want to know. Any chip will supply a line output, but a line driver is designed specifically to drive a "long" line or a long cable, so you first need to define the application and then choose the chip that will get that job done.
"Does it matter if I'm building a low-gain or no-gain line driver?"
Does "what" matter? Everything matters, but again, it depends on the application. If gain is an issue then it matters; if it's not an issue then it doesn't matter.
If your issue is that you have a couple of Buf's laying around and you want to build some type of line driver out of them then you should study up on line drivers to see what kind of specifications they have. Normally, a "line driver" can supply much more current than a "normal", for lack of a better term, chip or circuit. Some chips can provide upwards of 20mA of current, whereas some can only supply a couple of mA. A line driver can supply upwards of 100mA of current, question is, do you need that much?
A "buffer" can also supply a "high" output current and can also drive a long cable, but the first thing that comes to mind when I think "buffer" is when I need to drive multiple inputs (or provide multiple outputs) and don't want to load down my existing output.
If your question is about the sound quality of these two chips: BUF-03 versus LM6172, then I would say that the buf has probably not been optimized for audio applications. That is not to say that it won't sound good. If I had an "old school" audio stage that couldn't supply enough output current, then adding a Buf03 or some other one, could be an option to consider - it certainly won't degrade the existing sound that I already have.
So, basically, a buffer chip is a specialized device designed to add current drive capability to an existing design - not necessarily an audio design - but the 6172 has probably been designed from the ground up to handle audio applications.
So, the list goes on and on depending on what you need to do, so first figure that out and then post back.