Can someone trace my phone back to my address?

[Destiny]

A weirdo called my house yesterday which is a landline phone number. He began to get me into conversation with him by telling me he had lost a bet and he had to say the persons name and blank you rock. I went along with it, though suspiciously because I did feel uncomfortable. Well I eventually began to think he was masturbating and it really freaked me out. He asked my name when he first called, but I lied. He also tried to ask more questions and again, I lied. I know I should have hung up but for whatever reason, I find it very hard to hang up on people. Anyways, he obviously knows my number, but doesnt have my name or any other information that I'm aware of. Could he somehow find my address? I already tried the reverse phone number and it didnt work for my house so I know he cant use that. I also called my phone company and they said he couldnt find me but I'm still in fear. I dont want some rapist showing up at my door. I may sound like I'm worrying too much but this isnt my first run in with a weirdo and I have reason to be scared from my past experiences.