luring my cat home from neighbors house

Question
My neighbors are very much in love with my tabby cat.  He comes home from their house RARELY and I spot him in their driveway, call him, and he looks at me like "who are you?"  He comes in, eats and sleeps then returns to their house.  Also, I'm sure he's eating somewhere else.  I've discussed this with my neighbors.  They promised not to feed him.  What's more, he got sick and had to be in my house for a week and during that week, they got a look-alike tabby cat of their very own.  How can I lure him back home and keep him coming in and out of my cat door?
I play with him and feed him immediately when he walks in our house.  
Also, they are washing him, I think, because he smells very much like ladies' bath powder when he arrives at our house for the rare 15 minute appearance.

Answer
Of course, the best solution would be to keep him inside.  I used to allow my cats out until I experienced first-hand cats getting torn apart by racoons, eating by coyotes, stolen, lost, and catching fatal diseases.  I converted them all to indoor cats, and they are happier than ever.

At any rate, I find that when a cat decides to switch homes, it's often because he's getting tastier food!  Our neigbhor used to feed dry food to some wild cats but gave them no other care.  We started feeding them tasty canned food, and they all ended up waiting for us in the driveway at mealtime.  We had a talk with the neighbor, who said she hadn't seen the cats come to her house in weeks!  And it was strictly because they preferred our food.  Until you can find out who's feeding him, maybe try a special treat with his food?  Most cats love canned Fancy Feast or a few bites of tuna on top of their food.

The only other reason I've seen for a cat to switch homes is stress in his real home.  I had one cat decide to move to the neighbor's when we got a dog.  And my sister's cat refuses to stay home now because she's fostering a litter of kittens.  Cats will avoid noisy houses, too.

Aside from minimizing stress if there is any and trying to tempt him with special treats, I don't think there's any way to solve this situation besides making him an indoor-only cat.