soft water = low pH?

Question
To spare you the details of all the things I've done with this tank from hell, here's what's still alive: 1 male guppy, 1 female guppy, 1 catfish sort of thing that eats algae

the temperature's right, I have a filter and aerator, but everything keeps dying so I took a water sample to a store and they told me my pH was low and gave me some stuff called "pH up".  It works...for a few minutes...but then the pH drops right back down again to below 6 - and over the last 4 days, I've used half a bottle of the stuff!

I figured the water was buffered too low so I did some research on line and found out that can happen with water that's too soft.  

So I went looking for a nice decorative limestone rock or something to throw in there, but no one seems to cary them.  All the mineral substrates I found are for saltwater tanks and would make the water too hard.

I did finally find a product called "Wardley 3-in-1, 7.5 pH" which seems to at least be making a difference...I started using it today and the pH is up to 6.6 - I want to wait 2 weeks before trying to raise it more.  

But for the long term, I don't want to keep throwing chemicals in my tank every time I do a water change. Are there any other alternatives?

Answer
Well evryone has tank problems i have had the same problem so don't feel like your the only one this has happened too.
Their is not much you can do to raise the P.H., but it sounds like that Wardley product is working so i would stick with that. When you do water changes you only want to take out at the most 25% of the water once a week or else you will have problems like you are now. If their is any more questions please ask and don't forget to rate my answer!!!