I think there is a difference between identifying the letter of the law, and whether we should be worrying morally about our actions.

I would ask myself two question:

1. ) First of all, If it were illegal to convert my physical book into a text file on my computer (which i'm not convinced of), would I buy the digital copy, or would I live with the hassle of lugging around the physical book? If you're going to do the second, which can be hard to determine, but if you were going to then they are not losing any money on you having a digital copy because you wouldn't have boughten one anyway. That's what they really care about, so they're not losing any money and you're having an easier time carrying books around. However, If saving you the hassle of carrying the physical book around would result in you buying a digital copy (given you already have the book) then we must go to question 2.

2.) Is it illegal for me to convert on my own accord a physical book I own into a text file (for personal use) (And from there it's not hard to convert it into e-book format, there is free software that will let you). I don't know of any hidden agreement in buying book, that says you can't copy it word from word, on another piece of paper, or on a word file. As long as you keep it for yourself (don't let out multiple copies of it) then I find no issue with converting it, or in a less hassle free way downloading it for personal use. (given you own a legit physical copy).

As an aside question does anyone know the legality on printing an e-book you own?