I felt a similar conflict when I got a tomato plant^[1] I eventually concluded that, while I still am unsure how I feel about eating leaves and grains, it is at least ethical to eat fruit and nuts, since they are pretty much intended to be eaten as a means of seed dispersal, and for most kinds of fruit don't really cause any trauma to the plant when harvested - especially given they'd eventually fall off on their own accord anyway.

Although, I suspect I'm not much better in the general food ethics department, so take that with a grain of salt. I generally keep vegan at home, with some occasional egg consumption, but when eating out I'll eat meat because I can't digest dairy and it's very hard to find non-dairy vegetarian options.

[1] Which, sadly, I managed to kill, but that's another story.