It would be good, but rich and really starchy (not in taste, but I'm thinking health-wise).
You can get mixes of onion, mushroom, and some other vegetable based gravies. Or just make a gravy with vegetable stock.
I just checked a packet of McCormick brown gravy I have, and it's vegetarian. No animal anythings in it.
And those are really quick and easy. Easier than potato soup.
When I make a pot pie, it isn't usually a pot pie but closer to a shepard's pie. I just throw a bunch of vegetables, including potatoes, into a 9X13 pan, pour whatever kind of gravy over the top of it to just barely cover the veggies, and then I roll out homemade buttermilk biscuit dough and use that to cover the top. Then bake and eat. Edit: You can totally do the same thing with a pie crust, either just a crust on top or lining the pan first too. I just find that the crust in the pan the way I make it gets soggy. And biscuits are easier than crust.
I just thought, are you trying to duplicate a chicken pot pie? With white gravy? In that case I'd probably just make a thinnish white sauce (resources ALL OVER the google), and season it really well, maybe even add some cheese in there if you still eat that. Easier than making a whole separate soup for the gravy.