Yes, That is the part I don’t like. When typing you have no way to know when you have let someone down or at least someone thought you let them down lol I find that sometimes myself. You might look for a larger fleet if this is important to you. Monitor the fleets to see how many people are on at your play hours. I can say for our fleet when there are 10 people on Mumble there are about 15 people not on Mumble. The larger the fleet is the better chance of finding 4 or 5 others that like to non-voice chat. The only drawback is the larger the fleet gets the easier it is to get lost.
Another think you can do is see if you can listen in for a couple of days to see what they are all about.
Much luck in your quest and take the time to find the right home.