After having a private speech therapist evaluate him, I decided 20 minutes a week was not worth the hassle and there were many other things I couldn't drop.
I was all ready for disagreements to this, so I came prepared with the report from the private speech therapist and also my monthly schedule for the last two months. The open times were again presented (nothing had changed from our last conversation. I still couldn't do it). I explained that I was caring for two children with heavy medical needs and could not be leaving the house any more or drag all three with me to some grade school just because she serviced other students there. Finally, the director of the school suggested dropping Fencing. I haven't wanted to do this. I want him to follow through with something. He really is not liking it, but if we stopped everything just because he was frustrated, he wouldn't learn anything. The speech lady had an open time during his class. Since he's not enjoying fencing, I'm pulling him out and he'll do speech for the first 20-25 minutes. Then I'll find something for us to do while we wait for my nephews to finish the fencing class.
So, it's a pretty good compromise. I won't have to drive anywhere else and he can get free services (although I don't know how much good they'll do).