Well, not always. I made sure I either picked fast learners, or people who've already playtested for me before, and who already have some romhacking experience. In other instances, the job is big enough that the time spent coaching is well worth it.
In other words, you have to find people that have some clue.
What you are looking from people is to ask the right questions... it may be hard to differentiate them but ultimately it is like the "asking the smart question".
Maybe I should get back to the FF4 hack that I have on hold, but sometimes I've felt that I've been asked questions I should not have to answer/explain, but inevitably I should be expecting them... I probably have high expectations for random people to have read a guide I wrote about the game's mechanics, but I get asked anyways. It is a lot of info to digest after all, but if I asked people in the know, I'm sure I'd get better responses and questions. I like the challenge of a good question after all.
The problem with public-betatesting in general is that you will get varying degrees of issues... some of which not worth attending to. The worst thing that can happen is when attention to detail fails at some level. It can ultimately be a WTF moment...