If you guys are really worried about others or the rest of the town finding out , then i would suggest first feel out this couple you feel want the same so you know for sure, if things progress then i would plan a weekend with them for a hotel out of town, nothing extravagant or anything but if both couples chip in a nice suite will provide the safe room,safe from anyone noticing and provide you all plenty of time to do what every all like.
There is a good chance this other couple might know more in your quiet town than you all do and if all goes well provide you with names of those others, it basically is how referral system works and that provides you with no internet searching where it may not be safe. unless your running for mayor or something where it getting out will destroy your name just try worse they say is no, but it might be yes