It could be an added discomfort that since he has lied to his family so much, his brother communicating with you might make him nervous in terms of what would come out if the two of your spoke. He would probably be imagining the two of you scheming against him, and find this highly threatening as well.
No, it would not serve you at all to worry about all kinds of things where you don't see anything strange in your environment. Only to be safe and remain very prudent, particularly when some of these behaviors (projection) are very "N-common", so you would be able to immediately
recognize a sign of something strange should it come up, for what it could really mean.
I wonder there must be another reason like wanting control that he does this..
If he is accusing you of having affairs with even women that you might like to be friends with... Accusing you of that with just anyone and everyone that you speak with, then it sounds like it could also be a control and isolation issue.
Many abusers try to isolate their partners from all outside contacts (family, friends, co-workers) in order to attempt to maintain control and keep the status quo (how the abuser wants it) within the relationship. To them, outside influences in any form are threatening. They do not like anything that might cause you to change the way you think (they have you where they want you), or might cause you to change anything within relationship.
They also fear abandonment in that you will "see the light" in talking with others, and leave them.
More distant than their needs, is that they may not want others to hear about how the relationship is, either.
One way they are able to succeed with the isolation is to create sheer hell around the issue of their partner talking with anyone else, and it just becomes too emotionally traumatizing and destroying to face the abusive, vile, dumbfounding in irrationality, barrage of punishment that never tires of being dished out. So, the emotionally weakened partner complies in order to keep the peace

BT