The word "let" has multiple definitions, two of which are:
let verb
1 (with object and infinitive) Not prevent or forbid; allow.
3 British (with object) Allow someone to have the use of (a room or property) in return for regular payments.
-Lexico
In your example, the intended usage is likely to be definition 1. Note that "let" is used with an object ("the children") and an infinitive ("play games"). The infinitive play is a bare infinitive in this case:
After the object after certain verbs, such as hear, see, make, let, there is no to: ...
-bbc.co.uk
If you add "to", the sense of "let" switches to definition 3, with "to" heading up a purpose clause - but the context doesn't make sense. The phrase "to play games" becomes the purpose of "let[ting]" the children (hiring them out as if they were property). The letting is done as a form of entertainment for the "families". This would be a horrible thing were it to happen, and is most likely not the intent of the speaker.
Leaving out the word "to" worked in your favour this time.