A lot of cats have 2 homes. Some even go away for 2 weeks to another home then come back for 2 weeks. They’re weird. But a lot of cats just go to other houses for food/attention. It’s normal, so just keep playing with him outside because he obviously can’t go inside as your mum is allergic.