When you do allow him out - and I agree with what others have said about waiting longer - try to familiarise him with the sights, sounds and smells of his home on the outside before letting him go off on his own. If he will take to a harness, walk him round the garden, hold the end of the harness while you sit and allow him to explore on a long leash.
I moved house recently with five cats. I kept them in for a couple of months, let them out and never saw one of them again, another went missing the following week. I think Fred, who didn't return from his first outing, was confused because he expected to see all the clues he would have seen at the old address and he didn't recognise the new house from the outside. Why George went missing is a mystery.
But this is why I recommend going very slowly in introducing your cat to your outdoors. Even if he's closely bonded to you and to the indoors, outdoors and the sight of your house from outdoors will all be new, and he needs to learn to recognise it.
So try a harness or a large pen in the garden (out of direct sunlight) so that he can bond with your garden.