Dude, it has nothing to do with free market. It's the Government having control of what you can do or where you can go if not for the vaccine. This really isn't complicated.
You are making your point based on conjecture of what could happen, he is making his argument based on the released information about how the passport would be implemented. At this point, it has been said to be something that would be implemented by businesses, with information verified by the gov.
You may be correct that it will lead to the scenario you are afraid of, but it is wholly unknown at this point.