The difference is that passenger rail isn't an essential economic or military system like we view the highways and freight rail systems. It has always been the secondary purpose within the rail system itself, and currently only is cost-effective over relatively short distances.
Now that doesn't mean there shouldn't be investment in it, in regions that could benefit from passenger service. However, subsidizing it at a national level has always been something like the government taking over telegraph/gram service from Western Union. There are valuable services that the company provides (mostly wiring money) but overall it's a dying system that doesn't need to be supported in full.
As your first sentence intimates, we essentially chose to value cars and planes as the means for people to travel great distances around the country. But does that automatically mean that those are the most efficient ways to achieve that? Likewise, are there not sufficient ancillary benefits to building a national high-speed passenger rail system, like economic development in some of the more remote towns across the country?