In the 1957 case of Dayton v. Dulles, the U.S. Supreme Court ruled that it was unconstitutional for the Secretary of State to deny a passport on grounds of alleged Communist affiliation without providing due process rights as required by the Fifth Amendment. The appellant, Rockwell Kent, an artist and writer who had been denied a passport because he refused to submit an affidavit concerning his membership in organizations listed as subversive by Attorney General Herbert Brownell Jr., challenged this denial arguing that it violated his constitutional right to travel abroad. The court agreed with Kent's argument stating that freedom of movement is part and parcel of our liberty which cannot be taken away without due process under law.

In the dissenting opinion for Dayton v. Dulles, Justice Frankfurter argued that Congress had not explicitly stated in the Passport Act of 1926 that a passport could be denied due to ideological beliefs or affiliations. He believed this interpretation was an overreach by the Secretary of State and violated citizens' rights to travel internationally under the Fifth Amendment's Due Process Clause. Furthermore, he asserted that it is up to Congress, not administrative officials, to determine what constitutes "un-American activities" warranting denial of a passport application. Lastly, he emphasized his concern about potential abuses of power if government officials were allowed such broad discretion without clear legislative guidance.