CIA/State Dept. Cold War liberals and anti-Communist social democrats dominated the US foreign policy establishment in 1954.They thought the existing indigenous elite in Vietnam was too traditionalist and too corrupt to bring about the (non-Communist) social revolution they assumed Vietnam needed.
So they picked Diem , a man with impeccable anti-colonialist, anti-French credentials and a brother in the anti-communist wing of the international labor movement. It also didn't hurt that he was one of a handful of Vietnamese of any kind living in the US in 1954 and he didn't stint on presenting his case.
But when the CIA and the State Department gave control of the country to Diem and his family they presented it to a gang of thieves far worse than the ones that had preceded them. Also, Diem was far more concerned with crushing anti-Communist traditionalist power centers than he was with fighting the Viet Cong. He tried-using bribery made possible by generous US aid and police state terror administered by Diem's brother Nhu -to create a totalitarian regime while using all the US State Dept. approved rhetoric. Thus, when Diem was finally overthrown, there were no strong, coherent elements left in the South to counter the Vietcong threat.