The protest was criticised by many local residents and politicians. But they were outnumbered by hundreds of jeering counter protestors.
Police intervened after some of the demonstrators were sprayed with tear gas.
Some protesters who came for the "Unite the Right" rally were armed and dressed in military-like clothing, while others wore shirts with Nazi symbols and quotes from Adolf Hitler.
In the days leading up to Saturday's planned rally, there had been some back-and-forth about where it would be held.
Stocks slide on continued concern over tension between United States and North Korea
Crude briefly pared losses in late trading after USA inventories fell more than expected last week. The overall financials group, which accounts for roughly a third of the index slipped 0.9 percent.
Jason Kessler, a blogger who organized the rally, said in a phone interview that his group dispersed after police declared the assembly to be unlawful. Kessler originally planned to hold the rally at Emancipation Park but the city announced earlier this week that it should be held at a larger venue because of safety reasons.
"Our constitutional First Amendment rights were violated", Kessler said, declining to comment on whether his group would try to stage another rally in Charlottesville.
"There is no doubt that Mr. Kessler has a First Amendment right to hold a demonstration and to express his views", the city manager wrote in a statement on August 7.
After the ruling late Friday night, white nationalists and so-called "Unite the right" protesters descended on the University of Virginia campus with tiki torches, which they threw at counter-protesters. The Ku Klux Klan and supporters rallied in the city in July, causing concern at the university, but Friday night's march was on campus, and ended at the Rotunda, a hallowed space at the university. "Hate has no place in Va".
"The violence displayed on the grounds is intolerable and is entirely inconsistent with the university's values", the statement added.
Stocks slide on continued concern over tension between USA and North Korea
The Toronto Stock Exchange's S&P/TSX composite index fell 57.36 points, or 0.38 percent, to 15,159.97 shortly after the open. The Dow and S&P 500 inched higher on the day but they both posted their largest weekly percentage drops since late March.
Police have not released an official statement of how many people were arrested.
Charlottesville: Hours before a noon rally was set to begin on Saturday, violent skirmishes broke out between bands of white supremacists and counter protesters who have converged on the college town of Charlottesville around the issue of a Confederate statue.
Thomas said he was informed the National Guard would monitor the situation. However, that's only deterred a small fraction of the thousands of people expected to attend "Unite the Right". It says those people defy its community standards.
Conte on Pedro, Morata, Wenger's comments and Chelsea's search for players
I know he was at Juventus but I don't think he was ever completely their No. 1 striker, slightly in and out at different times. It is the best players from all over the world playing for some of the top clubs in the world.
- Was Lady Stoneheart in the latest episode of Game of Thrones?
- Futures lower as North Korea tensions linger
- Gold, silver gain on rising tension between US-North Korea
- North Korea Threat: US, Japanese troops prepare for war
- Markets slide, dollar slumps as North Korea fears spike
- If North Korea attacks United States , it 'will regret it fast'
- Cayetano: ASEAN won't oust N. Korea from ASEAN Regional Forum
- Guardiola: I'm Not Under Pressure, City Won't Spend Big Again
- FTSE ends down as shares hit by US-North Korea tensions
- NKorea Rebuffs Nuclear Talks, Says US to 'Pay Dearly'