CHARLOTTE AMALIE, US Virgin Islands (AP) — Ask about Jeffrey Epstein on St Thomas and rooms go quiet. Some people leave. Those who share stories speak in barely audible tones.

The 66-year-old billionaire bought Little St James Island off this US Caribbean territory more than two decades ago and began to transform it — clearing the native vegetation, ringing the property with towering palm trees and planting two massive US flags on either end. When guides took scuba divers to spots near the island, security guards would walk to the water’s edge.

It was off-putting to residents of St Thomas — a lush tropical island east of Puerto Rico with winding roads through mountains dotted with dainty Danish colonial-era homes. Then, when Epstein pleaded guilty in a 2008 to soliciting and procuring a minor for prostitution, his need for privacy began to appear more sinister.

“Everybody called it ‘Pedophile Island,’” said Kevin Goodrich, who is from St Thomas and operates boat charters. “It’s our dark corner.”

Many people who worked for Epstein told The Associated Press this week that they had signed long non-disclosure agreements, and refused to talk. One former employee, who declined to be identified said Epstein once had five boats, including a large ferry in which he transported up to 200 workers from St Thomas to his island every day for construction work.

The man said he saw a handful of young women when he was on Epstein’s property but he believed they were older than 18.

“When he was there, it was keep to yourself and do your thing,” the man recalled, adding that Epstein paid well and would give away older machinery and surplus, including lumber, to his employees.

Epstein built a stone mansion with cream-coloured walls and a bright turquoise roof surrounded by several other structures including the maids’ quarters and a massive, square-shaped white building on one end of the island. Workers told each other it was a music room fitted with a grand piano and acoustic walls. Its gold dome flew off during the deadly 2017 hurricane season. Locals recalled seeing Epstein’s black helicopter flying back and forth from the tiny international airport in St Thomas to his helipad on Little St James Island, a roughly 75-acre retreat a little over a mile (about 2 kilometres) south-east of St Thomas.

Government documents and ledgers show that Little St James Island was once known as Mingo Cay. In April 1998, a company called LSJ LLC bought it for $7.95 million. Additional documents showed that Epstein is the sole member of LSJ, with his signature reminiscent of an infinity loop.

Epstein later bought neighbouring Great St James Island, which once was popular with locals and tourists for its main attraction, Christmas Cove, a place where you could hang out and order pizza and have it delivered via boat.

“He wasn’t well-received,” recalled Spencer Consolvo, a St Thomas native who runs a tourist shop near a large marina. “People think he’s too rich to be policed properly.”

According to public records obtained by the AP, the island was once worth $3 million and owned by a woman who died in a chalet in Switzerland. The island had been divided into parcels and given to three people and was later sold to Epstein.

Federal authorities consider the smaller of the two islands to be Epstein’s primary residence in the United States, a place where at least one alleged victim said in a court affidavit that she participated in an orgy, as well as had sex with Epstein and other people. She said she saw former US President Bill Clinton on the island, but that she never saw him having sex with anyone. A Clinton spokesman issued a statement saying he never visited there.

A day after he pleaded not guilty in a New York courtroom to charges of sexually abusing dozens of underage girls, there was scant movement on the Caribbean island. Hurricane shutters covered the windows, locals hadn’t seen any lights at night and a lone worker drove a bright blue golf cart around the property.

At a nearby office that locals say Epstein owns in a seaside strip mall, a man in a T-shirt and sunglasses on his head opened the door a crack, shook his head vehemently when asked about Epstein and locked the door. The firm, Southern Trust Company Inc, hired Cecile de Jongh, wife of former Governor John de Jongh, as its office manager, according to records with the US Virgin Islands Economic Development Authority.

Meanwhile, Epstein’s arrest also prompted the US Virgin Islands representative in Congress, Stacey Plaskett, to announce she would give the money Epstein had donated to her campaigns to charitable groups.

Now that Epstein has been arrested a second time, locals say tourists are increasingly asking about his islands when they visit St Thomas. A woman, who did not want to be identified for fear of losing her job running a charter company, said she was elated when Epstein got arrested but is now vexed at tourists’ curiosity, saying she reluctantly shares whispered details of his case to prying adults if children are around.

Some of that fascination aggravates Vernon Morgan, a taxi driver and St Thomas native.

“It brought some kind of notoriety to the Virgin Islands,” he said. “We would much rather that the Virgin Islands be seen in a different light.”