The project, being developed by EDF and Fishermen’s Energy, was due to comprise three 8.3MW turbines from MHI Vestas.
It was formerly known as the six-turbine 24MW Atlantic City project, prior to EDF's involvement in the site.
Since being awarded a final construction permit in 2012, it has been mired in delays, with state officials questioning its feasibility.
"The New Jersey Board of Public Utilities’ decision on Nautilus Offshore Wind is disappointing to say the least," Doug Copeland, EDF Renewables’ regional project development manager.
"With permitting already in place, Nautilus is the only project capable of giving New Jersey an early lead in the [US] offshore wind race.
"This rejection will delay workers’ access to local and real-world training on offshore wind construction and slow early investment in critical supply chain infrastructure needed to support more large-scale projects," he added.
The BPU said under the Offshore Wind Economic Development Act, the project "must demonstrate positive economic and environmental net benefits to the state".
The developers "did not provide the necessary information" to allow the board to "validate the decisions" the regulatory body said.
"The price quoted by Nautilus was too high given the unsubstantiated benefits, and therefore an unacceptable burden for the state’s ratepayers," it added.
Under its guise as the Atlantic City project, numerous questions about the project’s feasibility, including its price, were raised. This caused it to lose US Department of Energy funding.
This latest decision stutters the momentum generated by the US offshore wind industry, following the Massachusetts offshore area lease sale, which attracted record bid prices.
Liz Burdock, the CEO of the Busines Network of Offshore Wind, said any project, regardless of size, would play an important role in the US offshore sector.
"Although Nautilus was a small project, it has always offered significant value beyond the clean power in the form of optimising the efficiency of the New Jersey offshore wind supply chain and business community," she said.
In the same announcement, however, New Jersey's BPU said it approved a funding mechanism to support the state's aim of procuring 3.5GW of offshore wind capacity by 2030.
"The funding mechanism, known as the Offshore Wind Renewable Energy Certificate (OREC), establishes the process by which all offshore wind projects will be funded and how revenues will flow back to ratepayers," the BPU said.
In September, the state opened a solicitation for applications of projects for up to 1.1GW of capacity. Applications will be accepted until the end of December.