These are the central conclusions of a review of the former energy department's management of the LCF, which capped renewable energy spending to prevent increases in household bills.
During the government reshuffle following Theresa May's appointment as Prime Minister in July, Decc was folded in to the business department, creating the new Business, Energy and Industrial Strategy (BEIS) office, which now has the responsibility of renewables deployment.
The internal review was prompted by projections in April 2015 suggesting that total spending on renewable energy was due to exceed the LCF's £7.6 billion (€8.9 billion) budget by around £1 billion (€1.2 billion) in 2020/21.
The projection saw a 26% increase in renewables obligation (RO) spending, 32% for feed-in tariffs (Fits), and a decrease of 8% for contracts for difference (CfDs) between November 2012 and April 2015.
This represented a 14% increase in LCF spending overall, and did not take into account that a £500 million contingency for wholesale prices, load factors and cost uncertainty was being eaten into.
Unlike a more sharply critical report by the National Audit Office in October 2016, which echoed concern over poor governance, the new report focuses on systemic causes of failure.
Two key drivers of overspend were a fundamental fall in the price of wholesale electricity following the drop in the global price of fossil fuels and a surge in demand for both the RO and Fits.
A third was technological change in both existing and new offshore wind turbines that increased load factor and hence pressure on subsidies without benefitting consumers, it said.
The report said the excessive pressure on the RO and FITs as fossil fuel prices started to fall was evident early on. Action to cut costs was taken from autumn 2013, but key assumptions were not changed until April 2015.
It pointed out that the more serious consequence of the LCF breach will be greater constraints on any new low-carbon initiatives before 2021, particularly CfDs.
It also blamed undue certainty attached to forecasts and technology assessments by often less experienced analysts "and an insufficient monitoring of, and response to, developing reality".
These underlying failures, together with several specific issues, had a "ratcheting effect" that increased risks.
Another factor is that CfD contracts aim to give certainty to investors, with little incentive to gather data and adjust as circumstances change for fear of legal challenge.
This is further reinforced by political unwillingness to focus on uncertainties in renewables policy, it added.
The review also noted there has been a tendency to focus on short-term, individual goals rather than overall policy goals, in this case meeting the 32% decarbonisation target. But it said a positive aspect of this has been government's desire to create a self-sustaining renewables sector.
In its recommendations, the report said the government must be more open about the tension between giving investors certainty and volatility of the market and technological assumptions as the low-carbon sector matures.
It also suggested BEIS should consider whether the various elements of the LCF including progress and cost pressures would be better overseen, and monitored continuously, in a more integrated way.
This could potentially be done by one person answerable to the secretary of state, the permanent secretary and the department's executive committee.
Quarterly reports should be produced with greater participation and openness, added the report.
They should make assumptions clear, log key changes in issues such as costs and technology, along with responses of ministers, and include options to deal with any unfavourable developments.
The review recommended assumptions should be published regularly, inviting comment internally and externally, and that a wider field of experts should be sought.
A more radical proposal is to introduce a 'change control mechanism' in complex projects to encourage the energy industry to innovate while returning a share of the benefits to consumers "who have, in effect, under-written it".
It was also recommended BEIS should ensure it has adequate resources and analysts to carry out longer-term scenario planning rather than being forced by resource limitations to focus only on short-term outcomes.
In its response, BEIS stressed a number of policy decisions the government took to control overspend in 2020/21 in solar, wind and biomass through the RO, reduced tariffs and a Fit cap.
Collectively, it said, these should cut spending by £500 million and reduce future overspend.
BEIS also pointed to an additional £730 million for post-2020 projects made available for three auctions by 2020.
Even so, its response confirmed BEIS had accepted all nine recommendations of the report. It added that competition for CfDs through auctions now ensures consumers "benefit through lower strike prices where developers' load factor assumptions are higher than BEIS'".