A US government agency says it has attained the “holy grail” of energy – the next-generation system of battery storage, that has has been hotly pursued by the likes of Bill Gates and Elon Musk.

Advanced Research Projects Agency-Energy (Arpa-E) – a branch of the Department of Energy – says it achieved its breakthrough technology in seven years.

Ellen Williams, Arpa-E’s director, said: “I think we have reached some holy grails in batteries – just in the sense of demonstrating that we can create a totally new approach to battery technology, make it work, make it commercially viable, and get it out there to let it do its thing,”

If that’s the case, Arpa-E has come out ahead of Gates and Musk in the multi-billion-dollar race to build the next generation battery for power companies and home storage.

Arpa-E was founded in 2009 under Barack Obama’s economic recovery plan to fund early stage research into the generation and storage of energy.

Such projects, or so-called moonshots, were widely seen as too risky for regular investors, but – if they succeed – could potentially be game-changing.

Many of the projects fostered by the agency were already in sight of getting funding, Williams said. Several have now secured private sector follow-on funding. Others are being taken up by the State Department or the Pentagon, she said.

But the biggest breakthrough is in the area of energy storage. “I think that’s one area where we have delivered big time,” Williams told the Guardian.

The battery storage systems developed with Arpa-E’s support are on the verge of transforming America’s electrical grid, a transformation that could unfold within the next five to 10 years, Williams said.

The most promising developments are in the realm of large-scale energy storage systems, which electricity companies need to put in place to bring more solar and wind power on to the grid.

She said projects funded by Arpa-E had the potential to transform utility-scale storage, and expand the use of micro-grids by the military and for disaster relief. Projects were also developing faster and more efficient super conductors, and relying on new materials beyond current lithium-ion batteries.

The companies incubated at Arpa-E have developed new designs for batteries, and new chemistries, which are rapidly bringing down the costs of energy storage, she said. 

“Our battery teams have developed new approaches to grid-scale batteries and moved them out,” Williams said. Three companies now have batteries on the market, selling grid-scale and back-up batteries. Half a dozen other companies are developing new batteries, she added. 

Williams was speaking on the sidelines of Arpa-E’s annual gathering of researchers and investors this week.

Arpa-E has been upbeat in the past about emerging technologies. But researchers have struggled and failed to replicate such successes at greater scale and lower cost outside the research lab - a challenge Gates describes as the valley if death for innovation.

The agency, until now, has focused on funding early-stage research – turning projects around every three years – a faster clip than academia but still longer than private investors. 

But Ernie Moniz, the energy secretary, said this week he was seeking to expand Arpa-E’s mission, in the light of a commitment from the US and 19 other governments on the sidelines of the Paris climate summit to double clean energy research funding over the next five years.

In a parallel initiative, Gates and two dozen other billionaires have committed to spend $2bn on energy research. 

In addition to energy storage, the agency is backing researchers trying to find a faster system of tracing natural gas leaks – and so reduce the risks of a massive leak like the one just plugged in California. 

Other Arpa-E backed researchers are using robots and drones to breed better biofuels, focusing on sorghum. Another team is responding to the risk of drought by working on industrial cooling systems that do not rely on water. Researchers are also working with the US military on low-energy air conditioning for battlefield conditions. 

But Williams said she anticipates many more projects will focus on exercising more personal control of energy use, especially through the internet.

“We want power to be easy,” she said. “So I think one of the things that we will see in the next five to 10 years is a whole set of ideas about the internet of things start[ing] to develop in ways to make it easier, or even make it fun, for people to optimise their energy use.”