New report into scheduling and ahead markets

Today we are pleased to publish a major report[i] the Australian Energy Council commissioned into the question of Scheduling and Ahead Markets, prepared by the highly experienced National Electricity Market (NEM) economists, David Smith of Creative Energy Consulting and Veronika Nemes of MarketWise Solutions.

This is a matter being deeply investigated by the Energy Security Board (ESB) as part of its “Post 2025 NEM review”[ii] as it has been advocated in many Australian Energy Market Operator (AEMO) technical reports[iii]. In fact, the March COAG Energy Council meeting[iv] went as far as directing the ESB to develop “a new framework for … ahead market arrangements” for the NEM.

But this was a very open direction with an infinite range of possible interpretations. Australian Energy Council (AEC) members were unsurprisingly keen to receive expert advice on:

• what it is
• why it is needed
• what in the NEM would have to change to accommodate it, and
• whether other options existed that resolved those needs.

These are the questions we put to our advisers.

We briefly discuss the report below, however we urge readers to invest the time to give the report full justice, which is a deeply engaging and enlightening read.

How the NEM works now

Much of Creative Energy’s report is dedicated to filling a gap in the material published to date promoting ahead markets, by explaining the NEM’s existing arrangements. The NEM has an elegantly simple scheduling and settlement process:

• AEMO forecasts demand;
• Generators submit initial bids to AEMO;
• AEMO combines the forecasts and the bids to produce a non-binding predispatch forecast of dispatch and prices;
• Generators review that forecast and adjust their individual plans to better suit their commercial drivers, which they must immediately rebid back into AEMO;
• AEMO recalculates the predispatch forecast, and the process re-iterates 92 times[v] until the moment of dispatch, converging towards an optimal final dispatch outcome;
• All generation and load is settled only once against one price – that calculated at the moment of dispatch. The 92 forecasts are discarded.

For those only experienced in the NEM, this scheduling and dispatch process is natural and intuitive. However many other electricity markets, particularly in the US, employ much more complex approaches using multiple settlement prices and a more direct role for the market operator in generators’ operations. Academics refer to the NEM as a “Single-pass/self-commitment” market, and its alternatives as forms of “Day-ahead and balancing/central-commitment” markets.

The NEM’s rather simple decentralised process remarkably produces a near-optimal solution to the incredibly complex problem of scheduling a big power system. The report describes this magic as a “Mega-Algorithm” but might also be considered an example of Adam Smith’s famous “invisible hand”. And, just like many countries in the twentieth century found when they tried it in their economies, attempts to replace the invisible hand with a central one can prove intractable.

Figure 1: The NEM’s predispatch optimisation presented as a “Mega-Algorithm”

Source: Creative Energy Consulting Report

What is it?

The consultants found our request to clarify the proposal perhaps the most challenging part of the project. Many forms of “aheadness” has been discussed and a recent ESB paper presented a continuum of options between what is little more than a small procedural tweak, through to very fundamental changes in the way the market is priced, hedged and dispatched.

At the minimalist end, it involves AEMO simply applying a more methodical approach to the circumstances where it needs to intervene in the market. This is described as “Unit Commitment for Security” (UCS). The report supports this concept and suggests it can be implemented immediately within the existing rules.

Half-way into the spectrum, there are options that are purely financial, and may be optional or mandatory. For example, it could involve a centralised clearing of willing buyers and sellers of short-term derivative instruments a few hours ahead of dispatch.

At the maximalist extreme, two fundamental changes are in play:

• Mandatory multi-pass settlement. This implies all the prices and volumes in one of the 92 predispatch iterations are “locked-in” and market settled between participants, i.e. we settle retailers’ forecast demands and generators’ forecast dispatch at the forecast price: an “ahead settlement”. Then, because things will subsequently change, we resettle the variations between those results and the final dispatch as a “balancing settlement” (see Figure 2 below). Examples of multiple settlement markets include the Western Australian Wholesale Electricity Market (WEM) and the Victorian Declared Wholesale Gas Market. (DWGM).

• Central commitment. This is where the market operator, rather than participants, determines when to start and stop generating units. The market operator uses an estimate of a plant’s true start costs, startup times, running costs and technical characteristics to determine when the plants should run and refunds these costs separately. This was attempted briefly in the 1990s in Victoria in “VicPool II”, a NEM predecessor which drew from early US market designs, but which was subsequently abandoned in favour of self-commitment, as was being used in the UK and has since been used in Australia.

Figure 2: Example of Multipass settlement

Source: Creative Energy Consulting Report

The problem

The rationale supporting the need for an ahead market is, as you may have guessed, the dramatically changing power system with its preponderance of variable, asynchronous technologies, and greater demand response and distributed resources. These are dramatic changes indeed, but the report does not find why they should lead to the maximalist options, and indeed the contrary may be the case.

Cited as evidence of a problem in the current approach are the numerous market interventions in the NEM in the last two years, almost all associated with the need to recruit system strength in South Australia. The report finds a very simple explanation – this is a service that the power system needs, but for which there is presently no market reward, so this outcome is entirely predictable. And, the intuitive solution - to create a market for it within the current scheduling arrangements - is also Creative Energy’s recommendation.

As it happens, these recent interventions will soon become moot, as monopoly network equipment is being installed to supply South Australian system strength from non-generation sources.

What would have to change

The implementation challenge is very much dependent on what solution on the minimalist – maximalist spectrum we are contemplating. The UCS described above seems to have near-zero cost and impact. Half-way approaches that are purely financial and optional have some costs, but if optional, participants themselves can decide whether or not they want to invest in order to participate in the new arrangements.

At the maximalist end of mandatory physical participation and multiple settlement passes, the impacts become profound. The dispatch process itself must fundamentally change, and the NEM would need to break new ground, because the US markets apply it with nodal pricing, and their systems are well behind Australia in the technology transition. Their central-commitment mechanisms have proved tractable as they operate largely within traditional, well-understood generation technology, but will soon face great challenges in centrally committing exotic technologies.

The proposal anticipates costs of startup and minimum load being paid at actual costs rather than a competitive bid price, it introduces a quite different paradigm for the way NEM competitive generators reveal costs, and confuses the incentive to maximise flexibility – something very much needed for the transition to renewables.

Retailers will need to provide their own customers’ forecasts to AEMO and will be held responsible for their accuracy with errors charged in the balancing market.

But perhaps the biggest impact is on the way markets are hedged. With multiple settlement passes and new costs associated with startups, participants will need to think differently as to how they structure their forward contracting. Exactly which of these many volatile settlement streams will they want to stabilise, who will be the counterparty, and how will they adapt their existing contract portfolios to the new world? This is an area which has yet to be considered in the ESB’s work.

Other solutions

Creative Energy’s report does not suggest the challenges of the transforming NEM will be easy to navigate with the existing dispatch and settlement processes. It concludes however, that the problems sometimes presented as justification of the more maximalist versions of ahead markets are more appropriately, and at much less risk and cost, resolved in more intuitive ways.

As discussed above, the most obvious solution is to create ancillary services arrangements for all services that the power system is likely to need from the competitive market, but does not presently price. This is thankfully already underway with a series of rule changes being worked on by the AEMC.[vi]

Where, despite these markets, AEMO still needs to intervene, then the UCS process would provide a more mechanical and repeatable way for this to occur at least cost than the current manual approach. Also, by incorporating the intervention itself into pre-dispatch (the report cheekily describes this as “applying good faith bidding rules to AEMO”) then the market will be able to optimise its decentralised commitment process around it.

In summary

The report provides a table of its main findings in the executive summary which is reproduced below.

[i] Creative Energy Consulting, Scheduling and Ahead Markets Design Options for post-2025 NEM, June 2020

[ii] http://www.coagenergycouncil.gov.au/energy-security-board/post-2025

[iii] See for example AEMO’s Renewable Integration Study: Stage 1 Report, April 2020 Action 2.3 p.8

[iv] Communique of 23^rd COAG Energy Council Meeting, available at http://www.coagenergycouncil.gov.au/publications/23rd-energy-council-meeting-communiqu%C3%A9

[v] 80 half hour plus 12 5 minute predispatch forecasts are published ahead of a real-time dispatch interval.

[vi] https://www.aemc.gov.au/market-reviews-advice/investigation-system-strength-frameworks-nem