Simplex orchestration layer preliminary implementation by yacovm · Pull Request #427 · ava-labs/Simplex

yacovm · 2026-06-29T12:40:44Z

This commit contains an implementation of an orchestration layer that orchestrates
epoch transition for validators and non-validators as they move through epochs.

The orchestration layer introduces an Instance that drives the epoch lifecycle,
switching between validator consensus and non-validator block tracking as the
node's role changes across epochs.

Key pieces:

Instance: manages the per-epoch lifecycle, ticks the active epoch or
non-validator, and handles the handoff at epoch boundaries.
epochChangeSupression: drops outgoing messages and cancels VM operations
while an epoch change is in progress, preventing side effects mid-transition.
EpochAwareStorage: ignores blocks from previous epochs and signals when a
new epoch is detected.
Config: wiring for the P-chain, storage, crypto, and networking dependencies.

samliok · 2026-06-29T20:09:30Z

+// Copyright (C) 2019-2025, Ava Labs, Inc. All rights reserved.
+// See the file LICENSE for licensing terms.
+
+package simplex


should the orchastration be in its own package?

I think it makes sense to put this in the top most package, to make it clear that it is the entry point to simplex.

samliok · 2026-06-29T20:14:40Z

+		// Parameter config
+		Epoch:                      epochNum,
+		ReplicationEnabled:         true,
+		StartTime:                  time.Now(),


should the orchestration layer have its own time field? i don't see where we forward/call advanceTime

samliok · 2026-06-29T20:18:37Z

+		MaxProposalWait:            i.Config.MaxProposalWait,
+		MaxRebroadcastWait:         i.Config.MaxRebroadcastWait,


i think we should change in a seperate pr these to be consistent with avalanche go

MaxNetworkDelay time.Duration `json:"maxNetworkDelay" yaml:"maxNetworkDelay"` MaxRebroadcastWait time.Duration `json:"maxRebroadcastWait" yaml:"maxRebroadcastWait"`

samliok · 2026-06-29T20:19:45Z

+)
+
+
+type Config struct {


maybe this type of structure is more organized? this way we can see exactly what is used for the MSM vs epoch vs shared

type Config struct { // Shared by both the metadata state machine and the epoch. Common CommonConfig // Used only to build the metadata state machine. MSM MSMConfig // Used only to build the simplex epoch. Epoch EpochParams } type CommonConfig struct { Logger common.Logger Signer common.Signer SignatureAggregatorCreator common.SignatureAggregatorCreator VM VM } type MSMConfig struct { GenesisValidatorSet metadata.NodeBLSMappings LastNonSimplexInnerBlock metadata.VMBlock GetPChainHeightForProposing func() uint64 GetPChainHeightForVerifying func() uint64 ComputeICMEpoch metadata.ICMEpochTransition } type EpochParams struct { ID common.NodeID Sender Sender Storage Storage QCDeserializer common.QCDeserializer BlockDeserializer common.BlockDeserializer Verifier common.SignatureVerifier MaxProposalWait time.Duration MaxRoundWindow uint64 MaxRebroadcastWait time.Duration FinalizeRebroadcastTimeout time.Duration MaxWALSize int WALCreator wal.Creator WALs []wal.DeletableWAL }

this could also help with the organization we talked about at Monday's consensus sync. The caller can see which interfaces it needs to implement and which purposes

samliok · 2026-06-29T20:24:11Z

+	PChainProgressListener metadata.PChainProgressListener
+	// LastNonSimplexBlockPChainHeight is the P-chain height of the last block built by a non-Simplex proposer.
+	// It is used to determine the validator set of the first ever Simplex h.
+	LastNonSimplexBlockPChainHeight uint64


dont see where this is uesd

samliok · 2026-06-29T20:24:34Z

+
+	ComputeICMEpoch metadata.ICMEpochTransition
+	// PChainProgressListener listens for changes in the P-chain height to trigger block building or h transitions.
+	PChainProgressListener metadata.PChainProgressListener


same, this isn't used anywhere

samliok · 2026-06-29T20:37:10Z

+
+func (e *EpochAwareStorage) Index(ctx context.Context, block common.VerifiedBlock, certificate common.Finalization) error {
+	if block.BlockHeader().Epoch < e.Epoch {
+		// This is a Telock from a previous h, so we ignore it and do not index it.


how does this mean the block is a telock? don't telocks have the same epoch as the sealing block?

this is a Telock from the previous epoch. We may collect a finalization on a Telock and then we first index the sealing block and then the Telocks. Once we index the sealing block, we increment our epoch, right?
We don't want to index the Telocks too, so this is a safeguard against that.

samliok

leaving what i have, was not able to review as much as i hoped today

samliok · 2026-07-08T16:03:40Z

+type epochChangeSupression struct {
+	lock            sync.RWMutex
+	sealingBlockSeq uint64
+	active          bool
+}
+
+func (ecs *epochChangeSupression) isSupressionActive() bool {
+	ecs.lock.RLock()
+	defer ecs.lock.RUnlock()
+	return ecs.active
+}
+
+func (ecs *epochChangeSupression) sendProhibited(seq uint64) bool {
+	ecs.lock.RLock()
+	defer ecs.lock.RUnlock()
+	if !ecs.active {
+		return false
+	}
+	return seq > ecs.sealingBlockSeq
+}
+
+func (ecs *epochChangeSupression) setSupression(sealingBlockSeq uint64) {
+	ecs.lock.Lock()
+	defer ecs.lock.Unlock()
+	ecs.sealingBlockSeq = sealingBlockSeq
+	ecs.active = true
+}
+
+func (ecs *epochChangeSupression) clearSupression() {
+	ecs.lock.Lock()
+	defer ecs.lock.Unlock()
+	ecs.active = false
+}
+
+type Communication struct {
+	epochChangeSupression *epochChangeSupression
+	nodes                 atomic.Value // common.Nodes
+	Sender
+	Broadcaster
+}
+
+func (c *Communication) SetValidators(nodes common.Nodes) {
+	c.nodes.Store(nodes)
+}
+
+func (c *Communication) Validators() common.Nodes {
+	nodes, ok := c.nodes.Load().(common.Nodes)
+	if !ok {
+		return nil
+	}
+	return nodes
+}
+
+func (c *Communication) Broadcast(msg *common.Message) {
+	if c.epochChangeSupression.sendProhibited(msg.Seq()) {
+		return
+	}
+
+	c.Broadcaster.Broadcast(msg)
+}
+
+func (c *Communication) Send(msg *common.Message, destination common.NodeID) {
+	if c.epochChangeSupression.sendProhibited(msg.Seq()) {
+		return
+	}
+
+	c.Sender.Send(msg, destination)
+}


i think we should remove this comm adapter completely.

IMO the epoch should be the one filtering these messages themselves. I think it would simplify the orchestrator code a bit, remove some weirdness with the msg.Seq function, and make the epoch more explicit about how it handles future epoch messages.

plus if we add other messages to the epoch we need to remember to properly adjust the msg.Seq which is not that intuitive. Because whether a message gets sent to the epoch is actually controlled by msg.Seq() which is not an ideal place.

For example the block digest request literally has a seq field and you would expect msg.seq to return BlockDigestRequest.seq but instead it returns 0.

it might make sense creating the epoch.go changes in a separate pr to be merged before this one?

samliok · 2026-07-08T21:11:26Z

 	// P-chain height referenced by the last block in the chain and until the provided P-chain height.
 	hasValidatorSetChanged func(pChainHeight uint64) (bool, NodeBLSMappings, error)
-	getPChainHeight        func() uint64
+	getPChainHeight        func(ctx context.Context) (uint64, error)


are these changes important to the orchestrator? if not can we separate them?

277eb0a#diff-6e7f518f13f3f50e665a925ee4fa645f1ad8d2a9c222dbd80950cb08f29e7d65

samliok · 2026-07-08T21:23:28Z

 }

+// mockBlockBuilder is a test BlockBuilder whose BuildBlock returns a preconfigured block/error.
+type mockBlockBuilder struct {


i think the use of the word mock will be triggering to some members of the avalanchego team

This commit contains an implementation of an orchestration layer that orchestrates epoch transition for validators and non-validators as they move through epochs. The orchestration layer introduces an Instance that drives the epoch lifecycle, switching between validator consensus and non-validator block tracking as the node's role changes across epochs. Key pieces: - Instance: manages the per-epoch lifecycle, ticks the active epoch or non-validator, and handles the handoff at epoch boundaries. - epochChangeSupression: drops outgoing messages and cancels VM operations while an epoch change is in progress, preventing side effects mid-transition. - EpochAwareStorage: ignores blocks from previous epochs and signals when a new epoch is detected. - Config: wiring for the P-chain, storage, crypto, and networking dependencies. Signed-off-by: Yacov Manevich <yacov.manevich@avalabs.org>

Signed-off-by: Yacov Manevich <yacov.manevich@avalabs.org>

samliok · 2026-07-09T15:45:33Z

 )

+const (
+	DefaultMaxSequenceWindow = 100


i think this should be a similar value to DefaultMaxRoundWindow, if not the same. Mainly because validators will reject replication that have more seqs than MaxRoundWindow.

BTW, I think we should change how we process replication requests, to not drop messages that request too many sequences but rather fill up the response with at most MaxRoundWindow. Also, we should ensure this message does not exceed the maximum number of bytes AvalancheGo clients can accept.

Yeah good catch, I removed and changed it here.

samliok · 2026-07-09T16:38:21Z

+	return lastBlock, numBlocks, nil
+}
+
+func (i *Instance) determineValidatorOrNot(nodes common.Nodes) bool {


thoughts on changing this to nodes.Contains(myNodeId)?

or making the function header a bit more clear

Suggested change

func (i *Instance) determineValidatorOrNot(nodes common.Nodes) bool {

func (i *Instance) isMyNodeValidator(validators common.Nodes) bool {

Changed it to iAmValidator before seeing this comment, I think it's equivalent.

samliok · 2026-07-09T16:41:42Z

+type MessageHandler interface {
+}
+
+type nodeRole byte


this nodeRole variable is confusing. It represents the previous role of the node right? Why do we need to know this?

maybe something like this works?
#434

We need to know this in order to know which node type to shutdown and whether to skip shutting down a non-validator in case it is still a non validator in the next epoch.

I changed the code significantly before looking at this suggestion so please let me know if you think it's still relevant.

samliok · 2026-07-09T16:55:03Z

+	config, err := i.createEpochConfig()
+	if err != nil {
+		i.Config.Logger.Error("Error creating epoch config on epoch change", zap.Error(err))
+		return
+	}
+
+	if config.Epoch != epochChange.epochNum {
+		i.Config.Logger.Error("Epoch number mismatch on epoch change", zap.Uint64("expected", epochChange.epochNum), zap.Uint64("actual", config.Epoch))
+		return
+	}
+
+	if err := i.startEpoch(config); err != nil {
+		i.Config.Logger.Error("Error starting new epoch on epoch change", zap.Error(err))
+	}


i think we should have one unified way of starting a validator. We have this logic here, but we have a different method startValidator

Agreed, I changed accordingly.

samliok · 2026-07-09T17:36:50Z

+	// First, figure out if I'm still a validator.
+	if i.determineValidatorOrNot(epochChange.validators) {
+		i.Config.Logger.Info("I am now a validator")
+		i.startValidator()


we are dropping the error here, shouldn't this be propagated up? Otherwise, we will just be dropping messages not knowing why we failed starting

whoops, good catch. Made us halting the node entirely if we fail.

samliok · 2026-07-09T17:46:26Z

+	cachedStorage := NewCachedStorage(i.Config.Storage)
+	i.cs = cachedStorage
+
+	lastBlock, numBlocks, err := i.lastBlock()


missing error check

fixed, thanks

samliok · 2026-07-09T18:01:00Z

+	msm           *metadata.StateMachine
+	OnEpochChange func(seq uint64, validators common.Nodes) error
+	Storage
+	Epoch uint64
+}


why are some of these fields exported while others aren't?

samliok · 2026-07-09T18:04:30Z

+	// GetBlock retrieves the block and finalization at [seq] with the given digest.
+	// If the digest is nil, the block with the given sequence number is returned.
+	// If [seq] the block cannot be found, returns ErrBlockNotFound.
+	GetBlock(seq uint64) (metadata.StateMachineBlock, *common.Finalization, error)


can we use the same terminology as simplex.Storage? maybe we can embed simplex.Storage in this API?

type Storage interface { simplex.Storage // CreateWAL creates a new Write-Ahead Log (WAL). CreateWAL() (wal.DeletableWAL, error)

why do couple the WAL with the storage anyways?

can we use the same terminology as simplex.Storage? maybe we can embed simplex.Storage in this API?

We can't, Retrieve conflicts because the parameters are different.

./instance.go:468:12: cannot use i.cs (variable of type *CachedStorage) as Storage value in struct literal: *CachedStorage does not implement Storage (wrong type for method Retrieve) have Retrieve(uint64, [32]byte) (common.VerifiedBlock, *common.Finalization, error) want Retrieve(uint64) (common.VerifiedBlock, common.Finalization, error)

why do couple the WAL with the storage anyways?

Yeah, so i was trying to minimize the number of dependencies. I was too frugal to dedicate an entire interface for a single method. I can do it though

yea i think it makes sense to separate it

https://github.com/ava-labs/Simplex/pull/427/changes#diff-f4581317ccb65231765f0fa1aec4aa2bb1c08b09fe9dc1faf095d9039483e4fdR39

samliok · 2026-07-09T18:46:32Z

+	defer i.lock.Unlock()
+
+	// Stop the non-validator before doing anything else, so that we don't process any more messages while we are changing epochs.
+	i.stopNonValidator()


i dont get why we need to stop and restart a non-validator for every epoch change. This actually seems like a problem during a non-validators bootstrapping phase.

Non-validators backwards hash-chain validate epochs. This process involves storing requested future sealing blocks in memory and then indexing in order. However, when we index a sealing block, we would restart the non-validator and therefore clear its cached memory of sealing blocks meaning it needs to restart the requesting phase.

The non-validator is epoch agnostic so I think if we transition from non-validator -> non-validator all we need to do is update the communication interface to return the latest validator set.

even further, if we go from non-validator -> validator, and we are aware of a future epoch we should still be in the non-validating phase?

Yeah, all very good callouts 👍

I was aware that the current approach isn't ideal but was in a hurry to meet the end of month deadline.

Take a look at the latest code version in the meantime. I will add a test a bit later that will check that we don't needlessly restart non-validators.

https://github.com/ava-labs/Simplex/pull/427/changes#diff-f4581317ccb65231765f0fa1aec4aa2bb1c08b09fe9dc1faf095d9039483e4fdR153-R165

https://github.com/ava-labs/Simplex/pull/427/changes#diff-f4581317ccb65231765f0fa1aec4aa2bb1c08b09fe9dc1faf095d9039483e4fdR523-R535

Signed-off-by: Yacov Manevich <yacov.manevich@avalabs.org>

samliok · 2026-07-09T19:08:05Z

+
+type CryptoOps interface {
+	Sign(message []byte) ([]byte, error)
+	AggregateKeys(keys ...[]byte) ([]byte, error)


instead of redefining the functions should we just inline the interfaces from msm

Suggested change

AggregateKeys(keys ...[]byte) ([]byte, error)

metadata.KeyAggregator

same with the others

common.Signer common.SignatureVerifier common.QCDeserializer metadata.KeyAggregator

samliok · 2026-07-09T19:14:50Z

+type ParameterConfig struct {
+	// WalMaxEntryCount is the maximum number of entries in the write-ahead log before it is closed.
+	WALMaxEntryCount int
+	// MaxnetworkDelay is the assumed upper bound on the network delay for messages to be delivered.


Suggested change

// MaxnetworkDelay is the assumed upper bound on the network delay for messages to be delivered.

// MaxNetworkDelay is the assumed upper bound on the network delay for messages to be delivered.

samliok · 2026-07-09T20:00:06Z

+	// GetBlock retrieves the block and finalization at [seq] with the given digest.
+	// If the digest is nil, the block with the given sequence number is returned.
+	// If [seq] the block cannot be found, returns ErrBlockNotFound.
+	GetBlock(seq uint64) (metadata.StateMachineBlock, *common.Finalization, error)


yea i think it makes sense to separate it

samliok · 2026-07-09T21:36:14Z

 	}
 }

+func (m *Message) Seq() uint64 {


wait why is this added back?

samliok reviewed Jun 29, 2026

View reviewed changes

yacovm force-pushed the orchestration branch 9 times, most recently from ed41178 to 55576be Compare July 6, 2026 22:46

yacovm force-pushed the orchestration branch from 55576be to cd2a874 Compare July 7, 2026 20:09

yacovm changed the title ~~Epoch orchestration layer~~ Simplex orchestration layer preliminary implementation Jul 7, 2026

yacovm force-pushed the orchestration branch from cd2a874 to 23ee8f0 Compare July 7, 2026 20:17

samliok reviewed Jul 8, 2026

View reviewed changes

yacovm force-pushed the orchestration branch 2 times, most recently from 1bfcbc4 to 5eeffd9 Compare July 9, 2026 16:05

yacovm added 5 commits July 9, 2026 18:27

Revert P-chain height getters function signature change

3d2625b

Signed-off-by: Yacov Manevich <yacov.manevich@avalabs.org>

Remove artificial silence to sealed epoch

fcef0a9

Signed-off-by: Yacov Manevich <yacov.manevich@avalabs.org>

Rename structs used in tests

9924691

Signed-off-by: Yacov Manevich <yacov.manevich@avalabs.org>

gofmt

e6b32ca

Signed-off-by: Yacov Manevich <yacov.manevich@avalabs.org>

yacovm force-pushed the orchestration branch from 5eeffd9 to e6b32ca Compare July 9, 2026 16:28

samliok reviewed Jul 9, 2026

View reviewed changes

Address code review comments

baadba5

Signed-off-by: Yacov Manevich <yacov.manevich@avalabs.org>

yacovm force-pushed the orchestration branch from a24b89d to baadba5 Compare July 9, 2026 19:49

Address code review comments II

f336772

Signed-off-by: Yacov Manevich <yacov.manevich@avalabs.org>

samliok reviewed Jul 9, 2026

View reviewed changes

		MaxProposalWait: i.Config.MaxProposalWait,
		MaxRebroadcastWait: i.Config.MaxRebroadcastWait,

	func (i *Instance) determineValidatorOrNot(nodes common.Nodes) bool {
	func (i *Instance) isMyNodeValidator(validators common.Nodes) bool {

	AggregateKeys(keys ...[]byte) ([]byte, error)
	metadata.KeyAggregator

	// MaxnetworkDelay is the assumed upper bound on the network delay for messages to be delivered.
	// MaxNetworkDelay is the assumed upper bound on the network delay for messages to be delivered.

Uh oh!

Conversation

yacovm commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samliok left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

yacovm commented Jun 29, 2026 •

edited

Loading