rfcs

node-graph/rfcs/editor-rework.md

@@ -0,0 +1,16 @@
+The editor has two types of state, persistent and UI state.
+
+Persistent state is everything that will be synced between clients, saved in the file, and saved to the document file. This data is all stored in the NodeNetwork, which is loaded into the runtime when a file is opened.
+
+UI state will be passed through the context in an `Arc<EditorMetadata>`. This allows the node graph to perform different operations based on the current state of the UI for different clients. For example, this can include returning data to the properties panel, or using the selection state to render a bbox overlay over a shape.
+
+The runtime's responsibility is to take request from the editor, and return a list of Frontend UI updates, editor state updates, and a list of updates to the NodeNetwork itself. For example, consider rendering a properties panel. First, the node has to be selected. This is done within the layer node, which stores the click target in the local layer space, transforms the mouse position, and if it intersects it sends an EditorMessage to select that layer. It also uses the &NodeNetwork in the Context to iterate upstream from the node and get NodeIds for the upstream nodes. It uses these upstream IDs to send EditorMessages which add the upstream nodes to the render_properties `HashSet<NodeId>`. Like selection state, this is stored in the editor, cloned when making an evaluation request, and passed as a reference. Then, an editor message is added which requests that the network is evaluated, but from the properties panel generator node. This node uses the upstream inputs to send the properties panel widgets to the editor.
+
+Since all persistent state is stored in the NodeNetwork, the undo/redo history can be implemented in the dynamic executor. The executor stores a hashmap representing the previous state for each node input that gets set between history steps, and a mapping of node ID to added/removed nodes. Then, when adding a history step, it pushes this list of changes into a vec. When undoing, a request is made which sets the network to this previous state, and pushes the previous inputs into the redo queue.
+
+For example, the requests:
+AddNode(1)
+SetInput(1, Value::Int(1))
+
+Then on a undo, this stored input/node is reapplied. This relies on the fact that setting an input is a reversable operation. It also means a CRDT is not necessary, since the client stores no stateful information regarding the node graph. Multiple clients make requests to the runtime, which is the single source of truth, and it returns data back to client based on the parameters passed in the request, which is used to evaluate the network.
+

node-graph/rfcs/incremental-compilation.md

@@ -7,13 +7,13 @@ Any time a NodeInput is changed in the network the following changes to the borr
 3. Update the SNI of the downstream nodes (Not necessary if ghostcell is used and downstream nodes can have their inputs mutated)
 4. Reset any downstream caches
 
-We currently clone and recompile the entire network, which is very expensive. As networks grow larger and functionality gets split into separate nodes, a reasonable network may have hundreds of thousands of nodes. I believe with the following data structure and algorithm changes, it will be possible to implement incremental compilation, where a diff of changes is sent to the borrow tree. The overall goal of this method is the editor "damages" the node network by performing a series of set_input requests that invalidate downstream SNI's, and get queued in the compiler. Then, a compilation request iterates over the network, and uses the input requests, global cached compilation metadata,and NODE_REGISTRY to send a diff of updates to to the executor. These updates are `Add(SNI, (Constructor, Vec<SNI>))` and `Remove (SNI)` requests. In the future, GhostCell could be used to make a new `Modify((SNI, usize), SNI)` request, which remaps a nodes input to a new SharedNodeContainer.
+We currently clone and recompile the entire network, which is very expensive. As networks grow larger and functionality gets split into separate nodes, a reasonable network may have hundreds of thousands of nodes. I believe with the following data structure and algorithm changes, it will be possible to implement incremental compilation, where only the nodes which change need to be recompiled, and a compilation can only be done upstream of a specified input. The overall system requires the editor to send topological changes to the runtime. These changes include adding nodes, removing nodes, and setting inputs. These changes get queued in the compiler. Then, a compilation request iterates over the network from a certain input, and uses the queued requests, global cached compilation metadata,and NODE_REGISTRY to update the borrow tree.
 
 # New Editor Workflow
-Each document still stores a NodeNetwork, but this is used just for serialization. When a document is opened, the NodeNetwork gets moved into the compiler (within NodeId(0), the render node is NodeId(1), and stays the same). When the document is closed, the network within NodeId(0) is taken from the compiler and moved into the document. All changes such as add node, remove node, and set input get sent to the compiler. Then on a compilation request, the set input requests are applied, the NodeNetwork is modified, and the diff of changes is generated to be applied to the borrow tree. 
+Each document still stores a NodeNetwork, but this is used just for serialization. When a document is opened, the NodeNetwork gets moved into the compiler (within NodeId(0), the render node is NodeId(1), and stays the same). When the document is closed, the network within NodeId(0) is taken from the compiler and moved into the document. All changes such as add node, remove node, and set input get sent to the compiler. Then on a compilation request, a traversal is performed from the specified input, the set input requests are applied, the NodeNetwork is modified, and the borrow tree gets updated.
 
 # Editor changes:
-The editor now performs all compilation, type resolution, and stable node id generation.
+The editor now just stores editor-specific information such as the metadata for each node,
 
 ```rust
 struct SNI(u64);
@@ -24,19 +24,35 @@ SNI represents the stable node id of a protonode, which is calculated based on t
 ProtonodeInput is the path to the protonode DocumentNode in the recursive NodeNetwork structure, as well as the input index.
 
 ```rust
-PortfolioMessageHandler { 
-    pub compiler_metadata: HashMap<SNI, CompilerMetadata>
-}
 
-// Used by the compiler and editor to send diff based changes to the runtime and cache the types so the constructor can be easily looked up
-pub struct NodeNetworkCompiler {
-    resolved_type: NodeIOTypes,
-    // How many document nodes with this SNI exist. Could possibly be moved to the executor
-    usages: usize,
+// Stored in the runtime
+pub struct DynamicExecutor {
+    // Stores cached state for each inserted protonode, which includes types, context dependencies of all upstream nodes, and how many usages exist in the NodeNetwork
+    borrow_tree: HashMap<SNI, InsertedProtonode>,
     // The previously compiled NodeNetwork, which represents the current state of the borrow tree.
     network: NodeNetwork
-    // A series of SetInput requests which are queued between compilations
+    // A series of set input, add node, and remove node requests which are queued between compilations
+    add_node: HashMap<NodeId, DocumentNode>,
     input_requests: HashMap<InputConnector, NodeInput>
+    remove_node: HashSet<NodeId>
+
+    // The queue of changes to apply between history steps
+    undo_queue: Vec<NetworkUpdateRequest>
+}
+
+// Sent from the editor
+pub enum NetworkUpdateRequest {
+    SetInput(InputConnector, NodeInput)
+    AddNode(NodeId, DocumentNode),
+    RemoveNode(NodeId),
+}
+
+pub struct InsertedProtonode {
+    // Either the value node, cache node if output is clone, or protonode if output is not clone
+	cached_protonode: SharedNodeContainer,
+    resolved_type: NodeIOTypes,
+    // How many document nodes with this SNI exist.
+    usages: usize,
 }
 ```