Introduction

This book describes the purpose of the Peace automation framework, and concepts used in its design.

How To Read This Book

Depending on your frame of mind:

What – Show me the framework in action:

Have a play with the live Examples.
Why – Show me the rationale guiding the decisions:

Head on through to the Background section.
How – Show me how it works:

Jump to Technical Concepts.

This is essential reading for framework consumers.

Background

Automation, like most subjects, has many questions to ask and answer. The following is a selection of issues that are common in this space:

Before running the automation:
- What is the current state?
- What will the automation do?
When it is running:
- Where is it up to?
- When will things be complete?
When it errs:
- What went wrong?
- Must I begin from scratch again?
- How can I solve this myself?
When it completes successfully:
- Can I see a report of what's changed?
- Can it send a notification?

A lot of the responsibility and weight of these questions is placed on automation teams. Issues that originate from many separate systems surface through the automation, and are often reported to the automation team, potentially resulting in burnout.

For users, it is frustrating when it is difficult to understand what the automation is doing, where one is in the process, and how long it takes. This is amplified when the process errs after a lot of waiting – if it ever ends. Can one recover from errors, or end up feeling lost?

Amidst the chaos, pressure, and expectation, have a little.. peace ✌️.

Purpose

The purpose of the Peace framework is to bring people joy and peace of mind in automation development and usage. This is an active effort to take away mental and emotional stress and frustration experienced with software.

For maintainers, Peace aims to guide developers to build and support resilient automation.

For users, Peace aims to make automation work surprisingly well, and reduce the feeling of being lost or frustrated when things don't go according to plan.

Strategy

Outcomes

The main priority is to take care of people. This is correlated to the following measurable dimensions:

Time optimization
Effort optimization
Cost optimization
Decision making

If these outputs trend in a favourable direction, then surveying people's well-being and satisfaction should show similar outcomes.

Input Dimensions

The following input dimensions are considered when designing the Peace framework:

Approach: Choose an optimal method.
Correctness: Take the right action, in success and in err.
Visibility: Show the user what will happen, is happening, and has happened.
Understandability: Show what is important. Don't overwhelm the user.
Recoverability: Continue where the work was stopped.
Performance: Use resources efficiently to execute the approach.
Psychology: Design for how people behave.

By keeping these dimensions in mind:

For development, the framework gives people confidence that what they build is of quality, by guiding them to consider and handle cases commonly experienced by users. Through feature gates, developers are protected from mental overload by allowing parts of the automation to be developed incrementally, without risk of diverging from the ideal.
For users, the framework provides tools to observe the state and have predictable outcomes, so that they can be confident in making decisions. It also lowers the chances and cost of failure. This reduces the fear of making mistakes, as the effort of recovery is also lowered.
From a business perspective, increased morale, increased performance, and decreased wastage are all benefits that allow results to be achieved faster and in a reliable manner.

Examples

Each page in this section demonstrates the examples compiled as WASM applications.

If you are running this book locally, you may compile all examples before visiting the pages using the following command:

for example in $(ls examples)
do wasm-pack build \
  --target web \
  --out-dir "../../doc/src/examples/pkg" \
  --release \
  "examples/${example}"
done

Download

This example manages a file download into the browser's memory.

Env Man

🚧 This example is a work in progress

Peace framework web application lifecycle example

This example demonstrates management of a web application's lifecycle. This includes:

Building the application.
Starting / stopping the application in development.
Deploying / upgrading / removing the application in test servers.
Configuration management of the application.
Deploying / upgrading / removing the application in live servers.
Diffing the application and configuration across environments.
Creating a replica environment from an existing environment.

Walkthrough

Technical Concepts

The following sections cover essential concepts to develop automation using the framework.

Resources: An any-map, whose values' read and write access are checked at runtime instead of compile time.
Function graph: Functions with logical and data-access-dependent concurrency.
Item: Specification that defines information and logic to manage an arbitrary item.
Item graph: Like a function graph, where each node is not just a function, but an item.

Resources

In Peace, Resources refers to an any-map – a map that can store one value of different types – whose borrowing rules are checked per entry at runtime, instead of compile time.

Example of an any-map:

Key	Value
`TypeId::of::<u32>()`	`1u32`
`TypeId::of::<bool>()`	`true`
`TypeId::of::<A>()`	`A { value: 1u32 }`

use peace::resources::Resources;

let mut resources = Resources::new();
resources.insert(1u32);
resources.insert(2u64);

// Change `resources` to be immutable.
let resources = resources;

// We can validly have two mutable borrows
// from the map!
let mut a = resources.borrow_mut::<u32>();
let mut b = resources.borrow_mut::<u64>();
*a = 2;
*b = 3;

// Accessing the same value while it is already
// mutably borrowed returns an error.
assert!(resources.try_borrow::<u32>().is_err())

For more information about the underlying type, see resman.

Java Equivalent

The conceptual Java equivalent looks like this:

var resources = new MagicMap(); // new HashMap<Class<?>, Object>();
resources.insert((Integer) 1);  // innerMap.put(Integer.class, 1);
resources.insert((Long)    2);  // innerMap.put(Long.class,    2L);

var anInt = (Integer) resources.get(Integer.class);
var aLong = (Long)    resources.get(Long.class);

// Can mutate both.
anInt = 2;
aLong = 3L;

Rust's does not allow access to multiple mutable entries at the same time with the built in HashMap, so Resources is an implementation to bypass the compilation strictness.

Function Graph

A fn_graph is a data structure that holds functions, tracks their dependencies, and streams the functions in dependency order.

This is the basis of a framework that enables:

Consumer provided logic and data.
Framework controlled invocation of logic.

When consumers can plug in their logic and data to the framework, control now resides with the framework on invoking that logic and presenting the data in a goal form.

Scenario

Consider the following program:

Initialize two inputs, a and b, and two outputs c and d.
Add a to c.
Add b to c.
Add c to d.
Print the values of a, b, and c.

Static Representation

The above program can be represented by the following code:

#![allow(unused)]
fn main() {
// Define data to work on.
let mut a = -1;
let mut b = -1;
let mut c = -1;
let mut d = -1;

type Fn1 = fn(&mut i32, &mut i32, &mut i32, &mut i32);
type Fn2 = fn(&i32, &mut i32);
type Fn3 = fn(&i32, &mut i32);
type Fn4 = fn(&i32, &mut i32);
type Fn5 = fn(&i32, &i32, &i32);

// Define logic.
let fn1: Fn1 = |a, b, c, d| { *a = 1; *b = 2; *c = 0; *d = 0; };
let fn2: Fn2 = |a,    c   | *c += *a;
let fn3: Fn3 = |   b, c   | *c += *b;
let fn4: Fn4 = |      c, d| *d += *c;
let fn5: Fn5 = |a, b, c   | println!("{a} + {b} = {c}");

// Invoke logic over data.
fn1(&mut a, &mut b, &mut c, &mut d);
fn2(&a, &mut c);
fn3(&b, &mut c);
fn4(&c, &mut d);
fn5(&a, &b, &c);
}

Dynamic Representation

The following shows the complete logic implemented using fn_graph:

// [dependencies]
// fn_graph = { version = "0.5.4", features = ["fn_meta", "resman"] }
// futures = "0.3.21"
// resman = { version = "0.15.0", features = ["fn_meta", "fn_res"] }
// tokio = { version = "1.20.0", features = ["rt", "macros", "time"] }
use std::{
    fmt,
    ops::{AddAssign, Deref, DerefMut},
};

use fn_graph::FnGraphBuilder;
use futures::stream::StreamExt;
use resman::{IntoFnRes, Resources};

// Define newtypes for each parameter.
#[derive(Debug)] struct A(u32);
#[derive(Debug)] struct B(u32);
#[derive(Debug)] struct C(u32);
#[derive(Debug)] struct D(u32);

fn main() {
// Initialize data.
let mut resources = Resources::new();
resources.insert(A(0));
resources.insert(B(0));
resources.insert(C(0));
resources.insert(D(0));
let resources = &resources; // Now the map is compile time immutable.

type Fn1 = fn(&mut A, &mut B, &mut C, &mut D);
type Fn2 = fn(&A, &mut C);
type Fn3 = fn(&B, &mut C);
type Fn4 = fn(&C, &mut D);
type Fn5 = fn(&A, &B, &C);

// Define logic and insert them into graph structure.
let fn_graph = {
    let fn1: Fn1 = |a, b, c, d| { a.0 = 1; b.0 = 2; c.0 = 0; d.0 = 0; };
    let fn2: Fn2 = |a,    c   | *c += a.0;
    let fn3: Fn3 = |   b, c   | *c += b.0;
    let fn4: Fn4 = |      c, d| *d += c.0;
    let fn5: Fn5 = |a, b, c   | println!("{a} + {b} = {c}");

    let mut fn_graph_builder = FnGraphBuilder::new();

    // Store functions in graph.
    let [fn_id1, fn_id2, fn_id3, fn_id4, fn_id5] = fn_graph_builder.add_fns([
        fn1.into_fn_res(),
        fn2.into_fn_res(),
        fn3.into_fn_res(),
        fn4.into_fn_res(),
        fn5.into_fn_res(),
    ]);

    // Define dependencies to control ordering.
    fn_graph_builder
        .add_edges([
            (fn_id1, fn_id2),
            (fn_id1, fn_id3),
            (fn_id2, fn_id4),
            (fn_id2, fn_id5),
            (fn_id3, fn_id4),
            (fn_id3, fn_id5),
        ])
        .unwrap();

    fn_graph_builder.build()
};

// Invoke logic over data.
//
// For synchronous sequential execution, you may uncomment the next line:
// fn_graph.iter().for_each(|fun| fun.call(resources));
//
// For asynchronous concurrent execution, use the following:
let rt = tokio::runtime::Builder::new_current_thread()
    .enable_time()
    .build()
    .unwrap();
rt.block_on(async move {
    fn_graph
        .stream()
        .for_each_concurrent(None, |fun| async move { fun.call(resources); })
        .await;
});

// prints:
// 1 + 2 = 3
}

macro_rules! u32_newtype {
    ($name:ident) => {
        impl fmt::Display for $name {
            fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
                self.0.fmt(f)
            }
        }
        impl Deref for $name {
            type Target = u32;
            fn deref(&self) -> &Self::Target {
                &self.0
            }
        }
        impl DerefMut for $name {
            fn deref_mut(&mut self) -> &mut Self::Target {
                &mut self.0
            }
        }
        impl AddAssign<u32> for $name {
            fn add_assign(&mut self, other: u32) {
                *self = Self(self.0 + other);
            }
        }
    };
}
u32_newtype!(A);
u32_newtype!(B);
u32_newtype!(C);
u32_newtype!(D);

Note that the knowledge of types is only needed at the point of definition for both data and logic. At the point of invocation, it's dynamic:

let resources: &Resources = /* .. */;

fn_graph
    .stream()
    .for_each_concurrent(None, |fun| async move { fun.call(resources); })
    .await;

How It Works

The remainder of this section shows how fn_graph is implemented.

Generic data storage and dynamic access is already solved by Resources from the previous page. The remaining parts to solve all relate to the logic:

Abstraction: Abstraction over different logic types to retrieve parameters and invoke the logic.
Ordering: Sort logic by their dependencies and how they access data.
Streaming: Concurrently stream logic once they are available without violating data access rules.

Abstraction

To abstract over the logic, the following needs to be solved:

Storage: Single type to be held together in a collection.
Instantiation: Easy to instantiate from functions or closures.
Invocation: Able to be invoked with Resources.

The concept applies to function with any number of parameters¹; the code snippets on this page shows implementations for a function with 2 parameters – one mutable parameter, and one immutable parameter.

Storage

We can store different types under a single abstraction by defining a trait, and implementing the trait for each of those different types.

// Trait for all logic types to implement.
pub trait FnRes {
    // ..
    /// Return type of the function.
    type Ret;
    /// Runs the function.
    fn call(&self, resources: &Resources) -> Self::Ret;
}

In order to name the logic type in code, the logic is stored in a wrapper type called FnResource.

FnRes is implemented for each type parameter combination of FnResource¹:

// Intermediate type so that traits can be implemented on this.
pub struct FnResource<Fun, Ret, Args> {
    pub func: Fun,
    marker: PhantomData<(Fun, Ret, Args)>,
}

// Implement `FnRes` for `FnResource` with different type parameters.
impl<Fun, Ret, C, A> FnRes for FnResource<F, Ret, (&mut C, &A)>
where
    Fun: Fn(&mut C, &A) -> Ret + 'static,
    Ret: 'static,
{
    // ..
}

Once a function or closure is wrapped in a FnResource, then we can hold different functions and closures as Box<dyn FnRes>.

Instantiation

To make it easy to transform functions and closures into Box<dyn FnRes>, the IntoFnResource and IntoFnRes traits are provided generically:

// Turns a function or closure into the `FnResource` wrapper type.
impl<Fun, Ret, C, A> IntoFnResource<Fun, Ret, (&mut C, &A)> for Fun
where
    Fun: Fn(&mut C, &A) -> Ret + 'static,
    Ret: 'static,
{
    fn into_fn_resource(self) -> FnResource<Fun, Ret, (&mut C, &A)> {
        FnResource {
            func: self,
            marker: PhantomData,
        }
    }
}

// Turns a function or closure into a `Box<dyn FnRes>`
impl<Fun, Ret, C, A> IntoFnRes<Fun, Ret, (&mut C, &A)> for Fun
where
    Fun: Fn(&mut C, &A) -> Ret + 'static,
    Ret: 'static,
    A: 'static,
    B: 'static,
    FnResource<Fun, Ret, (&mut C, &A)>: FnRes<Ret = Ret>,
{
    fn into_fn_res(self) -> Box<dyn FnRes<Ret = Ret>> {
        Box::new(self.into_fn_resource())
    }
}

Usage:

let fn_res = (|c: &mut u32, a: &u32| *c += *a).into_fn_res();

Invocation

Now that we can easily hold different types under the Box<dyn FnRes> abstraction, the remaining issue is to invoke the logic through that common abstraction.

To do this, FnRes has a method that takes a &Resources parameter, and each trait implementation will handle parameter fetching and function invocation:

// Trait for all logic types to implement.
pub trait FnRes {
    /// Return type of the function.
    type Ret;

    /// Runs the function.
    fn call(&self, resources: &Resources) -> Self::Ret;
}

// Implementation to fetch the parameters from `Resources`, and invoke the function.
impl<Fun, Ret, C, A> FnRes for FnResource<F, Ret, (&mut C, &A)>
where
    Fun: Fn(&mut C, &A) -> Ret + 'static,
    Ret: 'static,
{
    pub fn call(&self, resources: &Resources) -> Ret {
        let c = resources.borrow_mut::<C>();
        let a = resources.borrow::<A>();

        (self.func)(c, a)
    }
}

With this, consumers are able to invoke functions generically:

let mut resources = Resources::new();
resources.insert(A::new(1));
resources.insert(C::new(0));

let fn_res: Box<dyn FnRes> = (|c: &mut C, a: &A| *c += *a).into_fn_res();

// Generically invoke logic with generic data type.
fn_res.call(&resources);

Notably:

Parameters must be &T or &mut T, as T is owned by Resources.
Parameters must be distinct types, as it is invalid to borrow &mut T twice.
Borrowing and invocation is implemented by the function type – FnResource in this example. Out of the box, this lives in the resman crate instead of fn_graph.

¹ resman implements FnRes for functions with up to 6 parameters.

Ordering

To define the order of execution, fn_graph uses two kinds of dependencies – logical and data. Logical dependencies are defined by consumers of the crate. Data dependencies are calculated by inspecting the type and mutability of function parameters.

Using the example:

type Fn1 = fn(&mut i32, &mut i32, &mut i32, &mut i32);
type Fn2 = fn(&i32, &mut i32);
type Fn3 = fn(&i32, &mut i32);
type Fn4 = fn(&i32, &mut i32);
type Fn5 = fn(&i32, &i32, &i32);

let fn1: Fn1 = |a, b, c, d| { *a = 1; *b = 2; *c = 0; *d = 0; };
let fn2: Fn2 = |a,    c   | *c += *a;
let fn3: Fn3 = |   b, c   | *c += *b;
let fn4: Fn4 = |      c, d| *d += *c;
let fn5: Fn5 = |a, b, c   | println!("{a} + {b} = {c}");

Even though it is possible to order these sequentially, we also want to be able to achieve concurrency to increase performance.

Logical Dependencies

To encode this into a function graph, logical dependencies can be specified through edges:

For correctness, fn1 must be executed before both fn2 and fn3, and those must complete before fn4 and fn5 are executed.

Data Dependencies

Note that fn4 and fn5 can both be executed in parallel, as there is no overlap in accessed data.

However, fn2 and fn3 both require mutable access to C. Since it is not possible to safely provide both fn2 and fn3 with mutable access, fn_graph automatically introduces a data dependency between those functions to ensure consistent ordering:

Streaming

When ordering has been defined, consumers can use FnGraph::iter to iterate through the graph and invoke each function sequentially. When the "async" feature is enabled, which is on by default, FnGraph::stream will produce each function once all of its predecessors have been streamed and returned.

To know when a function is available to be streamed, fn_graph uses the following algorithm:

Track the number of predecessors of each node.
When streaming, each time a function is streamed and the closure returns, subtract 1 from the predecessor counts of each of that function's successors.
If the predecessor count of a function is 0, it is now available to be streamed.

Visualized

In the example scenario, each function has a number of predecessors:
When a function completes, subtract 1 from each of its successors' predecessor counts:
This applies to both logical and data dependencies:
Performance is gained when multiple functions can be executed concurrently:

This can be seen by timing the executions:

// [dependencies]
// fn_graph = { version = "0.5.4", features = ["fn_meta", "resman"] }
// futures = "0.3.21"
// resman = { version = "0.15.0", features = ["fn_meta", "fn_res"] }
// tokio = { version = "1.20.0", features = ["rt", "macros", "time"] }
use std::{
    fmt,
    ops::{AddAssign, Deref, DerefMut},
};

use fn_graph::FnGraphBuilder;
use futures::stream::StreamExt;
use resman::{IntoFnRes, Resources};

// Define newtypes for each parameter.
#[derive(Debug)]
struct A(u32);
#[derive(Debug)]
struct B(u32);
#[derive(Debug)]
struct C(u32);
#[derive(Debug)]
struct D(u32);

fn main() {
    // Initialize data.
    let mut resources = Resources::new();
    resources.insert(A(0));
    resources.insert(B(0));
    resources.insert(C(0));
    resources.insert(D(0));
    let resources = &resources; // Now the map is compile time immutable.

    // Define logic and insert them into graph structure.
    type Fn1 = fn(&mut A, &mut B, &mut C, &mut D);
    type Fn2 = fn(&A, &mut C);
    type Fn3 = fn(&B, &mut C);
    type Fn4 = fn(&C, &mut D);
    type Fn5 = fn(&A, &B, &C);

    let fn_graph = {
        let fn1: Fn1 = |a, b, c, d| { a.0 = 1; b.0 = 2; c.0 = 0; d.0 = 0; };
        let fn2: Fn2 = |a,    c   | *c += a.0;
        let fn3: Fn3 = |   b, c   | *c += b.0;
        let fn4: Fn4 = |      c, d| *d += c.0;
        let fn5: Fn5 = |a, b, c   | println!("{a} + {b} = {c}");

        let mut fn_graph_builder = FnGraphBuilder::new();

        // Store functions in graph.
        let [fn_id1, fn_id2, fn_id3, fn_id4, fn_id5] = fn_graph_builder.add_fns([
            fn1.into_fn_res(),
            fn2.into_fn_res(),
            fn3.into_fn_res(),
            fn4.into_fn_res(),
            fn5.into_fn_res(),
        ]);

        // Define dependencies to control ordering.
        fn_graph_builder
            .add_edges([
                (fn_id1, fn_id2),
                (fn_id1, fn_id3),
                (fn_id2, fn_id4),
                (fn_id2, fn_id5),
                (fn_id3, fn_id4),
                (fn_id3, fn_id5),
            ])
            .unwrap();

        fn_graph_builder.build()
    };

// Invoke logic over data.
let sequential_start = tokio::time::Instant::now();
fn_graph.iter().for_each(|fun| {
    fun.call(resources);
    std::thread::sleep(std::time::Duration::from_millis(10));
});
let sequential_elapsed = sequential_start.elapsed();
println!("sequential_elapsed: {sequential_elapsed:?}");

// prints:
// 1 + 2 = 3
// sequential_elapsed: 50.683709ms

let rt = tokio::runtime::Builder::new_current_thread()
    .enable_time()
    .build()
    .unwrap();
rt.block_on(async move {
    let concurrent_start = tokio::time::Instant::now();
    fn_graph
        .stream()
        .for_each_concurrent(None, |fun| async move {
            fun.call(resources);
            tokio::time::sleep(tokio::time::Duration::from_millis(10)).await;
        })
        .await;
    let concurrent_elapsed = concurrent_start.elapsed();
    println!("concurrent_elapsed: {concurrent_elapsed:?}");
});

// prints:
// 1 + 2 = 3
// concurrent_elapsed: 44.740638ms
}

macro_rules! u32_newtype {
    ($name:ident) => {
        impl fmt::Display for $name {
            fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
                self.0.fmt(f)
            }
        }
        impl Deref for $name {
            type Target = u32;

            fn deref(&self) -> &Self::Target {
                &self.0
            }
        }
        impl DerefMut for $name {
            fn deref_mut(&mut self) -> &mut Self::Target {
                &mut self.0
            }
        }
        impl AddAssign<u32> for $name {
            fn add_assign(&mut self, other: u32) {
                *self = Self(self.0 + other);
            }
        }
    };
}
u32_newtype!(A);
u32_newtype!(B);
u32_newtype!(C);
u32_newtype!(D);

Notably there is some overhead with the asynchronous execution, but as the number of functions grow, so should the concurrency, and the performance gains should increase proportionally.

Item

Note: 🚧 means the concept is not yet implemented.

An item is something that can be created, inspected, and cleaned up by automation.

An itemification defines data types and logic to manage that item.

The Item and associated types are how consumers integrate with the Peace framework. Consumers provide a unique ID, data types, and functions, which will be selectively executed by the framework to provide lean, robust automation, and a good user experience.

This logical breakdown guides automation developers to structure logic to handle cases that are not normally considered of when writing automation. Combined with trait requirements on data types, the framework is able to provide commands, workflow optimizations, and understandable output to ensure a pleasant automation experience.

Data Types

Error: The umbrella error type returned when anything goes wrong when managing the item.
State: Information about a managed item, can be divided into logical and physical state.
Logical state: Part of the state that can be controlled, e.g. application server's existence.
Physical state: Part of the state that cannot be controlled, e.g. application server's instance ID.
Current state: Current State of the managed item, both logical and physical.
Goal state: Logical State that one wants the item to be in.
State difference: Difference between the current Logical state and the Goal state.

Logic – Building Blocks

1. Define an ID.

Item::id

Provide the framework with a unique ID for this item.

These are intended to be safe to use as file names, as well as avoid surprises, and so have been limited to alphanumeric characters and underscores, and cannot begin with a number. This is validated at compile time by using the item_id!("..") macro.

The examples in the peace repository will use snake_case, but the rules are flexible enough to accept PascalCase or camelCase if that is preferred.

Examples

Item that manages a file download: "download".
Item that manages a server: "server_existence".

2. Initialize framework with parameters to manage item.

Item::setup

🚧 parameters are passed in for each command

Provide the framework with enough information to begin managing the item.

This function also instantiates the data types referenced by this Item into the Resources map.

Examples

Item that manages a file download:

Required parameters are the URL to download from, and the destination file path.
Item that manages a server:

Required parameters are the base image ID to launch the server with, and hardware specs.

3. Fetch current item state.

Item::state_current

This may not necessarily be a cheap function, for example if it needs to make web requests that take seconds to complete.

Examples

Item that manages a file download:

Current state is checking a file's existence and contents.
Item that manages a server:

Current state is checking a server's existence and its base image ID.

4. Fetch goal item state.

Item::StateGoalFn

This may not necessarily be a cheap function, for example if it needs to make web requests that take seconds to complete.

Examples

Item that manages a file download:

Goal state is file metadata retrieved from a remote server.
Item that manages a server:

Goal state is one server exists with the specified the base image ID.

5. Return the difference between current and goal states.

Item::StateDiffFn

It is important that both the from and to are shown for values that have changed, and values that have not changed or are not relevant, are not returned.

Examples

Item that manages a file download:

State difference is a change from a file that does not exist, to a file with contents "abc".
Item that manages a server:

State difference is a change from a non-existent server, to a server exists with the specified the base image ID.

6. Ensure that the item is in the goal state.

Transforms the current state to the goal state.

check: Returns whether exec needs to be run to transform the current state into the goal state.
exec: Actual logic to transform the current state to the goal state.
exec_dry: Dry-run transform of the current state to the goal state.

Like exec, but all interactions with external services, or writes to the file system should be substituted with mocks.

7. Clean up the item.

Cleans up the item from existence.

check: Returns whether exec needs to be run to clean up the item.
exec: Actual logic to clean up the item.
exec_dry: Dry-run clean up of the item.

Comparison with `git`

Readers may notice the function breakdown is git-like. The following table compares the concepts:

Subject	Peace	Git
Item	Any consumer defined item.	A directory of files.
Project initialization	`init` command takes in parameters to manage the item.	Uses the current directory or passed in directory.
State	Consumer defined information about the item.	Current state is the latest commit, goal state is the working directory.
State retrieval	On request by user using the `StatesDiscover` command.	Retrieved each time the `status` command is run, cheap since it is all local.
State display	On request using `state` and `goal` commands	`show $revision:path` command shows the state at a particular `$revision`.
State difference	On request using `diff` command	`status` command shows a summary, `show` and `diff` commands shows the state difference.
State application	On request through the `ensure` command.	On request through `commit` and `push` commands.
Environments	Handled by updating initialization parameters.	Defined through `remote` URLs.

Item Parameters

For an item to work with different values, the values must be passed in. These values are called item parameters.

There are a number of related concepts:

User needs to be able to specify how the values are defined:
- Plain Values: Use a value provided by the user.
- Referenced Values: Use a value produced by a predecessor item.
- Transformed Values: Take values produced by predecessor item(s), transform it, then use that.
Implementors:
- Need to define the parameters.
- Take in parameter values for state_current, state_goal.
- Take in Option<Field> for each field within the parameter for try_state_current, try_state_goal.
Peace should be able to store and load:
- The specification by the user.
- The actual values computed and used during command execution.

Value Specification

Item parameter values are eventually concrete values.

Some of those concrete values are not necessarily known until partway through a flow execution. When a flow is defined, a user needs a way to encode where a value comes from.

Plain Values

Plain values are values that a user provides before a command is executed.

A variation of plain values is to provide a lookup function that is evaluated at the point the value is needed, but that has potential negative effects for user experience:

Performance: Web service call(s) may take seconds to complete.
Consistency: Multiple executions may discover different values between different command executions.

In code, this may look like:

let app_params_spec = AppParams::spec()
    .repo_path(Path::from("/path/to/project"))
    .build();
let server_params_spec = ServerParams::spec()
    .image_id(image_id!("abcd1234"))
    .instance_size(InstanceSize::Large)
    .build();

cmd_ctx_builder
    .with_item_params(app_params_spec)
    .with_item_params(server_params_spec)
    .await?;

Referenced Values

Referenced values are values directly taken from a predecessor's state output.

In code, this may look like:

let file_upload_params_spec = FileUploadParams::spec()
    .src_from::<AppOutputPath>()
    // ..
    .build();

cmd_ctx_builder
    .with_item_params(file_upload_params_spec)
    .await?;

Transformed Values

In code, this may look like:

let file_upload_params_spec = FileUploadParams::spec()
    // ..
    .dest_from_map::<Server>(|server| {
        let ip = server.ip();
        format!("user@${ip}:/path/to/dest")
    })
    .build();

cmd_ctx_builder
    .with_item_params(file_upload_params_spec)
    .await?;

Params Definition

For an item to work with different values, the values must be passed in.

Item implementors define these as part of the Item trait:

trait Item {
    type Params: ..;
}

Use Cases

The following shows a number of use cases of these params:

State Apply: Param values must be known, and Peace should pass concrete values to the Item::{state_current, state_goal, apply} functions.

State Discovery (fallible):

Param values may be known, if predecessors have previously executed.

try_state_current: StateDiscoverCmd::current

e.g. Look up file contents on a remote host:

match params_partial.dest_ip() {
    Some(dest_ip) => Some(file_hash(dest_ip, path)),
    None => None, // or `Some(FileState::None)`
}

try_state_goal: StateDiscoverCmd::goal

e.g. Look up source file contents:

match params_partial.src_path() {
    Some(src_path) => file_hash(src_path),
    None => None, // or `Some(FileState::None)`
}

By Item Function

try_state_current: Should work with field_partials.
try_state_goal: Should work with field_partials.
state_current: Needs real concrete param values.
state_goal: Needs real concrete param values.
state_diff: Doesn't need parameters or data; everything should be captured in States.

But for presentation, it's useful to know what a file should be (current vs goal), or difference between params (multiple profile current vs current).
state_clean: Maybe always returns ItemState::None, and doesn't need parameters or data.

However, presenting state_clean with e.g. a file path, would mean the None state contains the value, which means state_clean needs params.

Arguably state_goal will show the path that would be created.

StateDiff for cleaning should also show the deletion of the path.
apply_check: Doesn't need parameters or data.
apply_dry: Needs concrete param values, even if they are fake.
apply: Needs real concrete param values.

Encoding: Serialization / Deserialization

Because:

It is convenient to serialize Item::Params::Spec and store it, and deserialize it for use at a later time.
It is useful to support config-based parameter specification (no compiler needed).
It is not possible to serialize closures.

Then there must be a way to encode the same functionality that Item::Params::Spec::field_from_map provides, as something serializable.

Possibilities:

ToString and FromStr impls that represent the logic
Serialized form uses enum variants, and when deserializing, map that back to functions.
Custom language.

Code Implications

From the implementor's perspective, item trait needs to change to support the above use cases.

The following snippets are here to show the changes that include the above concepts. These are:

non-compilable.
just enough to show where types are changed.
show certain trait bounds (non-exhaustive).
do not include the encoding / decoding of field_from_map concept.

Framework

// Traits in Peace Framework
trait Item {
    type Params: Params + Serialize + Deserialize;

    fn setup(&self, resources);
    fn try_state_current(fn_ctx, params_partial, data);
    fn try_state_goal(fn_ctx, params_partial, data);
    fn state_clean      (        params_partial, data);
    fn state_current    (fn_ctx, params,         data);
    fn state_goal    (fn_ctx, params,         data);
    fn apply_dry        (fn_ctx, params,         data, state_current, state_target, diff);
    fn apply            (fn_ctx, params,         data, state_current, state_target, diff);
    fn apply_check      (        params_partial, data, state_current, state_target, diff);
    fn state_diff       (        params_partial, data, state_a, state_b);

    // Once more, with types:
    fn setup(&self, &mut Resources<Empty>);
    fn try_state_current(FnCtx<'_>, Self::Params<'_>::Partial, Self::Data<'_>);
    fn try_state_goal(FnCtx<'_>, Self::Params<'_>::Partial, Self::Data<'_>);
    fn state_clean      (           Self::Params<'_>::Partial, Self::Data<'_>);
    fn state_current    (FnCtx<'_>, Self::Params<'_>         , Self::Data<'_>);
    fn state_goal    (FnCtx<'_>, Self::Params<'_>         , Self::Data<'_>);
    fn apply_dry        (FnCtx<'_>, Self::Params<'_>         , Self::Data<'_>, Self::State, Self::State, Self::StateDiff);
    fn apply            (FnCtx<'_>, Self::Params<'_>         , Self::Data<'_>, Self::State, Self::State, Self::StateDiff);
    fn apply_check      (           Self::Params<'_>::Partial, Self::Data<'_>, Self::State, Self::State, Self::StateDiff);
    fn state_diff       (           Self::Params<'_>::Partial, Self::Data<'_>, Self::State, Self::State);
}

/// For Peace to access <Item::Params as Params>::Spec
trait Params {
    type Spec: Serialize + Deserialize;
    type SpecBuilder: SpecBuilder<Output = Self::Spec>;
    type Partial: Serialize + Deserialize;
}

enum ValueSpec<T> {
    Value(T),
    From,
    FromMap(Box<dyn Fn(&Resources) -> Option<T>>),
}

Also need to provide a Params derive macro.

Design Note

The apply_check and state_diff functions usually are not expected to need params and data, but some items may use them, such as the ShCmdItem which accesses params to determine the script to run.

Regarding params will be Params or Params<'_>::Partial for state_diff, if we call state_diff for a file upload item, we must have both the current state and goal state. Then we need to ask, does having both states imply Params is fully resolvable?

If we call state_current for a file upload:

the destination server may not be there
so params may not have the IP
we may still return Some(State::Empty)
So params may still be partial, even if State is Some.

If we call state_goal for a file upload:

the source file may not be there
so params may not have the source content hash
we may still return Some(State::Empty)
So params may still be partial, even if State is Some.

Implementor

// Implementation
struct FileUploadItem;

impl Item for FileUploadItem {
    type Params = FileUploadParams;
}

#[derive(Clone, Debug, Params, Serialize, Deserialize)]
struct FileUploadParams {
    src: PathBuf,
    dest_ip: IpAddr,
    dest_path: PathBuf,
}

Auto generated by Params derive:

impl Params for FileUploadParams {
    type Spec = FileUploadParamsSpec;
    type SpecBuilder = FileUploadParamsSpecBuilder;
    type Partial = FileUploadParamsPartial;
}

// Serialize / Deserialize not needed.
struct FileUploadParamsPartial {
    src: Option<PathBuf>,
    dest_ip: Option<IpAddr>,
    dest_path: Option<PathBuf>,
}

#[derive(Clone, Debug, Serialize, Deserialize)]
struct FileUploadParamsSpec {
    src: ValueSpec<PathBuf>,
    dest_ip: ValueSpec<IpAddr>,
    dest_path: ValueSpec<PathBuf>,
}

#[derive(Clone, Debug)]
struct FileUploadParamsSpecBuilder {
    src: Option<ValueSpec<PathBuf>>,
    dest_ip: Option<ValueSpec<IpAddr>>,
    dest_path: Option<ValueSpec<PathBuf>>,
}

See:

Params Framework Support

For ergonomic usage and accurate data modelling, the following types need to exist for each item:

Params
ParamsPartial
ParamsSpec
ParamsSpecBuilder

Since Params is defined by the item implementor, the other three structs can only be defined after it, and can be generated by a proc macro.

Questions:

How does Peace know what fields to fill in, and how?

The fields are known by the ParamsSpec impl, though it resides in the implementor's crate.

Peace can reference the ParamsSpec through a Params::Spec associated type.

Does Peace need to know about the field, or can it be in the generated impl?

It needs to go, for each field on the ParamsSpec, fill in the ParamsPartial. This could be implemented by ParamsSpec::partial_from(&Resources) -> ParamsPartial:

impl ParamsSpec {
    fn partial_from(resources: &Resources) -> ParamsPartial {
        // for each field (generated by proc macro)
        let value_0 = value_0_spec.try_value_from(resources);
        let value_1 = value_1_spec.try_value_from(resources);

        ParamsPartial { value_0, value_1 }
    }
}

enum ValueSpec<T> {
    Value(T),
    From,
    FromMap(Box<dyn Fn(&Resources) -> Option<T>>),
}

impl<T> ValueSpec<T> {
    fn try_value_from(&self, resources: &Resources) -> Option<T> {
        match self {
            Self::Value(t) => Some(t.clone()),
            Self::From => resources.try_borrow::<T>().cloned(),
            Self::FromMap(f) => f(resources),
        }
    }
}

How does ValueSpec::FromMap get instantiated:

Probably something like this:

impl ParamsSpecBuilder {
    // for each field
    fn field_from_map<F, U>(mut self, f: F) -> Self
    where F: &U -> Field {
        let from_map_boxed = Box::new(move |resources| {
            resources.try_borrow::<Field>().map(f)
        });
        self.field_spec = ValueSpec::FromMap(from_map_boxed);
        self
    }
}

To allow Peace to reference those types, and for Peace to expose that to users, we need the following traits:

dyn Params: To know what type the ParamsPartial, ParamsSpec, and ParamsSpecBuilder are.
dyn ParamsSpec: Peace needs the spec to generate a ParamsPartial from Resources.

Implications On Serialization and Flow Params

It is inconvenient to have to register item params types as flow params, because the presence of the item could automatically tell Peace its params type.

However, users:

Must provide the param values / spec for each item for the first execution of a flow.
May have different preferences whether they have to provide the values / spec for subsequent executions, and whether it implicitly uses what was used before, or is explicit for each field.

What we need:

Flow params are serialized separately. Flow params may be values that control how things are presented, but not affect the actual functional execution, so they may be log verbosity.
Item params spec must be deserialized, and overwritten by provided values.
Item params spec must be serialized before executing a command.
Item params must be serialized as execution happens.
Item params partials do not need to be serialized, only presented, as they can be re-calculated from state for each discover / read command execution.

Item Graph

An item graph is a function graph, whose logic type is an item.

Instead of storing a single function in each node of the graph, each logic type is a collection of well-defined functions adhering to the Item trait.

Combining these concepts contributes to the strategy's outcomes, by enabling the framework to address common issues in each input dimension.

let graph = {
    let mut graph_builder = ItemGraphBuilder::<XError>::new();
    let [id_1, id_2, id_3, id_4] = graph_builder.add_fns([
        Item1::new(param1).into(),
        Item2::new(/* .. */).into(),
        Item3::new().into(),
        Item4::new().into(),
    ]);
    graph_builder.add_edges([
        (id_1, id_3),
        (id_2, id_3),
        (id_2, id_4),
    ])?;

    graph_builder.build()
};

The remainder of this section explains how using these concepts together allows sensible user facing commands to be created. Available commands are documented in the reference.

Initialization

Before a process is started, make sure all the necessary information is provided.

In function graph, data types in Resources are inserted by the consumer, separate from the graph.

In Peace, Item::setup is run for each item, which allows data types to be inserted into Resources.

let graph = /* .. */;

let resources = graph.setup(Resources::new()).await?;

// Item1::setup
resources.insert(param1);

// Item2::setup
resources.insert(param2);

// Item3::setup
// no-op

// Item4::setup
resources.insert(param3);
resources.insert(param4);

ℹ️ Note: Each initialization parameter should be specified in each item's setup method, even though the parameter is inserted by a predecessor item.

This is because when only a subset of the graph is executed, or if the item is used in a different graph, the parameter should still be inserted.

🚧 A wrapper type should conditionally insert the initialization parameter into Resources

State Inspection

Before applying a change, you should understand the current state, and what the change will do.

To discover the current state of all items, Item::try_state_current is run for each item concurrently.

let graph = /* .. */;
let resources = /* .. */;

let resources = StatesDiscoverCmd::current(graph, resources).await?;

// Item1::StateCurrentFn::try_exec
let exists = param1.path().exists();
Ok(State::new(exists, ()))

// Item2::StateCurrentFn::try_exec
let instance_url = discover(param2).await?;
Ok(State::new(Some(instance_url), ()))

// Item3::StateCurrentFn::try_exec
let version = reqwest::get(instance_url).await?;
Ok(State::new(version, ()))

// ..

When all states have been collected, they are presented to the user:

state1: exists
state2: 1.2.3.4
state3: 1.0.0
state4: abcdef0123456

State Ensure

When applying a change, the change should converge the current state to the goal state.

let graph = /* .. */;
let resources = /* .. */;

let resources = ApplyCmd::exec(graph, resources).await?;

Note that the Item::apply requires implementers to return StatePhysical, which is the state information generated during the exec logic, but not necessarily within the implementers' control.

Method

To discover the current state of all items, the following method is used:

ApplyFns::check is run for all items.
Of the ones that return ApplyCheck::ExecRequired, Item::apply is run.
Finally, Item::state_current is run so that the end state can be compared with the goal state to confirm that they match.

`ApplyFns::check`

// Item1
ApplyCheck::ExecRequired { .. }

// Item2
ApplyCheck::ExecNotRequired

// Item3
ApplyCheck::ExecRequired { .. }

// Item4
ApplyCheck::ExecRequired { .. }

`Item::apply`

Items 1, 3, and 4 need to be executed, but Item2's Item::apply is skipped as check indicated it isn't needed.

// Item1
()

// Item2
IpAddr::from_str("1.2.3.4")

// Item3
()  // Note: version is logical state

// Item4
Revision::new("abcdef0123456")

Dry Run

let resources = Item::apply_dry(graph, resources).await?;

Similar to the Item::apply, Item::apply_dry is meant to simulate what would happen, and allow users to correct mistakes before actual execution.

Implementers must replace all write logic with mocks. These include:

File writes
Web requests

It is also recommended that read requests to external services are minimized to decrease the time to return feedback to the user. If possible, move read request logic to Item::state_current so that it is stored by the StatesDiscoverCmd.

Convergence / Non-Transactional Execution Recovery

Since these processes happen over distributed systems, and errors can happen at any point in the process, it is realistic to assume that the process doesn't happen transactionally.

ApplyFns has been designed so that implementers will consider transitions from non-atomic states to the goal state. In simpler terms, if the goal state is "10 servers must exist", then:

When the current state is 0 servers, then 10 servers should be launched.
When the current state is 6 servers, then 4 servers should be launched.

As time and effort is saved by reusing existing useful state and not requiring complete clean up, recoverability and user experience is improved.

Clean

Partial Clean

Even though it is possible to clean up the server without uninstalling the application, it is recommended that the CleanOpSpec is implemented for every item.

Imagine one has a goal to test fresh installations of different versions of an application. One approach is to deploy each version in a separate environment, and clean up the whole environment, . have a command that only uninstalls the application from the server, .

Least-work approach, but needs additional engineering.

State

In peace, State represents the values of an item, and has the following usages:

Showing users the state of an item.
Allowing users to describe the state that an item should be.
Determining what needs to change between the current state and the goal state.

Therefore, State should be:

Serializable
Displayable in a human readable format
Relatively lightweight – e.g. does not necessarily contain file contents, but a hash of it.

Logical and Physical State

State can be separated into two parts:

Logical state: Information that is functionally important, and can be specified by the user ahead of time.

Examples of logical state are:
- File contents
- An application version
- Server operating system version
- Server CPU capacity
- Server RAM capacity
Physical state: Information that is discovered / produced when the automation is executed.

Examples of physical state are:
- ETag of a downloaded file.
- Execution time of a command.
- Server ID that is generated on launch.
- Server IP address.

Defining State

Fully Logical

If an item's state can be fully described before the item exists, and can be made to happen without interacting with an external service, then the state is fully logical.

For example, copying a file from one directory to another. The state of the file in the source directory and destination directories are fully discoverable, and there is no information generated during automation that is needed to determine if the states are equivalent.

Logical and Physical

If an item's goal state can be described before the item exists, but interacts with an external service which produces additional information to bring that goal state into existence, then the state has both logical and physical parts.

For example, launching a server or virtual machine. The operating system, CPU capacity, and RAM are logical information, and can be determined ahead of time. However, the server ID and IP address are produced by the virtual machine service provider, which is physical state.

Fully Physical

If an item's goal state is simply, "automation has been executed after these files have been modified", then the state has no logical component.

For example, running a compilation command only if the compilation artifact doesn't exist, or the source files have changed since the last time the compilation has been executed.

The remaining pages in this section explain how to define the logical and physical state types when implementing an item.

Logical State

Logical state is the part of an item that is:

Implementor / user definable
Controllable by automation
Deterministic

Uses

There are three uses of logical state:

Representing current state.
Representing goal state.
Computing state difference.

let params = data.params();

// `dest` references the actual item that is being managed.
// e.g. calculate content hash of actual file being written to.
let state_current = params.dest().read();

// `src` references the *specification* of what the item is intended to be.
// e.g. retrieve content hash from a source file.
let state_goal = params.src().read();

// We can only compute the `diff` when both `src` and `dest` are available.
let state_diff = state_goal - state_current;

Discovery Constraints

In an item's parameters, there must be the following categories of information:

src: information of what the item should be, or where to look up that information.

Thus, src is a reference to where to look up state_goal.
dest: reference to where the actual item should be.

dest is a reference to where to push state_current.

Both src and dest may reference resources that are ensured by predecessor items. Meaning sometimes state_goal and state_current cannot be discovered because they rely on the predecessors' completions.

Examples

A list of files in a zip file cannot be read, if the zip file is not downloaded.
A file on a server cannot be read, if the server doesn't exist.
A server cannot have a domain name assigned to it, if the server doesn't exist.

Implications

If dest is not available, then state_current may simply be "does not exist".
If src is not available, and we want to show state_goal that is not just "we can't look it up", then src must be defined in terms of something readable during discovery.
If that is not possible, or is too expensive, then one or more of the following has to be chosen:
1. Item::state_goal functions have to always cater for src not being available.
  
  It incurs mental effort to always cater for src not being available – i.e. implementing an item would need knowledge beyond itself.
2. the peace framework defaults to not running state_current_fn for items that have a logical dependency on things that Item::apply_check returns ExecRequired
  
  For this to work, when the current state is requested, peace will:
  1. For each non-parent item, run state_current, state_goal, state_diff, and apply_check.
  2. If apply_check returns ApplyCheck::ExecNotRequired, then successor items can be processed as well.
3. state_current could return Result<Option<Status>, E>:
  - Ok(None): State cannot be discovered, likely because predecessor hasn't run
  - Ok(Some(State<_, _>)): State cannot be discovered.
  - Err(E): An error happened when discovering state.
    
    May be difficult to distinguish some cases from Ok(None), e.g. failed to connect to server, is it because the server doesn't exist, or because the address is incorrect.
    
    Should we have two state_currents? Or pass in whether it's being called from Discover vs Ensure – i.e. some information that says "err when failing to connect because the predecessor has been ensured".
Option 2 may be something we have to do anyway – we will not be able to provide current state to run Item::apply for successors for the same reason.

Option 3 may coexist with option 2.

Note: State discovery may be expensive, so it is important to be able to show a stored copy of what is discovered.

Output

Output is providing information to a user.

Output can be categorized with number of input dimensions:

Disclosure:
- Push: Information is pushed to users once available.
- Pull: Information is requested by users, may be real time or historical.
Rate: (of requests)
- High: Information that constantly updates.
- Low: Information that once created isn't expected to change.
Consumer:
- Human: Person using the application.
- Software: ReST API consumer.
- Both: Readable and parseable text output.
Accumulation:
- Append: Information continuously increases.
- Replace: Latest information is relevant.

Example scenarios and solutions for different output combinations

#	Disclosure	Rate	Consumer	Accumulation	Example Scenario	Example solution
1	Push	High	Human	Append	Execution progress in CI / builds	Text
2	Push	High	Human	Replace	Execution in interactive CLI	Progress bars
3	Push	High	Software	Append	Execution progress delta web socket API	Serialized
4	Push	High	Software	Replace	Execution progress web socket API	Serialized
5	Push	High	Both	Append	none
6	Push	High	Both	Replace	none
7	Push	Low	Human	Append	Execution errors in CLI	Friendly errors
8	Push	Low	Human	Replace	Execution outcome in CLI	Markdown output
9	Push	Low	Software	Append	Execution errors web socket API	Serialized
10	Push	Low	Software	Replace	Execution outcome web socket API	Serialized
11	Push	Low	Both	Append	Execution errors in file	Friendly serialized
12	Push	Low	Both	Replace	Execution outcome in file	Friendly serialized
13	Pull	High	Human	Append	Historical execution progress logs	Text
14	Pull	High	Human	Replace	none
15	Pull	High	Software	Append	none	Serialized
16	Pull	High	Software	Replace	Historical execution progress ReST API	Serialized
17	Pull	High	Both	Append	none
18	Pull	High	Both	Replace	none
19	Pull	Low	Human	Append	Execution errors in web page	Formatted errors
20	Pull	Low	Human	Replace	Execution outcome in web page	Formatted report
21	Pull	Low	Software	Append	Execution errors ReST API	Serialized
22	Pull	Low	Software	Replace	Execution outcome ReST API	Serialized
23	Pull	Low	Both	Append	Historical execution errors in file	Friendly serialized
24	Pull	Low	Both	Replace	Historical execution outcome in file	Friendly serialized

From the above table, further thoughts include:

Information that changes at a high rate implies a high rate of requests.
Information requested at a high rate may need to be small (in size).
Progress output is frequent, and is most important in real time / as it happens.
If frequency is high and the output is transferred between hosts, then ideally the size of the output is reduced.
The point of serialized data without the friendliness requirement is space and time efficiency.
Friendly serialized data may be a format such as YAML / JSON / TOML.
Pulled data is either over an API, or viewing historical information, meaning pulled data needs to be serializable.
Web page output may be large, but mental overload can be avoided by hiding information through interactivity.

Outcome

Outcome Status

Success: Task has completed successfully.
Break: Task has stopped for manual action not managed by the command.
Error: Task has stopped with an error.

Outcome Messages

Nothing
Informatives: What the user needs to do next.
Warnings

Outcome Errors

What happened
Why it is considered an error
Source of the information that led to the error
Suggestions for fixing

Format

Output format can be optimized for different consumers. For example:

Interactive command line:
- Hide detail to reduce mental overload.
- Show enough to give an appropriate picture.
- Use colour to highlight / dim detail based on level of importance.
- Show commands to display additional detail.
- Use a format that makes sense when copying and pasting into a different application, e.g. markdown.
Web page:
- Use interactive elements to allow detail to be revealed when needed.
- Use consistent colour for different levels of detail.
Network transfer:
- Use a serialization format that is small to cater for latency.
- Use a serialization format that is not difficult to deserialize to reduce CPU utilization.

Execution Progress

Execution progress output is most useful as it happens. Execution progress output should fulfill the following concerns:

easy for a human to understand
easy to be sent from an item implementation
pushed to output in real time
pulled from client in real time

Information

Information that is likely useful to the consumer, and therefore should be captured, and sent if bandwidth is not an issue.

Progress Status

Whether a task has started, is in-progress, stalled, or completed.

Execution Pending: Execution is pending a predecessor's execution.
Running: Execution is in progress.

This status is best conveyed alongside the following information:
- Units total: Unknown (spinner) / known (progress bar).
- Units current
- Function: Item::{state_current, state_goal, apply}.
  
  Certain functions will not be applicable, e.g. when StateCurrent is feature gated, then the function won't be available when the feature is not enabled.
Running Stalled: Progress updates have not been received for a given period.

Item implementations are responsible for sending progress updates, but if there are no progress updates, or an identical "it's running" progress update is continuously received, then Peace may determine that the task may have stalled, and user attention is required.

Peace may also provide a hook for implementors to output a suggestion to the user.
User Pending: Task is pending user input.
Completed: Task has completed.

This status is best conveyed alongside the following information:
- Completion Status: Success, Failed.
- Function: Item::{state_current, state_goal, apply}.

The following variant is possible conceptually, but not applicable to the Peace framework:

Stopped: Task is not running, but can be started.

This is not applicable because Peace uses runtime borrowing to manage state, and a stopped task has potentially altered data non-atomically, so locking the data is not useful, and unlocking the data may cause undefined behaviour due to reasoning over inconsistent state.

For rate limiting tasks, the task in its entirety would be held back.

Progress Measurement

Unit of measurement: Steps, Bytes, Percentage, None
Units total: Known / Unknown
Units current: Numeric / Unknown
Units remaining: Numeric / Unknown. This is total - current.
Unit of progress tick: as it happens.

Progress Timings

Time started
Elapsed duration
Remaining duration estimate
Time of completion estimate
Time of completion

Information Production / Consumption

As much as possible, Peace should collect progress information without burdening the implementor.

Info	Produced / Captured by
Progress status	Framework
Unit of measurement	Implementor
Units total	Implementor
Units initially completed	Implementor
Units remaining	Framework
Unit of progress tick	Implementor
Time started	Framework
Elapsed duration	Framework
Remaining duration estimate	Framework
Time of completion estimate	Framework
Time of completion	Framework

Peace should support the following usages out-of-the-box¹:

Interactive CLI
Non-interactive CLI
CI logging
WASM
REST endpoint (pull)
REST endpoint (push)

¹ even if it is not yet implemented, it should be designed in a way that allows for it

API

Implementors define the following in ApplyFns::check:

Unit of measurement: Steps, Bytes, Percentage, None
Units total: Known / Unknown
Units initially completed: Numeric / Unknown

Units remaining is tracked by Peace as the unit of progress is ticked (updated).

There is a tradeoff between:

Implementors manually ticking progress

More accurate information
Code becomes messy -- progress ticks scattered throughout code.

zzz has a really good way of doing this without making things messy:

#![allow(unused)]
fn main() {
(0..1000).into_iter().progress()
//                    ^^^^^^^^^^
// This is all that's needed to render the progress bar
// assuming `size_hint()` is implemented.
}

Peace could provide something like:

#![allow(unused)]
fn main() {
impl ApplyFns for MyApplyFns {
    async fn exec(data: Data<'_>) -> Result<(), E> {
        [
            Self::fn_1,
            Self::fn_2,
            Self::fn_3,
            Self::fn_4,
        ].progress(data);
    }
}
}

But that constrains the signature of each sub function to be the same. A type parameterized function chain – where the return value of one function is the parameter to the next – may be messy, especially for error messages.

For spinners, manually ticking the progress bar can be cumbersome, but automatic ticking will give a false impression of progress happening. Somewhat mitigated by stall detection.

Codegen to add progress ticks

This could be done with a proc macro that counts the number of expressions, with some clever handling of conditionals, and inserts tick() expressions.

For now Peace will require implementors to manual tick the progress, and in the future, explore the following option:

Trait to automatically tick progress for iterators.
Proc macro to automatically insert ticks.

Presentation

For human readable output, it is desirable for consumers as well as the framework to be able to:

Output arbitrary types.
Have them formatted by the output implementation.
Not require the output implementation to know the specific type that is being output.

The OutputWrite trait was written to handle different output formats – human-readable vs CI logging vs structured text – but not how to present the output.

Peace should provide two traits:

OutputWrite: Maps between the information and the output format, e.g. human readable vs parseable.

Examples:
- CLI output: Writes progress as log messages (CI), or progress bar (interactive), or nothing (when piped).
- Web output: Write JSON.
- Native application: Update a UI component.
Presentable: Maps between the information and the presentation format, e.g. present this as an ID, short text, long text, a list, etcetera.

Examples:
- CLI output: Colour text with ANSI colour codes.
- Web output: Create and style web elements.
- Native application: Create and style UI components.

Current State

The OutputWrite trait has methods for:

writing progress: setting up state, updating progress, and ending.
writing states (current, goal, diff, etc.)
writing error

These methods are specific to States, and if we add methods per type, it doesn't allow any arbitrary type to be formatted and written.

Goal State

To be usable with arbitrary information, OutputWrite should have methods to output different kinds of information. These information kinds are based on the purpose of the information, not on how they should be grouped or presented.

Information Kinds

Progress: Information about the execution of automation.
Outcome: Information that the automation is purposed to produce.
Notes: Meta information about the outcome or progress -- informatives, warnings.

These can be used to refine the automation.

For each information kind, OutputWrite should be able to:

Write one or many of that information kind
Reason over the parameters of that information, and potentially pass it to a formatter.

For structured output, all information should be serializable.

Presentation / Formatting

For human readable output to be understandable, the OutputWrite implementation should improve clarity by adding styling or reducing information overload. For this to work with arbitrary types, the OutputWrite needs additional hints to determine how to format the information.

Examples:

An object may be presented as a list, and the type needs to define which fields that list is built from.
When presenting a list of named items, the type needs to define both the name and the description, which allows the names to be styled differently to the descriptions.
When presenting a large object, the density of information can be reduced through collapsible sections, and more detail displayed when the sections are expanded.

Implementation

To achieve this, we can:

Define a peace::fmt::Presentable trait, analogous to std::fmt::Display
Define a peace::fmt::Presenter trait, analogous to std::fmt::Formatter
Presenter has methods to format:
- short text descriptions
- long text descriptions (e.g. always split at \ns)
- names, e.g. always bold
- lists of Presentables
- groups, e.g. always collapsible, or presenter may choose to not display if level of detail is too high
Implementors will impl Presentable for MyType. This can be made easier with a derive macro.
Update OutputWrite to take in &dyn Presentable instead of concrete types, and the OutputWrite implementation can decide whether or not to delegate to Presenter for presentation information. e.g. a serializing output write may not need to.

Note: Structured output that is read by humans (e.g. prettified YAML or JSON) is not a peace::fmt::Presentable concern, but an OutputWrite parameter, as it is a standard format serialization parameter, not formatting hints that the output endpoint needs.

Instead of using &strs for what is presented, we could add type safety to:

Enforce certain constraints, e.g. short descriptions must be one line, less than 200 characters
For human readable output, instead of std::fmt::Display, types implement peace::fmt::Presentable trait where a peace::fmt::Presenter is passed in.

The type safety can be opt-in, e.g. allow &strs, but if using the type-safe wrappers, you get compilation errors when the constraints are not met.

Recursion

If the Presentable trait is recursive like Debug, then we need to make sure implementors understand that a "name" will always be styled as a name, unless one creates a wrapping type that does not delegate to the name's underlying Presentable implementation (just like Debug).

Workspace

A tool built using the Peace framework must execute in a workspace – a location that Peace reads and writes information when commands are executed.

In a native automation tool, the workspace is usually the repository in which the automation is run, so the workspace directory is the repository root. In a WASM tool, Peace currently is able to store information in the browser LocalStorage or SessionStorage.

Peace Data

The following shows the file structure for data stored by Peace.

Path Segments

Hierarchy

$workspace_dir  # usually the project repository
|
|  # .peace, single directory to store data from tools made with Peace
|- $peace_dir
    |
    |  # directory per tool
    |- $peace_app_dir
        |- 📝 $workspace_params_file
        |
        |  # profile name, directory per profile
        |- 🌏 $profile_dir
            |- 📝 $profile_params_file
            |
            |  # flow name, directory per flow
            |- 🌊 $flow_dir
                |- 📝 $flow_params_file
                |- 📋 $states_current_file
                |- 📋 $states_goal_file

Concrete Hierarchy Example

path/to/repo
|- .peace
    |- envman
        |- 📝 workspace_params.yaml
        |
        |- 🌏 internal_dev_a
        |   |- 📝 profile_params.yaml
        |   |
        |   |- 🌊 deploy
        |   |   |- 📝 flow_params.yaml
        |   |   |- 📋 states_goal.yaml
        |   |   |- 📋 states_current.yaml
        |   |
        |   |- 🌊 config
        |   |   |- 📝 flow_params.yaml
        |   |   |- 📋 states_goal.yaml
        |   |   |- 📋 states_current.yaml
        |   |
        |   |- 🌊 benchmark
        |       |- 📝 flow_params.yaml
        |       |- 📋 states_goal.yaml
        |       |- 📋 states_current.yaml
        |
        |- 🌏 customer_a_dev
        |   |- 📝 profile_params.yaml
        |   |
        |   |- 🌊 deploy - ..
        |   |- 🌊 config - ..
        |
        |- 🌏 customer_a_prod
            |- 📝 profile_params.yaml
            |
            |- 🌊 deploy - ..
            |- 🌊 config - ..

Commands

Commands in Peace are categorized based on how they interact with a workspace. Make sure you understand the Peace workspace model.

The information available within a command execution is called the command context. Depending on the scope of a command, the available information changes. Peace's command API adaptively exposes methods that are relevant to the chosen scope.

This section covers:

Scopes: What information is accessible for each command scope.
Use Cases: Which scope to use and how to create different kinds of commands.

Scopes

The scope of a command determines:

If it has access to zero, one, or many Profile directories.
If it has access to zero, one, or many Flow directories.
Whether it can deserialize profile params.
Whether it can deserialize flow params.
Whether it can deserialize states from each flow directory.

Scenario: envman is an example tool that manages server environments.

In the following directory structure:

internal_dev_a, customer_a_dev, customer_a_prod are all separate Profiles.
deploy, config, and benchmark are separate Flows.
Each flow tracks its own flow_params.yaml, states_goal.yaml, and states_current.yaml

# peace app dir
path/to/repo/.peace/envman
|- 📝 workspace_params.yaml
|
|- 🌏 internal_dev_a
|   |- 📝 profile_params.yaml
|   |
|   |- 🌊 deploy
|   |   |- 📝 flow_params.yaml
|   |   |- 📋 states_goal.yaml
|   |   |- 📋 states_current.yaml
|   |
|   |- 🌊 config
|   |- 🌊 benchmark
|
|- 🌏 customer_a_dev
|   |- 📝 profile_params.yaml
|   |
|   |- 🌊 deploy - ..
|   |- 🌊 config - ..
|
|- 🌏 customer_a_prod
    |- 📝 profile_params.yaml
    |
    |- 🌊 deploy - ..
    |- 🌊 config - ..

See each page for details of each scope:

No Profile No Flow: Commands that only work with workspace parameters.
Single Profile No Flow: Commands that work with a single profile, without any items.
Single Profile Single Flow: Commands that work with one profile and one flow.
Multi Profile No Flow: Commands that work with multiple profiles, without any items.
Multi Profile Single Flow: Commands that work with multiple profiles, and a single flow.

Reborrowing

Reborrowing is used in scopes to so that the View structs don't hold onto the borrow for the length of 'ctx, but only for the 'view lifetime.

#[derive(Debug)]
struct Life<'life> {
    value: &'life mut u8,
}

impl<'life> Life<'life> {
    pub fn reborrow<'short>(&'short mut self) -> Life<'short> {
        Life {
            value: &mut self.value,
        }
    }
}

// Used in
struct Scope<'life> {
    life: Life<'life>,
    other: u16,
}

struct ScopeView<'view> {
    life: Life<'view>,
    other: &'view mut u16,
}

impl<'life> Scope<'life> {
    pub fn view(&mut self) -> ScopeView<'_> {
        let Scope { life, other } = self;

        // Needed to shorten the lifetime of `'life`.
        let life = life.reborrow();

        ScopeView { life, other }
    }
}

fn main() {
    let mut value = 123;

    let life = Life { value: &mut value };
    let mut many_data = Scope { life, other: 456 };

    let ScopeView {
        life: _life,
        other: _other,
    } = many_data.view();
}

Use Cases

This section covers the kinds of commands that Peace is intended to support. While it is still a work in progress, the aim is for all of these commands to be easily constructed.

Workspace Initialization: Like git init.
Profile List: Like git branch.
Workspace Stored Active Profile: Like git switch -c $branch.
Workspace Active Profile Switch: Like git switch $branch.
State Discovery and Display: Like git fetch.
State Read and Display: Like git show.
State Diff: Like git status / git diff.
State Apply: Like git commit / git push / git clean, depending on your perspective.
State Clean: Like rm -rf.

Workspace Initialization

This is like git init.

When initializing a project workspace, the suitable scope is different depending on the amount of work done:

Command Creation

If it only stores parameters provided by the user, shared across profiles, and does not use any item, use NoProfileNoFlow.

To create this command:
1. Build the command context with the provided parameters. When the command context is built:
- Workspace parameters are written to workspace_params.yaml.
If it stores parameters for a default profile, and does not use any item, use SingleProfileNoFlow.

To create this command:
1. Build the command context with the provided parameters. When the command context is built:
- Workspace parameters are written to workspace_params.yaml.
- Profile parameters are written to profile_params.yaml.
If it does any repeatable work, such as download files or clone a repository, use SingleProfileSingleFlow.

To create this command:
1. Build the command context with the provided parameters.
2. Call StateDiscoverCmd::exec to discover the current and goal states.
3. Call EnsureCmd::exec to execute the flow. When the command context is built:
- Workspace parameters are written to workspace_params.yaml.
- Profile parameters are written to profile_params.yaml.
- Flow parameters are written to flow_params.yaml.

Profile List

This is like git branch.

When listing profiles, the most suitable scope is MultiProfileNoFlow.

There are a number of variations of profile listings:

Simple: Simple list of profiles within the PeaceAppDir.
Filtered: List profiles within the PeaceAppDir for some criteria, e.g. excluding a workspace_init profile.

This is especially relevant when profile parameters are loaded, as the ProfileParams must be the same across the listed profiles to be able to be loaded.
Augmented: List profiles and some profile parameters.

This is valuable when the user needs to see the profile parameter, such as whether the profile is managing a development environment or a production environment.

Command Creation

To create this command:

Build the command context.
- Make sure to register the workspace / profile params types, even if the value for all of them is None.
- Provide the filter function if necessary.
Present the list of profiles using the goal output.

Workspace Stored Active Profile

This is like git switch -c $branch.

This stores the profile to use in workspace_params, so the use does not have to provide the profile on every command invocation.

Suitable scopes for this command are NoProfileNoFlow or SingleProfileNoFlow.

Command Creation

To create this command:

When building the command context:
- Include a Profile as a workspace param.
- If setting profile parameters, use the SingleProfileNoFlow scope, and set the current profile using with_profile.
When building the command context, load the profile from workspace params using: with_profile_from_workspace_params.

It is best practice to set the active profile during workspace initialization, so that it is never None when subsequent commands are invoked.

Workspace Active Profile Switch

This is like git switch $branch.

This changes the stored profile in workspace_params, used in conjunction with the above.

Suitable scopes for this command are NoProfileNoFlow or SingleProfileNoFlow.

Command Creation

To create this command:

When building the command context:
- Include a Profile as a workspace param.
- Also set the current profile using with_profile.

When the command context is built, the default profile is stored in workspace_params.yaml.

State Discovery and Display

This is like git fetch.

This kind of command is intended for state discovery that is expensive – takes more than 500 milliseconds.

Suitable scopes for this command are:

SingleProfileSingleFlow: For discovering state for one profile.
MultiProfileSingleFlow: For discovering state for multiple profiles.

Command Creation

To create this command:

When building the command context:
- Provide the profile.
- Provide the flow ID.
Call the StatesDiscoverCmd depending on the intended use:

These will store the discovered states under the corresponding $profile/$flow_id directory as states_current.yaml or states_goal.yaml.
If the discovered states are to be displayed, call the relevant state display command(s):
- StatesCurrentStoredDisplayCmd: For stored current states to be displayed.
- StatesGoalDisplayCmd: For goal states to be displayed.

State Read and Display

This is like git show.

This kind of command is intended to display state that has previously been discovered.

Suitable scopes for this command are:

SingleProfileSingleFlow: For reading state for one profile.
MultiProfileSingleFlow: For reading state for multiple profiles.

Command Creation

To create this command:

When building the command context:
- Provide the profile.
- Provide the flow ID.
Call one of the state read commands depending on the intended use:

These will store the discovered states under the corresponding $profile/$flow_id directory as states_current.yaml or states_goal.yaml.
- StatesCurrentReadCmd: For current states to be discovered.
- StatesGoalReadCmd: For goal states to be discovered.
Call the relevant state display command(s):
- StatesCurrentStoredDisplayCmd: For current states to be displayed.
- StatesGoalDisplayCmd: For goal states to be displayed.

State Diff

This is like git status / git diff.

git status is a diff between the latest commit, and the working directory, summarized as one line per file.

git diff is that same diff, but shown with more detail.

This kind of command shows the difference between two States for each managed item.

Typically this is the difference between the current or current stored state, and its goal state. It can also be the difference between the current states between two profiles.

Suitable scopes for this command are:

SingleProfileSingleFlow: For diffing state within one profile.
MultiProfileSingleFlow: For diffing state across multiple profiles.

Command Creation

To create this command for a single profile:

When building the command context:
- Provide the profile.
- Provide the flow ID.
Determine the "from" and "to" states:

The "from" state is usually one of:
- Discovering the current state.
- Reading states_current.yaml. The "to" state is usually one of:
- Discovering the goal state.
- Reading states_goal.yaml.
- The clean state.
Call the state DiffCmd.

To create this command for multiple profiles:

When building the command context:
- Filter the profiles if necessary.
- Provide the flow ID.
Determine the "from" and "to" states:

The "from" state is usually one of:
- Discovering the current state.
- Reading states_current.yaml. The "to" state is usually one of:
- Discovering the goal state.
- Reading states_goal.yaml.
- The clean state.
Call the state DiffCmd for each pair of states.

State Apply

This is like git commit / git push / git clean, depending on your perspective.

This kind of command applies the goal state over the current state.

This generally requires what is stored in states_current.yaml to match the newly discovered current state.

The only suitable scope for this command is SingleProfileSingleFlow.

Command Creation

To create this command:

When building the command context:
- Provide the profile.
- Provide the flow ID.
Call the EnsureCmd.

This will call ApplyFns::exec for each item, beginning from the first item, until the last item.

State Clean

This is like rm -rf.

This kind of command applies the clean state over the current state.

This generally requires what is stored in states_current.yaml to match the newly discovered current state.

The only suitable scope for this command is SingleProfileSingleFlow.

Command Creation

To create this command:

When building the command context:
- Provide the profile.
- Provide the flow ID.
Call the CleanCmd.

This will call CleanOpSpec::exec for each item, beginning from the last item, until the first item.

Cmd Execution

When we run a CmdExecution, possible outcomes are:

In a CmdBlock, failure within one or more items.
In a CmdBlock, failure within block code.
Interruption / partial execution.
Successful execution of all blocks.

Each CmdBlock has its own block outcome type, and its own .

Framework

Use Cases: Framework Implementor

`*Cmd` Impls

It would be nice to not have to write interruption handling code in every CmdBlock.
Also, having CmdBlocks abstracted allows timings to be collected per block in a common place.

What does a `CmdExecution` implementor want to do?

DiffCmd:
- StateDiscoveryBlock: Return item failures to caller.
- StateDiscoveryBlock: Return block failure / interruption to caller.
- DiffBlock: Return item failures to caller.
- DiffBlock: Return block failure / interruption to caller.
EnsureCmd:
- StateDiscoveryBlock: Return item failures to caller.
- StateDiscoveryBlock: Return block failure / interruption to caller.
- DiffBlock: Return item failures to caller.
- DiffBlock: Return block failure / interruption to caller.
- EnsureBlock: Serialize States on item failure, return States and item failures to caller.
- EnsureBlock: Return block failure / interruption to caller.
- CleanBlock: Serialize States on item failure, return States and item failures to caller.
- CleanBlock: Return block failure / interruption to caller.
- EnsureBlock: ditto.
- CleanBlock: ditto. The desired return type would be:
- StatesEnsured on success.
How should we reply to the caller?
- CmdBlock block level error.
- IndexMap<ItemId, E> item level error. Error handlers per CmdBlock will do any serialization work if desired.

Use Cases: Automation Developer / End User

Store, retrieve, and display execution report.

How do we do this in code?

What it could look like conceptually:

`DiffCmd`

#![allow(unused)]
fn main() {
let cmd_execution = CmdExecution::builder()
    .with_block(StatesDiscoverCmdBlock::<CurrentAndGoal>::new())
    .with_block(DiffCmdBlock::new())
    .build()
    .await;
}

`EnsureCmd` -- "simple" version

#![allow(unused)]
fn main() {
let cmd_execution = CmdExecution::builder()
    .with_block(StatesCurrentReadCmdBlock::new())
    .with_block(StatesDiscoverCmdBlock::new(Discover::Current))
    // Compares current with stored, and fails if
    // current stored doesn't match current discovered.
    .with_block(ApplyGuardCmdBlock::new())  // `Outcome` and `OutcomeAcc` are `()`
    .with_block(ApplyCmdBlock::new())       // state_goal, diff and apply.
    .build()
    .await;
}

`EnsureCmd` -- "complex" version

#![allow(unused)]
fn main() {
let cmd_execution = CmdExecution::builder()
    .with_block(StatesCurrentReadCmdBlock::new())
    .with_block(StatesDiscoverCmdBlock::new(Discover::Current))
    .with_block(ApplyGuardCmdBlock::new())
    .with_block(EnsureCmdBlock::new(item_ids_no_blockage))  // state_goal, diff and apply.
    .with_block(CleanCmdBlock::new(item_ids_blocking))    // state_clean, diff and apply.
    .with_block(EnsureCmdBlock::new(item_ids_unblocked))  // state_goal, diff and apply.
    .with_block(CleanCmdBlock::new(item_ids_obsolete))    // state_clean, diff and apply.
    .build()
    .await;

// `item_ids_unblocked` is a superset of the `item_ids_blocking`.
//
// ⚠️ We could have a more granular design version that starts ensuring unblocked items
// while `cleaning` is happening.
}

Error Handling and Interruptions

When a CmdBlock fails, we need to consider what to do with:

CmdBlock::OutcomeAcc.
item errors.
CmdBlock error.
interruptions.

Originally an additional closure argument next to each CmdBlock argument was considered:

|_input, _outcome_acc, cmd_block_error| async move { cmd_block_error }

For example:

|(_states_current, _states_goal), states_ensured_mut, apply_error| async move {
    let serialize_result = StatesSerializer::serialize(states_ensured_mut).await;

    apply_error.or(serialize_result)
},

However, most usages would just pass through the error.

Before this CmdExecution design, ApplyCmd would:

In the producer, send the intermediate result to the outcome collator and stop execution (apply_cmd.rs#L699-L710).
In the collator:
- Intermediate CmdBlock errors are returned as is. (apply_cmd.rs#L521-L525).
- Intermediate discovery errors are collated into the CmdExecution outcome type (apply_cmd.rs#L526-L558).

From the use cases, what we care about are:

Returning CmdBlock errors to the caller.
Returning item errors to the caller.
Collating intermediate CmdBlock item values into the CmdExecution outcome type if possible.
Serializing States to storage: EnsureCmdBlock and CleanCmdBlock should still serialize each item's current state, on failure / interruption / success.
Serializing the CmdBlock::Outcome in the execution outcome report.

Deferred:

Serializing the CmdBlock::OutcomeAcc in the execution outcome report.
Deserializing the CmdBlock::OutcomeAcc to be displayed.

Interruptions

Interrupt before execution.
Interrupt within a block.
Interrupt between blocks.
Interrupt after last block.

Interruptibility

How should the interrupt channel be initialized and stored?

Use Cases

When automation software is used as:

CLI interactive / non-interactive.
Web service + browser access.
CLI + web service + browser access.

In all cases, we need to:

Initialize the CmdCtx with the interrupt_rx
Spawn the listener that will send InterruptSignal in interrupt_tx.

Imagined Code

CLI Interactive / Non-interactive

Both interactive and non-interactive can listen for SIGINT:

Interactive: SIGINT will be sent by the user pressing Ctrl + C.
Non-interactive: SIGINT could be sent by a CI thread.

#![allow(unused)]
fn main() {
let (interrupt_tx, interrupt_rx) = oneshot::channel::<InterruptSignal>();

tokio::task::spawn(async move {
    // Note: Once tokio takes over the process' `SIGINT` handler, it cannot be undone.
    //
    // This limitation is due to how Linux currently works.
    tokio::signal::ctrl_c()
        .await
        .expect("Failed to initialize signal handler for SIGINT");

    let (Ok(()) | Err(InterruptSignal)) = interrupt_tx.send(InterruptSignal);
});

let mut cmd_ctx = CmdCtx::single_profile_single_flow(output, workspace, interrupt_rx)
    .build();

let cmd_outcome = EnsureCmd::exec(&mut cmd_ctx).await?;
}

Web Service

The interrupt_tx must be accessible from a separate web request.

#![allow(unused)]
fn main() {
async fn cmd_exec_start_handler(params: Params) -> CmdExecutionId {
    let (interrupt_tx, interrupt_rx) = oneshot::channel::<InterruptSignal>();
    let mut cmd_ctx = CmdCtx::single_profile_single_flow(output, workspace, interrupt_rx)
        .build();

    let cmd_execution_id = EnsureCmd::exec_bg(cmd_ctx);

    let cmd_execution_by_id = cmd_execution_by_id
        .lock()
        .await;
    cmd_execution_by_id.insert(cmd_execution_id, interrupt_tx);

    cmd_execution_id
}

/// Returns the progress of the `CmdExecution`.
async fn cmd_exec_progress_handler(cmd_execution_id: CmdExecutionId) -> Result<CmdProgress, E> {
    self.cmd_progress_storage.get(cmd_execution_id).await
}

async fn cmd_exec_interrupt_handler(cmd_execution_id: CmdExecutionId) -> Result<(), E> {
    let cmd_execution_by_id = cmd_execution_by_id
        .lock()
        .await;

    if let Some(interrupt_tx) = cmd_execution_by_id.get(cmd_execution_id) {
        let (Ok(()) | Err(InterruptSignal)) = interrupt_tx.send(InterruptSignal);

        Ok(())
    } else {
        Err(E::from(Error::CmdExecutionIdNotFound { cmd_execution_id }))
    }
}
}

CLI + Web Service

There are two variants of CLI and web service:

CLI command running on the user's machine, web service that is a UI for that one command execution.
CLI client to a web service, so the CLI is just a REST client.

CLI on User's Machine + Web UI

For the first variant, the CmdExecution invocation is similar to Web Service, with the following differences:

Output progress is pushed to both CLI and CmdProgress storage.
Interruptions are received from both process SIGINT and client requests.

#![allow(unused)]
fn main() {
async fn cmd_exec_start(params: Params) {
    let (interrupt_tx, interrupt_rx) = oneshot::channel::<InterruptSignal>();
    let mut cmd_ctx = CmdCtx::single_profile_single_flow(output, workspace, interrupt_rx)
        .build();

    let cmd_execution_id = EnsureCmd::exec_bg(cmd_ctx);

    // We store an `interrupt_tx` per `CmdExecutionId`,
    // as well as spawn a Ctrl C handler.
    let cmd_execution_by_id = cmd_execution_by_id
        .lock()
        .await;
    cmd_execution_by_id.insert(cmd_execution_id, interrupt_tx.clone());

    tokio::task::spawn(async move {
        tokio::signal::ctrl_c()
            .await
            .expect("Failed to initialize signal handler for SIGINT");

        let (Ok(()) | Err(InterruptSignal)) = interrupt_tx.send(InterruptSignal);
    });

    // TODO: store `cmd_execution_id` as the only running `CmdExecution`.
}

/// Returns the progress of the `CmdExecution`.
async fn cmd_exec_progress_handler(cmd_execution_id: CmdExecutionId) -> Result<CmdProgress, E> {
    self.cmd_progress_storage.get(cmd_execution_id).await
}

async fn cmd_exec_interrupt_handler(cmd_execution_id: CmdExecutionId) -> Result<(), E> {
    let cmd_execution_by_id = cmd_execution_by_id
        .lock()
        .await;

    if let Some(interrupt_tx) = cmd_execution_by_id.get(cmd_execution_id) {
        let (Ok(()) | Err(InterruptSignal)) = interrupt_tx.send(InterruptSignal);

        Ok(())
    } else {
        Err(E::from(Error::CmdExecutionIdNotFound { cmd_execution_id }))
    }
}
}

CLI as Rest Client to Web Service

This is essentially the Web Service implementation, but rendering the progress on the machine with the CLI.

Flow Versioning

A flow's version needs to be increased whenever:

Item: An associated data type is changed -- State, Diff, Params.
Flow: An item is added.
Flow: An item is removed.
Flow: An item is renamed.

Flow Version Maintenance

For developers, maintaining multiple versions of flows is a cost, and usually one that is necessary. As items are created, modified, and removed throughout a flow's evolution, automation software needs to manage stored information from previous versions of the flow.

There are a number of ways to manage and ship flow versions:

One version: Only ship with one flow version.

Items never persist beyond the lifetime of the automation software.

Suitable for once off flows where ensure (and clean up) are done with the same version of the automation software.
Blue-green versioning: Ship with the previous and the next version.

Automation software is compatible with the previous version of each flow, as well as the next.

All stored information must be upgraded to the newer version before a new flow version is shipped.

Suitable for flows that do not evolve often, and not all previous versions of flows need to be supported.
n-versioning: Ship with n-versions of flows.

n may be all flows.

Each flow's state will need to adhere to the following, mainly for deserialization:
- Either never evolve its data type, or be an enum, with a variant per version / evolution.
- Ship with upgrade code from old versions to the current version. This is the highest cost in terms of code maintenance, and the shipped binary may grow unreasonably large over time.

Flow Upgrades

Upgrade Strategies

For blue-green / n versioning, there are a number of strategies for upgrading stored information in the flow:

Automatic upgrade:
- Implementors and developers need to ship with v1 -> v2 migration code.
- Simplest for users, assuming migration path is well understood / no surprises.
- Developers may want to diff this.
Manual upgrade:
- Implementors and developers need to ship with v1 -> v2 migration code.
- Users have to choose to migrate.
- Developers and users may want to diff things.

For n-versioning, upgrading from v1 -> v3 should use v1 -> v2 -> v3, so that developers don't have to write migration code form every version to the latest.

Execution History

To render old either we have one standard format that doesn't need old data types to present, or we ship those types.

Makes sense to have a standard format -- shipping with previous data types creates bloat, and makes it impossible to support execution history for shipping single versions of flows.

Seamless Execution

Seamless execution when multiple flow versions exist requires handling the following:

Reading and writing (serialization) stored state / outcomes from different flow versions.
Reading and writing to (adding, removing, modifying) the items from different flow versions.

Stored State

Stored state is accessed when:

CmdCtx is built: Read and write.
Commands are run:
- StatesCurrentStoredReadCmd, StatesGoalReadCmd, ExecutionHistoryListCmd / ExecutionHistoryShowCmd: Read.
- StatesDiscoverCmd, EnsureCmd, CleanCmd: Write.

`CmdCtx` Build

For cmd ctx build, because state could be in any version (current or any previous version), deserialization needs to be stateful -- like how type_reg::untagged::TypeReg is done.

fn envman_cmd_ctx() {
    let flow_versions = FlowVersions::builder()
        .add_flow(flow_v1)
        .add_flow(flow_v2)
        .add_flow(flow_v3)
        .build();

    let cmd_ctx = CmdCtx::builder_single_profile_single_flow(&mut output, &workspace)
        // .with_flow(flow)
        .with_flow_versions(flow_versions)
        .await?;
}

Cmd States and Outcome: Read

StatesCurrentStoredReadCmd and StatesGoalReadCmd needs to read states from any supported flow version.

ExecutionHistoryListCmd / ExecutionHistoryShowCmd (not implemented yet) need to read outcomes (and hence states and diffs) from any previous flow version.

Cmd States and Outcome: Write

StatesDiscoverCmd: Some overlap with migrating stored states to latest format.
EnsureCmd / CleanCmd: Write new version after ensuring.

What about interruptions?

We'd probably have to write the version of each item state then. Enum variant per version solves this.

/// Indicates when stored state can be upgraded.
///
/// Note: Stored outcomes also contain state, so for an upgrade between two
/// given versions, an `OutcomeUpgradeReq` will be the same as this.
///
/// However, should we mutate stored outcomes? I imagine not.
enum StateUpgradeReq {
    /// Data can be upgraded without discovering state.
    None,
    /// State needs to be discovered in order to store the new version.
    ///
    /// Existing stored state doesn't carry enough information to construct the
    /// new version, but the existing item does.
    Discover,
    /// Item needs an ensure / clean to be run to store the new version.
    ///
    /// The existing item doesn't carry enough information to construct the new
    /// state, and needs `apply` to be run to do so.
    ///
    /// This means `ApplyCmd`, and hence `Item{,Rt,Wrapper}`, need to work with
    /// different state versions.
    Apply,
}

Items: Use Latest / Migrate to Latest

An item upgrade may be one of:

Only modifying the data type, and changing the information stored about the item (see StateUpgradeReq::Discover above).
Modifying the item in place.
Cleaning up the item then re-ensuring it.

Notes:

It may not be possible to migrate an item to a newer version without cleaning up its successors.
It is possible if successors don't "live inside" the item, but only need their parameters changed.
it is not possible if successors live inside the item, and need to be deleted in order to modify the item.

Whether or not a successor needs to be cleaned up likely should be encoded into the Edge type, not part of the item upgrade parameters -- an item is unable to know if its successors live inside it or point to it.

/// Indicates what is needed for an instance of an item to be upgraded.
enum ItemUpgradeReq {
    /// Item instance does not need to be upgraded, i.e. only state data type is changed.
    None,
    /// Item can and needs to be re-ensured as part of this upgrade.
    Modify,
    /// Item needs to be cleaned and re-ensured as part of this upgrade.
    Replace,
}

Running The Upgrade Code

The following use cases seem natural for developers / users:

Manually upgrade state after automation software has been updated.
Automatically upgrade state as part of using regular commands.
Manually upgrade items after automation software has been updated.
Automatically upgrade items as part of using regular commands.

Some use cases are mutually exclusive, though mutually exclusive may still exist in the same automation software, for different version or kinds of upgrades.

For example, automatic item upgrades as part of using regular commands are appropriate when nothing needs to be cleaned up, but manual item upgrades should be used when some items need to be cleaned as part of the upgrade.

Manual State Upgrade

Developers must provide an upgrade command, to call StateUpgradeCmd::exec.

impl StateUpgradeCmd {
    /// # Note
    ///
    /// This is single profile single flow.
    ///
    /// To upgrade the states of multiple profiles and multiple flows, this
    /// must be called per flow, and maybe per profile as well.
    ///
    /// `MultiProfileSingleFlow` may only be safe for `StateUpgradeReq::None`
    /// and `StateUpgradeReq::Discover`.
    async fn state_upgrade_exec(cmd_ctx: &mut CmdCtx<SingleProfileSingleFlow>)
    -> Result<(), Error> {
        match state_upgrade_req {
            StateUpgradeReq::None => {
                let states = resources.get::<StatesCurrentStored>();
                let states_upgraded = flow
                    .graph()
                    .iter()
                    .for_each(|item| item.state_upgrade(states));
                states_serializer.serialize(&states_upgraded).await?;
            }
            StateUpgradeReq::Discover => {
                let CmdOutcome {
                    value: (states_current, states_goal),
                    errors,
                } = StatesDiscoverCmd::current_and_goal(&mut cmd_ctx).await?;
                states_serializer.serialize_current(&states_upgraded).await?;
                states_serializer.serialize_goal(&states_upgraded).await?;
            }
            StateUpgradeReq::Apply => {
                let CmdOutcome {
                    value: states_upgraded,
                    errors,
                } = EnsureCmd::exec_with(&mut cmd_ctx.as_sub_cmd()).await?;
            }
        }
    }
}

Automatic State Upgrade

The above method, but called automatically from a set of chosen commands: EnsureCmd, CleanCmd.

May need to think about this more carefully.

Manual Item Upgrade

Developers must provide an upgrade command.

For actual item modification, it may be one of:

// Only discovery needed
fn env_ensure_cmd() {
    env_flow_current_version_ensure();
    env_flow_ensure();
}

// apply needed
fn env_ensure_cmd() {
    // Item implementation needs to handle upgrade from whatever version the
    // state is in, to the current version.
    //
    // Meaning, there is no API support / constraints from the `peace`
    // framework. It's all handled from within the item implementation.
    env_flow_ensure();
}

Automatic Item Upgrade

Same as manual item upgrade, except the upgrade command is not separate to the regular ensure command.

For item upgrades, state could always be an enum, and peace doesn't include API support, constraints, or special handling for migrating between item versions -- the current building blocks are enough to implement seamless upgrades if items are implemented well.

For flow upgrades, we still need to handle clean up of old items no longer needed in subsequent flow versions, meaning the items must still be included in the development tool for their clean up logic.

For the developer, this could be either:

Passing in the previous and current flow versions, and the framework working out which items need to be cleaned up.
Passing in the current flow version, and old items and params for the framework to clean up those items.

Is this enough to start working on StatesSerde?

Yes:

StatesSerde should hold unknown entries (removed items).
States should be mapped From<StatesSerde> after deserialization.
Users should be informed when there are unknown entries.
States should be mapped Into<StatesSerde>, with an entry for each item in the flow.

Upgrade Actions

There are 4 levels of upgrade actions that can be applied to items.

Reserialization with existing stored values.
State discovery: current / goal.
Item ensure.
Replacement (clean and ensure) required.

The following table shows what is required for ensuring an environment to be in sync with the change. If multiple changes are involved, then the highest level of upgrade action needs to be applied to the environment for params and state, and the environment to be in sync, i.e. max(upgrade_actions).

Notes:

"Data" refers to an item's Params, State, or StateDiff.

Apply is associated with params, state, and diff.

apply_clean needs to know state_clean based on the parameters used to compute state_goal at the time of the previous apply_goal.

"Predecessor action" means what upgrade action is needed for this item, given a predecessor has had action applied to it. In other words, the change for an item can imply an upgrade action for successors.

Change	Upgrade action
Param value	`apply_goal`
Data type field rename	`reserialization`
Data type field addition	`state_discovery`
Data type field removal	`state_discovery`
Data type field modification	`apply_goal`
Flow item addition	`apply_goal`
Flow item removal	`apply_clean`
Flow item replacement	`apply_clean`, `apply_goal`
Predecessor replacement, `Edge::Link`	`apply_goal`
Predecessor replacement, `Edge::Contains`	`apply_clean`, `apply_goal`
Predecessor addition / modification	`apply_goal` (in case of update)
Predecessor removal	`apply_clean` (successor first)

User Facing Automation - Part 1

Set Up

Commands before demoing (ZSH)

# Create demo directory and source file
demo_dir=/tmp/automation_demo
test -d "${demo_dir}" || mkdir "${demo_dir}"
cd "${demo_dir}"
src="${demo_dir}/a.txt"
dest="${demo_dir}/b.txt"

# Change time format. This is for zsh.
#
# Functions must also be run in subshells in zsh for `time` to work:
# <https://superuser.com/questions/688128/how-to-run-time-on-a-function-in-zsh>
TIMEFMT=$'%*E seconds'

echo hi > "$src"

For the download example:

# Requires Rust
cargo install --git https://github.com/azriel91/peace download --all-features

In a separate shell:

cd /tmp/automation_demo
case "$(uname)" in
    Linux)  watch -n 0.4 -c -d "stat --format='%y' b.txt | cut -b 12-23 ; bat b.txt" ;;
    Darwin) watch -n 0.4 -c -d "stat -f '%y' b.txt | cut -b 12-23 ; bat b.txt" ;;
esac

Scenario

Copy a file from one place to another.

For demonstration, this section will use the sleep and cp (copy) commands to emulate a task:

function slow_cp {
    sleep 1;
    cp "$1" "$2";
}

Basics

Concept 1: Repeatable in One Action

What: Do it again without thinking.

Value: Save time and mental effort.

Example

# Hard coded values
function slow_cp {
    sleep 1;
    cp a.txt b.txt;
}

slow_cp

Concept 2: Parameterized

What: Do the same action on different things.

Value: Multiply the automation gain per thing.

Example

function slow_cp {
    sleep 1;
    cp "$1" "$2";
}

src="/tmp/automation_demo/a.txt"
dest="/tmp/automation_demo/b.txt"

slow_cp "${src}" "${dest}"

Efficiency

Concept 3: Idempotence

What: Don't do it if it's already done.

Value: Save time.

Example

Execution 1:

Execution 2:

function idempotent_cp {
    if   ! test -f "${dest}"
    then slow_cp "$1" "$2"
    fi;
}

rm -f "${dest}"
time (idempotent_cp "${src}" "${dest}")
time (idempotent_cp "${src}" "${dest}")

echo updated > "${src}"
time (idempotent_cp "${src}" "${dest}")

Concept 4: Smart Idempotence

What: Handle updates.

Value: Do what's expected.

Example

function idempotent_cp {
    local src_hash;
    local dest_hash;
     src_hash=$(md5sum <(cat "$1"))
    dest_hash=$(md5sum <(cat "$2" 2>/dev/null))
    if   ! test "${src_hash}" = "${dest_hash}"
    then slow_cp "$1" "$2"
    fi;
}

rm -f "${dest}"
time (idempotent_cp "${src}" "${dest}")
time (idempotent_cp "${src}" "${dest}")

Concept 5: Parallelism

What: Run tasks at the same time.

Value: Elapsed duration to execute process is decreased.

Example

dest_1="/tmp/automation_demo/b1.txt"
dest_2="/tmp/automation_demo/b2.txt"
dest_3="/tmp/automation_demo/b3.txt"

Serial

# Serial execution
rm -f "${dest_1}" "${dest_2}" "${dest_3}"
time (
    idempotent_cp "${src}" "${dest_1}";
    idempotent_cp "${src}" "${dest_2}";
    idempotent_cp "${src}" "${dest_3}";
)
time (
    idempotent_cp "${src}" "${dest_1}";
    idempotent_cp "${src}" "${dest_2}";
    idempotent_cp "${src}" "${dest_3}";
)

# Remove one file
rm -f "${dest_2}"
time (
    idempotent_cp "${src}" "${dest_1}";
    idempotent_cp "${src}" "${dest_2}";
    idempotent_cp "${src}" "${dest_3}";
)
time (
    idempotent_cp "${src}" "${dest_1}";
    idempotent_cp "${src}" "${dest_2}";
    idempotent_cp "${src}" "${dest_3}";
)

Parallel

# Parallel execution
rm -f "${dest_1}" "${dest_2}" "${dest_3}"
time (
    idempotent_cp "${src}" "${dest_1}" &;
    idempotent_cp "${src}" "${dest_2}" &;
    idempotent_cp "${src}" "${dest_3}" &;
    wait;
)
time (
    idempotent_cp "${src}" "${dest_1}" &;
    idempotent_cp "${src}" "${dest_2}" &;
    idempotent_cp "${src}" "${dest_3}" &;
    wait;
)

# Remove one file
rm -f "${dest_2}"
time (
    idempotent_cp "${src}" "${dest_1}" &;
    idempotent_cp "${src}" "${dest_2}" &;
    idempotent_cp "${src}" "${dest_3}" &;
    wait;
)
time (
    idempotent_cp "${src}" "${dest_1}" &;
    idempotent_cp "${src}" "${dest_2}" &;
    idempotent_cp "${src}" "${dest_3}" &;
    wait;
)

Concept 6: Logical Dependencies

What: Wait if you have to.

Value: Correctness.

Example

# Logical dependency
rm -f "${dest_1}" "${dest_2}" "${dest_3}"
time (
    idempotent_cp "${src}" "${dest_1}" &;
    (
        idempotent_cp "${src}" "${dest_2}";
        idempotent_cp "${dest_2}" "${dest_3}";
    ) &;
    wait;
)
time (
    idempotent_cp "${src}" "${dest_1}" &;
    (
        idempotent_cp "${src}" "${dest_2}";
        idempotent_cp "${dest_2}" "${dest_3}";
    ) &;
    wait;
)

Output

Concept 7: Progress Information

What: Tell users what's going on.

Value: Users know what's happening, or if it's stalled.

Example

Code

function slow_cp {
    sleep 1;
    cp "$1" "$2";
}

function hash_file {
    1>&2 printf "hashing file: ${1}\n"
    test -f "${1}" &&
      md5sum <(cat "$1" 2>/dev/null) ||
      printf '00000000000000000000000000000000'
}

function informational_idempotent_cp {
     src_hash=$(hash_file "${src}")
    dest_hash=$(hash_file "${dest}")
    1>&2 printf " src_hash: ${src_hash}\n"
    1>&2 printf "dest_hash: ${dest_hash}\n"

    if   ! test "${src_hash}" = "${dest_hash}"
    then
        1>&2 printf "contents don't match, need to copy.\n"
        slow_cp "$1" "$2"
        1>&2 printf "file copied.\n"
    else
        1>&2 printf "contents match, don't need to copy.\n"
    fi;
}

rm -f "${dest}"

informational_idempotent_cp "${src}" "${dest}"

# if we don't care about the verbose information, we can hide it
informational_idempotent_cp "${src}" "${dest}" 2>/dev/null

Concept 8: Provide Relevant Information

What: Show as little and as much information as necessary.

Value: Reduce effort to understand.

Example

Code

delay=0.1
pb_lines=3

function slow_cp_with_progress {
    progress_update 50 20 'copying file'; sleep $delay
    progress_update 50 25 'copying file'; sleep $delay
    progress_update 50 30 'copying file'; sleep $delay
    progress_update 50 35 'copying file'; sleep $delay
    progress_update 50 40 'copying file'; sleep $delay
    progress_update 50 45 'copying file'; sleep $delay
    cp "$1" "$2";
}

function hash_file {
    test -f "${1}" &&
      md5sum <(cat "$1" 2>/dev/null) ||
      printf '00000000000000000000000000000000'
}

function clear_lines {
    local lines_to_clear
    local i

    lines_to_clear=$(($1 - 1))  # subtract 1 because bash loop range is inclusive

    if test "${lines_to_clear}" -ge 0
    then
        for i in {1..${lines_to_clear}}
        do
            1>&2 printf "\033[2K\r" # clear message line
            1>&2 printf "\033[1A"   # move cursor up one line
        done
    fi
}

function progress_write {
    local progress_total
    local progress_done
    local progress_message

    local progress_remaining
    local printf_format

    progress_total=$1
    progress_done=$2
    progress_message="${3}"
    progress_remaining=$(($progress_total - $progress_done))
    if test $progress_total -eq $progress_done
    then printf_format="\e[48;5;35m%${progress_done}s\e[48;5;35m%${progress_remaining}s\e[0m\n" # green
    else printf_format="\e[48;5;33m%${progress_done}s\e[48;5;18m%${progress_remaining}s\e[0m\n" # blue
    fi

    1>&2 printf "$printf_format" ' ' ' '
    1>&2 printf "$progress_message"
    1>&2 printf '\n'
}

function progress_update {
    clear_lines $pb_lines # message line, progress bar line, extra line
    progress_write "$@"
}

function informational_idempotent_cp {
    local src_hash;
    local dest_hash;
    progress_write 50 0 'hashing source file'; sleep $delay
    src_hash=$(hash_file "${src}")

    progress_update 50 5 'hashing destination file'; sleep $delay
    dest_hash=$(hash_file "${dest}")

    progress_update 50 10 'comparing hashes'; sleep $delay
    if   ! test "${src_hash}" = "${dest_hash}"
    then
        progress_update 50 15 'copying file'; sleep $delay
        slow_cp_with_progress "$1" "$2"

        progress_update 50 50 '✅ file copied!'
        1>&2 printf "\n"
    else
        progress_update 50 50 '✅ contents match, nothing to do!'
        1>&2 printf "\n"
    fi;
}

rm -f "${dest}"

informational_idempotent_cp "${src}" "${dest}"

delay=0.7
rm -f "${dest}"

Concept 9: Information Format

What: Change information format specifically for how it is consumed.

Value: Makes using the API ergonomic.

Tip: Progress information can be more than a string.

For a human: output a progress bar, replace status text
For continuous integration: append status text
For a web request: output json

Example

Code

output_format=pb # pb, text, json

function progress_write {
    local progress_total
    local progress_done
    local progress_message
    progress_total=$1
    progress_done=$2
    progress_message="${3}"

    local progress_remaining
    local printf_format
    progress_remaining=$(($progress_total - $progress_done))

    case "${output_format}" in
        pb)
            if test $progress_total -eq $progress_done
            then printf_format="\e[48;5;35m%${progress_done}s\e[48;5;35m%${progress_remaining}s\e[0m\n" # green
            else printf_format="\e[48;5;33m%${progress_done}s\e[48;5;18m%${progress_remaining}s\e[0m\n" # blue
            fi

            1>&2 printf "$printf_format" ' ' ' '
            1>&2 printf "$progress_message"
            1>&2 printf '\n'
            ;;
        text)
            1>&2 printf "$progress_message"
            1>&2 printf '\n'
            ;;
        json)
            cat << EOF
{ "progress_total": $progress_total, "progress_done": $progress_done, "progress_remaining": $progress_remaining, "message": "$progress_message" }
EOF
            ;;
    esac
}

function progress_update {
    case "${output_format}" in
        pb)
            clear_lines $pb_lines # message line, progress bar line, extra line
            ;;
        text)
            ;;
        json)
            ;;
    esac

    progress_write "$@"
}

output_format=pb
rm -f "${dest}"
time (informational_idempotent_cp "${src}" "${dest}")
echo '---'
time (informational_idempotent_cp "${src}" "${dest}")

output_format=text
rm -f "${dest}"
time (informational_idempotent_cp "${src}" "${dest}")
echo '---'
time (informational_idempotent_cp "${src}" "${dest}")

output_format=json
rm -f "${dest}"
time (informational_idempotent_cp "${src}" "${dest}")
echo '---'
time (informational_idempotent_cp "${src}" "${dest}")

informational_idempotent_cp "${src}" "${dest}" | jq

informational_idempotent_cp "${src}" "${dest}" | jq '.progress_remaining'

Error Handling

This section uses the download example.

Concept 10: Accumulate And Summarize

What: When an error happens, save it, then display it at the very end.

Value: User doesn't have to spend time and effort investigating.

Example

Don't Do This:

# Per subprocess
log_info "${id}: Start."
log_info "${id}: Processing."

download_file
download_result=$?

if [[ "$download_result" -eq 0 ]]
then
    log_info "${id}: Successful."
    log_info "${id}: Notifying service."
    return 0
else
    log_error "${id}: Download failed: ${download_result}"
    return 1
fi

Info : Starting process.
Info : b1: Start.
Info : b2: Start.
Info : b3: Start.
Info : main: Waiting for results.
Info : b1: Processing.
Info : b2: Processing.
Info : main: Waiting for results.
Info : b3: Processing.
Error: b2: Download failed: 12
Info : b1: Processing complete.
Info : main: Waiting for results.
Info : b3: Processing complete.
Info : b1: Successful.
Info : main: Waiting for results.
Info : b3: Successful.
Info : b3: Notifying service.
Info : main: Waiting for results.
Info : b1: Notifying service.
Info : main: Collected results.
Info : main: Analyzing.
Error: Process failed: b2.

Do This:

# Per subprocess
download_file
download_result=$?

if [[ "$download_result" -eq 0 ]]
then
    printf "{ \"id\": ${id} \"success\": true }"
    return 0
else
    printf "{ \"id\": ${id} \"success\": false, \"error_code\": 12 }"
    return 1
fi

Info : Starting process.
Info : b1: Start.
Info : b2: Start.
Info : b3: Start.
Info : main: Waiting for results.
Info : b1: Processing.
Info : b2: Processing.
Info : main: Waiting for results.
Info : b3: Processing.
Error: b2: Download failed: 12
Info : b1: Processing complete.
Info : main: Waiting for results.
Info : b3: Processing complete.
Info : b1: Successful.
Info : main: Waiting for results.
Info : b3: Successful.
Info : b3: Notifying service.
Info : main: Waiting for results.
Info : b1: Notifying service.
Info : main: Collected results.
Info : main: Analyzing.
Error: Process failed: b2.

Error: b2 failed with error code: 12

So that:

Info : Starting process.
Info : b1: Start.
Info : b2: Start.
Info : b3: Start.
Info : main: Waiting for results.
Info : b1: Processing.
Info : b2: Processing.
Info : main: Waiting for results.
Info : b3: Processing.
Error: b2: Download failed: 12
Info : b1: Processing complete.
Info : main: Waiting for results.
Error: b3: Download failed: 13
Info : b1: Successful.
Info : main: Waiting for results.
Info : main: Waiting for results.
Info : b1: Notifying service.
Info : main: Collected results.
Info : main: Analyzing.
Error: Process failed: b2, b3.

Error:
  b2 failed with error code: 12
  b3 failed with error code: 13

Concept 11: Understandable Error Messages

What: Translate the technical terms into spoken language terms.

Value: User can understand the message and take action.

Example

download -v init http://non_existent_domain file.txt

Error: peace_item_file_download::src_get

  × Failed to download file.
  ├─▶ error sending request for url (http://non_existent_domain/): error
  │   trying to connect: dns error: failed to lookup address information:
  │   Temporary failure in name resolution
  ├─▶ error trying to connect: dns error: failed to lookup address
  │   information: Temporary failure in name resolution
  ├─▶ dns error: failed to lookup address information: Temporary failure in
  │   name resolution
  ╰─▶ failed to lookup address information: Temporary failure in name
      resolution

Error: peace_item_file_download::src_get

  × Failed to download file.
   ╭────
 1 │ download init http://non_existent_domain/ file.txt
   ·               ─────────────┬─────────────
   ·                            ╰── defined here
   ╰────
  help: Check that the URL is reachable: `curl http://non_existent_domain/`
        Are you connected to the internet?

Concept 12: Capture the Source of Information

What: Show where the information came from.

Value: User doesn't have to spend time and effort investigating.

Example

download init http://localhost:3000/ peace_book.html

Stop the server, then:

download ensure

Error: peace_item_file_download::src_get

  × Failed to download file.
   ╭────
 1 │ download init http://localhost:3000/ peace_book.html
   ·               ───────────┬──────────
   ·                          ╰── defined here
   ╰────
  help: Check that the URL is reachable: `curl http://localhost:3000/`
        Are you connected to the internet?

Workflow Concepts

Concept 13: Clean Up

What: Leave a place in a state, no worse than when you found it.

Value: Don't waste resources.

Example

ls

download clean

Bash

function cp_flow {
    sub_cmd=$1

    case ${sub_cmd} in
        ensure)
            informational_idempotent_cp "$2" "$3"
            ;;
        clean)
            if   test -f "$2"
            then rm -f "$2"
            fi
            ;;
    esac ;
}

rm -f "${dest}"
time (cp_flow ensure "${src}" "${dest}")
time (cp_flow ensure "${src}" "${dest}")

time (cp_flow clean "${dest}")

User Facing Automation - Part 2

Workflow Concepts

Concept 14: Storing Parameters

What: Take in parameters, recall them automatically.

Value: Don't require users to pass in parameters repeatedly.

Example

download init https://ifconfig.me ip.json

download ensure
download clean

Concept 15: Current State, Goal State, Difference

What: Show users what is, and what will be, before doing anything.

Value: Give users clarity on what would happen, before doing it.

Example

download clean
download init https://ifconfig.me ip.json

download status
download goal
download diff

download ensure
download diff

download clean
download diff

Concept 16: Profiles

What: Use the same workspace input for logically separate environments.

Value: Ease the creation of cloned environments.

Concept 17: Flows

What: Perform different workflows within an environment.

Value: Workflows automatically share parameters with each other.

Concept 18: Parameter Limits

What: Guard automation from executing with unusual values.

Value: Don't accidentally incur large costs.

Efficiency Concepts (Again!)

Concept 19: Do It Before It's Asked

If a process cannot take less than 10 minutes,
then to do it in 5,
you must begin in the past.

What: Execute the automation in the background, show where it is when it is asked to be executed.

Value: Reduce waiting time.

Cost: Background work consumes resources, and may be redundant.

Concept 20: Reverse Execution

At the beginning of a process, what's the fastest way to get to step 9?

Go through steps 1 through 9.

What if you're on step 10?

Transition from step 10 to 9.

What: The beginning and destination state are not necessarily the start and end of a process.

Value: Allows inspection / mutation of state at a particular point, or re-testing of automation steps after that point.

Empathetic Code Design

Communicating Clearly In Code

Today's talk is about empathetic code design.
The project I will use to demonstrate this is Peace, a framework to create empathetic and forgiving software automation.
Peace provides constraints over logic and data types, and common functionality.
By using the framework, logic and data types will be shaped to produce an automation tool that is clear in its communication.
A user uses that tool to get their work done, with good user experience.
Here's a 3 step process to upload a file to the internet.
Download the file, create an S3 bucket, upload the file to that bucket.
We can use a shell script to execute this process. (live demo)
We can also use a tool, created using the Peace framework, to do the same thing.
envman is a tool I created earlier to do this process. (live demo)
This is a short demonstration of why one would use this framework.
When changing any environment, first we want to understand where we are.
Second, we also want to know, where we are headed.
Third, we want to know the cost of getting there.
Finally, when we understand enough about the change, we can decide to do it.
Now, when I say clear communication, it should be quite easy to see that A B C here maps to 1 2 3 there.
By presenting information in the right form, at the right level of detail, it's much easier to map between the concept and what is seen.
This is how Peace takes care of the user.
Today, we are going to look at how Peace takes care of the automation tool developer.
How clearly can we communicate between the mental model of a process, and code.
When designing a flow, we need to define the items in the flow, and the ordering between those items.
The reason these are called "items" instead of "steps", is because in Peace, the emphasis is on the outcome, rather than the actions.
Said another way, the purpose of technological processes is the end state, not the work undertaken to get there.
That statement does not apply to other kinds of processes, where the work is meaningful.
In code, adding items and ordering looks like this.
This shows linear ordering; it is possible to have work done in parallel when there is no dependency between items.
So here we can adjust the ordering such that A and B are both predecessors of C. A comes before C, and B comes before C.
So A and B can both begin at the same time, reducing the overall duration.
We'll adjust this in the example, I want you to see that both the file download and the s3 bucket turn blue, and when both are done, then the upload begins. We'll slow down our connection so it is easier to see. (live demo)
Try doing that in a shell script.
After a developer defines the items in a flow, they also need to define how data passes through these items.
If you've written automation using bash scripts or YAML, usually data is just strings.
For the non-devs among us, using "strings" everywhere is like saying, "I took stuff and did stuff, and here is stuff".
It would be much better to say, "I grab my camera, and took a shot, and here's a picture."
To better communicate with developers, Peace uses type safety.
Because Peace is a framework, each item's automation logic should be shareable.
This means, the input to each item, and its output, are API.
In Peace, we call the input "parameters", and we use call the output "state".
For a file download item, the parameters are the source URL to download from and the path to write to.
and the state, can be either "the file's not there", or "it's there, and here is its content hash".
Given each item defines its parameters and state, we need a way for developers to pass values into the parameters.
Usually, at the start of a process, we know the values of some parameters, and we can pass them straight in.
However, sometimes we need part of the process to be completed, to have the information to pass to a subsequent item.
For example, if we had a server instead of an S3 bucket, we cannot know what its IP address will be until we the server is created.
But we still want to encode into the flow, that the state from an earlier item, is needed to determine the parameter value for a subsequent item.
For the simple case, where we know the parameter values up front, we can just plug them in.
But we want to be able to express, "get the state from step two, and pass it into step three".
So I came up with this, for every item's parameters, the framework will generate a specification type that records how to figure out what values to use.
and you can pass it direct values, or you can tell it "here's what states to read, and here's how to extract the value to use for this field".
Here's how it looks like in code.
Yes, we can read states from multiple items, and use them to compute a value.
Best of all, it's type safe -- so within the code you can ask "what parameters do I need to pass in, and get them right".
If you make a mistake, you have a really big safety net called the compiler, that will save you.
The error message needs improvement, though I haven't yet figured that out.
If the parameter type changes, such as when you upgrade your item version, then you get a compilation error, which means you don't have to work to discover where to change -- the compiler tells you.
Now that we have our logic and our data, we need a way to interact with it.
Peace is designed to separate the automation logic and data, from how it is interacted with and presented.
Whether it is interactive shell, uninteractive shell, web API, or web page, there is no rendering code in any item.
This allows automation developers to use different front ends to the same automation backend.
In code, the developer can choose which output to use.
Currently the only user facing output implementation is the CliOutput, which you've seen.
There are a number of other output implementations used in tests,
such as the NoOpOutput which discards all information, and the FnTrackerOutput which stores what output methods were called, with what parameters.
Developers can choose to write their own output implementation as well, but
It is within the scope of Peace provide implementations for common use cases.
So far, we've seen Items, Parameters, Specifications, and Output.
As a developer, this should be sufficient to run some commands.
First we group all of the above into a CmdCtx, then we call the command we want to run for the flow.
StatesCurrentReadCmd and EnsureCmd are provided by the framework.
The real code still contains code to format the output, but this is something that the framework can provide.
For any automation software, the user should only need to pass in item parameters once.
Commands run using that flow should be able to recall parameters for the user.
How can we make this easy for the automation software developer?
The easiest thing, is for them to do nothing.
So whenever a command context is built, the following happens.
Previous parameter specifications will attempt to be loaded from disk.
The provided parameter specification will be merged with those.
Validation happens, to make sure there is enough information to resolve parameters for all items.
The merged parameter specifications will be serialized to disk.
If the parameter specifications don't pass validation, we should get a useful error message.
When a command is run, Peace stores information about that execution.
It stores this in a .peace directory within a workspace directory.
The automation tool developer needs to tell Peace how to determine the workspace directory.
Because if the tool is run in a sub directory, information should still be stored in the workspace directory.
If you think about git, if you run git status in any subdirectory of a repository, it will still find the .git directory somewhere up the chain.
One undeniable way to tell if software works, is to run it.
And to tell if it will work in production, we usually want to run it in a replica environment first.
Out of the box, Peace provides support for logically separate environments, through profiles.
So you can have multiple environments under different names, just like git branching.
Let's see a demo.
How do we communicate clearly in code?
We make sure the API matches the mental model.
We make sure we have good error messages.
When we require something of the developer, make it easy for them to provide it.
and when we adhere to these constraints, it will improve the automation development experience.

Background

Peace

Constraints and
common functionality

💻

Code

Logic and data types

⚙️

Nice Tool

Automation software

⚙️

Nice Tool

Automation software

🔡

Data

Values for parameters

🚀

Work Done

App built and published

Environment deployed

Scenario

Shell Script

bucket_name='azriel-peace-demo-bash'
file_path='web_app.tar'
object_key='web_app.tar'

curl --fail \
  -o "${file_path}" \
  --location \
  https://github.com/azriel91/web_app/releases/download/0.1.1/web_app.tar

aws s3api create-bucket \
  --bucket "${bucket_name}" \
  --acl private \
  --create-bucket-configuration LocationConstraint=ap-southeast-2 |
  bat -l json

aws s3api put-object \
  --bucket "${bucket_name}" \
  --key "${object_key}" \
  --body "${file_path}" |
  bat -l json

aws s3api delete-object \
  --bucket "${bucket_name}" \
  --key "${object_key}" |
  bat -l json

aws s3api delete-bucket --bucket "${bucket_name}" | bat -l json

rm -f "${file_path}"

What does automation look like, from the perspective of an automation tool developer, or a workflow designer.

Clarity between concept and code.
Easy to write.
Fast feedback when developing automation.
Provide good UX without needing to write UI code.

Command Context

At the bottom of every page, we will be building up a cmd_context, which holds the information about a workflow. Different commands can be invoked with the cmd_context to gather information about the workflow.

let mut cmd_context = CmdContext::builder()
    /* ?? */
    .build();

StatusCmd(&mut cmd_context).await?;
DeployCmd(&mut cmd_context).await?;

Flow Definition

Items: Steps of a process.
Ordering: Sequence between steps.

// examples/envman/src/flows/app_upload_flow.rs
let flow = {
    let graph = {
        let mut graph_builder = ItemGraphBuilder::<EnvManError>::new();

        let [a, b, c] = graph_builder.add_fns([
            FileDownloadItem::<WebApp>::new(item_id!("app_download")).into(),
            S3BucketItem::<WebApp>::new(item_id!("s3_bucket")).into(),
            S3ObjectItem::<WebApp>::new(item_id!("s3_object")).into(),
        ]);

        graph_builder.add_edges([(a, b), (b, c)])?;
        graph_builder.build()
    };

    Flow::new(flow_id!("app_upload"), graph)
};

Non-linear Ordering

 graph_builder.add_edges([
-    (a, b),
+    (a, c),
     (b, c),
 ])?;

Inputs and Outputs

For items to be reusable, its inputs and outputs are API.

Item API

impl<Id> Item for FileDownloadItem<Id>
{
    type Params<'exec> = FileDownloadParams<Id>;
    type State = FileDownloadState;
    // ..
}

Input:

pub struct FileDownloadParams<Id> {
    src: Url,
    dest: PathBuf,
    marker: PhantomData<Id>,
}

Output:

pub enum FileDownloadState {
    None,
    Some {
        path: PathBuf,
        md5: Md5Sum,
    },
}

Parameters Specification

Specify where to get the value for each item's input.

The value may not necessarily exist until the flow is executed.

let app_download_params = FileDownloadParams::<WebApp> {
    src: Url::parse("https://example.com/web_app.tar")?,
    dest: PathBuf::from("/tmp/path/to/web_app.tar"),
    marker: PhantomData,
};

let s3_bucket_params = S3BucketParams::<WebApp>::new(bucket_name);

let s3_object_params = S3ObjectParams::<WebApp> {
    file_path: PathBuf::from("/tmp/path/to/web_app.tar"),
    object_key: String::from("web_app.tar"),
    bucket_name: !?, /* Somehow get the bucket name from `b` */
};

Deferred Values

// examples/envman/src/flows/app_upload_flow.rs
let s3_object_params_spec = S3ObjectParams::<WebApp>::field_wise_spec()
    .with_file_path(PathBuf::from("/tmp/path/to/web_app.tar"))
    .with_object_key(String::from("web_app.tar"))
    .with_bucket_name_from_map(|s3_bucket_state: &S3BucketState| {
        match s3_bucket_state {
            S3BucketState::None => None,
            S3BucketState::Some {
                name,
                creation_date: _,
            } => Some(name.clone()), // type safe!
        }
    })
    .build();

let file_download_params_spec = file_download_params.into();
let server_instance_params_spec = server_instance_params.into();

Output and Presentation

When the workflow is defined, we want to interact with it in different ways.

Presentation

Command Line Interface

> ./envman deploy
Using profile ⦗demo⦘ -- type `development`

✅ 1. app_download ▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰ done!
⏳ 2. s3_bucket    ▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱ 0/1 creating bucket (el: 6s, eta: 0s)
⏳ 3. s3_object    ▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱ (el: 0s, eta: 0s)

Continuous Integration

2023-05-07T01:30:31.135Z | app_download | downloaded at /tmp/.a9efb2/web_app.tar
2023-05-07T01:30:31.754Z | s3_bucket    | creating bucket
2023-05-07T01:30:32.687Z | s3_bucket    | bucket created

Web API

// GET /api/v1/demo/env_deploy/progress
{
    "item_id": "s3_bucket",
    "progress_update": { "Delta": "Tick" },
    "msg_update": { "Set": "creating bucket" }
    // ..
}

Web page

Current
State

Deploy
(preview)

Clean
(preview)

▶

// outputs to CLI
let mut output = CliOutput::new();

// === Used in tests === //
// discards all output
let mut output = NoOpOutput;

// records which output functions were called
// and what parameters they were called with
let mut output = FnTrackerOutput::new();

// === Write your own === //
// writes JSON to HTTP response
let mut output = JsonResponseOutput::new();

Command Invocation

// examples/envman/src/cmds/profile_init_cmd.rs
// fn app_upload_flow_init
let cmd_ctx = CmdCtx::builder_single_profile_single_flow
    ::<EnvManError, _>(output, workspace)
    .with_profile_from_workspace_param(profile_key)
    .with_flow(flow)
    .with_item_params::<FileDownloadItem<WebApp>>(
        item_id!("app_download"),
        app_download_params_spec,
    )
    .with_item_params::<S3BucketItem<WebApp>>(
        item_id!("s3_bucket"),
        s3_bucket_params_spec,
    )
    .with_item_params::<S3ObjectItem<WebApp>>(
        item_id!("s3_object"),
        s3_object_params_spec,
    )
    .await?;

// examples/envman/src/cmds/env_status_cmd.rs
// envman status
StatesCurrentReadCmd::exec(&mut cmd_ctx).await?;

// examples/envman/src/cmds/env_deploy_cmd.rs
// envman deploy
EnsureCmd::exec(&mut cmd_ctx).await?;

Parameters Recall

./envman init demo --type development azriel91/web_app 0.1.1

# For good user experience, parameters
# should not have to be passed in on
# every command invocation

./envman status  # demo --type development azriel91/web_app 0.1.1
./envman deploy  # demo --type development azriel91/web_app 0.1.1

We need to store and load the parameters passed in previously.

First command invocation:

// examples/envman/src/cmds/profile_init_cmd.rs
// fn app_upload_flow_init
cmd_ctx_builder
    .with_profile_from_workspace_param(profile_key)
    .with_flow(flow)
    .with_item_params::<FileDownloadItem<WebApp>>(
        item_id!("app_download"),
        app_download_params_spec,
    )
    .with_item_params::<S3BucketItem<WebApp>>(
        item_id!("s3_bucket"),
        s3_bucket_params_spec,
    )
    .with_item_params::<S3ObjectItem<WebApp>>(
        item_id!("s3_object"),
        s3_object_params_spec,
    )
    .await?

Subsequent command invocations:

// examples/envman/src/cmds/app_upload_cmd.rs
cmd_ctx_builder
    .with_profile_from_workspace_param(&profile_key)
    .with_flow(&flow)
    // * file_download params spec not specified
    // * s3_bucket params spec not specified
    .with_item_params::<S3ObjectItem<WebApp>>(
        item_id!("s3_object"),
        s3_object_params_spec,
    )
    .await?

let s3_object_params_spec = S3ObjectParams::<WebApp>::field_wise_spec()
    // Note:
    //
    // * file_path not specified
    // * object key not specified
    // * Function logic cannot be deserialized,
    //   so needs to be provided
    .with_bucket_name_from_map(|s3_bucket_state: &S3BucketState| {
        match s3_bucket_state {
            S3BucketState::None => None,
            S3BucketState::Some {
                name,
                creation_date: _,
            } => Some(name.clone()),
        }
    })
    .build();

Workspace

When a command is run, Peace stores information about that execution.

It stores this in a .peace directory within a workspace directory. The automation tool developer needs to tell Peace how to determine the workspace directory.

let cmd_ctx = CmdCtx::builder_single_profile_single_flow
    ::<EnvManError, _>(output, workspace)
    // ..
    .await?;

pub enum WorkspaceSpec {
    /// Use the exe working directory as the workspace directory.
    ///
    /// The working directory is the directory that the user ran the program in.
    WorkingDir,
    /// Use a specified path.
    Path(PathBuf),
    /// Traverse up from the working directory until the given file is found.
    ///
    /// The workspace directory is the parent directory that contains a file or
    /// directory with the provided name.
    FirstDirWithFile(OsString),
}

Profiles

Profiles are a way of logically separating environments, aka namespacing.

First execution / init:

let cmd_ctx_builder = CmdCtx::builder_single_profile_no_flow
    ::<EnvManError, _>(output, &workspace)
    .with_profile(profile!("demo"))

    // for recall
    .with_workspace_param_value(
        WorkspaceParamsKey::Profile,
        Some(profile!("demo")),
    );

Subsequent executions:

let cmd_ctx_builder = CmdCtx::builder_single_profile_single_flow
    ::<EnvManError, _>(output, workspace)
    .with_profile_from_workspace_param(WorkspaceParamsKey::Profile)
    .with_flow(flow);

Summary

How do we communicate clearly in code?

Consistency between the conceptual model and the written code.
Make use of type safety with strong types.
When we require something of the developer, make it easy to be provided.
Prefer errors at compile time than runtime.
Error messages should show what is wrong and how to fix it.

Q & A

Interruptible Software

▶️

⏸️

⏹️

Processes

Build Process

Deployment Process

Execution

Manual

Automatic

Bus Analogy

🚏
🚌🚌

Designing Interruptibility

🛑 Stop Button

🚏 Stopping Point

🔀 Parallelism

📃 Reporting

🗺️ Strategy

🛑 Stop Button

Command Line

Web

Desktop

        > deploy_
    

🗄️🗄️🌐

x

go

❌ Don't

let outcome = tokio::select! {
    value = process.next()          => Outcome::Done(value),
        _ = tokio::signal::ctrl_c() => Outcome::Interrupted,
};

✅ Do

struct InterruptSignal;
let (interrupt_tx, interrupt_rx) = tokio::sync::mpsc::<InterruptSignal>(8);

let outcome = tokio::select! {
    value = process.next()      => Outcome::Done(value),
        _ = interrupt_rx.recv() => Outcome::Interrupted,
};

and:

// CLI
tokio::spawn(async move {
    tokio::signal::ctrl_c().await.unwrap();
    let _ = interrupt_tx.send(InterruptSignal).await;
});

// Web / Desktop
async fn handle_request(request: Request) -> Response {
    match request {
        Request::Interrupt { execution_id } => {
            let executions = executions.lock().await;
            let execution = executions.get(execution_id);
            let interrupt_tx = execution.interrupt_tx();
            let _ = interrupt_tx.send(InterruptSignal).await;
        }
        // ..
    }
}

🚏 Stopping Point

🛑 Interrupt

🚏 Stop

No Interruptibility

fn execute(
    params: Params,
) -> Outcome {
    let output_1 = step_1(params);
    let output_2 = step_2(output_1);
    let outcome = step_3(output_2);

    return outcome;
}

Basic

fn execute(
    interrupt_rx: mut Receiver<InterruptSignal>,
    params: Params,
) -> ControlFlow<InterruptSignal, Outcome> {
    let () = interruptibility_check(&mut interrupt_rx)?;
    let output_1 = step_1(params);

    let () = interruptibility_check(&mut interrupt_rx)?;
    let output_2 = step_2(output_1);

    let () = interruptibility_check(&mut interrupt_rx)?;
    let outcome = step_3(output_2);

    ControlFlow::Continue(outcome)
}

fn interruptibility_check(receiver: &mut Receiver<InterruptSignal>)
-> ControlFlow<InterruptSignal, ()> {
    if let Ok(interrupt_signal) = interrupt_rx.try_recv() {
        ControlFlow::Continue(())
    } else {
        ControlFlow::Break(interrupt_signal)
    }
}

Fine Grained Interruptibility

fn execute(
    interrupt_rx: mut Receiver<InterruptSignal>,
    params: Params,
) -> ControlFlow<InterruptSignal, Outcome> {
    let () = interruptibility_check(&mut interrupt_rx)?;
    let output_1 = step_1(&mut interrupt_rx, params);

    // ..
}

fn step_1(
    interrupt_rx: mut Receiver<InterruptSignal>,
    params: Params,
) -> ControlFlow<Output1, Outcome> {
    let mut output_1 = Output1::new();
    for i in 0..1_000_000 {
        if i % 1_000 == 0 {
            let () = interruptibility_check(&mut interrupt_rx)?;
        }
        do_something(output_1, params, i);
    }

    ControlFlow::Continue(output_1)
}

🔀 Parallelism

Parallelism and concurrency

🛑 Interrupt

🚏 Stop

Safe Interruption Rules

🔵 Finish everything in progress.
⚫ Don't start anything new.

See fn_graph on Github.

Queuer:
- Sends IDs of steps that can be executed.
- Receives IDs of steps that are complete.
- Checks for interruption.
Scheduler
- Receives IDs of steps that can be executed.
- Sends IDs of steps that are complete.

📃 Reporting

Non-Interruptible Software

fn execute(params: Params) -> Value {
    // ..
}

Interruptible Software

fn execute(params: Params) -> Outcome {
    // ..
}

Outcome

Basic

enum Outcome {
    /// Execution completed, here is the return value.
    Complete(Value),
    /// Execution was interrupted.
    Interrupted,
}

Step Values

enum Outcome {
    /// Execution completed, here is the return value.
    Complete(Value),
    /// Execution was interrupted.
    ///
    /// Here's the information we collected so far.
    Interrupted {
        step_1_value: Option<Step1Value>,
        step_2_value: Option<Step2Value>,
        step_3_value: Option<Step3Value>,
    },
}

Step Values and Execution Info

enum Outcome {
    /// Execution completed, here is the return value.
    Complete(Value),
    /// Execution was interrupted.
    ///
    /// Here's the information we collected so far.
    Interrupted {
        step_1_value: Option<Step1Value>,
        step_2_value: Option<Step2Value>,
        step_3_value: Option<Step3Value>,
        steps_processed: Vec<StepId>,
        steps_not_processed: Vec<StepId>,
    },
}

🗺️ Strategy

🛑 Interrupt

🚏 Stop

/// How to poll an underlying stream when an interruption is received.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum InterruptStrategy {
    /// On interrupt, keep going.
    IgnoreInterruptions,
    /// On interrupt, wait for the current future's to complete and yield its
    /// output, but do not poll the underlying stream for any more futures.
    FinishCurrent,
    /// On interrupt, continue polling the stream for the next `n` futures.
    ///
    /// `n` is an upper bound, so fewer than `n` futures may be yielded if the
    /// underlying stream ends early.
    PollNextN(u64),
}

Interruptible Implementation

No Interruptibility

fn execute(
    params: Params,
) -> Outcome {
    let output_1 = step_1(params);
    let output_2 = step_2(output_1);
    let outcome = step_3(output_2);

    return outcome;
}

Basic

fn execute(
    interrupt_rx: mut Receiver<InterruptSignal>,
    params: Params,
) -> ControlFlow<InterruptSignal, Outcome> {
    let () = interruptibility_check(&mut interrupt_rx)?;
    let output_1 = step_1(params);

    let () = interruptibility_check(&mut interrupt_rx)?;
    let output_2 = step_2(output_1);

    let () = interruptibility_check(&mut interrupt_rx)?;
    let outcome = step_3(output_2);

    ControlFlow::Continue(outcome)
}

fn interruptibility_check(receiver: &mut Receiver<InterruptSignal>)
-> ControlFlow<InterruptSignal, ()> {
    if let Ok(interrupt_signal) = interrupt_rx.try_recv() {
        ControlFlow::Continue(())
    } else {
        ControlFlow::Break(interrupt_signal)
    }
}

"Better"

fn execute<T>(
    interrupt_rx: mut Receiver<InterruptSignal>,
    params: Params,
) -> Result<T, InterruptSignal> {
    [step_1, step_2, step_3]
        .into_iter()
        .try_fold(params, |(mut last_param, step)| {
            interruptibility_check(&mut interrupt_rx)?;
            step(last_param)
        })
}

Real Code

Playground

use std::any::Any;
use std::iter::IntoIterator;
use std::marker::PhantomData;

fn main() {
    match execute::<String, _>(Receiver::new(3), true) {
        Ok(value) => println!("main: {}", value),
        Err(InterruptSignal) => println!("main: interrupted!"),
    }
}

fn execute<T, I>(
    mut interrupt_rx: Receiver<InterruptSignal>,
    params: I,
) -> Result<T, InterruptSignal>
where
    T: 'static,
    I: 'static,
{
    [step(step_1), step(step_2), step(step_3)]
        .into_iter()
        .try_fold(Box::new(params) as Box<dyn Any>, |last_param, step| {
            interruptibility_check(&mut interrupt_rx)?;
            Ok(step.exec(last_param))
        })
        .map(|boxed_any| *boxed_any.downcast().unwrap())
}

fn step_1(takes_bool: bool) -> u32 {
    let number = 123;
    println!("step_1 ({takes_bool}: bool) -> {number}");
    number
}

fn step_2(takes_u32: u32) -> f64 {
    let float = 1.5;
    println!("step_2 ({takes_u32}: u32) -> {float}");
    float
}

fn step_3(takes_f64: f64) -> String {
    let string = String::from("magic");
    println!("step_3 ({takes_f64}: f64) -> {string}");
    string
}

fn step<F, I, O>(f: F) -> Box<dyn StepErased>
where
    F: Fn(I) -> O + 'static,
    I: 'static,
    O: 'static,
{
    Box::new(StepWrapper::new(f))
}

trait Step {
    type Input: Any;
    type Output: Any;
    fn exec(&self, input: Self::Input) -> Self::Output;
}

impl<F, I, O> Step for StepWrapper<F, I, O>
where
    F: Fn(I) -> O,
    I: 'static,
    O: 'static,
{
    type Input = I;
    type Output = O;

    fn exec(&self, input: Self::Input) -> Self::Output {
        (self.0)(input)
    }
}

trait StepErased {
    fn exec(&self, input: Box<dyn Any>) -> Box<dyn Any>;
}

struct StepWrapper<F, I, O>(F, PhantomData<(I, O)>);
impl<F, I, O> StepWrapper<F, I, O> {
    fn new(f: F) -> Self {
        Self(f, PhantomData)
    }
}

impl<F, I, O> StepErased for StepWrapper<F, I, O>
where
    StepWrapper<F, I, O>: Step<Input = I, Output = O>,
    I: 'static,
    O: 'static,
{
    fn exec(&self, input: Box<dyn Any>) -> Box<dyn Any> {
        let input = *input.downcast::<I>().unwrap();
        let output = Step::exec(self, input);
        Box::new(output)
    }
}

struct Receiver<T>(u32, PhantomData<T>);
impl<T> Receiver<T> {
    pub fn new(n: u32) -> Self {
        Self(n, PhantomData)
    }
}
#[derive(Debug)]
struct InterruptSignal;
fn interruptibility_check(rx: &mut Receiver<InterruptSignal>) -> Result<(), InterruptSignal> {
    if (rx.0) == 0 {
        Err(InterruptSignal)
    } else {
        rx.0 -= 1;
        Result::Ok(())
    }
}

Resumability

🟢 Start 1

🚏 Stop

🟢 Start 2

Designing Resumability

🕸️ Executor Resumability

💮 Stepwise Resumability

🕸️ Executor Resumability

 fn execute<T>(
    execution_state: ExecutionState, // Add this
    interrupt_rx: mut Receiver<InterruptSignal>,
//    params: Params,                // Remove this
) -> Result<T, InterruptSignal> {
    let mut steps = [step_1, step_2, step_3];

    let (steps, params) = match &execution_state {
        ExecutionState::CleanSlate { params } => (steps, params),
        ExecutionState::PreviousState { step_ids_not_done, step_values, .. } => {
            steps.retain(|step| step_ids_not_done.contains(step.id()));
            let params = step_values.last().unwrap();

            (steps, params)
        }
    };

    steps
        .into_iter()
        .try_fold(params, |(mut last_param, step)| {
            interruptibility_check(&mut interrupt_rx)?;
            step(last_param)
        })
}

#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)]
pub enum ExecutionState {
    CleanSlate {
        params: Params,
    },
    PreviousState {
        step_ids_done: Vec<StepId>,
        step_ids_not_done: Vec<StepId>,
        step_values: Vec<Value>,
    },
}

Pros / Cons

Simpler API: Step logic does not need to know about interruptibility concepts.
Stale Reasoning: Saved state may be stale, and the executor does not have the means to check for staleness.

💮 Stepwise Resumability

 fn execute<T>(
    execution_state: ExecutionState, // Add this
    interrupt_rx: mut Receiver<InterruptSignal>,
) -> Result<T, InterruptSignal> {
    [step_1, step_2, step_3]
        .into_iter()
        .try_fold(params, |(mut last_param, step)| {
            interruptibility_check(&mut interrupt_rx)?;

            let step_state = execution_state.get(step.id());
            step(step_state, last_param)
        })
}

#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)]
pub struct ExecutionState { .. }

impl ExecutionState {
    pub fn get<T>(step_id: StepId) -> T { .. }
}

Pros / Cons

Heavier API: Step logic needs to cater for initial state.
Usage Safety: Step can determine if the saved initial state is in-sync with actual state, and warn user.

Resumable Implementation

Playground

use std::any::Any;
use std::fmt::Debug;
use std::iter::IntoIterator;
use std::marker::PhantomData;

fn main() {
    let _execution_state = ExecutionState::clean_slate(true);
    let execution_state = ExecutionState::PreviousState {
        step_ids_done: vec![1, 2],
        step_ids_not_done: vec![3],
        step_values: vec![Box::new(234u32), Box::new(3.5f64)],
    };

    match execute::<String>(execution_state, Receiver::new(3)) {
        Ok(value) => println!("main: {}", value),
        Err(InterruptSignal) => println!("main: interrupted!"),
    }
}

fn execute<T>(
    execution_state: ExecutionState,
    mut interrupt_rx: Receiver<InterruptSignal>,
) -> Result<T, InterruptSignal>
where
    T: 'static,
{
    let steps = vec![step(1, step_1), step(2, step_2), step(3, step_3)];
    let (steps_to_execute, params) = filter_what_to_do(execution_state, steps);

    steps_to_execute
        .into_iter()
        .try_fold(params, |last_param, step| {
            interruptibility_check(&mut interrupt_rx)?;
            Ok(step.exec(last_param))
        })
        .map(|boxed_any| *boxed_any.downcast().unwrap())
}

fn filter_what_to_do(
    execution_state: ExecutionState,
    mut steps: Vec<Box<dyn StepErased>>,
) -> (Vec<Box<dyn StepErased>>, Box<dyn Any>) {
    match execution_state {
        ExecutionState::CleanSlate { params } => (steps, params),
        ExecutionState::PreviousState {
            step_ids_not_done,
            mut step_values,
            ..
        } => {
            steps.retain(|step| step_ids_not_done.contains(&step.id()));
            let params = step_values.pop().unwrap();

            (steps, params)
        }
    }
}

fn step_1(takes_bool: bool) -> u32 {
    let number = 123;
    println!("step_1 ({takes_bool}: bool) -> {number}");
    number
}

fn step_2(takes_u32: u32) -> f64 {
    let float = 1.5;
    println!("step_2 ({takes_u32}: u32) -> {float}");
    float
}

fn step_3(takes_f64: f64) -> String {
    let string = String::from("magic");
    println!("step_3 ({takes_f64}: f64) -> {string}");
    string
}

fn step<F, I, O>(id: u32, f: F) -> Box<dyn StepErased>
where
    F: Fn(I) -> O + 'static,
    I: 'static,
    O: 'static,
{
    Box::new(StepWrapper::new(id, f))
}

trait Step {
    type Input: Any;
    type Output: Any;
    fn id(&self) -> u32;
    fn exec(&self, input: Self::Input) -> Self::Output;
}

impl<F, I, O> Step for StepWrapper<F, I, O>
where
    F: Fn(I) -> O,
    I: 'static,
    O: 'static,
{
    type Input = I;
    type Output = O;

    fn id(&self) -> u32 {
        self.0
    }

    fn exec(&self, input: Self::Input) -> Self::Output {
        (self.1)(input)
    }
}

trait StepErased {
    fn id(&self) -> u32;
    fn exec(&self, input: Box<dyn Any>) -> Box<dyn Any>;
}

struct StepWrapper<F, I, O>(u32, F, PhantomData<(I, O)>);
impl<F, I, O> StepWrapper<F, I, O> {
    fn new(id: u32, f: F) -> Self {
        Self(id, f, PhantomData)
    }
}

impl<F, I, O> StepErased for StepWrapper<F, I, O>
where
    StepWrapper<F, I, O>: Step<Input = I, Output = O>,
    I: 'static,
    O: 'static,
{
    fn id(&self) -> u32 {
        self.0
    }

    fn exec(&self, input: Box<dyn Any>) -> Box<dyn Any> {
        let input = *input.downcast::<I>().unwrap();
        let output = Step::exec(self, input);
        Box::new(output)
    }
}

struct Receiver<T>(u32, PhantomData<T>);
impl<T> Receiver<T> {
    pub fn new(n: u32) -> Self {
        Self(n, PhantomData)
    }
}
#[derive(Debug)]
struct InterruptSignal;
fn interruptibility_check(rx: &mut Receiver<InterruptSignal>) -> Result<(), InterruptSignal> {
    if (rx.0) == 0 {
        Err(InterruptSignal)
    } else {
        rx.0 -= 1;
        Result::Ok(())
    }
}

#[allow(dead_code)]
#[derive(Debug)]
enum ExecutionState {
    CleanSlate {
        params: Box<dyn Any>,
    },
    PreviousState {
        step_ids_done: Vec<u32>,
        step_ids_not_done: Vec<u32>,
        step_values: Vec<Box<dyn Any>>,
    },
}

#[allow(dead_code)]
impl ExecutionState {
    fn clean_slate<I>(i: I) -> Self
    where
        I: 'static,
    {
        ExecutionState::CleanSlate {
            params: Box::new(i),
        }
    }
}

Real Crates

📦🦀

🗂️ interruptible: Stops a stream from producing values when interrupted.
🧬 fn_graph: Dynamically managed function graph execution. Integrates with interruptible.
🕊️ peace: Framework to build empathetic and forgiving software automation.

Summary

Q & A

🦂🦞🦀⁉️

Ideas

This page records ideas that I'd like, but there isn't enough mental capacity and time to design and work on them yet.

1. Abstraction over native storage and web storage – use IndexedDB instead of `WebStorage` APIs.

2. Graphical user interface that renders each flow's graph.

Each item is a node.
User can select which nodes to run – these may be a subset of the flow.
User can select beginning and ending nodes – and these can be in reverse order.

Note: Graphviz is compiled to WASM and published by hpcc-systems/hpcc-js-wasm. May be able to use that to render.

graphviz-visual-editor is a library that allows basic editing of a graphviz graph. It's not yet developed to a point that is intuitive for users.

3. Tool that uses `peace` to check consumer code whether it adheres to best practices.

4. Clean Command Retains History

End users may want to see what was previously deployed.

If we retain a ${profile}/.history directory with all previous execution information, it allows:

Re-attempting clean up.
Reporting on what was cleaned up.
Computing costs of all executions

Perhaps we should make the API be, on visit, return a list of identifiers for things to clean up.

5. Types / proc macros to place constraints at compile time for aesthetic reports.

short summary sentences
2 ~ 3 sentence paragraphs / word limit

#![allow(unused)]
fn main() {
/// An ID
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Id<'s>(Cow<'s, str>);

/// Single line description, hard limit of 200 characters.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct DescShort<'s>(Cow<'s, str>);
}

6. MultiProfile Cmd scope profile sort function

Users may want to sort profiles in the profile directory differently to their alphabetical / lexicographical sorting.

This may be dependent on profile params – sort env based on env type, last execution time – profile history.

7. Cancel-safe state storage

When an item ensure does multiple writes, there is a possibility of not all of those writes occur during execution:

user interrupts the execution.
internet connection drops.
machine loses power.

In the last case, we cannot safely write state to disk, so a StateCurrent discover is needed to bring StatesCurrentStored up to date. However, the previous two cases, it is possible for Items to return State that has been partially ensured, without making any further outgoing calls -- i.e. infer StatesEnsured based on the successful writes so far.

Note that this places a burden on the Item implementor to return the partial state ensured (which may conflict with keeping the State simple), as well as make the ApplyFns::exec return value more complex.

The trade off may not be worthwhile.

8. Adding / removing / modifying items in flows

Implementors may add/remove/modify items in flows.

Peace needs to be designed such that these changes do not cause already-existent flows to not be loadable, i.e. when:

states_*.yaml contains state for which an item no longer exists in the flow.
states_*.yaml does not contain state for an item that is newly added to the flow.
states_*.yaml contains state whose fields are different to a new version of an item.

This one can be addressed by having State be an enum, with versioned variants.

Also see 14.

9. Item expiry

For items that cost, it is useful to have an expiry time that causes it to be deleted.

This would have to be supported by the service that hosts the item.
There should be a way to notify the user of items that are about to expire.
There should also be a way to extend the item expiry times easily.

10. Interrupt / cancel safety

The tokio-graceful-shutdown library can be used to introduce interrupt safety into item executions. This is particularly useful for write functions.

See the is_shutdown_requested method in particular.

11. Diffable item params

DiffCmd originally was written to diff the current and goal states. However, with the second use case of "diff states between two profiles", it is also apparent that other related functionality is useful:

Diff profile params / flow params.
Diff item params between profiles for a given flow.

Because of diffable params, and [#94], the Item should likely have:

type Params: ItemParams + Serialize + DeserializeOwned.
feature gated fn item_params_diff(..).

fn item_params_diff(..) should likely have a similar signature to fn state_diff(..), whereby if one uses XData<'_>, the other should as well for consistency:

For MultiProfileSingleFlow commands, a diff for item params which contains a referential value (e.g. "use the some_predecessor.ip_address()") may(?) need information about some_predecessor through Resources / Data.

We should work out the design of that before settling on what state_diff and item_params_diff's function parameters will be. See Design Thoughts on [#94] for how it may look like.

12. Default item params

An Item's params may not necessarily be mandatory. From the Params type (and corresponding trait), Peace may:

Insert default param values, if the Item implementor provides a default
Still make the params required if there is no default.
Provide a way for ParamsSpec for each field to be the default, or a mapping function.

13. Upgrades: Tolerate optional or different workspace params / item params / states

When new workspace params are added, or new items are added to a flow, existing *_params.yaml, item_params.yaml, and states_*.yaml may not contain values for those newly added params / items.

Automation software should be able to:

Work with missing parameters.
Work with changed parameter types.

When workspace params / items are removed from a flow, leftover params / state are no longer used. However, we may want to do one of:

Notify the user to clean up unused params
Peace should ignore it
Inform the automator to still register the item, so that old execution may be loaded.

14. Store params per execution, pass previous execution's params to clean cmd

Instead of requiring Item::State to store the params used when applied, maybe we should store the params used in the last ensure alongside that item's state.

Users are concerned with the current state of the item. They also may be concerned with the parameters used to produce that state. Requiring item implementors to store paths / IP addresses within the state that has been ensured feels like unnecessary duplication.

However, when comparing diffs, we would hope either:

The params used to discover the current and goal states are the same, or
The "params and states" pairs are both compared.

Also:

apply_check needs to have both the old and new params to determine whether apply needs to be executed.
State as the output API, should not necessarily include params.
When parameters change, and an apply is interrupted, then we may have earlier items using the new parameters, and later items still on the previous parameters. More complicated still, is if parameters change in the middle of an interruption, and re-applied.

Perhaps there should be a (dest_parameters, Item::State) current state, and a (src_parameters, Item::State) goal state. That makes sense for file downloads if we care about cleaning up the previous dest_path, to move a file to the new dest_path.

Or, all dest parameters should be in Item::State, because that's what's needed to know if something needs to change.

Another thought:

states_*.yaml should store this per item:

params used for apply
values resolved for those params
current state

Also see 8.

15. Use openlayers for tiling level of detail

Generate dot diagram using graphviz with full resolution, and then convert to tiles, then display different styling depending on the state of each item.

16. Combine data and params{,_partial} into FnCtx

Item functions take in FnCtx, data, and item params as separate arguments.

This was done to:

Reduce the additional layer to get Item::Params, or Item::ParamsPartial.
Avoid progress sender from being passed in to function that didn't need it.

However, functions don't necessarily need runtime fn_ctx or data, making it noise in the signature.

Should we combine all 3 into FnCtx? It would make FnCtx type parameterized over Params and ParamsPartial.

Notes

1. SSH-like on Windows

psexec
Windows powershell and WinRM

2. Learnings from envman end-to-end implementation.

Referential lookup of values in state / item params. ([#94])
AWS SDK is not WASM ready -- includes mio unconditionally through tokio (calls UDP). (aws-sdk-rust#59)
AWS SDK does not always include error detail -- S3 head_object. (aws-sdk-rust#227)
Progress output should enable-able for state current / goal discover / clean functions.
Flow params are annoying to register every time we add another item. Maybe split end user provided params from item params.
Blank item needs a lot of rework to be easier to implement an item. ([67], [#96])
For ApplyCmd, collect StateCurrent, StateGoal, StateDiff in execution report.
AWS errors' code and message should be shown to the user.
Progress limit should not be returned in ApplyFns::check, but sent through progress_sender.limit(ProgressLimit). This simplifies check, and allows state current/goal discovery to set the limits easily.
Consolidate StatesDiscoverCmd and ApplyCmd, so the outcome of a command is generic. Maybe use a trait and structs, instead of enum variants and hardcoded inlined functions, so that it is extendable.
Add an ListKeysAligned presentable type so Presenters can align keys of a list dynamically.
Remove the peace_cfg::State type.
Contextual presentable strings, for states and diffs.

What command is this called for:
- state current: "is .."
- goal state: "should be .."
- diff between current and goal: "will change from .. to .."
- diff between current and cleaned: "will change from .. to .."
- diff between two profiles' current states: : "left is .., right is .." Maybe we don't burden the presenter implementation, but Peace will insert the contextual words
Easy API functions for diffing -- current vs goal, between profiles' current states.
What about diffing states of different state versions?

Maybe this is already taken care of -- state_diff is already passed in both States, so implementors had to manage it already.

fd -Ftd 'app_cycle' -x bash -c 'mv $0 ${0/app_cycle/envman}' {}
fd -Ftf 'app_cycle' -x bash -c 'mv $0 ${0/app_cycle/envman}' {}
sd -s 'app_cycle' 'envman' $(fd -tf)
sd -s 'app cycle' 'envman' $(fd -tf)
sd -s 'App Cycle' 'Env Man' $(fd -tf)
sd -s 'AppCycle' 'EnvMan' $(fd -tf)
cargo fmt --all

Feature Gated Incremental Functionality

Tried the following parameter type for Item::apply:

use std::marker::PhantomData;

/// Parameters to `Item::apply`.
#[derive(Debug)]
pub struct EnsureExecParams<
    'params,
    Data,
    #[cfg(feature = "state_current")] StateCurrent,
    #[cfg(feature = "state_goal")] StateGoal,
    #[cfg(feature = "state_diff")] StateDiff,
> {
    /// Data accessed by the apply fns.
    pub data: Data,
    /// Current state of the item.
    #[cfg(feature = "state_current")]
    pub state_current: &'params StateCurrent,
    /// Goal state of the item.
    #[cfg(feature = "state_goal")]
    pub state_goal: &'params StateGoal,
    /// Diff between current and goal states.
    #[cfg(feature = "state_diff")]
    pub diff: &'params StateDiff,
    /// Marker.
    marker: PhantomData<&'params ()>,
}

But the following produces a compile error when used:

async fn exec(
    ensure_exec_params: EnsureExecParams<
        '_,
        Self::Data<'_>,
        #[cfg(feature = "state_current")]
        Self::State,
        #[cfg(feature = "state_diff")]
        Self::StateDiff,
    >,
) -> Result<Self::StatePhysical, Self::Error>;

The #[cfg(..)] attributes are not supposed in function parameter type parameters: See the Attributes Galore RFC and rfc#2602.

Perhaps it is possible to define the type separately, but we probably need to define this in a separate trait:

#[cfg(all(
    not(feature = "state_current"),
    not(feature = "state_goal"),
    not(feature = "state_diff"),
))]
pub type EnsureExecParams<'params> = EnsureExecParams<
    'params,
    Self::Data<'params>,
>

#[cfg(all(
    feature = "state_current",
    not(feature = "state_goal"),
    not(feature = "state_diff"),
))]
pub type EnsureExecParams<'params> = EnsureExecParams<
    'params,
    Self::Data<'params>,
    Self::State,
>

#[cfg(all(
    feature = "state_current",
    feature = "state_goal",
    not(feature = "state_diff"),
))]
pub type EnsureExecParams<'params> = EnsureExecParams<
    'params,
    Self::Data<'params>,
    Self::State,
>

#[cfg(all(
    feature = "state_current",
    feature = "state_goal",
    feature = "state_diff",
))]
pub type EnsureExecParams<'params> = EnsureExecParams<
    'params,
    Self::Data<'params>,
    Self::State,
    Self::StateDiff,
>

Third Party Licenses

This page lists the licenses of the projects used in Peace.

    <h2>Overview of licenses:</h2>
    <ul class="licenses-overview">
        <li><a href="#Apache-2.0">Apache License 2.0</a> (368)</li>
        <li><a href="#MIT">MIT License</a> (99)</li>
        <li><a href="#BSD-3-Clause">BSD 3-Clause &quot;New&quot; or &quot;Revised&quot; License</a> (5)</li>
        <li><a href="#ISC">ISC License</a> (4)</li>
        <li><a href="#Zlib">zlib License</a> (3)</li>
        <li><a href="#BSL-1.0">Boost Software License 1.0</a> (1)</li>
        <li><a href="#NOASSERTION">NOASSERTION</a> (1)</li>
        <li><a href="#OpenSSL">OpenSSL License</a> (1)</li>
        <li><a href="#Unicode-DFS-2016">Unicode License Agreement - Data Files and Software (2016)</a> (1)</li>
    </ul>

    <h2>All license text:</h2>
    <ul class="licenses-list">
        <li class="license">
            <h3 id="Apache-2.0">Apache License 2.0</h3>
            <h4>Used by:</h4>
            <ul class="license-used-by">
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-config 1.1.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-credential-types 1.1.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-http 0.60.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-runtime 1.1.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-smithy-async 1.1.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-smithy-checksums 0.60.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-smithy-eventstream 0.60.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-smithy-http 0.60.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-smithy-json 0.60.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-smithy-query 0.60.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-smithy-runtime 1.1.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-smithy-runtime-api 1.1.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-smithy-types 1.1.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-smithy-xml 0.60.1</a></li>
                <li><a href=" https://github.com/smithy-lang/smithy-rs ">aws-types 1.1.1</a></li>
            </ul>
            <pre class="license-text">
                             Apache License
                       Version 2.0, January 2004
                    http://www.apache.org/licenses/

TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

Definitions.

"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.

"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.

"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.

"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.

"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.

"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.

"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).

"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.

"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."

"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:

(a) You must give any other recipients of the Work or Derivative Works a copy of this License; and

(b) You must cause any modified files to carry prominent notices stating that You changed the files; and

(c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and

(d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.

You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
Apache License 2.0

Used by:
```
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
```

TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

Definitions.

"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.

"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.

"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.

"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.

"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.

"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.

"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).

"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.

"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."

"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:

(a) You must give any other recipients of the Work or Derivative Works a copy of this License; and

(b) You must cause any modified files to carry prominent notices stating that You changed the files; and

(c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and

(d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.

You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.

END OF TERMS AND CONDITIONS

The Peace Framework