Introduction

This book describes the purpose of the Peace automation framework, and concepts used in its design.

How To Read This Book

Depending on your frame of mind:

• What – Show me the framework in action:

  Have a play with the live Examples.

• Why – Show me the rationale guiding the decisions:

  Head on through to the Background section.

• How – Show me how it works:

  Jump to Technical Concepts.

  This is essential reading for framework consumers.

Background

Automation, like most subjects, has many questions to ask and answer. The following is a selection of issues that are common in this space:

• Before running the automation:
  • What is the current state?
  • What will the automation do?
• When it is running:
  • Where is it up to?
  • When will things be complete?
• When it errs:
  • What went wrong?
  • Must I begin from scratch again?
  • How can I solve this myself?
• When it completes successfully:
  • Can I see a report of what's changed?
  • Can it send a notification?

A lot of the responsibility and weight of these questions is placed on automation teams. Issues that originate from many separate systems surface through the automation, and are often reported to the automation team, potentially resulting in burnout.

For users, it is frustrating when it is difficult to understand what the automation is doing, where one is in the process, and how long it takes. This is amplified when the process errs after a lot of waiting – if it ever ends. Can one recover from errors, or end up feeling lost?

Amidst the chaos, pressure, and expectation, have a little.. peace ✌️.

Purpose

The purpose of the Peace framework is to bring people joy and peace of mind in automation development and usage. This is an active effort to take away mental and emotional stress and frustration experienced with software.

For maintainers, Peace aims to guide developers to build and support resilient automation.

For users, Peace aims to make automation work surprisingly well, and reduce the feeling of being lost or frustrated when things don't go according to plan.

Strategy

Outcomes

The main priority is to take care of people. This is correlated to the following measurable dimensions:

• Time optimization
• Effort optimization
• Cost optimization
• Decision making

If these outputs trend in a favourable direction, then surveying people's well-being and satisfaction should show similar outcomes.

Input Dimensions

The following input dimensions are considered when designing the Peace framework:

• Approach: Choose an optimal method.
• Correctness: Take the right action, in success and in err.
• Visibility: Show the user what will happen, is happening, and has happened.
• Understandability: Show what is important. Don't overwhelm the user.
• Recoverability: Continue where the work was stopped.
• Performance: Use resources efficiently to execute the approach.
• Psychology: Design for how people behave.

By keeping these dimensions in mind:

• For development, the framework gives people confidence that what they build is of quality, by guiding them to consider and handle cases commonly experienced by users. Through feature gates, developers are protected from mental overload by allowing parts of the automation to be developed incrementally, without risk of diverging from the ideal.

• For users, the framework provides tools to observe the state and have predictable outcomes, so that they can be confident in making decisions. It also lowers the chances and cost of failure. This reduces the fear of making mistakes, as the effort of recovery is also lowered.

• From a business perspective, increased morale, increased performance, and decreased wastage are all benefits that allow results to be achieved faster and in a reliable manner.

Examples

Each page in this section demonstrates the examples compiled as WASM applications.

If you are running this book locally, you may compile all examples before visiting the pages using the following command:

for example in $(ls examples)
do wasm-pack build \
  --target web \
  --out-dir "../../doc/src/examples/pkg" \
  --release \
  "examples/${example}"
done

Download

This example manages a file download into the browser's memory.

Env Man

🚧 This example is a work in progress

Peace framework web application lifecycle example

This example demonstrates management of a web application's lifecycle. This includes:

1. Building the application.
2. Starting / stopping the application in development.
3. Deploying / upgrading / removing the application in test servers.
4. Configuration management of the application.
5. Deploying / upgrading / removing the application in live servers.
6. Diffing the application and configuration across environments.
7. Creating a replica environment from an existing environment.

Walkthrough

Technical Concepts

The following sections cover essential concepts to develop automation using the framework.

1. Resources: An any-map, whose values' read and write access are checked at runtime instead of compile time.
2. Function graph: Functions with logical and data-access-dependent concurrency.
3. Item: Specification that defines information and logic to manage an arbitrary item.
4. Item graph: Like a function graph, where each node is not just a function, but an item.

Resources

In Peace, Resources refers to an any-map – a map that can store one value of different types – whose borrowing rules are checked per entry at runtime, instead of compile time.

Example of an any-map:

use peace::resource_rt::Resources;

let mut resources = Resources::new();
resources.insert(1u32);
resources.insert(2u64);

// Change `resources` to be immutable.
let resources = resources;

// We can validly have two mutable borrows
// from the map!
let mut a = resources.borrow_mut::<u32>();
let mut b = resources.borrow_mut::<u64>();
*a = 2;
*b = 3;

// Accessing the same value while it is already
// mutably borrowed returns an error.
assert!(resources.try_borrow::<u32>().is_err())

For more information about the underlying type, see resman.

Java Equivalent

The conceptual Java equivalent looks like this:

var resources = new MagicMap(); // new HashMap<Class<?>, Object>();
resources.insert((Integer) 1);  // innerMap.put(Integer.class, 1);
resources.insert((Long)    2);  // innerMap.put(Long.class,    2L);

var anInt = (Integer) resources.get(Integer.class);
var aLong = (Long)    resources.get(Long.class);

// Can mutate both.
anInt = 2;
aLong = 3L;

Rust's does not allow access to multiple mutable entries at the same time with the built in HashMap, so Resources is an implementation to bypass the compilation strictness.

Function Graph

A fn_graph is a data structure that holds functions, tracks their dependencies, and streams the functions in dependency order.

This is the basis of a framework that enables:

• Consumer provided logic and data.
• Framework controlled invocation of logic.

When consumers can plug in their logic and data to the framework, control now resides with the framework on invoking that logic and presenting the data in a goal form.

Scenario

Consider the following program:

1. Initialize two inputs, a and b, and two outputs c and d.
2. Add a to c.
3. Add b to c.
4. Add c to d.
5. Print the values of a, b, and c.

Static Representation

The above program can be represented by the following code:

#![allow(unused)]
fn main() {
// Define data to work on.
let mut a = -1;
let mut b = -1;
let mut c = -1;
let mut d = -1;

type Fn1 = fn(&mut i32, &mut i32, &mut i32, &mut i32);
type Fn2 = fn(&i32, &mut i32);
type Fn3 = fn(&i32, &mut i32);
type Fn4 = fn(&i32, &mut i32);
type Fn5 = fn(&i32, &i32, &i32);

// Define logic.
let fn1: Fn1 = |a, b, c, d| { *a = 1; *b = 2; *c = 0; *d = 0; };
let fn2: Fn2 = |a,    c   | *c += *a;
let fn3: Fn3 = |   b, c   | *c += *b;
let fn4: Fn4 = |      c, d| *d += *c;
let fn5: Fn5 = |a, b, c   | println!("{a} + {b} = {c}");

// Invoke logic over data.
fn1(&mut a, &mut b, &mut c, &mut d);
fn2(&a, &mut c);
fn3(&b, &mut c);
fn4(&c, &mut d);
fn5(&a, &b, &c);
}

Dynamic Representation

The following shows the complete logic implemented using fn_graph:

// [dependencies]
// fn_graph = { version = "0.5.4", features = ["fn_meta", "resman"] }
// futures = "0.3.21"
// resman = { version = "0.15.0", features = ["fn_meta", "fn_res"] }
// tokio = { version = "1.20.0", features = ["rt", "macros", "time"] }
use std::{
    fmt,
    ops::{AddAssign, Deref, DerefMut},
};

use fn_graph::FnGraphBuilder;
use futures::stream::StreamExt;
use resman::{IntoFnRes, Resources};

// Define newtypes for each parameter.
#[derive(Debug)] struct A(u32);
#[derive(Debug)] struct B(u32);
#[derive(Debug)] struct C(u32);
#[derive(Debug)] struct D(u32);

fn main() {
// Initialize data.
let mut resources = Resources::new();
resources.insert(A(0));
resources.insert(B(0));
resources.insert(C(0));
resources.insert(D(0));
let resources = &resources; // Now the map is compile time immutable.

type Fn1 = fn(&mut A, &mut B, &mut C, &mut D);
type Fn2 = fn(&A, &mut C);
type Fn3 = fn(&B, &mut C);
type Fn4 = fn(&C, &mut D);
type Fn5 = fn(&A, &B, &C);

// Define logic and insert them into graph structure.
let fn_graph = {
    let fn1: Fn1 = |a, b, c, d| { a.0 = 1; b.0 = 2; c.0 = 0; d.0 = 0; };
    let fn2: Fn2 = |a,    c   | *c += a.0;
    let fn3: Fn3 = |   b, c   | *c += b.0;
    let fn4: Fn4 = |      c, d| *d += c.0;
    let fn5: Fn5 = |a, b, c   | println!("{a} + {b} = {c}");

    let mut fn_graph_builder = FnGraphBuilder::new();

    // Store functions in graph.
    let [fn_id1, fn_id2, fn_id3, fn_id4, fn_id5] = fn_graph_builder.add_fns([
        fn1.into_fn_res(),
        fn2.into_fn_res(),
        fn3.into_fn_res(),
        fn4.into_fn_res(),
        fn5.into_fn_res(),
    ]);

    // Define dependencies to control ordering.
    fn_graph_builder
        .add_logic_edges([
            (fn_id1, fn_id2),
            (fn_id1, fn_id3),
            (fn_id2, fn_id4),
            (fn_id2, fn_id5),
            (fn_id3, fn_id4),
            (fn_id3, fn_id5),
        ])
        .unwrap();

    fn_graph_builder.build()
};

// Invoke logic over data.
//
// For synchronous sequential execution, you may uncomment the next line:
// fn_graph.iter().for_each(|fun| fun.call(resources));
//
// For asynchronous concurrent execution, use the following:
let rt = tokio::runtime::Builder::new_current_thread()
    .enable_time()
    .build()
    .unwrap();
rt.block_on(async move {
    fn_graph
        .stream()
        .for_each_concurrent(None, |fun| async move { fun.call(resources); })
        .await;
});

// prints:
// 1 + 2 = 3
}

macro_rules! u32_newtype {
    ($name:ident) => {
        impl fmt::Display for $name {
            fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
                self.0.fmt(f)
            }
        }
        impl Deref for $name {
            type Target = u32;
            fn deref(&self) -> &Self::Target {
                &self.0
            }
        }
        impl DerefMut for $name {
            fn deref_mut(&mut self) -> &mut Self::Target {
                &mut self.0
            }
        }
        impl AddAssign<u32> for $name {
            fn add_assign(&mut self, other: u32) {
                *self = Self(self.0 + other);
            }
        }
    };
}
u32_newtype!(A);
u32_newtype!(B);
u32_newtype!(C);
u32_newtype!(D);

Note that the knowledge of types is only needed at the point of definition for both data and logic. At the point of invocation, it's dynamic:

let resources: &Resources = /* .. */;

fn_graph
    .stream()
    .for_each_concurrent(None, |fun| async move { fun.call(resources); })
    .await;

How It Works

The remainder of this section shows how fn_graph is implemented.

Generic data storage and dynamic access is already solved by Resources from the previous page. The remaining parts to solve all relate to the logic:

• Abstraction: Abstraction over different logic types to retrieve parameters and invoke the logic.
• Ordering: Sort logic by their dependencies and how they access data.
• Streaming: Concurrently stream logic once they are available without violating data access rules.

Abstraction

To abstract over the logic, the following needs to be solved:

• Storage: Single type to be held together in a collection.
• Instantiation: Easy to instantiate from functions or closures.
• Invocation: Able to be invoked with Resources.

The concept applies to function with any number of parameters¹; the code snippets on this page shows implementations for a function with 2 parameters – one mutable parameter, and one immutable parameter.

Storage

We can store different types under a single abstraction by defining a trait, and implementing the trait for each of those different types.

// Trait for all logic types to implement.
pub trait FnRes {
    // ..
    /// Return type of the function.
    type Ret;
    /// Runs the function.
    fn call(&self, resources: &Resources) -> Self::Ret;
}

In order to name the logic type in code, the logic is stored in a wrapper type called FnResource.

FnRes is implemented for each type parameter combination of FnResource¹:

// Intermediate type so that traits can be implemented on this.
pub struct FnResource<Fun, Ret, Args> {
    pub func: Fun,
    marker: PhantomData<(Fun, Ret, Args)>,
}

// Implement `FnRes` for `FnResource` with different type parameters.
impl<Fun, Ret, C, A> FnRes for FnResource<F, Ret, (&mut C, &A)>
where
    Fun: Fn(&mut C, &A) -> Ret + 'static,
    Ret: 'static,
{
    // ..
}

Once a function or closure is wrapped in a FnResource, then we can hold different functions and closures as Box<dyn FnRes>.

Instantiation

To make it easy to transform functions and closures into Box<dyn FnRes>, the IntoFnResource and IntoFnRes traits are provided generically:

// Turns a function or closure into the `FnResource` wrapper type.
impl<Fun, Ret, C, A> IntoFnResource<Fun, Ret, (&mut C, &A)> for Fun
where
    Fun: Fn(&mut C, &A) -> Ret + 'static,
    Ret: 'static,
{
    fn into_fn_resource(self) -> FnResource<Fun, Ret, (&mut C, &A)> {
        FnResource {
            func: self,
            marker: PhantomData,
        }
    }
}

// Turns a function or closure into a `Box<dyn FnRes>`
impl<Fun, Ret, C, A> IntoFnRes<Fun, Ret, (&mut C, &A)> for Fun
where
    Fun: Fn(&mut C, &A) -> Ret + 'static,
    Ret: 'static,
    A: 'static,
    B: 'static,
    FnResource<Fun, Ret, (&mut C, &A)>: FnRes<Ret = Ret>,
{
    fn into_fn_res(self) -> Box<dyn FnRes<Ret = Ret>> {
        Box::new(self.into_fn_resource())
    }
}

Usage:

let fn_res = (|c: &mut u32, a: &u32| *c += *a).into_fn_res();

Invocation

Now that we can easily hold different types under the Box<dyn FnRes> abstraction, the remaining issue is to invoke the logic through that common abstraction.

To do this, FnRes has a method that takes a &Resources parameter, and each trait implementation will handle parameter fetching and function invocation:

// Trait for all logic types to implement.
pub trait FnRes {
    /// Return type of the function.
    type Ret;

    /// Runs the function.
    fn call(&self, resources: &Resources) -> Self::Ret;
}

// Implementation to fetch the parameters from `Resources`, and invoke the function.
impl<Fun, Ret, C, A> FnRes for FnResource<F, Ret, (&mut C, &A)>
where
    Fun: Fn(&mut C, &A) -> Ret + 'static,
    Ret: 'static,
{
    pub fn call(&self, resources: &Resources) -> Ret {
        let c = resources.borrow_mut::<C>();
        let a = resources.borrow::<A>();

        (self.func)(c, a)
    }
}

With this, consumers are able to invoke functions generically:

let mut resources = Resources::new();
resources.insert(A::new(1));
resources.insert(C::new(0));

let fn_res: Box<dyn FnRes> = (|c: &mut C, a: &A| *c += *a).into_fn_res();

// Generically invoke logic with generic data type.
fn_res.call(&resources);

Notably:

• Parameters must be &T or &mut T, as T is owned by Resources.
• Parameters must be distinct types, as it is invalid to borrow &mut T twice.
• Borrowing and invocation is implemented by the function type – FnResource in this example. Out of the box, this lives in the resman crate instead of fn_graph.

¹ resman implements FnRes for functions with up to 6 parameters.

Ordering

To define the order of execution, fn_graph uses two kinds of dependencies – logical and data. Logical dependencies are defined by consumers of the crate. Data dependencies are calculated by inspecting the type and mutability of function parameters.

Using the example:

type Fn1 = fn(&mut i32, &mut i32, &mut i32, &mut i32);
type Fn2 = fn(&i32, &mut i32);
type Fn3 = fn(&i32, &mut i32);
type Fn4 = fn(&i32, &mut i32);
type Fn5 = fn(&i32, &i32, &i32);

let fn1: Fn1 = |a, b, c, d| { *a = 1; *b = 2; *c = 0; *d = 0; };
let fn2: Fn2 = |a,    c   | *c += *a;
let fn3: Fn3 = |   b, c   | *c += *b;
let fn4: Fn4 = |      c, d| *d += *c;
let fn5: Fn5 = |a, b, c   | println!("{a} + {b} = {c}");

Even though it is possible to order these sequentially, we also want to be able to achieve concurrency to increase performance.

Logical Dependencies

To encode this into a function graph, logical dependencies can be specified through edges:

For correctness, fn1 must be executed before both fn2 and fn3, and those must complete before fn4 and fn5 are executed.

Data Dependencies

Note that fn4 and fn5 can both be executed in parallel, as there is no overlap in accessed data.

However, fn2 and fn3 both require mutable access to C. Since it is not possible to safely provide both fn2 and fn3 with mutable access, fn_graph automatically introduces a data dependency between those functions to ensure consistent ordering:

Streaming

When ordering has been defined, consumers can use FnGraph::iter to iterate through the graph and invoke each function sequentially. When the "async" feature is enabled, which is on by default, FnGraph::stream will produce each function once all of its predecessors have been streamed and returned.

To know when a function is available to be streamed, fn_graph uses the following algorithm:

• Track the number of predecessors of each node.
• When streaming, each time a function is streamed and the closure returns, subtract 1 from the predecessor counts of each of that function's successors.
• If the predecessor count of a function is 0, it is now available to be streamed.

Visualized

1. In the example scenario, each function has a number of predecessors:

2. When a function completes, subtract 1 from each of its successors' predecessor counts:

3. This applies to both logical and data dependencies:

4. Performance is gained when multiple functions can be executed concurrently:

This can be seen by timing the executions:

// [dependencies]
// fn_graph = { version = "0.5.4", features = ["fn_meta", "resman"] }
// futures = "0.3.21"
// resman = { version = "0.15.0", features = ["fn_meta", "fn_res"] }
// tokio = { version = "1.20.0", features = ["rt", "macros", "time"] }
use std::{
    fmt,
    ops::{AddAssign, Deref, DerefMut},
};

use fn_graph::FnGraphBuilder;
use futures::stream::StreamExt;
use resman::{IntoFnRes, Resources};

// Define newtypes for each parameter.
#[derive(Debug)]
struct A(u32);
#[derive(Debug)]
struct B(u32);
#[derive(Debug)]
struct C(u32);
#[derive(Debug)]
struct D(u32);

fn main() {
    // Initialize data.
    let mut resources = Resources::new();
    resources.insert(A(0));
    resources.insert(B(0));
    resources.insert(C(0));
    resources.insert(D(0));
    let resources = &resources; // Now the map is compile time immutable.

    // Define logic and insert them into graph structure.
    type Fn1 = fn(&mut A, &mut B, &mut C, &mut D);
    type Fn2 = fn(&A, &mut C);
    type Fn3 = fn(&B, &mut C);
    type Fn4 = fn(&C, &mut D);
    type Fn5 = fn(&A, &B, &C);

    let fn_graph = {
        let fn1: Fn1 = |a, b, c, d| { a.0 = 1; b.0 = 2; c.0 = 0; d.0 = 0; };
        let fn2: Fn2 = |a,    c   | *c += a.0;
        let fn3: Fn3 = |   b, c   | *c += b.0;
        let fn4: Fn4 = |      c, d| *d += c.0;
        let fn5: Fn5 = |a, b, c   | println!("{a} + {b} = {c}");

        let mut fn_graph_builder = FnGraphBuilder::new();

        // Store functions in graph.
        let [fn_id1, fn_id2, fn_id3, fn_id4, fn_id5] = fn_graph_builder.add_fns([
            fn1.into_fn_res(),
            fn2.into_fn_res(),
            fn3.into_fn_res(),
            fn4.into_fn_res(),
            fn5.into_fn_res(),
        ]);

        // Define dependencies to control ordering.
        fn_graph_builder
            .add_logic_edges([
                (fn_id1, fn_id2),
                (fn_id1, fn_id3),
                (fn_id2, fn_id4),
                (fn_id2, fn_id5),
                (fn_id3, fn_id4),
                (fn_id3, fn_id5),
            ])
            .unwrap();

        fn_graph_builder.build()
    };

// Invoke logic over data.
let sequential_start = tokio::time::Instant::now();
fn_graph.iter().for_each(|fun| {
    fun.call(resources);
    std::thread::sleep(std::time::Duration::from_millis(10));
});
let sequential_elapsed = sequential_start.elapsed();
println!("sequential_elapsed: {sequential_elapsed:?}");

// prints:
// 1 + 2 = 3
// sequential_elapsed: 50.683709ms

let rt = tokio::runtime::Builder::new_current_thread()
    .enable_time()
    .build()
    .unwrap();
rt.block_on(async move {
    let concurrent_start = tokio::time::Instant::now();
    fn_graph
        .stream()
        .for_each_concurrent(None, |fun| async move {
            fun.call(resources);
            tokio::time::sleep(tokio::time::Duration::from_millis(10)).await;
        })
        .await;
    let concurrent_elapsed = concurrent_start.elapsed();
    println!("concurrent_elapsed: {concurrent_elapsed:?}");
});

// prints:
// 1 + 2 = 3
// concurrent_elapsed: 44.740638ms
}

macro_rules! u32_newtype {
    ($name:ident) => {
        impl fmt::Display for $name {
            fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
                self.0.fmt(f)
            }
        }
        impl Deref for $name {
            type Target = u32;

            fn deref(&self) -> &Self::Target {
                &self.0
            }
        }
        impl DerefMut for $name {
            fn deref_mut(&mut self) -> &mut Self::Target {
                &mut self.0
            }
        }
        impl AddAssign<u32> for $name {
            fn add_assign(&mut self, other: u32) {
                *self = Self(self.0 + other);
            }
        }
    };
}
u32_newtype!(A);
u32_newtype!(B);
u32_newtype!(C);
u32_newtype!(D);

Notably there is some overhead with the asynchronous execution, but as the number of functions grow, so should the concurrency, and the performance gains should increase proportionally.

Item

Note: 🚧 means the concept is not yet implemented.

An item is something that can be created, inspected, and cleaned up by automation.

"Itemification" is to define data types and logic to manage that item.

The Item trait and associated types are how consumers integrate with the Peace framework. Consumers provide a unique ID, data types, and functions, which will be selectively executed by the framework to provide lean, robust automation, and a good user experience.

This logical breakdown guides automation developers to structure logic to handle cases that are not normally considered of when writing automation. Combined with trait requirements on data types, the framework is able to provide commands, workflow optimizations, and understandable output to ensure a pleasant automation experience.

Data Types

1. Error: The umbrella error type returned when anything goes wrong when managing the item.
2. State: Information about a managed item, can be divided into logical and physical state.
3. Logical state: Part of the state that can be controlled, e.g. application server's existence.
4. Physical state: Part of the state that cannot be controlled, e.g. application server's instance ID.
5. Current state: Current State of the managed item, both logical and physical.
6. Goal state: Logical State that one wants the item to be in.
7. State difference: Difference between the current Logical state and the Goal state.

Logic – Building Blocks

Comparison with git

Readers may notice the function breakdown is git-like. The following table compares the concepts:

Item Parameters

For an item to work with different values, the values must be passed in. These values are called item parameters.

There are a number of related concepts:

• User needs to be able to specify how the values are defined:

  • Plain Values: Use a value provided by the user.
  • Referenced Values: Use a value produced by a predecessor item.
  • Transformed Values: Take values produced by predecessor item(s), transform it, then use that.

• Implementors:

  • Need to define the parameters.
  • Take in parameter values for state_current, state_goal.
  • Take in Option<Field> for each field within the parameter for try_state_current, try_state_goal.

• Peace should be able to store and load:

  • The specification by the user.
  • The actual values computed and used during command execution.

Value Specification

Item parameter values are eventually concrete values.

Some of those concrete values are not necessarily known until partway through a flow execution. When a flow is defined, a user needs a way to encode where a value comes from.

Plain Values

Plain values are values that a user provides before a command is executed.

A variation of plain values is to provide a lookup function that is evaluated at the point the value is needed, but that has potential negative effects for user experience:

• Performance: Web service call(s) may take seconds to complete.
• Consistency: Multiple executions may discover different values between different command executions.

In code, this may look like:

let app_params_spec = AppParams::spec()
    .repo_path(Path::from("/path/to/project"))
    .build();
let server_params_spec = ServerParams::spec()
    .image_id(image_id!("abcd1234"))
    .instance_size(InstanceSize::Large)
    .build();

cmd_ctx_builder
    .with_item_params(app_params_spec)
    .with_item_params(server_params_spec)
    .await?;

Referenced Values

Referenced values are values directly taken from a predecessor's state output.

In code, this may look like:

let file_upload_params_spec = FileUploadParams::spec()
    .src_from::<AppOutputPath>()
    // ..
    .build();

cmd_ctx_builder
    .with_item_params(file_upload_params_spec)
    .await?;

Transformed Values

In code, this may look like:

let file_upload_params_spec = FileUploadParams::spec()
    // ..
    .dest_from_map::<Server>(|server| {
        let ip = server.ip();
        format!("user@${ip}:/path/to/dest")
    })
    .build();

cmd_ctx_builder
    .with_item_params(file_upload_params_spec)
    .await?;

Params Specification

For an item to work with different values, the values must be passed in.

Item implementors define these as part of the Item trait:

trait Item {
    type Params: ..;
}

Use Cases

The following shows a number of use cases of these params:

• State Apply: Param values must be known, and Peace should pass concrete values to the Item::{state_current, state_goal, apply} functions.

• State Discovery (fallible):

  Param values may be known, if predecessors have previously executed.

  • try_state_current: StateDiscoverCmd::current

    e.g. Look up file contents on a remote host:

    match params_partial.dest_ip() {
        Some(dest_ip) => Some(file_hash(dest_ip, path)),
        None => None, // or `Some(FileState::None)`
    }

  • try_state_goal: StateDiscoverCmd::goal

    e.g. Look up source file contents:

    match params_partial.src_path() {
        Some(src_path) => file_hash(src_path),
        None => None, // or `Some(FileState::None)`
    }

By Item Function

• try_state_current: Should work with field_partials.

• try_state_goal: Should work with field_partials.

• state_current: Needs real concrete param values.

• state_goal: Needs real concrete param values.

• state_diff: Doesn't need parameters or data; everything should be captured in States.

  But for presentation, it's useful to know what a file should be (current vs goal), or difference between params (multiple profile current vs current).

• state_clean: Maybe always returns ItemState::None, and doesn't need parameters or data.

  However, presenting state_clean with e.g. a file path, would mean the None state contains the value, which means state_clean needs params.

  Arguably state_goal will show the path that would be created.

  StateDiff for cleaning should also show the deletion of the path.

• apply_check: Doesn't need parameters or data.

• apply_dry: Needs concrete param values, even if they are fake.

• apply: Needs real concrete param values.

Encoding: Serialization / Deserialization

Because:

• It is convenient to serialize Item::Params::Spec and store it, and deserialize it for use at a later time.
• It is useful to support config-based parameter specification (no compiler needed).
• It is not possible to serialize closures.

Then there must be a way to encode the same functionality that Item::Params::Spec::field_from_map provides, as something serializable.

Possibilities:

• ToString and FromStr impls that represent the logic
• Serialized form uses enum variants, and when deserializing, map that back to functions.
• Custom language.

Code Implications

From the implementor's perspective, item trait needs to change to support the above use cases.

The following snippets are here to show the changes that include the above concepts. These are:

• non-compilable.
• just enough to show where types are changed.
• show certain trait bounds (non-exhaustive).
• do not include the encoding / decoding of field_from_map concept.

Framework

// Traits in Peace Framework
trait Item {
    type Params: Params;

    fn setup(&self, resources);
    fn try_state_current(fn_ctx, params_partial, data);
    fn try_state_goal(fn_ctx, params_partial, data);
    fn state_clean      (        params_partial, data);
    fn state_current    (fn_ctx, params,         data);
    fn state_goal    (fn_ctx, params,         data);
    fn apply_dry        (fn_ctx, params,         data, state_current, state_target, diff);
    fn apply            (fn_ctx, params,         data, state_current, state_target, diff);
    fn apply_check      (        params_partial, data, state_current, state_target, diff);
    fn state_diff       (        params_partial, data, state_a, state_b);

    // Once more, with types:
    fn setup(&self, &mut Resources<Empty>);
    fn try_state_current(FnCtx<'_>, Self::Params<'_>::Partial, Self::Data<'_>);
    fn try_state_goal(FnCtx<'_>, Self::Params<'_>::Partial, Self::Data<'_>);
    fn state_clean      (           Self::Params<'_>::Partial, Self::Data<'_>);
    fn state_current    (FnCtx<'_>, Self::Params<'_>         , Self::Data<'_>);
    fn state_goal    (FnCtx<'_>, Self::Params<'_>         , Self::Data<'_>);
    fn apply_dry        (FnCtx<'_>, Self::Params<'_>         , Self::Data<'_>, Self::State, Self::State, Self::StateDiff);
    fn apply            (FnCtx<'_>, Self::Params<'_>         , Self::Data<'_>, Self::State, Self::State, Self::StateDiff);
    fn apply_check      (           Self::Params<'_>::Partial, Self::Data<'_>, Self::State, Self::State, Self::StateDiff);
    fn state_diff       (           Self::Params<'_>::Partial, Self::Data<'_>, Self::State, Self::State);
}

/// For Peace to access <Item::Params as Params>::Spec
trait Params {
    type Spec: Serialize + Deserialize;
    type SpecBuilder: SpecBuilder<Output = Self::Spec>;
    type Partial: Serialize + Deserialize;
}

enum ValueSpec<T> {
    Value(T),
    From,
    FromMap(Box<dyn Fn(&Resources) -> Option<T>>),
}

Also need to provide a Params derive macro.

Design Note

The apply_check and state_diff functions usually are not expected to need params and data, but some items may use them, such as the ShCmdItem which accesses params to determine the script to run.

Regarding params will be Params or Params<'_>::Partial for state_diff, if we call state_diff for a file upload item, we must have both the current state and goal state. Then we need to ask, does having both states imply Params is fully resolvable?

If we call state_current for a file upload:

• the destination server may not be there
• so params may not have the IP
• we may still return Some(State::Empty)
• So params may still be partial, even if State is Some.

If we call state_goal for a file upload:

• the source file may not be there
• so params may not have the source content hash
• we may still return Some(State::Empty)
• So params may still be partial, even if State is Some.

Implementor

// Implementation
struct FileUploadItem;

impl Item for FileUploadItem {
    type Params = FileUploadParams;
}

#[derive(Clone, Debug, Params, Serialize, Deserialize)]
struct FileUploadParams {
    src: PathBuf,
    dest_ip: IpAddr,
    dest_path: PathBuf,
}

Auto generated by Params derive:

impl Params for FileUploadParams {
    type Spec = FileUploadParamsSpec;
    type SpecBuilder = FileUploadParamsSpecBuilder;
    type Partial = FileUploadParamsPartial;
}

// Serialize / Deserialize not needed.
struct FileUploadParamsPartial {
    src: Option<PathBuf>,
    dest_ip: Option<IpAddr>,
    dest_path: Option<PathBuf>,
}

#[derive(Clone, Debug, Serialize, Deserialize)]
struct FileUploadParamsSpec {
    src: ValueSpec<PathBuf>,
    dest_ip: ValueSpec<IpAddr>,
    dest_path: ValueSpec<PathBuf>,
}

#[derive(Clone, Debug)]
struct FileUploadParamsSpecBuilder {
    src: Option<ValueSpec<PathBuf>>,
    dest_ip: Option<ValueSpec<IpAddr>>,
    dest_path: Option<ValueSpec<PathBuf>>,
}

See:

• optfield
• partial_derive
• optional-field

Implementation

See the params_derive crate for code gen.

The following command helps to see what's been generated.

cargo expand --package peace_item_blank blank_params \
  | sd -f cm '^    #\[doc\(hidden\)\][\n](^[ ]{4}[a-z# ].+[\n])+^[ ]{4}\};\n' '' \
  | sd -f cm '^    #\[automatically_derived\][\n](^[ ]{4}[# a-z{].*[\n])+^[ ]{4}\{?\}\n' '' \
  | sd -f cm '^    #\[allow\(unused_qualifications\)\][\n](^[ ]{4}[# a-z{].*[\n])+^[ ]{4}\}\n' '' \
  | sd -f cm '^    #\[serde\(bound = ""\)\]' '    #[derive(Serialize, Deserialize)]\n    #[serde(bound = "")]' \
  | sd -f cm '^    extern crate test;[\n](^[ ]{4}.*[\n])+^\}' '}' \
  | sd -f cm '^(    pub struct [A-Za-z0-9_]+Partial)' '    #[derive(PartialEq, Eq)]\n$1' \
  | sd -f cm '^(    #\[derivative\()' '    #[derive(derivative::Derivative)]\n$1' \
  | xclip -se c

Params Framework Support

For ergonomic usage and accurate data modelling, the following types need to exist for each item:

• Params
• ParamsPartial
• ParamsSpec
• ParamsSpecBuilder

Since Params is defined by the item implementor, the other three structs can only be defined after it, and can be generated by a proc macro.

Questions:

• How does Peace know what fields to fill in, and how?

  The fields are known by the ParamsSpec impl, though it resides in the implementor's crate.

  Peace can reference the ParamsSpec through a Params::Spec associated type.

• Does Peace need to know about the field, or can it be in the generated impl?

  It needs to go, for each field on the ParamsSpec, fill in the ParamsPartial. This could be implemented by ParamsSpec::partial_from(&Resources) -> ParamsPartial:

  impl ParamsSpec {
      fn partial_from(resources: &Resources) -> ParamsPartial {
          // for each field (generated by proc macro)
          let value_0 = value_0_spec.try_value_from(resources);
          let value_1 = value_1_spec.try_value_from(resources);
  
          ParamsPartial { value_0, value_1 }
      }
  }
  
  enum ValueSpec<T> {
      Value(T),
      From,
      FromMap(Box<dyn Fn(&Resources) -> Option<T>>),
  }
  
  impl<T> ValueSpec<T> {
      fn try_value_from(&self, resources: &Resources) -> Option<T> {
          match self {
              Self::Value(t) => Some(t.clone()),
              Self::From => resources.try_borrow::<T>().cloned(),
              Self::FromMap(f) => f(resources),
          }
      }
  }

• How does ValueSpec::FromMap get instantiated:

  Probably something like this:

  impl ParamsSpecBuilder {
      // for each field
      fn field_from_map<F, U>(mut self, f: F) -> Self
      where F: &U -> Field {
          let from_map_boxed = Box::new(move |resources| {
              resources.try_borrow::<Field>().map(f)
          });
          self.field_spec = ValueSpec::FromMap(from_map_boxed);
          self
      }
  }

To allow Peace to reference those types, and for Peace to expose that to users, we need the following traits:

• dyn Params: To know what type the ParamsPartial, ParamsSpec, and ParamsSpecBuilder are.
• dyn ParamsSpec: Peace needs the spec to generate a ParamsPartial from Resources.

Implications On Serialization and Flow Params

It is inconvenient to have to register item params types as flow params, because the presence of the item could automatically tell Peace its params type.

However, users:

• Must provide the param values / spec for each item for the first execution of a flow.
• May have different preferences whether they have to provide the values / spec for subsequent executions, and whether it implicitly uses what was used before, or is explicit for each field.

What we need:

• Flow params are serialized separately. Flow params may be values that control how things are presented, but not affect the actual functional execution, so they may be log verbosity.
• Item params spec must be deserialized, and overwritten by provided values.
• Item params spec must be serialized before executing a command.
• Item params must be serialized as execution happens.
• Item params partials do not need to be serialized, only presented, as they can be re-calculated from state for each discover / read command execution.

Mapping Functions

Github Issue: #156

Mapping functions could be passed to a Params's FieldWiseSpec. There are implications whether or not it is implemented.

Summary

With It

Developers

1. Define an enum to name the function keys:

   #![allow(unused)]
   fn main() {
   enum MappingFunctions {
       BucketNameFromBucketState,
       IamPolicyArnFromIamPolicyState,
   }
   }

2. Define the mappings:

   #![allow(unused)]
   fn main() {
   let mapping_functions = {
       let mut mapping_functions = MappingFunctions::new();
       mapping_functions.insert(BucketNameFromBucketState, S3BucketState::bucket_name);
       // ..
   
       mapping_functions
   };
   }

   Note:

   If we want to have a compilation error here, the MappingFunctions::insert function needs to have a type parameter that tells it the Item > Params > Field that the mapping function is for.

   However, developers use #[derive(Params)] to derive the <ItemParams>FieldWiseSpec, and requiring them to specify something like the following is arduous:

   #![allow(unused)]
   fn main() {
   MappingFunction::insert<FromType, ToType>(BucketNameFromBucketState, S3BucketState::bucket_name)
   }

3. Pass MappingFunctions to CmdCtxBuilder, for each code instantiation (may be just one):

   #![allow(unused)]
   fn main() {
   cmd_ctx_builder.with_mapping_functions(mapping_functions);
   }

4. Not have to call .with_item_params::<TheItem>(..) in subsequent calls.

Users

1. Get runtime error if the mapping function type doesn't match, but it should be caught by tests.

Framework Maintainers

1. MappingFunctions map will have magic logic to store the function argument types and return type.
2. Error reporting when types are mismatched.

Without It

Developers

1. Define the item params spec:

   #![allow(unused)]
   fn main() {
   // First execution
   let s3_object_params_spec = S3ObjectParams::<WebApp>::field_wise_spec()
       .with_file_path(web_app_path_local)
       .with_object_key(object_key)
       .with_bucket_name_from_map(S3BucketState::bucket_name)
       .build();
   
   // Subsequent executions
   let s3_object_params_spec = S3ObjectParams::<WebApp>::field_wise_spec()
       .with_bucket_name_from_map(S3BucketState::bucket_name)
       .build();
   }

2. Pass the item params spec to CmdCtxBuilder, for every separate code instantiation:

   #![allow(unused)]
   fn main() {
   cmd_ctx_builder
       .with_item_params::<S3ObjectItem<WebApp>>(
           item_id!("s3_object"),
           s3_object_params_spec,
       )
   }

   This is somewhat of an inconvenience, because if this isn't done, the user / developer will have a runtime error, which looks like this:

   peace_rt_model::params_specs_mismatch
   
     × Item params specs do not match with the items in the flow.
     help: The following items either have not had a params spec provided previously,
           or had contained a mapping function, which cannot be loaded from disk.
   
           So the params spec needs to be provided to the command context for:
   
           * s3_object
   

When the closure passed to with_*_from_map doesn't have the argument type specified, or mismatches, the compilation error is still unclear. rust#119888 will allow us to return a useful compilation error.

Users

No runtime error, because it will be caught at compile time.

Framework Maintainers

1. Error messages / diagnostics showing which CmdCtx is missing which item spec for which field, should be made clear.

Item Graph

An item graph is a function graph, whose logic type is an item.

Instead of storing a single function in each node of the graph, each logic type is a collection of well-defined functions adhering to the Item trait.

Combining these concepts contributes to the strategy's outcomes, by enabling the framework to address common issues in each input dimension.

let graph = {
    let mut graph_builder = ItemGraphBuilder::<XError>::new();
    let [id_1, id_2, id_3, id_4] = graph_builder.add_fns([
        Item1::new(param1).into(),
        Item2::new(/* .. */).into(),
        Item3::new().into(),
        Item4::new().into(),
    ]);
    graph_builder.add_logic_edges([
        (id_1, id_3),
        (id_2, id_3),
        (id_2, id_4),
    ])?;

    graph_builder.build()
};

The remainder of this section explains how using these concepts together allows sensible user facing commands to be created. Available commands are documented in the reference.

Initialization

Before a process is started, make sure all the necessary information is provided.

In function graph, data types in Resources are inserted by the consumer, separate from the graph.

In Peace, Item::setup is run for each item, which allows data types to be inserted into Resources.

let graph = /* .. */;

let resources = graph.setup(Resources::new()).await?;

// Item1::setup
resources.insert(param1);

// Item2::setup
resources.insert(param2);

// Item3::setup
// no-op

// Item4::setup
resources.insert(param3);
resources.insert(param4);

ℹ️ Note: Each initialization parameter should be specified in each item's setup method, even though the parameter is inserted by a predecessor item.

This is because when only a subset of the graph is executed, or if the item is used in a different graph, the parameter should still be inserted.

🚧 A wrapper type should conditionally insert the initialization parameter into Resources

State Inspection

Before applying a change, you should understand the current state, and what the change will do.

To discover the current state of all items, Item::try_state_current is run for each item concurrently.

let graph = /* .. */;
let resources = /* .. */;

let resources = StatesDiscoverCmd::current(graph, resources).await?;

// Item1::StateCurrentFn::try_exec
let exists = param1.path().exists();
Ok(State::new(exists, ()))

// Item2::StateCurrentFn::try_exec
let instance_url = discover(param2).await?;
Ok(State::new(Some(instance_url), ()))

// Item3::StateCurrentFn::try_exec
let version = reqwest::get(instance_url).await?;
Ok(State::new(version, ()))

// ..

When all states have been collected, they are presented to the user:

state1: exists
state2: 1.2.3.4
state3: 1.0.0
state4: abcdef0123456

State Ensure

When applying a change, the change should converge the current state to the goal state.

let graph = /* .. */;
let resources = /* .. */;

let resources = ApplyCmd::exec(graph, resources).await?;

Note that the Item::apply requires implementers to return StatePhysical, which is the state information generated during the exec logic, but not necessarily within the implementers' control.

Method

To discover the current state of all items, the following method is used:

1. ApplyFns::check is run for all items.
2. Of the ones that return ApplyCheck::ExecRequired, Item::apply is run.
3. Finally, Item::state_current is run so that the end state can be compared with the goal state to confirm that they match.

ApplyFns::check

// Item1
ApplyCheck::ExecRequired { .. }

// Item2
ApplyCheck::ExecNotRequired

// Item3
ApplyCheck::ExecRequired { .. }

// Item4
ApplyCheck::ExecRequired { .. }

Item::apply

// Item1
()

// Item2
IpAddr::from_str("1.2.3.4")

// Item3
()  // Note: version is logical state

// Item4
Revision::new("abcdef0123456")

Dry Run

let resources = Item::apply_dry(graph, resources).await?;

Similar to the Item::apply, Item::apply_dry is meant to simulate what would happen, and allow users to correct mistakes before actual execution.

Implementers must replace all write logic with mocks. These include:

• File writes
• Web requests

It is also recommended that read requests to external services are minimized to decrease the time to return feedback to the user. If possible, move read request logic to Item::state_current so that it is stored by the StatesDiscoverCmd.

Convergence / Non-Transactional Execution Recovery

Since these processes happen over distributed systems, and errors can happen at any point in the process, it is realistic to assume that the process doesn't happen transactionally.

ApplyFns has been designed so that implementers will consider transitions from non-atomic states to the goal state. In simpler terms, if the goal state is "10 servers must exist", then:

• When the current state is 0 servers, then 10 servers should be launched.
• When the current state is 6 servers, then 4 servers should be launched.

As time and effort is saved by reusing existing useful state and not requiring complete clean up, recoverability and user experience is improved.

Clean

Partial Clean

Even though it is possible to clean up the server without uninstalling the application, it is recommended that the CleanOpSpec is implemented for every item.

Imagine one has a goal to test fresh installations of different versions of an application. One approach is to deploy each version in a separate environment, and clean up the whole environment, . have a command that only uninstalls the application from the server, .

Least-work approach, but needs additional engineering.

State

In peace, State represents the values of an item, and has the following usages:

• Showing users the state of an item.
• Allowing users to describe the state that an item should be.
• Determining what needs to change between the current state and the goal state.

Therefore, State should be:

• Serializable
• Displayable in a human readable format
• Relatively lightweight – e.g. does not necessarily contain file contents, but a hash of it.

Logical and Physical State

State can be separated into two parts:

• Logical state: Information that is functionally important, and can be specified by the user ahead of time.

  Examples of logical state are:

  • File contents
  • An application version
  • Server operating system version
  • Server CPU capacity
  • Server RAM capacity

• Physical state: Information that is discovered / produced when the automation is executed.

  Examples of physical state are:

  • ETag of a downloaded file.
  • Execution time of a command.
  • Server ID that is generated on launch.
  • Server IP address.

Defining State

Fully Logical

If an item's state can be fully described before the item exists, and can be made to happen without interacting with an external service, then the state is fully logical.

For example, copying a file from one directory to another. The state of the file in the source directory and destination directories are fully discoverable, and there is no information generated during automation that is needed to determine if the states are equivalent.

Logical and Physical

If an item's goal state can be described before the item exists, but interacts with an external service which produces additional information to bring that goal state into existence, then the state has both logical and physical parts.

For example, launching a server or virtual machine. The operating system, CPU capacity, and RAM are logical information, and can be determined ahead of time. However, the server ID and IP address are produced by the virtual machine service provider, which is physical state.

Fully Physical

If an item's goal state is simply, "automation has been executed after these files have been modified", then the state has no logical component.

For example, running a compilation command only if the compilation artifact doesn't exist, or the source files have changed since the last time the compilation has been executed.

The remaining pages in this section explain how to define the logical and physical state types when implementing an item.

Logical State

Logical state is the part of an item that is:

• Implementor / user definable
• Controllable by automation
• Deterministic

Uses

There are three uses of logical state:

1. Representing current state.
2. Representing goal state.
3. Computing state difference.

let params = data.params();

// `dest` references the actual item that is being managed.
// e.g. calculate content hash of actual file being written to.
let state_current = params.dest().read();

// `src` references the *specification* of what the item is intended to be.
// e.g. retrieve content hash from a source file.
let state_goal = params.src().read();

// We can only compute the `diff` when both `src` and `dest` are available.
let state_diff = state_goal - state_current;

Discovery Constraints

In an item's parameters, there must be the following categories of information:

• src: information of what the item should be, or where to look up that information.

  Thus, src is a reference to where to look up state_goal.

• dest: reference to where the actual item should be.

  dest is a reference to where to push state_current.

Both src and dest may reference resources that are ensured by predecessor items. Meaning sometimes state_goal and state_current cannot be discovered because they rely on the predecessors' completions.

Examples

• A list of files in a zip file cannot be read, if the zip file is not downloaded.
• A file on a server cannot be read, if the server doesn't exist.
• A server cannot have a domain name assigned to it, if the server doesn't exist.

Implications

• If dest is not available, then state_current may simply be "does not exist".

• If src is not available, and we want to show state_goal that is not just "we can't look it up", then src must be defined in terms of something readable during discovery.

• If that is not possible, or is too expensive, then one or more of the following has to be chosen:

  1. Item::state_goal functions have to always cater for src not being available.

     It incurs mental effort to always cater for src not being available – i.e. implementing an item would need knowledge beyond itself.

  2. the peace framework defaults to not running state_current_fn for items that have a logical dependency on things that Item::apply_check returns ExecRequired

     For this to work, when the current state is requested, peace will:

     1. For each non-parent item, run state_current, state_goal, state_diff, and apply_check.
     2. If apply_check returns ApplyCheck::ExecNotRequired, then successor items can be processed as well.

  3. state_current could return Result<Option<Status>, E>:

     • Ok(None): State cannot be discovered, likely because predecessor hasn't run

     • Ok(Some(State<_, _>)): State cannot be discovered.

     • Err(E): An error happened when discovering state.

       May be difficult to distinguish some cases from Ok(None), e.g. failed to connect to server, is it because the server doesn't exist, or because the address is incorrect.

       Should we have two state_currents? Or pass in whether it's being called from Discover vs Ensure – i.e. some information that says "err when failing to connect because the predecessor has been ensured".

  Option 2 may be something we have to do anyway – we will not be able to provide current state to run Item::apply for successors for the same reason.

  Option 3 may coexist with option 2.

  Note: State discovery may be expensive, so it is important to be able to show a stored copy of what is discovered.

Output

Output is providing information to a user.

Output can be categorized with number of input dimensions:

• Disclosure:
  • Push: Information is pushed to users once available.
  • Pull: Information is requested by users, may be real time or historical.
• Rate: (of requests)
  • High: Information that constantly updates.
  • Low: Information that once created isn't expected to change.
• Consumer:
  • Human: Person using the application.
  • Software: ReST API consumer.
  • Both: Readable and parseable text output.
• Accumulation:
  • Append: Information continuously increases.
  • Replace: Latest information is relevant.

From the above table, further thoughts include:

• Information that changes at a high rate implies a high rate of requests.
• Information requested at a high rate may need to be small (in size).
• Progress output is frequent, and is most important in real time / as it happens.
• If frequency is high and the output is transferred between hosts, then ideally the size of the output is reduced.
• The point of serialized data without the friendliness requirement is space and time efficiency.
• Friendly serialized data may be a format such as YAML / JSON / TOML.
• Pulled data is either over an API, or viewing historical information, meaning pulled data needs to be serializable.
• Web page output may be large, but mental overload can be avoided by hiding information through interactivity.

Outcome

Outcome Status

• Success: Task has completed successfully.
• Break: Task has stopped for manual action not managed by the command.
• Error: Task has stopped with an error.

Outcome Messages

• Nothing
• Informatives: What the user needs to do next.
• Warnings

Outcome Errors

• What happened
• Why it is considered an error
• Source of the information that led to the error
• Suggestions for fixing

Format

Output format can be optimized for different consumers. For example:

• Interactive command line:

  • Hide detail to reduce mental overload.
  • Show enough to give an appropriate picture.
  • Use colour to highlight / dim detail based on level of importance.
  • Show commands to display additional detail.
  • Use a format that makes sense when copying and pasting into a different application, e.g. markdown.

• Web page:

  • Use interactive elements to allow detail to be revealed when needed.
  • Use consistent colour for different levels of detail.

• Network transfer:

  • Use a serialization format that is small to cater for latency.
  • Use a serialization format that is not difficult to deserialize to reduce CPU utilization.

Execution Progress

Execution progress output is most useful as it happens. Execution progress output should fulfill the following concerns:

• easy for a human to understand
• easy to be sent from an item implementation
• pushed to output in real time
• pulled from client in real time

Information

Information that is likely useful to the consumer, and therefore should be captured, and sent if bandwidth is not an issue.

Progress Status

Whether a task has started, is in-progress, stalled, or completed.

• Execution Pending: Execution is pending a predecessor's execution.

• Running: Execution is in progress.

  This status is best conveyed alongside the following information:

  • Units total: Unknown (spinner) / known (progress bar).

  • Units current

  • Function: Item::{state_current, state_goal, apply}.

    Certain functions will not be applicable, e.g. when StateCurrent is feature gated, then the function won't be available when the feature is not enabled.

• Running Stalled: Progress updates have not been received for a given period.

  Item implementations are responsible for sending progress updates, but if there are no progress updates, or an identical "it's running" progress update is continuously received, then Peace may determine that the task may have stalled, and user attention is required.

  Peace may also provide a hook for implementors to output a suggestion to the user.

• User Pending: Task is pending user input.

• Completed: Task has completed.

  This status is best conveyed alongside the following information:

  • Completion Status: Success, Failed.
  • Function: Item::{state_current, state_goal, apply}.

The following variant is possible conceptually, but not applicable to the Peace framework:

• Stopped: Task is not running, but can be started.

  This is not applicable because Peace uses runtime borrowing to manage state, and a stopped task has potentially altered data non-atomically, so locking the data is not useful, and unlocking the data may cause undefined behaviour due to reasoning over inconsistent state.

  For rate limiting tasks, the task in its entirety would be held back.

Progress Measurement

• Unit of measurement: Steps, Bytes, Percentage, None
• Units total: Known / Unknown
• Units current: Numeric / Unknown
• Units remaining: Numeric / Unknown. This is total - current.
• Unit of progress tick: as it happens.

Progress Timings

• Time started
• Elapsed duration
• Remaining duration estimate
• Time of completion estimate
• Time of completion

Information Production / Consumption

As much as possible, Peace should collect progress information without burdening the implementor.

Peace should support the following usages out-of-the-box¹:

• Interactive CLI
• Non-interactive CLI
• CI logging
• WASM
• REST endpoint (pull)
• REST endpoint (push)

¹ even if it is not yet implemented, it should be designed in a way that allows for it

API

Implementors define the following in ApplyFns::check:

• Unit of measurement: Steps, Bytes, Percentage, None
• Units total: Known / Unknown
• Units initially completed: Numeric / Unknown

Units remaining is tracked by Peace as the unit of progress is ticked (updated).

There is a tradeoff between:

• Implementors manually ticking progress

  • More accurate information

  • Code becomes messy -- progress ticks scattered throughout code.

  • zzz has a really good way of doing this without making things messy:

    #![allow(unused)]
    fn main() {
    (0..1000).into_iter().progress()
    //                    ^^^^^^^^^^
    // This is all that's needed to render the progress bar
    // assuming `size_hint()` is implemented.
    }

    Peace could provide something like:

    #![allow(unused)]
    fn main() {
    impl ApplyFns for MyApplyFns {
        async fn exec(data: Data<'_>) -> Result<(), E> {
            [
                Self::fn_1,
                Self::fn_2,
                Self::fn_3,
                Self::fn_4,
            ].progress(data);
        }
    }
    }

    But that constrains the signature of each sub function to be the same. A type parameterized function chain – where the return value of one function is the parameter to the next – may be messy, especially for error messages.

  • For spinners, manually ticking the progress bar can be cumbersome, but automatic ticking will give a false impression of progress happening. Somewhat mitigated by stall detection.

• Codegen to add progress ticks

  This could be done with a proc macro that counts the number of expressions, with some clever handling of conditionals, and inserts tick() expressions.

For now Peace will require implementors to manual tick the progress, and in the future, explore the following option:

• Trait to automatically tick progress for iterators.
• Proc macro to automatically insert ticks.

Presentation

For human readable output, it is desirable for consumers as well as the framework to be able to:

• Output arbitrary types.
• Have them formatted by the output implementation.
• Not require the output implementation to know the specific type that is being output.

The OutputWrite trait was written to handle different output formats – human-readable vs CI logging vs structured text – but not how to present the output.

Peace should provide two traits:

• OutputWrite: Maps between the information and the output format, e.g. human readable vs parseable.

  Examples:

  • CLI output: Writes progress as log messages (CI), or progress bar (interactive), or nothing (when piped).
  • Web output: Write JSON.
  • Native application: Update a UI component.

• Presentable: Maps between the information and the presentation format, e.g. present this as an ID, short text, long text, a list, etcetera.

  Examples:

  • CLI output: Colour text with ANSI colour codes.
  • Web output: Create and style web elements.
  • Native application: Create and style UI components.

Current State

The OutputWrite trait has methods for:

• writing progress: setting up state, updating progress, and ending.
• writing states (current, goal, diff, etc.)
• writing error

These methods are specific to States, and if we add methods per type, it doesn't allow any arbitrary type to be formatted and written.

Goal State

To be usable with arbitrary information, OutputWrite should have methods to output different kinds of information. These information kinds are based on the purpose of the information, not on how they should be grouped or presented.

Information Kinds

• Progress: Information about the execution of automation.

• Outcome: Information that the automation is purposed to produce.

• Notes: Meta information about the outcome or progress -- informatives, warnings.

  These can be used to refine the automation.

For each information kind, OutputWrite should be able to:

• Write one or many of that information kind
• Reason over the parameters of that information, and potentially pass it to a formatter.

For structured output, all information should be serializable.

Presentation / Formatting

For human readable output to be understandable, the OutputWrite implementation should improve clarity by adding styling or reducing information overload. For this to work with arbitrary types, the OutputWrite needs additional hints to determine how to format the information.

Examples:

• An object may be presented as a list, and the type needs to define which fields that list is built from.
• When presenting a list of named items, the type needs to define both the name and the description, which allows the names to be styled differently to the descriptions.
• When presenting a large object, the density of information can be reduced through collapsible sections, and more detail displayed when the sections are expanded.

Implementation

To achieve this, we can:

• Define a peace::fmt::Presentable trait, analogous to std::fmt::Display

• Define a peace::fmt::Presenter trait, analogous to std::fmt::Formatter

• Presenter has methods to format:

  • short text descriptions
  • long text descriptions (e.g. always split at \ns)
  • names, e.g. always bold
  • lists of Presentables
  • groups, e.g. always collapsible, or presenter may choose to not display if level of detail is too high

• Implementors will impl Presentable for MyType. This can be made easier with a derive macro.

• Update OutputWrite to take in &dyn Presentable instead of concrete types, and the OutputWrite implementation can decide whether or not to delegate to Presenter for presentation information. e.g. a serializing output write may not need to.

Note: Structured output that is read by humans (e.g. prettified YAML or JSON) is not a peace::fmt::Presentable concern, but an OutputWrite parameter, as it is a standard format serialization parameter, not formatting hints that the output endpoint needs.

Instead of using &strs for what is presented, we could add type safety to:

• Enforce certain constraints, e.g. short descriptions must be one line, less than 200 characters
• For human readable output, instead of std::fmt::Display, types implement peace::fmt::Presentable trait where a peace::fmt::Presenter is passed in.

The type safety can be opt-in, e.g. allow &strs, but if using the type-safe wrappers, you get compilation errors when the constraints are not met.

Recursion

If the Presentable trait is recursive like Debug, then we need to make sure implementors understand that a "name" will always be styled as a name, unless one creates a wrapping type that does not delegate to the name's underlying Presentable implementation (just like Debug).

Diagrams

This section explores how various diagrams help to provide clarity to the user for:

• Progress: What's going on.
• Outcome: What does it look like.
• Interaction: What actions can be taken.
• Understanding: Hiding / providing layers of detail.

Progress

Outcome

An outcome diagram should:

• Show the "physical" items that exist / yet-to-exist / to-be-cleaned.
• Show which steps are linked to it, e.g. clicking on a file shows the steps that write to / read from the file.

Determining Information for Rendering

To render the outcome diagram, we need to deduce the physical things from Items, and determine:

1. Source: where data comes from, whether completely from parameters, or whether parameters are a reference to the data.
2. Destination: where data moves to or the system work is done to.
3. Whether the source or destination are declared in parameters.

As of 2024-02-18, Item::Params is a single type, which would take in both source and destination parameters, so we cannot (realistically) determine the source/destination/host/cloud from the Item::Params type.

Source and Destination

An Item is actually a step that encapsulates:

1. Source location.
2. Destination location.

This can be further broken down into:

/// Can be source / destination
enum ItemLocation {
    File {
        host: Host,
        path: PathBuf,
    },
    Cloud {
        cloud_provider: CloudProvider,
        global_or_region: GlobalOrRegion,
        subnet: Option<Subnet>,
        url: URL,
        name: String,
    },
}

Instead of using a flat enum, we may want to use a fixed set of data structures for simplicity, or a trait for extensible implementations.

enum Realm {
    Host {
        address: String,
        port: Option<u16>,
    },
    Cloud {
        cloud_provider: CloudProvider,
        global_or_region: GlobalOrRegion,
        subnet: Option<Subnet>,
        url: URL,
        name: String,
    },
}

struct ItemLocation {
    /// Host or Cloud provider
    realm: Realm,
    /// Not necessarily a file path, but segments denoting hierarchy 
    path: Option<String>,
}

trait Name {
    fn realm(&self) -> Realm;
}

Note that we want to support nesting, e.g. an application server is "just a host", which may be nested within a network, or a cloud provider, or nothing.

How should we do the nesting:

1. Strictly defined levels, implementor selects:

   1. Cloud
   2. Availability Zone
   3. Network
   4. Subnet
   5. Host
   6. Path

2. Arbitrary levels, levels are not necessarily defined by network-hierarchy.

   Note: This doesn't allow consistency across item implementations, or, it requires the tool developer to pass the level into the item.

   Actually, what we can do is:

   1. Allow the item implementor to specify the list of parents that this item resides in.
   2. Each parent may be an arbitrary layer around the current item.
   3. The peace framework will have a struct Layer(IndexMap<LayerId, Layer>); -- this is essentially NodeHierarchy from dot_ix.
   4. When the LayerIds from different items match up, those are inserted into this map.
   5. When we render, we render layer by layer as the hierarchy.

What we want

1. Tool developer passes in parameters for the item.
2. The correct realm for the item is automatically deduced.

e.g.

Tool Developer

let file_download_params_spec = FileDownloadParams::new(..).into();

Item Implementor

impl Item for FileDownload {
    fn location_from(
        params_partial: &<Self::Params<'_> as Params>::Partial,
        data: Self::Data<'_>,
    ) -> Vec<ItemLocation> {
        let host = params_partial
            .src()
            .map(Url::host)
            .map(Host::to_owned);
        let port = params_partial
            .src()
            .map(Url::port_or_known_default);
        let url = params_partial.src();

        let mut item_locations = Vec::new();
        if let Some(host) = host {
            item_locations.push(ItemLocation::Host { host, port });

            if let Some(url) = url {
                // May be rendered using the last segment of the URL as the node name.
                // The full URL may be used as the tooltip.
                item_locations.push(ItemLocation::Url(url));
            }
        }

        item_locations
    }

    fn location_to(params: &<Self::Params<'_> as Params>::Partial) -> Vec<ItemLocation> {
        let path = params_partial
            .dest()
            .map(Path::to_string_lossy)
            .map(Cow::to_owned);

        vec![
            ItemLocation::Localhost,
            ItemLocation::Path(path),
        ]
    }
}

impl Item for S3Bucket {
    fn location_from(
        params_partial: &<Self::Params<'_> as Params>::Partial,
        data: Self::Data<'_>,
    ) -> Vec<ItemLocation> {
        vec![ItemLocation::Localhost]
    }

    fn location_to(
        params: &<Self::Params<'_> as Params>::Partial,
        data: Self::Data<'_>,
    ) -> Vec<ItemLocation> {
        let region = data.region();
        let availability_zone = params.availability_zone();
        let name = params.name();

        let mut item_locations = Vec::with_capacity(10);
        item_locations.push(ItemLocation::CloudProvider(CloudProvider::Aws));
        item_locations.push(ItemLocation::Group(region.to_string()));

        if let Some(availability_zone) = availability_zone {
            item_locations.push(ItemLocation::Group(availability_zone.to_string()));
            if let Some(name) = name {
                item_locations.push(ItemLocation::Name(name.to_string()));
            }
        }
    }
}

Framework Maintainer

Go through all items' Item::location_* and add to outcome diagram -- see flow_spec_info.rs.

Problems

A: How do we get items to be consistent with the item locations they publish?

Cloud provider name, region, availability zone, etc.

Options:

1. 🔴 We could provide a crate with common enums.

   This requires:

   1. The framework to have knowledge of many many types, or be inaccurate.
   2. Item implementors to always be up to date.

   It's a lot of maintenance, so probably not a good idea.

2. 🟡 Item provides default ItemLocations, developer can pass in names.

   This isn't too bad, as it allows tool developers control over names of the items.

   It still requires item implementors to use the same ItemLocation variant.

   It could mean additional development burden.

   • If we go with sensible defaults, then we hope that item implementors use the same conventions.
   • Tool developers passing in names to the items is an escape hatch in case the conventions are different between item implementations.
   • Item implementors have to take in the names instead of only default them.

   If item implementors have consistent values for their ItemLocations, then the inconsistency downside is alleviated partially.

   For example, extracting the host / port from a Url could be offloaded to the framework:

   #[derive(Debug)]
   enum ItemLocation {
       Host(ItemLocationHost),
       Url(Url),
   }
   
   struct ItemLocationHost {
       host: Host<String>,
       port: Option<u16>,
   }
   
   impl ItemLocation {
       fn from_url(url: &Url) -> Self {
           Self::Url(url.clone())
       }
   }
   
   impl From<&Url> for ItemLocationHost {
       fn from(url: &Url) -> Self {
           let host = url
               .map(Url::host)
               .map(Host::to_owned)
               .expect("Expected URL to contain a host.");
           let port = url
               .map(Url::port_or_known_default);
   
           Self { host, port }
       }
   }

   Item implementors will implement Item::location_from and Item::location_to like so:

   impl Item for FileDownload {
       fn location_from(
           params_partial: &<Self::Params<'_> as Params>::Partial,
           data: Self::Data<'_>,
       ) -> Vec<ItemLocation> {
           params_partial
               .src()
               .as_ref()
               .map(|src| {
                   vec![
                       ItemLocation::from_server(src),
                       ItemLocation::from_url(src),
                   ]
               })
               .unwrap_or_else(Vec::new)
       }
   
       fn location_to(
           params: &<Self::Params<'_> as Params>::Partial,
       ) -> Vec<ItemLocation> {
           let path = params_partial
               .dest()
               .map(Path::to_string_lossy)
               .map(Cow::to_owned);
   
           vec![
               ItemLocation::Localhost,
               ItemLocation::Path(path),
           ]
       }
   }

3. 🟡 Developer specifies ItemLocations (from and to) for every item.

   Removes the item location inconsistency issue, but also means every tool developer has to specify item locations for every item, where this work could be commonized.

   This also addresses the problem B below, where the developer has control over the level of detail.

   Essentially we are placing the burden of defining the outcome diagram onto the developer.

   This isn't "nice" when the developer has to extract host names from URLs (common burden).

   Maybe the Item implementor provides a method like this:

   impl Item for MyItem {
       fn item_location_from(
           &self,
           level: MyItemLocationLevel,
           params_partial: &<Self::Params<'_> as Params>::Partial,
       ) -> ItemLocation {
   
           match level {
               MyItemLocationLevel::Host => {
                   let host = params_partial
                       .src()
                       .map(Url::host)
                       .map(Host::to_owned);
                   let port = params_partial
                       .src()
                       .map(Url::port_or_known_default);
                   ItemLocation::Host { host, port }
               }
               _ => todo!(),
           }
       }
   }

   This interface is awkward -- we don't know the params_partial at the point of flow definition (for the developer).

   It could be the developer that defines that function above, combined with framework helpers:

   let my_item_locations = MyItem::item_locations_spec()
       .with_locations_from(|my_item, params_partial| {
           // with the `From` implementations for 2. above:
           params_partial
               .src()
               .as_ref()
               .map(|src| {
                   vec![
                       ItemLocation::from_server(src),
                       ItemLocation::from_url(src),
                   ]
               })
               .unwrap_or_else(Vec::new)
       })
       .with_locations_to(todo!("similar to above"));

4. 🔴 Magically infer based on parameter names.

   Too inaccurate.

5. 🔴 Item implementors use a derive proc-macro.

   Too many annotations -- probably easier to write the trait directly.

B: How does control over whether a group is drawn get defined?

1. Users may not want so many levels of detail -- it can be overwhelming.
2. Developers may want sensible defaults / not require them to set whether a group is drawn.
3. Developers may want to set whether a group is drawn.

Options:

1. 🟡 They can't, everything is always shown.

   Not the best user experience -- too much detail can overwhelm.

2. 🟡 Item implementor guidelines to not include too many layers.

   May be okay? It means either we have too much / too little information sometimes.

   Perhaps this is diminishing returns, and it doesn't matter too much.

3. 🟡 Developer specifies ItemLocations (from and to) for every item.

   Same as A:3 above.

4. 🟡 Item takes in parameter whether each layer is visible.

   Probably in the Item constructor, so we introduce a compilation error:

   fn new(
       /* existing params */
       #[cfg(feature = "web")]
       layer_visibility: MyItemLayerVisibility,
   ) -> Self { /* .. */ }
   
   struct MyItemLayerVisibility {
       cloud: bool,
       availability_zone: bool,
       subnet: bool,
   }

   In tandem with A:2 above, maybe it's not worth the complexity.

5. 🟡 Draw everything, developer provides separate information for layers.

   1. Probably want the Item to export what layers it draws.
   2. Developer passes a Map<ItemLocation, DrawEnabled> to the framework.

   Feels weird / unintuitive as a design, not too weird as a user: "I see everything, I want to turn that one off.".

C: Consistent Diagram Layout

If the ItemLocations changes, such as the number of ItemLocations change (hierarchy level change), or the ItemLocation variants change (different rendering style applied), then the layout of the diagram can change:

1. Position of each box can change: pushing what used to be "here", "there".
2. Width / height of each box can expand: it doesn't fit in the screen.
3. Colours can change: different ItemLocation type.
4. Edges can change position.

It can be confusing to follow if these keep changing, which is counter productive to the intent of communicating clearly which physical systems are interacting.

Options:

1. 🟡 Don't solve it now, we don't know how much of a problem it is.

   dot_ix's HTML DivDiag is not very mature anyway. We may end up developing this a lot more.

   This is taking a bet that we won't have to do a large refactor to the Item API later. Probably alright since there aren't any consumers of Peace just yet.

2. 🔴 Try and match nodes, and animate.

   Too magic / inaccurate?

3. 🟡 Require / request Item implementors to always provide the same number of ItemLocations.

   May mean every ItemLocation that is unknown, is still populated:

   let item_location_server = item_location_server.unwrap_or(ItemLocation::HostUnknown);
   let item_location_url = item_location_url.unwrap_or(ItemLocation::UrlUnknown);
   
   vec![
       item_location_server,
       item_location_url,
   ]

Solution

We'll go with:

• A:2: Item implementors provide ItemLocations

• B:1: Everything is shown.

  There is still the possibility to migrate to B:4 (Items take in visibility flags) or B:5 (developers pass in visibility to framework) later.

• C:1: Don't solve the consistent diagram layout now.

What Rendering Makes Sense

Conceptually, Items can be thought of either an edge or a node:

• Edge: The item represents an action / work to between the source(s) and the destination(s).
• Node: The item represents the destination thing.

Use Cases

1. Upload a file -- one source, one dest.
2. Download a file -- one source, one dest.
3. Launch servers -- one source (localhost), one dest (AWS).
4. Wait for servers to start up -- multiple within (do we need the ItemLocationTree for the cloud provider / subnet context? or leverage previous resource tracking to work it out?).
5. Wait for endpoints to become available -- one source, multiple dest (query each endpoint).
6. Do we want ItemInteractions to be queried multiple times while Apply is happening? -- i.e. some servers may have started up, and so we need the Item to report that to us.
7. Notably, we want these ItemInteractions to be queryable without the items actually existing -- so we can generate diagrams to demonstrate what would happen upon execution.

Naive

Consider the following diagram, which is the first attempt at rendering an outcome diagram on 2024-02-18 -- this uses edges / hierarchy to draw nodes and edges:

Notes

1. It is not clear where the app_download Item transfers the file from, or to.
2. It is not clear that the iam_policy, iam_role, and instance_profile are defined in AWS.
3. The links between iam_policy, iam_role, and instance_profile should probably be reversed, to indicate what references what.
4. The s3_object item transfers the downloaded application file, to the S3 bucket created by s3_bucket, but it isn't clear whether we should be highlighting the link, or the nested node.
5. If we highlight the link, then note that it is a forward link (data is pushed), compared to point 3 which are backward links (references).

Hosts, Realms, and Edges

If we could show:

1. The hosts of all network communication
2. The "realm" where resources live in, which may be a cloud provider or a region within
3. A node for each resource that is created/modified/deleted
4. For each item, the edges and nodes that interact

then what is happening becomes slightly clearer:

dot_ix

Notes

1. There is only one level of detail.
2. It is useful to have expandable detail, e.g. hide a full URL, and allow the user to expand it if they need to.
3. It is useful to show animated edges while that step is in progress, and hide them after.

impl Item for FileDownload {
    fn location_from(&self, params_partial: &<Self::Params<'_> as Params>::Partial)
    -> ItemLocation {
        let address = params_partial
            .src()
            .map(Url::host_str)
            .unwrap_or("<to_be_determined>")
            .to_string();
        let path = params_partial
            .src()
            .map(Url::path);

        ItemLocation {
            realm: Realm::Host { address },
            path,
        }
    }

    fn location_to(&self, params: &<Self::Params<'_> as Params>::Partial) -> ItemLocation {
        let address = "localhost".to_string();
        let path = params_partial
            .dest()
            .map(Path::to_string_lossy)
            .map(Cow::to_owned);

        ItemLocation {
            realm: Realm::Host { address },
            path,
        }
    }
}

impl Item for S3Bucket {
    fn location_from(&self, params_partial: &<Self::Params<'_> as Params>::Partial)
    -> ItemLocation {
        let address = "localhost"; // or None
        ItemLocation::Host { address }
    }

    fn location_to(&self, params: &<Self::Params<'_> as Params>::Partial) -> ItemLocation {
        ItemLocation::Cloud {
            cloud_provider: CloudProvider::Aws,
            global_or_region: GlobalOrRegion::Region(Region::ApSoutheast2),
            subnet: None,
            url: URL::from(".."),
            name: params.name.unwrap_or("<to_be_determined>"),
        }
    }
}

Render Technology

Currently:

1. Graphviz dot is used to render the SVG.
2. Tailwind CSS is used to define and generate styles.
3. leptos is used with axum to serve the graph in a web application.
4. dot_ix connects those together.

There are other use cases and desirable features that are difficult to implement with the current technology, namely:

1. Rendering the outcome on a command line interface.
2. Richer formatting, also by item implementors.
3. Controllable / predictable / consistent node positioning.
4. Serializing the outcome rendering, for later display.
5. Diffing two outcome diagrams.
6. Rendering errors and recovery hints.

Command Line Interface

Rendering the outcome on the CLI is not really necessary when the user is able to use a web browser to see the outcome. If the user is connected to a server via SSH, the tool could technically serve the web interface locally, and an SSH tunnel used to forward the traffic.

dioxus may allow us to have single source for both the web and CLI, but this is not known for sure -- plasmo, the crate that takes HTML and renders it on the terminal does not handle <input> elements on the terminal, so exploration is needed to determine if this is suitable.

Rich Formatting

For richer formatting, the Presentable trait was intended to capture this. However, serialization of different types makes the code complex -- I haven't figured out a way to transfer arbitrary types across the server to a client.

One option is to use markdown, and transfer the plain markdown to the target output, which can render it in its own means, e.g. syntect for CLI, and comrak or pulldown-cmark (what mdbook uses) to generate HTML for the web.

We would have to automatically nest markdown by indenting inner types' markdown.

Controllable Node Positioning

layout is a Rust port of a subset of dot. Essentially there are no HTML-like labels, but it is written in Rust. It likely has consistent node positioning, so using it over dot has that advantage, with the added benefits of performance and portability. vizdom uses it to generate graphs in real time.

Instead of using a dot-like library, generating elements with a flexbox layout, and drawing arrows may be the way to go.

Serializing Outcome Rendering

As long as we have a serializable form, which is stable, we can re-render the diagram later.

This means all information about the flow, parameters, and errors need to be serializable.

Diffing Outcome Diagrams

vizdom does this really nicely with dot graphs, see its examples.

Should we do a visual diff? Or clever node matching, then a styling diff?

Rendering Errors and Recovery Hints

For rendering errors, we need to know whether the error is to do with:

• a host / a function run by a host that failed
• a connection between hosts

then styling it.

For recovery hints, we need to make it clear where the error shown on the the outcome diagram is related to input parameters, whether it is a file, or a value produced by a host.

If we are using dot-like or a CSS-enabled technology, then we can style the relevant edge / node with Tailwind CSS styles.

Tailwind CSS generation

encre-css is likely what we will use, as it is TailwindCSS compatible.

HTML + Flexbox

<style>
.diagram {
    display: flex;
    flex: 0 1;
    align-items: stretch;
}
.focusable:focus {
    outline: 2px dashed #aabbee;
    outline-offset: 2px;
}
.node {
    border: 2px solid #999999;
    border-radius: 1.5rem;
    background-color: #dddddd;
    padding: 1.0rem 1.0rem;
}
.node:hover:not(:has(.node:hover)) {
    border-color: #bbbbbb;
    background-color: #f0f0f0;
}
.node:focus {
    border-color: #aaaaaa;
    background-color: #eeeeee;
}
.realm > .node {
    flex-grow: 1;
}
.node_vertical {
    display: flex;
    flex: 1 1;
    flex-wrap: wrap;
    flex-direction: column;
    align-content: stretch;
    gap: 1.0rem; /* Needs to match `.realm` `gap`. */
}
.realm {
    border: 2px dotted;
    border-radius: 1.5rem;
    display: flex;
    flex: 1 1 1;
    flex-wrap: wrap;
    align-content: flex-start;
    gap: 1.0rem; /* Needs to match `.node_vertical` `gap`. */
    padding: 1.0rem 1.0rem;
}
.realm_label {
    flex-basis: 100%;
    padding-bottom: 1.0rem;
}
.realm_localhost {
    border-color: #99bbcc;
    background-color: #cceeff;
}
.realm_localhost:hover:not(:has(.node:hover)) {
    border-color: #aaddee;
    background-color: #eeffff;
}
.realm_aws {
    border-color: #bbcc99;
    background-color: #eeffcc;
}
.realm_aws:hover:not(:has(.node:hover)) {
    border-color: #ddeeaa;
    background-color: #ffffee;
}
.realm_github {
    border-color: #cacaca;
    background-color: #fafafa;
}
.realm_github:hover:not(:has(.node:hover)) {
    border-color: #dbdbdb;
    background-color: #ffffff;
}
</style>

<div tabindex="0" class="focusable diagram">
    <div tabindex="0" class="focusable realm realm_github">
        <div class="realm_label" id="github" title="github">🐙 Github</div>
        <div tabindex="0" class="focusable node" id="github_app_download" title="app_download::src">📁 app.zip</div>
        <div tabindex="0" class="focusable node" id="other" title="other">..</div>
    </div>
    <div tabindex="0" class="focusable realm realm_localhost">
        <div class="realm_label" id="localhost" title="localhost">💻 Your computer</div>
        <div class="node_vertical">
            <div tabindex="0" class="focusable node" id="app_download" title="app_download">📥 app.zip</div>
            <div tabindex="0" class="focusable node" id="app_extract" title="app_extract">📂 /opt/app</div>
        </div>
    </div>
    <div tabindex="0" class="focusable realm realm_aws">
        <div class="realm_label" id="aws" title="aws">☁️ Amazon Web Services</div>
        <div class="node_vertical">
            <div tabindex="0" class="focusable node" id="s3_bucket" title="s3_bucket">
                <div class="realm_label">🪣 demo-artifacts</div>
                <div tabindex="0" class="focusable node" id="s3_object" title="s3_object">📁 app.zip</div>
            </div>
        </div>
        <div class="node_vertical">
            <div tabindex="0" class="focusable node" id="iam_policy" title="iam_policy">📝 EC2: Allow S3 Read</div>
            <div tabindex="0" class="focusable node" id="iam_role" title="iam_role">🔰 EC2 IAM policy attachment</div>
            <div tabindex="0" class="focusable node" id="instance_profile" title="instance_profile">🏷️ EC2 instance role attachment</div>
        </div>
    </div>
</div>

<script src="https://cdn.jsdelivr.net/npm/leader-line/leader-line.min.js"></script>
<script>
    new LeaderLine(
        document.getElementById('app_download'),
        document.getElementById('s3_object'),
        {
            color: '#336699',
            dash: {
                animation: true
            },
            size: 3,
            startSocketGravity: 20, // 100 by default
            endSocketGravity: 40,
            endPlugSize: 1.2
        }
    );
</script>

It is possible to produce a diagram in a similar style to dot using HTML elements and JS to draw an SVG arrow. The above uses leader-line, which has been archived by the original author.

It doesn't support adding an ID or CSS classes, so either we use its many configuration options, or we find another library which supports rendering arrows and setting classes.

Div Diag

Attempt at using dot_ix to generate a diagram using the same InfoGraph input model.

Desireables

1. Consistent layout.
2. Rendering arrows between nodes.
3. Styling nodes.
4. Styling arrows.
5. Consistent styling input with the DotSvg component.
6. Rendered SVG so that it can be uploaded as attachments and rendered inline, e.g. GitHub comments, mdbook.

Solutioning

Options

• A: Layout.
• B: Arrows.
• C: Styling.

🟢 A1: Flexbox + HTML to SVG Translation

1. HTML elements with Flexbox, easy to integrate rendered Markdown
2. Translate HTML to SVG.

🔴 A2: Render SVG directly

1. Need to implement layout algorithm that a browser already does.

   e.g. Flexbox, or somehow computer width/height with font metrics, unicode widths, potentially images.

🟡 B1: Arrow Overlay

1. Separately draw arrows as SVG over the laid out diagram as an overlay layer.
2. Redraw arrows when layout changes.

🟢 B2: Draw Arrows Inline

1. Layout diagram.
2. Convert to SVG.
3. Draw arrows within the SVG.

🟡 C1: CSS Utility Classes, e.g. TailwindCSS

1. Take class names directly from consumer / user.
2. Apply those to the HTML / SVG elements.

Requires element structure to be stable, and consumers to know the structure.

🟢 C2: CSS Utility Classes Keyed

1. Take in colours and border styles etc.
2. Prefix these with the appropriate structure, e.g. [>path]:, hover:, etc.
3. Apply these to the HTML / SVG elements.

Provides a stabler interface for users, and less knowledge of internals needed. Also a bit more freedom for maintainers.

Existing Tech

HTML to SVG

• ⚪ dom-to-svg: Typescript. Still need to try this.
• ⚪ vertopal: Python. Still need to try this.
• ⚪ Build one, with a cut down version of the diagram.

Online tools have not been able to produce an accurate SVG rendering.

Drawing Arrows

• 🟡 leader-line: Generates SVG arrows, with good configuration options. Out of the box, the SVG structure is not suitable for TailwindCSS classes.
• 🟡 Modify leader-line: Restructure the elements in the SVG, and add classes.
• ⚪ Build one, so that the elements are structured to match graphviz, and have classes added.

Styling

• 🟡 TailwindCSS: Versatile CSS library. See utility-first rationale.
• 🟢 encre-css: This is a TailwindCSS compatible Rust library.

API Design

Designed around 2024-07-13

Requirements

1. Be able to render a diagram, with/without the item existing.
2. Framework should be able to determine if an ItemLocation from B is:
   1. The same as an ItemLocation from A.
   2. Nested within an ItemLocation from A.
   3. Completely different.

Information Sources

Known information before / during deployment:

1. Flow edges -- Edge::Contains / Edge::Logic. Though maybe we want to reduce this to just Edge::Logic.
2. Flow item params that are specified.

Missing information before / during deployment:

1. State generated / produced / discovered through execution.
2. Parameters calculated from state.

Desired Outcome

What it should look like.

1. Item returns a Vec<ItemInteraction>, where an ItemInteraction denotes a push/pull/within for a given source/dest.
2. We need Item implementors to render a diagram with ParamsPartial, or some generated.
3. However, partial params means Item implementors may not have the information to instantiate the ItemLocation::{group, host, path}s for the Item.

Single Item Example

With values

From this item:

items:
  file_download: ! FileDownloadParams
    src: "https://example.com/file.zip"
    dest: "./target/abc/file.zip"

One of these diagrams is generated:

Which one is generated depends on how easy/difficult it is to figure out which node ports to use -- we don't necessarily know the node positions.

Also, we may not necessarily use dot, but a div_diag.

Without values

items:
  file_download: { src: "??", dest: "??" }

Multiple Items

With values

From these item params:

items:
  app_file_download: !FileDownloadParams
    src: "https://github.com/my_work_org/app/app_v1.zip"
    dest: "./target/customer/app_v1/app_v1.zip"

  config_file_download: !FileDownloadParams
    src: "https://my_work_org.internal/customer/app_v1/config.yaml"
    dest: "./target/customer/app_v1/config.yaml"

  app_file_upload: !S3ObjectParams
    file_path: "./target/customer/app_v1/app_v1.zip"
    bucket_name: "my-work-org-12345-ap-southeast-2"
    object_key: "/customer/solution/app/app_v1.zip"

  config_file_upload: !S3ObjectParams
    file_path: "./target/customer/app_v1/config.yaml"
    bucket_name: "my-work-org-12345-ap-southeast-2"
    object_key: "/customer/solution/app/config.yaml"

source

Without values

TODO

Learnings

State (Values) For Non-Existent Items

Options:

1. 🟡 A: Always get item implementors to return a value, but tagged as unknown or generated.

   Pros:

   1. 🟢 Can always generate a diagram and show what might be, even if we don't actually have values to do so.

   2. 🟢 Whether using dot or FlexDiag, the diagram layout will more consistent from the clean state to the final state if nodes are styled to be invisible, rather than not present. Cons:

   3. 🔴 What is generated can depend on input values, and if we have fake input values, we may generate an inaccurate diagram.

   4. 🟡 Choosing a default example value of "application version X" may cause a subsequent item to fail, because the application version doesn't exist.

      Item implementors would be constrained to not make any external calls.

      1. If we made every parameter value tagged with !Example vs !Real, then it can avoid this problem, but it is high maintenance on the item implementor.
      2. Maybe we pass in an additional parameter in Item::apply_dry so it isn't missed.
      3. Any !Example value used in a calculation can only produce !Example values.
      4. 🔵 If a dry run is intended to detect issues that would happen from an actual run, then we do want external systems to be called with the parameters passed in. e.g. detect credentials that are incorrect.
      5. 🔵 If a dry run is NOT intended to detect issues, then we will have simpler code implementation -- never make any external system calls (network, file IO).

   5. 🟡 Choosing a default cloud provider in one item, could make a nonsensical diagram if another item uses a different cloud provider.

      Note that an Item may still generate sensible example state based on example parameter values.

   6. 🔴 Code has complexity of either another derive macro generated type with RealOrExample<T> for each state field, or a wrapper for RealOrExample<Item::State>.

2. 🟡 B: Add Item::state_example, which provide fake state.

   Pros:

   1. 🟢 Can always generate a diagram and show what might be, even if we don't actually have values to do so.

   2. 🟢 Whether using dot or FlexDiag, the diagram layout will more consistent from the clean state to the final state if nodes are styled to be invisible, rather than not present. Cons:

   3. 🔴 Even more functions in the Item trait, creating more burden on item implementors.

   4. 🟡 Choosing a default cloud provider in one item, could make a nonsensical diagram if another item uses a different cloud provider.

      Note that an Item may still generate sensible example state based on example parameter values.

3. 🟡 C: Return Option<T> for each value that is unknown.

   Pros:

   1. 🟢 Never have false information in diagrams.

   2. 🟢 Code can put None for unknown values. Cons:

   3. 🔴 Unable to generate useful diagram when starting from a clean state. i.e. cannot visualize the fully deployed state before deploying anything.

   4. 🔴 Code has complexity of another derive macro generated type with Option<T> for each state field.

Let's go with B.

Notes From Designing Diagrams

1. The node ID must be fully derivable from the ItemLocation / parameter / state, i.e. we cannot randomly insert a middle group's name in the middle of the ID.
2. The node ID must be namespaced as much as possible, so two same paths on different hosts / groups don't collide.
3. The node ID must NOT use the flow's graph edges (sequence / nesting) in its construction. This is because flows may evolve -- more items inserted before/after, and the node ID should be unaffected by those changes.

Interaction Merging

1. Begin with multiple items' ItemLocations

source

2. Merge matching ItemLocations

source

3. Hide Edges (ItemInteractions)

source

4. Assign Nodes (ItemLocations) and (ItemInteractions) Edges to Tags

source

5. Animate and Show Edges (ItemInteractions) For Each Step

source

This can be used for both the example deployed state, as well as realtime execution.

Aesthetics and Clarity

Aesthetics helps reduce the "ick" in a diagram, reducing the impedance that a user experiences when viewing a diagram. Examples of icks are:

1. Visual clutter.
2. A node's dimensions being significantly different compared to other nodes.
3. Oversaturated colour.
4. Colours that are excessively solid for a less relevant concept.

Clarity is how well a user understands the information that a diagram is presenting. Examples of adding clarity are:

1. (Understandable) visual cues such as emojis in place of text.
2. Reducing detail to what is most relevant.

Consider the last diagram from Interaction Merging:

source

The following things that could make the diagram more digestable:

1. Aesthetic: Reduce the visual length of the presented URL.

   1. For a github repository, separating the username/repo into a separate group can be informative.
   2. Showing the initial and last characters of the URL, while hiding the middle using ellipses may make it more aesthetic at a glance, though it may hinder if a user wants to see the full URL.

2. Clarity: Add an emoji indicating that 012345678901-ap-southeast-2-releases is an S3 bucket.

Compare the above with the following:

source

Notes:

1. The URL is shortened into the path after the username/repo This requires the FileDownload item to know that the first 2 segments is a group namespace.
2. The 🪣 bucket emoji clarified that the 012345678901-ap-southeast-2-releases node represents an S3 bucket.

Endpoints and Interaction

Background

For a web UI, we want:

1. A way to invoke *Cmds -- invoke and return an ID.
2. Command invocation request is idempotent / does not initiate another invocation when one is in progress.
3. A way to interrupt CmdExecutions -- get the Sender<InterruptSignal> by execution ID, send.
4. A way to pull progress from the client -- close / reopen tab, send URL to another person.
5. A way to push progress to the client -- for efficiency. web sockets?
6. For both a local and shared server, a way to open a particular env/Flow/CmdExecution by default.
7. For a shared server, a way to list CmdExecutions.
8. For a local server, to automatically use different ports when running executions in different projects.

Glossary

Cmd Invocation

Need to cater for:

1. CLI usage: Invocation returns the result.
2. Web usage: Invocation returns an ID.
3. Any usage: For a given profile, two CmdExecutions cannot run at the same time, even for different flows.
4. Web usage: For a given profile, re-invocation returns existing in-progress CmdExecution ID. This can be deferred if two browser tabs for the same workspace + profile combination both disable the deploy button when a CmdExecution is initiated.

Option A1: exec delegates to request_exec

For the CLI usage, to reduce code duplication *Cmds can provide function that return the Result<CmdOutcome, ..>, where internally it calls the method that returns an execution ID, but immediately waits for that execution's completion.

#![allow(unused)]
fn main() {
pub async fn exec<'ctx>(
    cmd_ctx: &mut CmdCtx<SingleProfileSingleFlow<'_, CmdCtxTypesT>>,
) -> Result<CmdOutcome<_, CmdCtxTypesT::AppError>, CmdCtxTypesT::AppError>
where
    CmdCtxTypesT: 'ctx,
{
    let execution_id = Self::request_exec(cmd_ctx);

    executions.get(execution_id).await
}
}

Option A2: request_exec delegates to exec

pub async fn request_exec<'ctx>(
    cmd_ctx: &mut CmdCtx<SingleProfileSingleFlow<'_, CmdCtxTypesT>>,
) -> ExecutionId
where
    CmdCtxTypesT: 'ctx,
{
    if let Some(execution_id) = executions.get((workspace, profile)) {
        return execution_id;
    };

    let execution_id = server.generate_execution_id(workspace, profile).await;
    let cmd_execution = Self::exec(cmd_ctx);
    // or send(..) the execution request to a queue, and the queue receiver calls the `exec`.

    executions.put(execution_id, cmd_execution).await

    execution_id
}

Web Interface

Web Server:

• Needs to hold a collection of all executions.

• Needs to hold mapping from Execution ID to CmdExecution, and/or parts of the CmdExecution.

  Storing parts separately can with access and extensibility:

  • Sometimes we don't want to borrow the full CmdExecution, only part of it.
  • Adding new things gets stored in a different server context state, so components that are not concerned with the new state don't need to access it.

  Need to make sure all context is added in the same place, otherwise it is difficult to track "what makes up a CmdExecution".

1. Web Server CmdExecutions Tracking

• CmdExecutions is the collection of in-progress executions, not just their serializable info.

  Possibly a LinkedHashMap<ExecutionId, Box<dyn CmdExecutionRt>>, where CmdExecutionRt is a trait over the concrete CmdExecutions which are type parameterized.

• CmdExecutionsInfo is a serializable collection of both in-progress and historical execution infos.

  Possibly a LinkedHashMap<ExecutionId, CmdExecutionInfo>.

// Web Server set up needs to track everything
// or, link to a database that tracks everything
let cmd_executions = CmdExecutions::default();
let router = Router::new()
    // ..
    .leptos_routes_with_context(
        &leptos_options,
        routes,
        move || {
            // ..
            leptos::provide_context(Arc::clone(cmd_executions));
        },
        move || view! {  <Home /> },
    )
    // ..
    ;

2. Web Component CmdExecutionInfos Access For Display

CmdExecutionInfos is the serializable type used to represent CmdExecutions for display:

/// Returns the list of `CmdExecutions` that have run / are in-progress on the server.
#[leptos::server(endpoint = "/cmd_execution_infos")]
pub async fn cmd_execution_infos(
) -> Result<CmdExecutionInfos, ServerFnError<NoCustomError>> {
    let cmd_execution_infos = leptos::use_context::<CmdExecutionInfos>()
        .ok_or_else(|| {
            ServerFnError::<NoCustomError>::ServerError(
                "`CmdExecutionInfos` was not set.".to_string()
            )
        })?;

    Ok(cmd_execution_infos)
}

#[component]
pub fn CmdExecutionsList() -> impl IntoView {
    let cmd_execution_infos_resource = leptos::create_resource(
        || (),
        move |()| async move { cmd_execution_infos().await.unwrap() },
    );
    let cmd_execution_infos = move || {
        cmd_execution_infos_resource
            .get()
            .expect("Expected `cmd_execution_infos` to always be generated successfully.")
    };

    view! {
        <Transition fallback=move || view! { <p>"Loading..."</p> }>
            <For each=cmd_execution_infos /* display the info */ />
        </Transition>
    }
}

3. Web Component CmdExecutions Access For *Cmd Invocation

Given a workspace, profile, flow_id, a *Cmd and *Cmd parameters, a user should be able to send a CmdExecutionRequest. Some of these parameters should be able to be defaulted, e.g. for a local automation server which is run from the workspace directory.

Should create_action or create_server_action be used?

#[server(endpoint = "check_code")]
pub async fn check_code(s: Uuid, c: Code) -> Result<UserMetadata, ServerFnError> {
    todo!();
}

let check_action = create_server_action::<CheckCode>();
<ActionForm action=check_action class="mx-auto px-6 py-4 rounded-xl bg-white max-w-[400]">
    <input type="hidden" id="s" name="s" value=s />
    <div class="mb-4">
        <label for="c" class="block text-md text-gray-700">
            Verification Code
        </label>
        <input
            class="various tailwind"
            id="c" name="c" prop:value=c required type="number" placeholder="6 digit code"
            on:input=move |ev| c.set(event_target_value(&ev))/>
    </div>
    <ErrorDisplay res=check_action />
    <div class="mb-6">
        <p class="text-sm my-1 text-grey-600" hidden=move || email().is_none()>
            Sent code to {move || email()}
        </p>
        <a class="text-sm my-1 text-grey-600 hover:underline" href="loginhelp">
            "Didn't get an email?"
        </a>
    </div>
    <button type="submit" disabled=check_action.pending()
        class="various tailwind">
        SUBMIT
    </button>
</ActionForm>

Example

#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)]
pub struct EnsureCmdArgs {
    workspace: Workspace,
    profile: Profile,
    flow_id: FlowId,
}

pub struct CmdExecutionQueues(HashMap<(Workspace, Profile), Sender<CmdExecutionRequest>>);

#[leptos::server(endpoint = "/ensure_cmd")]
pub async fn ensure_cmd(
    ensure_cmd_args: EnsureCmdArgs,
) -> Result<ExecutionId, ServerFnError<NoCustomError>> {
    let cmd_execution_queues = leptos::use_context::<CmdExecutionQueues>()
        .ok_or_else(|| {
            ServerFnError::<NoCustomError>::ServerError(
                "`Sender<CmdExecutionRequest>` was not set.".to_string()
            )
        })?;
    let cmd_execution_infos = leptos::use_context::<CmdExecutionInfos>()
        .ok_or_else(|| {
            ServerFnError::<NoCustomError>::ServerError(
                "`Sender<CmdExecutionInfos>` was not set.".to_string()
            )
        })?;

    let execution_id = cmd_execution_queues.get(&(workspace, profile))
        .map(|cmd_execution_req_tx| {
            let execution_id = ExecutionId::new_rand();
            let cmd_execution_req = CmdExecutionReq {
                execution_id,
                ensure_cmd_args,
            };

            let cmd_execution_info = CmdExecutionInfo::new(execution_id, ensure_cmd_args);
            cmd_execution_infos.insert(execution_id, cmd_execution_info);

            cmd_execution_req_tx.send(cmd_execution_req).await;

            execution_id
        })
        .ok_or_else(|| {
            ServerFnError::<NoCustomError>::ServerError(
                format!("No `CmdExecutionQueue` for {workspace} {profile}.")
            )
        });

    Ok(execution_id)
}

#[component]
pub fn EnsureButton() -> impl IntoView {
    let ensure_cmd = leptos::create_action(
        |workspace: Workspace, profile: Profile, flow_id: FlowId| {
            let execution_id = execution_id.clone();
            async move { ensure_cmd(EnsureCmdParams { workspace, profile, flow_id }).await }
        },
    );
    let submitted = ensure_cmd.input(); // RwSignal<Option<String>>
    let pending = ensure_cmd.pending(); // ReadSignal<bool>
    let todo_id = ensure_cmd.value(); // RwSignal<Option<Uuid>>

    view! {
        <form
            on:submit=move |ev| {
                ev.prevent_default(); // don't reload the page.
                ensure_cmd.dispatch();
            }
        >
            // Execution ID
            <button type="submit">"Deploy"</button>
        </form>
        // use our loading state
        <p>{move || pending().then("Loading...")}</p>
    }
}

Interruption

User has sent an interruption request. These are received differently depending on the Output that presented the interface to the user.

• CLI:

  • The developer needs to define the SIGINT handler, and pass it into Peace (1, 2).
  • The user presses Ctrl C to interrupt the command execution.

• Web:

  • The framework needs to track an execution ID to the CmdExecution's interrupt sender.
  • User sends an interrupt request with the execution ID.

Implementation

This follows on from the Cmd Invocation > Web Interface design.

4. Web Component InterruptSenders Access

InterruptSenders may be a newtype for Map<ExecutionId, Sender<InterruptSignal>>

See:

• leptos: actions.
• leptos_router::ActionForm
• leptos::create_server_action
• todo_app_sqlite

#[leptos::server(endpoint = "/cmd_exec_interrupt")]
pub async fn cmd_exec_interrupt(
    execution_id: &ExecutionId,
) -> Result<(), ServerFnError<NoCustomError>> {
    let interrupt_senders = leptos::use_context::<InterruptSenders>()
        .ok_or_else(|| {
            ServerFnError::<NoCustomError>::ServerError(
                "`InterruptSenders` was not set.".to_string()
            )
        })?;
    if let Some(interrupt_sender) = interrupt_senders.get(execution_id) {
        interrupt_sender.send(InterruptSignal).await;
    }

    Ok(())
}

#[component]
pub fn InterruptButton(
    execution_id: ReadSignal<ExecutionId>,
) -> impl IntoView {
    let cmd_exec_interrupt_action = leptos::create_action(
        |execution_id: &ExecutionId| {
            let execution_id = execution_id.clone();
            async move { cmd_exec_interrupt(&execution_id).await }
        },
    );
    let submitted = cmd_exec_interrupt_action.input(); // RwSignal<Option<String>>
    let pending = cmd_exec_interrupt_action.pending(); // ReadSignal<bool>
    let todo_id = cmd_exec_interrupt_action.value(); // RwSignal<Option<Uuid>>

    view! {
        <form
            on:submit=move |ev| {
                ev.prevent_default(); // don't reload the page.
                cmd_exec_interrupt_action.dispatch(execution_id.get());
            }
        >
            // Execution ID
            <button type="submit">"Interrupt"</button>
        </form>
        // use our loading state
        <p>{move || pending().then("Loading...")}</p>
    }
}

Progress Output

See also Execution Progress.

The way progress is transferred to the user varies based on the OutputWrite and build target.

1. CLI: This is pushed straight to the terminal
2. Web: This is pulled by the client from the server based on execution ID.
3. WASM: This is pushed by the WASM binary within the client.

Implementation

Web Interface

#[component]
pub fn FlowGraph(execution_id: ReadSignal<ExecutionId>) -> impl IntoView {
    let progress_dot_resource = leptos::create_resource(
        || (),
        move |()| async move { progress_dot_graph(execution_id.get()).await.unwrap() },
    );
    let progress_dot_graph = move || {
        let progress_dot_graph = progress_dot_resource
            .get()
            .expect("Expected `progress_dot_graph` to always be generated successfully.");

        Some(progress_dot_graph)
    };

    view! {
        <form
            on:submit=move |ev| {
                ev.prevent_default(); // don't reload the page.
                cmd_exec_interrupt_action.dispatch(execution_id.get());
            }
        >
            // Execution ID
            <button type="submit">"Interrupt"</button>
        </form>
        // use our loading state
        <p>{move || pending().then("Loading...")}</p>
    }
}

/// Returns the graph representing item execution progress.
#[leptos::server(endpoint = "/flow_graph")]
pub async fn progress_dot_graph(execution_id: ExecutionId) -> Result<DotSrcAndStyles, ServerFnError<NoCustomError>> {
    use dot_ix::{
        model::common::{graphviz_dot_theme::GraphStyle, GraphvizDotTheme},
        rt::IntoGraphvizDotSrc,
    };
    use peace_flow_model::FlowSpecInfo;

    let flow_spec_info = leptos::use_context::<FlowSpecInfo>().ok_or_else(|| {
        ServerFnError::<NoCustomError>::ServerError("`FlowSpecInfo` was not set.".to_string())
    })?;
    let cmd_progress_trackers = leptos::use_context::<CmdProgressTrackers>().ok_or_else(|| {
        ServerFnError::<NoCustomError>::ServerError("`CmdProgressTrackers` was not set.".to_string())
    })?;
    if let Some(cmd_progress_tracker) = cmd_progress_trackers.get(&execution_id) {
        // TODO: adjust styles on graph.
    }

    let progress_info_graph = flow_spec_info.into_progress_info_graph();
    Ok(IntoGraphvizDotSrc::into(
        &progress_info_graph,
        &GraphvizDotTheme::default().with_graph_style(GraphStyle::Circle),
    ))
}

Workspace

A tool built using the Peace framework must execute in a workspace – a location that Peace reads and writes information when commands are executed.

In a native automation tool, the workspace is usually the repository in which the automation is run, so the workspace directory is the repository root. In a WASM tool, Peace currently is able to store information in the browser LocalStorage or SessionStorage.

Peace Data

The following shows the file structure for data stored by Peace.

Path Segments

Hierarchy

$workspace_dir  # usually the project repository
|
|  # .peace, single directory to store data from tools made with Peace
|- $peace_dir
    |
    |  # directory per tool
    |- $peace_app_dir
        |- 📝 $workspace_params_file
        |
        |  # profile name, directory per profile
        |- 🌏 $profile_dir
            |- 📝 $profile_params_file
            |
            |  # flow name, directory per flow
            |- 🌊 $flow_dir
                |- 📝 $flow_params_file
                |- 📋 $states_current_file
                |- 📋 $states_goal_file

Concrete Hierarchy Example

path/to/repo
|- .peace
    |- envman
        |- 📝 workspace_params.yaml
        |
        |- 🌏 internal_dev_a
        |   |- 📝 profile_params.yaml
        |   |
        |   |- 🌊 deploy
        |   |   |- 📝 flow_params.yaml
        |   |   |- 📋 states_goal.yaml
        |   |   |- 📋 states_current.yaml
        |   |
        |   |- 🌊 config
        |   |   |- 📝 flow_params.yaml
        |   |   |- 📋 states_goal.yaml
        |   |   |- 📋 states_current.yaml
        |   |
        |   |- 🌊 benchmark
        |       |- 📝 flow_params.yaml
        |       |- 📋 states_goal.yaml
        |       |- 📋 states_current.yaml
        |
        |- 🌏 customer_a_dev
        |   |- 📝 profile_params.yaml
        |   |
        |   |- 🌊 deploy - ..
        |   |- 🌊 config - ..
        |
        |- 🌏 customer_a_prod
            |- 📝 profile_params.yaml
            |
            |- 🌊 deploy - ..
            |- 🌊 config - ..

Commands

Commands in Peace are categorized based on how they interact with a workspace. Make sure you understand the Peace workspace model.

The information available within a command execution is called the command context. Depending on the scope of a command, the available information changes. Peace's command API adaptively exposes methods that are relevant to the chosen scope.

This section covers:

• Scopes: What information is accessible for each command scope.
• Use Cases: Which scope to use and how to create different kinds of commands.

Scopes

The scope of a command determines:

• If it has access to zero, one, or many Profile directories.
• If it has access to zero, one, or many Flow directories.
• Whether it can deserialize profile params.
• Whether it can deserialize flow params.
• Whether it can deserialize states from each flow directory.

Scenario: envman is an example tool that manages server environments.

In the following directory structure:

• internal_dev_a, customer_a_dev, customer_a_prod are all separate Profiles.
• deploy, config, and benchmark are separate Flows.
• Each flow tracks its own flow_params.yaml, states_goal.yaml, and states_current.yaml

# peace app dir
path/to/repo/.peace/envman
|- 📝 workspace_params.yaml
|
|- 🌏 internal_dev_a
|   |- 📝 profile_params.yaml
|   |
|   |- 🌊 deploy
|   |   |- 📝 flow_params.yaml
|   |   |- 📋 states_goal.yaml
|   |   |- 📋 states_current.yaml
|   |
|   |- 🌊 config
|   |- 🌊 benchmark
|
|- 🌏 customer_a_dev
|   |- 📝 profile_params.yaml
|   |
|   |- 🌊 deploy - ..
|   |- 🌊 config - ..
|
|- 🌏 customer_a_prod
    |- 📝 profile_params.yaml
    |
    |- 🌊 deploy - ..
    |- 🌊 config - ..

See each page for details of each scope:

• No Profile No Flow: Commands that only work with workspace parameters.
• Single Profile No Flow: Commands that work with a single profile, without any items.
• Single Profile Single Flow: Commands that work with one profile and one flow.
• Multi Profile No Flow: Commands that work with multiple profiles, without any items.
• Multi Profile Single Flow: Commands that work with multiple profiles, and a single flow.

Reborrowing

Reborrowing is used in scopes to so that the View structs don't hold onto the borrow for the length of 'ctx, but only for the 'view lifetime.

#[derive(Debug)]
struct Life<'life> {
    value: &'life mut u8,
}

impl<'life> Life<'life> {
    pub fn reborrow<'short>(&'short mut self) -> Life<'short> {
        Life {
            value: &mut self.value,
        }
    }
}

// Used in
struct Scope<'life> {
    life: Life<'life>,
    other: u16,
}

struct ScopeView<'view> {
    life: Life<'view>,
    other: &'view mut u16,
}

impl<'life> Scope<'life> {
    pub fn view(&mut self) -> ScopeView<'_> {
        let Scope { life, other } = self;

        // Needed to shorten the lifetime of `'life`.
        let life = life.reborrow();

        ScopeView { life, other }
    }
}

fn main() {
    let mut value = 123;

    let life = Life { value: &mut value };
    let mut many_data = Scope { life, other: 456 };

    let ScopeView {
        life: _life,
        other: _other,
    } = many_data.view();
}

Use Cases

This section covers the kinds of commands that Peace is intended to support. While it is still a work in progress, the aim is for all of these commands to be easily constructed.

• Workspace Initialization: Like git init.
• Profile List: Like git branch.
• Workspace Stored Active Profile: Like git switch -c $branch.
• Workspace Active Profile Switch: Like git switch $branch.
• State Discovery and Display: Like git fetch.
• State Read and Display: Like git show.
• State Diff: Like git status / git diff.
• State Apply: Like git commit / git push / git clean, depending on your perspective.
• State Clean: Like rm -rf.

Workspace Initialization

This is like git init.

When initializing a project workspace, the suitable scope is different depending on the amount of work done:

Command Creation

• If it only stores parameters provided by the user, shared across profiles, and does not use any item, use NoProfileNoFlow.

  To create this command:

  1. Build the command context with the provided parameters. When the command context is built:
  • Workspace parameters are written to workspace_params.yaml.

• If it stores parameters for a default profile, and does not use any item, use SingleProfileNoFlow.

  To create this command:

  1. Build the command context with the provided parameters. When the command context is built:
  • Workspace parameters are written to workspace_params.yaml.
  • Profile parameters are written to profile_params.yaml.

• If it does any repeatable work, such as download files or clone a repository, use SingleProfileSingleFlow.

  To create this command:

  1. Build the command context with the provided parameters.
  2. Call StateDiscoverCmd::exec to discover the current and goal states.
  3. Call EnsureCmd::exec to execute the flow. When the command context is built:
  • Workspace parameters are written to workspace_params.yaml.
  • Profile parameters are written to profile_params.yaml.
  • Flow parameters are written to flow_params.yaml.

Profile List

This is like git branch.

When listing profiles, the most suitable scope is MultiProfileNoFlow.

There are a number of variations of profile listings:

• Simple: Simple list of profiles within the PeaceAppDir.

• Filtered: List profiles within the PeaceAppDir for some criteria, e.g. excluding a workspace_init profile.

  This is especially relevant when profile parameters are loaded, as the ProfileParams must be the same across the listed profiles to be able to be loaded.

• Augmented: List profiles and some profile parameters.

  This is valuable when the user needs to see the profile parameter, such as whether the profile is managing a development environment or a production environment.

Command Creation

To create this command:

1. Build the command context.

   • Make sure to register the workspace / profile params types, even if the value for all of them is None.
   • Provide the filter function if necessary.

2. Present the list of profiles using the goal output.

Workspace Stored Active Profile

This is like git switch -c $branch.

This stores the profile to use in workspace_params, so the use does not have to provide the profile on every command invocation.

Suitable scopes for this command are NoProfileNoFlow or SingleProfileNoFlow.

Command Creation

To create this command:

1. When building the command context:

   • Include a Profile as a workspace param.
   • If setting profile parameters, use the SingleProfileNoFlow scope, and set the current profile using with_profile.

2. When building the command context, load the profile from workspace params using: with_profile_from_workspace_params.

It is best practice to set the active profile during workspace initialization, so that it is never None when subsequent commands are invoked.

Workspace Active Profile Switch

This is like git switch $branch.

This changes the stored profile in workspace_params, used in conjunction with the above.

Suitable scopes for this command are NoProfileNoFlow or SingleProfileNoFlow.

Command Creation

To create this command:

1. When building the command context:

   • Include a Profile as a workspace param.
   • Also set the current profile using with_profile.

When the command context is built, the default profile is stored in workspace_params.yaml.

State Discovery and Display

This is like git fetch.

This kind of command is intended for state discovery that is expensive – takes more than 500 milliseconds.

Suitable scopes for this command are:

• SingleProfileSingleFlow: For discovering state for one profile.
• MultiProfileSingleFlow: For discovering state for multiple profiles.

Command Creation

To create this command:

1. When building the command context:

   • Provide the profile.
   • Provide the flow ID.

2. Call the StatesDiscoverCmd depending on the intended use:

   These will store the discovered states under the corresponding $profile/$flow_id directory as states_current.yaml or states_goal.yaml.

3. If the discovered states are to be displayed, call the relevant state display command(s):

   • StatesCurrentStoredDisplayCmd: For stored current states to be displayed.
   • StatesGoalDisplayCmd: For goal states to be displayed.

State Read and Display

This is like git show.

This kind of command is intended to display state that has previously been discovered.

Suitable scopes for this command are:

• SingleProfileSingleFlow: For reading state for one profile.
• MultiProfileSingleFlow: For reading state for multiple profiles.

Command Creation

To create this command:

1. When building the command context:

   • Provide the profile.
   • Provide the flow ID.

2. Call one of the state read commands depending on the intended use:

   These will store the discovered states under the corresponding $profile/$flow_id directory as states_current.yaml or states_goal.yaml.

   • StatesCurrentReadCmd: For current states to be discovered.
   • StatesGoalReadCmd: For goal states to be discovered.

3. Call the relevant state display command(s):

   • StatesCurrentStoredDisplayCmd: For current states to be displayed.
   • StatesGoalDisplayCmd: For goal states to be displayed.

State Diff

This is like git status / git diff.

git status is a diff between the latest commit, and the working directory, summarized as one line per file.

git diff is that same diff, but shown with more detail.

This kind of command shows the difference between two States for each managed item.

Typically this is the difference between the current or current stored state, and its goal state. It can also be the difference between the current states between two profiles.

Suitable scopes for this command are:

• SingleProfileSingleFlow: For diffing state within one profile.
• MultiProfileSingleFlow: For diffing state across multiple profiles.

Command Creation

To create this command for a single profile:

1. When building the command context:

   • Provide the profile.
   • Provide the flow ID.

2. Determine the "from" and "to" states:

   The "from" state is usually one of:

   • Discovering the current state.

   • Reading states_current.yaml. The "to" state is usually one of:

   • Discovering the goal state.

   • Reading states_goal.yaml.

   • The clean state.

3. Call the state DiffCmd.

To create this command for multiple profiles:

1. When building the command context:

   • Filter the profiles if necessary.
   • Provide the flow ID.

2. Determine the "from" and "to" states:

   The "from" state is usually one of:

   • Discovering the current state.

   • Reading states_current.yaml. The "to" state is usually one of:

   • Discovering the goal state.

   • Reading states_goal.yaml.

   • The clean state.

3. Call the state DiffCmd for each pair of states.

State Apply

This is like git commit / git push / git clean, depending on your perspective.

This kind of command applies the goal state over the current state.

This generally requires what is stored in states_current.yaml to match the newly discovered current state.

The only suitable scope for this command is SingleProfileSingleFlow.

Command Creation

To create this command:

1. When building the command context:

   • Provide the profile.
   • Provide the flow ID.

2. Call the EnsureCmd.

   This will call ApplyFns::exec for each item, beginning from the first item, until the last item.

State Clean

This is like rm -rf.

This kind of command applies the clean state over the current state.

This generally requires what is stored in states_current.yaml to match the newly discovered current state.

The only suitable scope for this command is SingleProfileSingleFlow.

Command Creation

To create this command:

1. When building the command context:

   • Provide the profile.
   • Provide the flow ID.

2. Call the CleanCmd.

   This will call CleanOpSpec::exec for each item, beginning from the last item, until the first item.

Cmd Execution

When we run a CmdExecution, possible outcomes are:

1. In a CmdBlock, failure within one or more items.
2. In a CmdBlock, failure within block code.
3. Interruption / partial execution.
4. Successful execution of all blocks.

Each CmdBlock has its own block outcome type, and its own .

Framework

Use Cases: Framework Implementor

*Cmd Impls

• It would be nice to not have to write interruption handling code in every CmdBlock.
• Also, having CmdBlocks abstracted allows timings to be collected per block in a common place.

What does a CmdExecution implementor want to do?

• DiffCmd:

  • StateDiscoveryBlock: Return item failures to caller.
  • StateDiscoveryBlock: Return block failure / interruption to caller.
  • DiffBlock: Return item failures to caller.
  • DiffBlock: Return block failure / interruption to caller.

• EnsureCmd:

  • StateDiscoveryBlock: Return item failures to caller.

  • StateDiscoveryBlock: Return block failure / interruption to caller.

  • DiffBlock: Return item failures to caller.

  • DiffBlock: Return block failure / interruption to caller.

  • EnsureBlock: Serialize States on item failure, return States and item failures to caller.

  • EnsureBlock: Return block failure / interruption to caller.

  • CleanBlock: Serialize States on item failure, return States and item failures to caller.

  • CleanBlock: Return block failure / interruption to caller.

  • EnsureBlock: ditto.

  • CleanBlock: ditto. The desired return type would be:

  • StatesEnsured on success.

• How should we reply to the caller?

  • CmdBlock block level error.
  • IndexMap<ItemId, E> item level error. Error handlers per CmdBlock will do any serialization work if desired.

Use Cases: Automation Developer / End User

• Store, retrieve, and display execution report.

How do we do this in code?

What it could look like conceptually:

DiffCmd

#![allow(unused)]
fn main() {
let cmd_execution = CmdExecution::builder()
    .with_block(StatesDiscoverCmdBlock::<CurrentAndGoal>::new())
    .with_block(DiffCmdBlock::new())
    .build()
    .await;
}

EnsureCmd -- "simple" version

#![allow(unused)]
fn main() {
let cmd_execution = CmdExecution::builder()
    .with_block(StatesCurrentReadCmdBlock::new())
    .with_block(StatesDiscoverCmdBlock::new(Discover::Current))
    // Compares current with stored, and fails if
    // current stored doesn't match current discovered.
    .with_block(ApplyGuardCmdBlock::new())  // `Outcome` and `OutcomeAcc` are `()`
    .with_block(ApplyCmdBlock::new())       // state_goal, diff and apply.
    .build()
    .await;
}

EnsureCmd -- "complex" version

#![allow(unused)]
fn main() {
let cmd_execution = CmdExecution::builder()
    .with_block(StatesCurrentReadCmdBlock::new())
    .with_block(StatesDiscoverCmdBlock::new(Discover::Current))
    .with_block(ApplyGuardCmdBlock::new())
    .with_block(EnsureCmdBlock::new(item_ids_no_blockage))  // state_goal, diff and apply.
    .with_block(CleanCmdBlock::new(item_ids_blocking))    // state_clean, diff and apply.
    .with_block(EnsureCmdBlock::new(item_ids_unblocked))  // state_goal, diff and apply.
    .with_block(CleanCmdBlock::new(item_ids_obsolete))    // state_clean, diff and apply.
    .build()
    .await;

// `item_ids_unblocked` is a superset of the `item_ids_blocking`.
//
// ⚠️ We could have a more granular design version that starts ensuring unblocked items
// while `cleaning` is happening.
}

Error Handling and Interruptions

When a CmdBlock fails, we need to consider what to do with:

• CmdBlock::OutcomeAcc.
• item errors.
• CmdBlock error.
• interruptions.

Originally an additional closure argument next to each CmdBlock argument was considered:

|_input, _outcome_acc, cmd_block_error| async move { cmd_block_error }

For example:

|(_states_current, _states_goal), states_ensured_mut, apply_error| async move {
    let serialize_result = StatesSerializer::serialize(states_ensured_mut).await;

    apply_error.or(serialize_result)
},

However, most usages would just pass through the error.

Before this CmdExecution design, ApplyCmd would:

1. In the producer, send the intermediate result to the outcome collator and stop execution (apply_cmd.rs#L699-L710).

2. In the collator:

   • Intermediate CmdBlock errors are returned as is. (apply_cmd.rs#L521-L525).
   • Intermediate discovery errors are collated into the CmdExecution outcome type (apply_cmd.rs#L526-L558).

From the use cases, what we care about are:

• Returning CmdBlock errors to the caller.
• Returning item errors to the caller.
• Collating intermediate CmdBlock item values into the CmdExecution outcome type if possible.
• Serializing States to storage: EnsureCmdBlock and CleanCmdBlock should still serialize each item's current state, on failure / interruption / success.
• Serializing the CmdBlock::Outcome in the execution outcome report.

Deferred:

• Serializing the CmdBlock::OutcomeAcc in the execution outcome report.
• Deserializing the CmdBlock::OutcomeAcc to be displayed.

Interruptions

1. Interrupt before execution.
2. Interrupt within a block.
3. Interrupt between blocks.
4. Interrupt after last block.

Interruptibility

How should the interrupt channel be initialized and stored?

Use Cases

When automation software is used as:

1. CLI interactive / non-interactive.
2. Web service + browser access.
3. CLI + web service + browser access.

In all cases, we need to:

1. Initialize the CmdCtx with the interrupt_rx
2. Spawn the listener that will send InterruptSignal in interrupt_tx.

Imagined Code

CLI Interactive / Non-interactive

Both interactive and non-interactive can listen for SIGINT:

• Interactive: SIGINT will be sent by the user pressing Ctrl + C.
• Non-interactive: SIGINT could be sent by a CI thread.

#![allow(unused)]
fn main() {
let (interrupt_tx, interrupt_rx) = oneshot::channel::<InterruptSignal>();

tokio::task::spawn(async move {
    // Note: Once tokio takes over the process' `SIGINT` handler, it cannot be undone.
    //
    // This limitation is due to how Linux currently works.
    tokio::signal::ctrl_c()
        .await
        .expect("Failed to initialize signal handler for SIGINT");

    let (Ok(()) | Err(InterruptSignal)) = interrupt_tx.send(InterruptSignal);
});

let mut cmd_ctx = CmdCtx::single_profile_single_flow(output, workspace, interrupt_rx)
    .build();

let cmd_outcome = EnsureCmd::exec(&mut cmd_ctx).await?;
}

Web Service

The interrupt_tx must be accessible from a separate web request.

#![allow(unused)]
fn main() {
async fn cmd_exec_start_handler(params: Params) -> CmdExecutionId {
    let (interrupt_tx, interrupt_rx) = oneshot::channel::<InterruptSignal>();
    let mut cmd_ctx = CmdCtx::single_profile_single_flow(output, workspace, interrupt_rx)
        .build();

    let cmd_execution_id = EnsureCmd::exec_bg(cmd_ctx);

    let cmd_execution_by_id = cmd_execution_by_id
        .lock()
        .await;
    cmd_execution_by_id.insert(cmd_execution_id, interrupt_tx);

    cmd_execution_id
}

/// Returns the progress of the `CmdExecution`.
async fn cmd_exec_progress_handler(cmd_execution_id: CmdExecutionId) -> Result<CmdProgress, E> {
    self.cmd_progress_storage.get(cmd_execution_id).await
}

async fn cmd_exec_interrupt_handler(cmd_execution_id: CmdExecutionId) -> Result<(), E> {
    let cmd_execution_by_id = cmd_execution_by_id
        .lock()
        .await;

    if let Some(interrupt_tx) = cmd_execution_by_id.get(cmd_execution_id) {
        let (Ok(()) | Err(InterruptSignal)) = interrupt_tx.send(InterruptSignal);

        Ok(())
    } else {
        Err(E::from(Error::CmdExecutionIdNotFound { cmd_execution_id }))
    }
}
}

CLI + Web Service

There are two variants of CLI and web service:

1. CLI command running on the user's machine, web service that is a UI for that one command execution.
2. CLI client to a web service, so the CLI is just a REST client.

CLI on User's Machine + Web UI

For the first variant, the CmdExecution invocation is similar to Web Service, with the following differences:

• Output progress is pushed to both CLI and CmdProgress storage.
• Interruptions are received from both process SIGINT and client requests.

#![allow(unused)]
fn main() {
async fn cmd_exec_start(params: Params) {
    let (interrupt_tx, interrupt_rx) = oneshot::channel::<InterruptSignal>();
    let mut cmd_ctx = CmdCtx::single_profile_single_flow(output, workspace, interrupt_rx)
        .build();

    let cmd_execution_id = EnsureCmd::exec_bg(cmd_ctx);

    // We store an `interrupt_tx` per `CmdExecutionId`,
    // as well as spawn a Ctrl C handler.
    let cmd_execution_by_id = cmd_execution_by_id
        .lock()
        .await;
    cmd_execution_by_id.insert(cmd_execution_id, interrupt_tx.clone());

    tokio::task::spawn(async move {
        tokio::signal::ctrl_c()
            .await
            .expect("Failed to initialize signal handler for SIGINT");

        let (Ok(()) | Err(InterruptSignal)) = interrupt_tx.send(InterruptSignal);
    });

    // TODO: store `cmd_execution_id` as the only running `CmdExecution`.
}

/// Returns the progress of the `CmdExecution`.
async fn cmd_exec_progress_handler(cmd_execution_id: CmdExecutionId) -> Result<CmdProgress, E> {
    self.cmd_progress_storage.get(cmd_execution_id).await
}

async fn cmd_exec_interrupt_handler(cmd_execution_id: CmdExecutionId) -> Result<(), E> {
    let cmd_execution_by_id = cmd_execution_by_id
        .lock()
        .await;

    if let Some(interrupt_tx) = cmd_execution_by_id.get(cmd_execution_id) {
        let (Ok(()) | Err(InterruptSignal)) = interrupt_tx.send(InterruptSignal);

        Ok(())
    } else {
        Err(E::from(Error::CmdExecutionIdNotFound { cmd_execution_id }))
    }
}
}

CLI as Rest Client to Web Service

This is essentially the Web Service implementation, but rendering the progress on the machine with the CLI.

Flow Versioning

A flow's version needs to be increased whenever:

• Item: An associated data type is changed -- State, Diff, Params.
• Flow: An item is added.
• Flow: An item is removed.
• Flow: An item is renamed.

Flow Version Maintenance

For developers, maintaining multiple versions of flows is a cost, and usually one that is necessary. As items are created, modified, and removed throughout a flow's evolution, automation software needs to manage stored information from previous versions of the flow.

There are a number of ways to manage and ship flow versions:

• One version: Only ship with one flow version.

  Items never persist beyond the lifetime of the automation software.

  Suitable for once off flows where ensure (and clean up) are done with the same version of the automation software.

• Blue-green versioning: Ship with the previous and the next version.

  Automation software is compatible with the previous version of each flow, as well as the next.

  All stored information must be upgraded to the newer version before a new flow version is shipped.

  Suitable for flows that do not evolve often, and not all previous versions of flows need to be supported.

• n-versioning: Ship with n-versions of flows.

  n may be all flows.

  Each flow's state will need to adhere to the following, mainly for deserialization:

  • Either never evolve its data type, or be an enum, with a variant per version / evolution.
  • Ship with upgrade code from old versions to the current version. This is the highest cost in terms of code maintenance, and the shipped binary may grow unreasonably large over time.

Flow Upgrades

Upgrade Strategies

For blue-green / n versioning, there are a number of strategies for upgrading stored information in the flow:

• Automatic upgrade:

  • Implementors and developers need to ship with v1 -> v2 migration code.
  • Simplest for users, assuming migration path is well understood / no surprises.
  • Developers may want to diff this.

• Manual upgrade:

  • Implementors and developers need to ship with v1 -> v2 migration code.
  • Users have to choose to migrate.
  • Developers and users may want to diff things.

For n-versioning, upgrading from v1 -> v3 should use v1 -> v2 -> v3, so that developers don't have to write migration code form every version to the latest.

Execution History

To render old either we have one standard format that doesn't need old data types to present, or we ship those types.

Makes sense to have a standard format -- shipping with previous data types creates bloat, and makes it impossible to support execution history for shipping single versions of flows.

Seamless Execution

Seamless execution when multiple flow versions exist requires handling the following:

• Reading and writing (serialization) stored state / outcomes from different flow versions.
• Reading and writing to (adding, removing, modifying) the items from different flow versions.

Stored State

Stored state is accessed when:

• CmdCtx is built: Read and write.
• Commands are run:
  • StatesCurrentStoredReadCmd, StatesGoalReadCmd, ExecutionHistoryListCmd / ExecutionHistoryShowCmd: Read.
  • StatesDiscoverCmd, EnsureCmd, CleanCmd: Write.

CmdCtx Build

For cmd ctx build, because state could be in any version (current or any previous version), deserialization needs to be stateful -- like how type_reg::untagged::TypeReg is done.

fn envman_cmd_ctx() {
    let flow_versions = FlowVersions::builder()
        .add_flow(flow_v1)
        .add_flow(flow_v2)
        .add_flow(flow_v3)
        .build();

    let cmd_ctx = CmdCtx::builder_single_profile_single_flow(&mut output, &workspace)
        // .with_flow(flow)
        .with_flow_versions(flow_versions)
        .await?;
}

Cmd States and Outcome: Read

StatesCurrentStoredReadCmd and StatesGoalReadCmd needs to read states from any supported flow version.

ExecutionHistoryListCmd / ExecutionHistoryShowCmd (not implemented yet) need to read outcomes (and hence states and diffs) from any previous flow version.

Cmd States and Outcome: Write

• StatesDiscoverCmd: Some overlap with migrating stored states to latest format.

• EnsureCmd / CleanCmd: Write new version after ensuring.

  What about interruptions?

  We'd probably have to write the version of each item state then. Enum variant per version solves this.

/// Indicates when stored state can be upgraded.
///
/// Note: Stored outcomes also contain state, so for an upgrade between two
/// given versions, an `OutcomeUpgradeReq` will be the same as this.
///
/// However, should we mutate stored outcomes? I imagine not.
enum StateUpgradeReq {
    /// Data can be upgraded without discovering state.
    None,
    /// State needs to be discovered in order to store the new version.
    ///
    /// Existing stored state doesn't carry enough information to construct the
    /// new version, but the existing item does.
    Discover,
    /// Item needs an ensure / clean to be run to store the new version.
    ///
    /// The existing item doesn't carry enough information to construct the new
    /// state, and needs `apply` to be run to do so.
    ///
    /// This means `ApplyCmd`, and hence `Item{,Rt,Wrapper}`, need to work with
    /// different state versions.
    Apply,
}

Items: Use Latest / Migrate to Latest

An item upgrade may be one of:

• Only modifying the data type, and changing the information stored about the item (see StateUpgradeReq::Discover above).
• Modifying the item in place.
• Cleaning up the item then re-ensuring it.

Notes:

• It may not be possible to migrate an item to a newer version without cleaning up its successors.
• It is possible if successors don't "live inside" the item, but only need their parameters changed.
• it is not possible if successors live inside the item, and need to be deleted in order to modify the item.

Whether or not a successor needs to be cleaned up likely should be encoded into the Edge type, not part of the item upgrade parameters -- an item is unable to know if its successors live inside it or point to it.

/// Indicates what is needed for an instance of an item to be upgraded.
enum ItemUpgradeReq {
    /// Item instance does not need to be upgraded, i.e. only state data type is changed.
    None,
    /// Item can and needs to be re-ensured as part of this upgrade.
    Modify,
    /// Item needs to be cleaned and re-ensured as part of this upgrade.
    Replace,
}

Running The Upgrade Code

The following use cases seem natural for developers / users:

1. Manually upgrade state after automation software has been updated.
2. Automatically upgrade state as part of using regular commands.
3. Manually upgrade items after automation software has been updated.
4. Automatically upgrade items as part of using regular commands.

Some use cases are mutually exclusive, though mutually exclusive may still exist in the same automation software, for different version or kinds of upgrades.

For example, automatic item upgrades as part of using regular commands are appropriate when nothing needs to be cleaned up, but manual item upgrades should be used when some items need to be cleaned as part of the upgrade.

Manual State Upgrade

Developers must provide an upgrade command, to call StateUpgradeCmd::exec.

impl StateUpgradeCmd {
    /// # Note
    ///
    /// This is single profile single flow.
    ///
    /// To upgrade the states of multiple profiles and multiple flows, this
    /// must be called per flow, and maybe per profile as well.
    ///
    /// `MultiProfileSingleFlow` may only be safe for `StateUpgradeReq::None`
    /// and `StateUpgradeReq::Discover`.
    async fn state_upgrade_exec(cmd_ctx: &mut CmdCtx<SingleProfileSingleFlow>)
    -> Result<(), Error> {
        match state_upgrade_req {
            StateUpgradeReq::None => {
                let states = resources.get::<StatesCurrentStored>();
                let states_upgraded = flow
                    .graph()
                    .iter()
                    .for_each(|item| item.state_upgrade(states));
                states_serializer.serialize(&states_upgraded).await?;
            }
            StateUpgradeReq::Discover => {
                let CmdOutcome {
                    value: (states_current, states_goal),
                    errors,
                } = StatesDiscoverCmd::current_and_goal(&mut cmd_ctx).await?;
                states_serializer.serialize_current(&states_upgraded).await?;
                states_serializer.serialize_goal(&states_upgraded).await?;
            }
            StateUpgradeReq::Apply => {
                let CmdOutcome {
                    value: states_upgraded,
                    errors,
                } = EnsureCmd::exec_with(&mut cmd_ctx.as_sub_cmd()).await?;
            }
        }
    }
}

Automatic State Upgrade

The above method, but called automatically from a set of chosen commands: EnsureCmd, CleanCmd.

May need to think about this more carefully.

Manual Item Upgrade

Developers must provide an upgrade command.

For actual item modification, it may be one of:

// Only discovery needed
fn env_ensure_cmd() {
    env_flow_current_version_ensure();
    env_flow_ensure();
}

// apply needed
fn env_ensure_cmd() {
    // Item implementation needs to handle upgrade from whatever version the
    // state is in, to the current version.
    //
    // Meaning, there is no API support / constraints from the `peace`
    // framework. It's all handled from within the item implementation.
    env_flow_ensure();
}

Automatic Item Upgrade

Same as manual item upgrade, except the upgrade command is not separate to the regular ensure command.

For item upgrades, state could always be an enum, and peace doesn't include API support, constraints, or special handling for migrating between item versions -- the current building blocks are enough to implement seamless upgrades if items are implemented well.

For flow upgrades, we still need to handle clean up of old items no longer needed in subsequent flow versions, meaning the items must still be included in the development tool for their clean up logic.

For the developer, this could be either:

• Passing in the previous and current flow versions, and the framework working out which items need to be cleaned up.
• Passing in the current flow version, and old items and params for the framework to clean up those items.

Is this enough to start working on StatesSerde?

Yes:

• StatesSerde should hold unknown entries (removed items).
• States should be mapped From<StatesSerde> after deserialization.
• Users should be informed when there are unknown entries.
• States should be mapped Into<StatesSerde>, with an entry for each item in the flow.

Upgrade Actions

There are 4 levels of upgrade actions that can be applied to items.

• Reserialization with existing stored values.
• State discovery: current / goal.
• Item ensure.
• Replacement (clean and ensure) required.

The following table shows what is required for ensuring an environment to be in sync with the change. If multiple changes are involved, then the highest level of upgrade action needs to be applied to the environment for params and state, and the environment to be in sync, i.e. max(upgrade_actions).

Notes:

• "Data" refers to an item's Params, State, or StateDiff.

• Apply is associated with params, state, and diff.

  apply_clean needs to know state_clean based on the parameters used to compute state_goal at the time of the previous apply_goal.

• "Predecessor action" means what upgrade action is needed for this item, given a predecessor has had action applied to it. In other words, the change for an item can imply an upgrade action for successors.

Topics

Benchmarks

Compiled vs Interpreted

Compiled

Interpreted

Native Code

Rust is compiled to target CPU code via LLVM. So you get similar speeds to C/C++.

Memory Usage

Stack memory is used when possible, heap when required / requested.

A field in a struct does not necessarily mean another pointer dereference.

Java

class Inner { int value;   }
class Outer { Inner inner; }

public static void main(String[] ag) {
    var s = "hello";

    var outer = new Outer();
    outer.inner = new Inner();
    inner.value = 123;

    System.out.println(
        "" + outer.inner.value
    );
}

Rust

struct Inner { value: i32   }
struct Outer { inner: Inner }

fn main() {
    let mut s = String::new();
    s.push_str("hello");

    let outer = Outer {
        inner: Inner {
            value: 123
        }
    };

    println!("{}", outer.inner.value);
}

Memory Freeing

As owned memory goes out of scope, it is freed.

No separate garbage collection process.

https://discord.com/blog/why-discord-is-switching-from-go-to-rust

Performance Summary

1. Runtime performance depends on different factors, such as CPU usage and memory.

2. LLVM languages compile to CPU specific instructions, which run very quickly on that CPU.

3. Memory is split into the stack and the heap.

4. Rust makes use of the stack, reducing the time taken to fetch information.

5. Rust has no garbage collection, so applications perform consistently.

🚧 Constraints

Correct Programs

Unconstrained

#! /bin/bash
echo rara # will execute, works
rara      # will execute, fails

> ./script.sh_

Constrained

fn main() {
    let a = 1;
    a = a + 1;
    println!("a is: {a}");
}

Constrained - Fixed

fn main() {
    let mut a = 1;
    a = a + 1;
    println!("a is: {a}");
}

Less vs More

Nullability - JS

function print(s) {
    console.log(s);
    console.log(s.length);
}

// incorrect usage, valid
print();          // `undefined`, Uncaught TypeError: s is undefined
print(undefined); // `undefined`, Uncaught TypeError: s is undefined
print(null);      // `null`,      Uncaught TypeError: s is null

// correct usage, valid
print("hello");  // hello, 5

Nullability - C#

static void print(String s) {
    Console.WriteLine(s);
    Console.WriteLine(s.Length);
}

public static void Main() {
print();        // incorrect usage, invalid
print(null);    // incorrect usage, valid
print("hello"); // correct usage,   valid
}

Nullability - Rust

fn print(s: String) {
    println!("{s}");
    println!("{}", s.len());
}

fn print(s: String) {
    println!("{s}");
    println!("{}", s.len());
}
fn main() {
print();
}

fn print(s: String) {
    println!("{s}");
    println!("{}", s.len());
}

fn main() {
print(None);
}

fn print(s: String) {
    println!("{s}");
    println!("{}", s.len());
}


fn main() {
print("hello".into());
}

Accepting "nothing"

fn print(s_opt: Option<String>) {
    match s_opt {
        None => println!("nothing!"),
        Some(s) => {
            println!("{s}");
            println!("{}", s.len());
        }
    }
}

fn main() {
    print(None);
    print(Some("hello".into()));
}

Data Race - C#

using System;
using System.Threading;

public class Program {
    class Data {
        public int Value { get; set; }
    }

    public static void Main() {
Data data = new Data { Value = 0 };

Action inc0 = () => { for (int i = 0; i < 50000; i++) { data.Value += 1; } };
Action inc1 = () => { for (int i = 0; i < 50000; i++) { data.Value += 1; } };

Thread thread0 = new Thread(new ThreadStart(inc0));
Thread thread1 = new Thread(new ThreadStart(inc1));

thread0.Start();
thread1.Start();

thread0.Join();
thread1.Join();

Console.WriteLine($"value: {data.Value}");
    }
}

# build and execute
(
  mkdir -p /tmp/target/cs
  mono-csc \
    doc/src/learning_material/why_rust/constraints/data_race.cs \
    -out:/tmp/target/cs/data_race.exe
  /tmp/target/cs/data_race.exe
)

Data Race - Rust 1

use std::thread;

#[derive(Debug)]
struct Data {
    value: u32,
}

fn main() {

let mut data = Data { value: 0 };
let data = &mut data;
let work_0 = || (0..50000).for_each(|_| data.value += 1);
let work_1 = || (0..50000).for_each(|_| data.value += 1);

let thread_0 = thread::spawn(work_0);
let thread_1 = thread::spawn(work_1);

thread_0.join().unwrap();
thread_1.join().unwrap();

println!("value: {}", data.value);

}

# build and execute
(
  mkdir -p /tmp/target/rust
  rustc \
    doc/src/learning_material/why_rust/constraints/data_race_1.rs \
    -o /tmp/target/rust/data_race_1
  /tmp/target/rust/data_race_1
)

Data Race - Rust 2

use std::{
    sync::{Arc, Mutex},
    thread,
};

#[derive(Debug)]
struct Data {
    value: u32,
}

fn main() -> thread::Result<()> {
let data = Data { value: 0 };
let arc_mutex = Arc::new(Mutex::new(data));
let arc_mutex_0 = arc_mutex.clone();
let arc_mutex_1 = arc_mutex.clone();

let work_0 = move || {
    (0..50000).for_each(|_| {
        if let Ok(mut data) = arc_mutex_0.lock() {
            data.value += 1;
        }
    });
};
let work_1 = move || {
    (0..50000).for_each(|_| {
        if let Ok(mut data) = arc_mutex_1.lock() {
            data.value += 1;
        }
    })
};

let thread_0 = thread::spawn(work_0);
let thread_1 = thread::spawn(work_1);

thread_0.join()?;
thread_1.join()?;

if let Ok(Ok(data)) = Arc::try_unwrap(arc_mutex).map(Mutex::into_inner) {
    println!("value: {}", data.value);
}

Ok(())
}

# build and execute
(
  mkdir -p /tmp/target/rust
  rustc \
    doc/src/learning_material/why_rust/constraints/data_race_2.rs \
    -o /tmp/target/rust/data_race_2
  /tmp/target/rust/data_race_2
)

Constraints in Rust

🔥

Rust has over 9000 error codes.

Constraints Summary

1. Constraints mark programs as invalid.

2. Well designed constraints mark incorrect programs as invalid.

3. Rust has strict constraints, so more incorrect programs are marked invalid.

---

Switch from:

"It's so hard to get anything to compile."

🫠

to:

"The compiler is doing so much work for me."

🙇‍♂️

💬 Expressive

Co/Co Mapping

• 🏠 Concept to Code 🗞️

• 🗞️ Code to Concept 🏚️

Traits

use std::time::Duration;
fn main() {
let duration =
    Duration::from_secs(3 * 60)
    + Duration::from_secs(15)
    + Duration::from_secs(47);

println!(
    "{} seconds",
    duration.as_secs()
);
}

fn main() {
let duration =
    3.minutes()
    + 15.seconds()
    + 47.seconds();

println!(
    "{}",
    duration.for_humans()
);
}

trait IntoDuration {
    fn seconds(self) -> std::time::Duration;
    fn minutes(self) -> std::time::Duration;
}

impl IntoDuration for u64 {
    fn seconds(self) -> std::time::Duration {
        std::time::Duration::from_secs(self)
    }
    fn minutes(self) -> std::time::Duration {
        std::time::Duration::from_secs(self * 60)
    }
}

trait ReadableExt {
    fn for_humans(&self) -> String;
}

impl ReadableExt for std::time::Duration {
    fn for_humans(&self) -> String {
        use std::fmt::Write;
        let mut buffer = String::with_capacity(128);
        let total = self.as_secs();
        let mins = total.div_euclid(60);
        let seconds = total.rem_euclid(60);
        match mins {
            1 => write!(&mut buffer, "{mins} minute").unwrap(),
            _ => write!(&mut buffer, "{mins} minutes").unwrap(),
        }
        match seconds {
            1 => write!(&mut buffer, " {seconds} second").unwrap(),
            _ => write!(&mut buffer, " {seconds} seconds").unwrap(),
        }
        buffer
    }
}

Multiple Values

fn fetch() -> String {
    "azriel".into()
}

fn main() {
let name = fetch();
println!("{name}");
}

fn fetch() -> (String, u32) {
    ("azriel".into(), 123)
}

fn main() {
let (name, number) = fetch();
println!("{name} {number}");
}

Cloning

🐑🐑

Problem: I want a deep copy of this object.

Cloning - Others

C#

var serialized = JsonConvert.SerializeObject(source);
return JsonConvert.DeserializeObject<T>(serialized);

JS

JSON.parse(JSON.stringify(original));

💎 Ruby

Marshal.load(Marshal.dump(@object))

Cloning - Rust

#![allow(unused)]
fn main() {
#[derive(Clone, Copy)] // <-- just add this "procedural macro"
pub struct Data {
    value: u32,
}
}

Copy

#![allow(unused)]
fn main() {
#[derive(Clone, Copy)]
pub struct Data {
    value: u32,
}

let data_0 = Data { value: 123 };
let mut data_1 = data_0; // bitwise

data_1.value = 456;

println!("data_0: {}", data_0.value);
println!("data_1: {}", data_1.value);
}

Clone

#![allow(unused)]
fn main() {
#[derive(Clone, Debug)]
pub struct DataOnHeap {
    num: u32,
    value: String,
}

let data_0 = DataOnHeap {
    num: 123,
    value: "hello".into()
};
let mut data_1 = data_0.clone();

data_1.value.make_ascii_uppercase();

println!("data_0: {data_0:#?}");
println!("data_1: {data_1:#?}");
}

Expressive Summary

1. Expressiveness is a measure of how easy it is to:

   1. Take a design and map it to code.
   2. Read code, and understand what it means.

2. Rust has constructs that make it easy to say what you mean.

3. When the code is read, it is clear to see what it is doing.

😵‍💫 Unambiguous

Equality

🥤 == 🥤

Equality - Java

class Data {
    public int value;
    public Data(int value) {
        this.value = value;
    }
}
String ab = "ab", c = "c";
boolean[] equality = new boolean[] {

// Which of the following are true / false?
"abc" == "abc",
"abc" == "ab" + "c",
"abc" == ab + c,
123 == 123,
new Data(123) == new Data(123)

};

for (boolean equal: equality) {
    System.out.println(equal);
}

https://dev.java/playground/

C#

https://dotnetfiddle.net/#

using System;

public class Program
{
class Data {
    public int value { get; set; }
    public Data(int value) { this.value = value; }
}
    public static void Main()
    {

String ab = "ab";
String c = "c";
bool[] equality = new bool[] {
    "abc" == "abc",
    "abc" == "ab" + "c",
    "abc" == ab + c,
    123 == 123,
    new Data(123) == new Data(123)
};

foreach (bool equal in equality) {
    Console.WriteLine(equal);
}

    }
}

Equality - Rust

struct Data(u32);

fn main() {

let ab = "ab".to_string();
let c = "c";
let equality = [

// Which of the following are true / false?
"abc" == "abc",
"abc" == "ab".to_string() + "c",
"abc" == ab + c,
123 == 123,
Data(123) == Data(123),

];

for equal in equality.iter() {
    println!("{equal}");
}

}

Equality - Rust

#[derive(PartialEq)]
struct Data(u32);

fn main() {

let ab = "ab".to_string();
let c = "c";
let equality = [

// Which of the following are true / false?
"abc" == "abc",
"abc" == "ab".to_string() + "c",
"abc" == ab + c,
123 == 123,
Data(123) == Data(123),

];

for equal in equality.iter() {
    println!("{equal}");
}

}

Referential Equality

#![allow(unused)]
fn main() {
#[derive(Clone, PartialEq)]
struct Data(String);

let data_0 = Data(String::from("hello"));
let data_1 = data_0.clone();

assert!(data_0 == data_1);
assert!(
    std::ptr::eq(&data_0, &data_1) == false
);
println!("ok!");
}

Control Flow

Control Flow - Java

public Success download(
    final String path
)
throws
    UnknownHostException,
    ConnectionLostException,
    OutOfDiskSpaceException
{
    //
}

public enum Success {
    DOWNLOADED,
    CACHED;
}

Usage

// Use `download()`
try {
    download("a_file.txt");
    download("b_file.txt");
    upload("a_file.txt");
}
catch (OutOfDiskSpaceException e) {}
catch (UnknownHostException e) {}
catch (ConnectionLostException e) {}

Concept

Unambiguous

try {
    download("a_file.txt");
}
catch (UnknownHostException e) {}
catch (ConnectionLostException e) {}
catch (OutOfDiskSpaceException e) {}

try {
    download("b_file.txt");
}
catch (UnknownHostException e) {}
catch (ConnectionLostException e) {}
catch (OutOfDiskSpaceException e) {}

try {
    upload("a_file.txt");
}
catch (UnknownHostException e) {}
catch (ConnectionLostException e) {}

Concept

Control Flow - Rust

pub fn download(path: &str)
-> Result<Success, Fail> {
    // ..
}

pub enum Success {
    Downloaded,
    Cached,
}

pub enum Fail {
    UnknownHost(io::Error),
    ConnectionLost(io::Error),
    OutOfDiskSpace {
        path: PathBuf,
        error: io::Error,
    },
}

Usage

#![deny(unused_must_use)]
fn main() {
// Use `download()`
let result = download("/tmp/a_file.txt");
match result {
    Ok(Success::Downloaded) => {}
    Ok(Success::Cached) => {}
    Err(Fail::UnknownHost(_)) => {}
    Err(Fail::ConnectionLost(_)) => {}
    Err(Fail::OutOfDiskSpace { .. }) => {}
}

match download("/tmp/b_file.txt") {
    Ok(_) => {}
    Err(Fail::UnknownHost(_)) => {}
    Err(Fail::ConnectionLost(_)) => {}
    Err(Fail::OutOfDiskSpace { .. }) => {}
}

upload("/tmp/a_file.txt");
// match upload_outcome {
//     Ok(_) => {}
//     Err(FailUpload::UnknownHost(_)) => {}
//     Err(FailUpload::ConnectionLost(_)) => {}
// }
}

pub fn download(_s: &str) -> Result<Success, Fail> {
    Ok(Success::Downloaded)
}
pub fn upload(_s: &str) -> Result<Success, FailUpload> {
    Ok(Success::Downloaded)
}

pub enum Success {
    Downloaded,
    Cached,
}

pub enum Fail {
    UnknownHost(std::io::Error),
    ConnectionLost(std::io::Error),
    OutOfDiskSpace {
        path: std::path::PathBuf,
        error: std::io::Error,
    },
}

pub enum FailUpload {
    UnknownHost(std::io::Error),
    ConnectionLost(std::io::Error),
}

Unambiguous Summary

1. Ambiguity happens when there are multiple meanings, and you have to guess which is the right one.

2. Expressiveness lets you say what you want.

3. Unambiguity means based on what is said, you can tell what was wanted.

4. Rust guides you towards writing unambiguous code.

🛠️ Tooling

Formatter

    fn main() {
      let a = 1;
    let b = 2;
let   sum=
    a+    b;

      println!(
"{a} + {b} = {sum}");
           }

🧹 RustFmt

➡️

fn main() {
    let a = 1;
    let b = 2;
    let sum = a + b;

    println!("{a} + {b} = {sum}");
}

Linter

fn main() {
let maybe_n = Some(123);

match maybe_n {
    Some(n) => println!("{n}"),
    _ => (),
}
}

📎 Clippy

➡️

fn main() {
let maybe_n = Some(123);

if let Some(n) = maybe_n {
    println!("{n}")
}
}

warning: you seem to be trying to use match for destructuring a single pattern.
Consider using if let

fn main() {
let maybe_n = Some(123);

+#[allow(clippy::single_match)] // deliberately allow this once
 match maybe_n {
     Some(n) => println!("{n}"),
     _ => (),
 }
}

Clippy lints index

1. Rust ships with a linter

2. It's called clippy, also a very original name.

3. If you write the code on the left, it will tell you to write code on the right.

4. This is really nice because it's teaching you how to write cleaner code.

5. Clippy has over 9000 lints.

Package Manager

We have one: cargo.

All commands are standard, you don't have to define scripts to build or run:

cargo init
cargo build
cargo run
cargo test
cargo publish

[package]
name = "scratch"
version = "0.1.0"

[dependencies]
# yes we have comments!
serde = { version = "1.0.197", features = ["derive"] }
serde_json = "1.0.115"

Others

Many tools by the community:

• 🛡️ audit
• 🙅 deny
• ❓ about
• more in lib.rs #cargo-subcommand

All integratable into your build pipeline.

Tooling Summary

1. Rust ships with standard tooling.

2. Everyone uses the same tooling to adhere to best practices.

3. You can focus on development, and not the flavour of the day.

🚛 Ecosystem

Variety

🐚 CLI Applications

• bat
• ripgrep
• nushell

🖥️ Native Applications

• warp
• zed

🌐 Web Development -- WASM

• dot_ix
• leptos
• dioxus

🎮 Game Development

• bevy

🤖 Embedded Systems

probe.rs

1. Rust doesn't really limit itself to any particular domain.

2. Here are some examples in different domains.

3. We'll do a 3 second demo of each.

4. CLI applications, there are plenty of tools that run in the shell.

5. There is also a shell.

6. Native applications, warp is the terminal application you see here.

7. We can run commands (ls and ls --tree), and navigate between command blocks.

8. There are web server frameworks and frontend frameworks.

Stable

🏰

Build something that lasts.

3. Rust is committed to stability --

4. what you built with Rust 1.0 should still build with Rust 1.78.

5. and I had my own experience of this -- I wrote a log parser in my first workplace, left, and years later when I rejoined, it still compiled and ran!

6. Can you expect that kind of backward compatibility from most software? or your phone?

7. There are a handful of things that can make that claim.

🦋 Second Order

Chain of Events

Topics Again

1. ⚡ Performance
2. 🚧 Constraints
3. 💬 Expressive
4. 😵‍💫 Unambiguous
5. 🛠️ Tooling
6. 🚛 Ecosystem
7. 🦋 Second Order

Try Rust

❓

Q & A

User Facing Automation - Part 1

Set Up

Commands before demoing (ZSH)

# Create demo directory and source file
demo_dir=/tmp/automation_demo
test -d "${demo_dir}" || mkdir "${demo_dir}"
cd "${demo_dir}"
src="${demo_dir}/a.txt"
dest="${demo_dir}/b.txt"

# Change time format. This is for zsh.
#
# Functions must also be run in subshells in zsh for `time` to work:
# <https://superuser.com/questions/688128/how-to-run-time-on-a-function-in-zsh>
TIMEFMT=$'%*E seconds'

echo hi > "$src"

For the download example:

# Requires Rust
cargo install --git https://github.com/azriel91/peace download --all-features

In a separate shell:

cd /tmp/automation_demo
case "$(uname)" in
    Linux)  watch -n 0.4 -c -d "stat --format='%y' b.txt | cut -b 12-23 ; bat b.txt" ;;
    Darwin) watch -n 0.4 -c -d "stat -f '%y' b.txt | cut -b 12-23 ; bat b.txt" ;;
esac

Scenario

Copy a file from one place to another.

For demonstration, this section will use the sleep and cp (copy) commands to emulate a task:

function slow_cp {
    sleep 1;
    cp "$1" "$2";
}

Basics

Concept 1: Repeatable in One Action

What: Do it again without thinking.

Value: Save time and mental effort.

Example

# Hard coded values
function slow_cp {
    sleep 1;
    cp a.txt b.txt;
}

slow_cp

Concept 2: Parameterized

What: Do the same action on different things.

Value: Multiply the automation gain per thing.

Example

function slow_cp {
    sleep 1;
    cp "$1" "$2";
}

src="/tmp/automation_demo/a.txt"
dest="/tmp/automation_demo/b.txt"

slow_cp "${src}" "${dest}"

Efficiency

Concept 3: Idempotence

What: Don't do it if it's already done.

Value: Save time.

Example

Execution 1:

Execution 2:

function idempotent_cp {
    if   ! test -f "${dest}"
    then slow_cp "$1" "$2"
    fi;
}

rm -f "${dest}"
time (idempotent_cp "${src}" "${dest}")
time (idempotent_cp "${src}" "${dest}")

echo updated > "${src}"
time (idempotent_cp "${src}" "${dest}")

Concept 4: Smart Idempotence

What: Handle updates.

Value: Do what's expected.

Example

function idempotent_cp {
    local src_hash;
    local dest_hash;
     src_hash=$(md5sum <(cat "$1"))
    dest_hash=$(md5sum <(cat "$2" 2>/dev/null))
    if   ! test "${src_hash}" = "${dest_hash}"
    then slow_cp "$1" "$2"
    fi;
}

rm -f "${dest}"
time (idempotent_cp "${src}" "${dest}")
time (idempotent_cp "${src}" "${dest}")

Concept 5: Parallelism

What: Run tasks at the same time.

Value: Elapsed duration to execute process is decreased.

Example

dest_1="/tmp/automation_demo/b1.txt"
dest_2="/tmp/automation_demo/b2.txt"
dest_3="/tmp/automation_demo/b3.txt"

Serial

# Serial execution
rm -f "${dest_1}" "${dest_2}" "${dest_3}"
time (
    idempotent_cp "${src}" "${dest_1}";
    idempotent_cp "${src}" "${dest_2}";
    idempotent_cp "${src}" "${dest_3}";
)
time (
    idempotent_cp "${src}" "${dest_1}";
    idempotent_cp "${src}" "${dest_2}";
    idempotent_cp "${src}" "${dest_3}";
)

# Remove one file
rm -f "${dest_2}"
time (
    idempotent_cp "${src}" "${dest_1}";
    idempotent_cp "${src}" "${dest_2}";
    idempotent_cp "${src}" "${dest_3}";
)
time (
    idempotent_cp "${src}" "${dest_1}";
    idempotent_cp "${src}" "${dest_2}";
    idempotent_cp "${src}" "${dest_3}";
)

Parallel

# Parallel execution
rm -f "${dest_1}" "${dest_2}" "${dest_3}"
time (
    idempotent_cp "${src}" "${dest_1}" &;
    idempotent_cp "${src}" "${dest_2}" &;
    idempotent_cp "${src}" "${dest_3}" &;
    wait;
)
time (
    idempotent_cp "${src}" "${dest_1}" &;
    idempotent_cp "${src}" "${dest_2}" &;
    idempotent_cp "${src}" "${dest_3}" &;
    wait;
)

# Remove one file
rm -f "${dest_2}"
time (
    idempotent_cp "${src}" "${dest_1}" &;
    idempotent_cp "${src}" "${dest_2}" &;
    idempotent_cp "${src}" "${dest_3}" &;
    wait;
)
time (
    idempotent_cp "${src}" "${dest_1}" &;
    idempotent_cp "${src}" "${dest_2}" &;
    idempotent_cp "${src}" "${dest_3}" &;
    wait;
)

Concept 6: Logical Dependencies

What: Wait if you have to.

Value: Correctness.

Example

# Logical dependency
rm -f "${dest_1}" "${dest_2}" "${dest_3}"
time (
    idempotent_cp "${src}" "${dest_1}" &;
    (
        idempotent_cp "${src}" "${dest_2}";
        idempotent_cp "${dest_2}" "${dest_3}";
    ) &;
    wait;
)
time (
    idempotent_cp "${src}" "${dest_1}" &;
    (
        idempotent_cp "${src}" "${dest_2}";
        idempotent_cp "${dest_2}" "${dest_3}";
    ) &;
    wait;
)

Output

Concept 7: Progress Information

What: Tell users what's going on.

Value: Users know what's happening, or if it's stalled.

Example

Code

function slow_cp {
    sleep 1;
    cp "$1" "$2";
}

function hash_file {
    1>&2 printf "hashing file: ${1}\n"
    test -f "${1}" &&
      md5sum <(cat "$1" 2>/dev/null) ||
      printf '00000000000000000000000000000000'
}

function informational_idempotent_cp {
     src_hash=$(hash_file "${src}")
    dest_hash=$(hash_file "${dest}")
    1>&2 printf " src_hash: ${src_hash}\n"
    1>&2 printf "dest_hash: ${dest_hash}\n"

    if   ! test "${src_hash}" = "${dest_hash}"
    then
        1>&2 printf "contents don't match, need to copy.\n"
        slow_cp "$1" "$2"
        1>&2 printf "file copied.\n"
    else
        1>&2 printf "contents match, don't need to copy.\n"
    fi;
}

rm -f "${dest}"

informational_idempotent_cp "${src}" "${dest}"

# if we don't care about the verbose information, we can hide it
informational_idempotent_cp "${src}" "${dest}" 2>/dev/null

Concept 8: Provide Relevant Information

What: Show as little and as much information as necessary.

Value: Reduce effort to understand.

Example

Code

delay=0.1
pb_lines=3

function slow_cp_with_progress {
    progress_update 50 20 'copying file'; sleep $delay
    progress_update 50 25 'copying file'; sleep $delay
    progress_update 50 30 'copying file'; sleep $delay
    progress_update 50 35 'copying file'; sleep $delay
    progress_update 50 40 'copying file'; sleep $delay
    progress_update 50 45 'copying file'; sleep $delay
    cp "$1" "$2";
}

function hash_file {
    test -f "${1}" &&
      md5sum <(cat "$1" 2>/dev/null) ||
      printf '00000000000000000000000000000000'
}

function clear_lines {
    local lines_to_clear
    local i

    lines_to_clear=$(($1 - 1))  # subtract 1 because bash loop range is inclusive

    if test "${lines_to_clear}" -ge 0
    then
        for i in {1..${lines_to_clear}}
        do
            1>&2 printf "\033[2K\r" # clear message line
            1>&2 printf "\033[1A"   # move cursor up one line
        done
    fi
}

function progress_write {
    local progress_total
    local progress_done
    local progress_message

    local progress_remaining
    local printf_format

    progress_total=$1
    progress_done=$2
    progress_message="${3}"
    progress_remaining=$(($progress_total - $progress_done))
    if test $progress_total -eq $progress_done
    then printf_format="\e[48;5;35m%${progress_done}s\e[48;5;35m%${progress_remaining}s\e[0m\n" # green
    else printf_format="\e[48;5;33m%${progress_done}s\e[48;5;18m%${progress_remaining}s\e[0m\n" # blue
    fi

    1>&2 printf "$printf_format" ' ' ' '
    1>&2 printf "$progress_message"
    1>&2 printf '\n'
}

function progress_update {
    clear_lines $pb_lines # message line, progress bar line, extra line
    progress_write "$@"
}

function informational_idempotent_cp {
    local src_hash;
    local dest_hash;
    progress_write 50 0 'hashing source file'; sleep $delay
    src_hash=$(hash_file "${src}")

    progress_update 50 5 'hashing destination file'; sleep $delay
    dest_hash=$(hash_file "${dest}")

    progress_update 50 10 'comparing hashes'; sleep $delay
    if   ! test "${src_hash}" = "${dest_hash}"
    then
        progress_update 50 15 'copying file'; sleep $delay
        slow_cp_with_progress "$1" "$2"

        progress_update 50 50 '✅ file copied!'
        1>&2 printf "\n"
    else
        progress_update 50 50 '✅ contents match, nothing to do!'
        1>&2 printf "\n"
    fi;
}

rm -f "${dest}"

informational_idempotent_cp "${src}" "${dest}"

delay=0.7
rm -f "${dest}"

Concept 9: Information Format

What: Change information format specifically for how it is consumed.

Value: Makes using the API ergonomic.

Tip: Progress information can be more than a string.

• For a human: output a progress bar, replace status text
• For continuous integration: append status text
• For a web request: output json

Example

Code

output_format=pb # pb, text, json

function progress_write {
    local progress_total
    local progress_done
    local progress_message
    progress_total=$1
    progress_done=$2
    progress_message="${3}"

    local progress_remaining
    local printf_format
    progress_remaining=$(($progress_total - $progress_done))

    case "${output_format}" in
        pb)
            if test $progress_total -eq $progress_done
            then printf_format="\e[48;5;35m%${progress_done}s\e[48;5;35m%${progress_remaining}s\e[0m\n" # green
            else printf_format="\e[48;5;33m%${progress_done}s\e[48;5;18m%${progress_remaining}s\e[0m\n" # blue
            fi

            1>&2 printf "$printf_format" ' ' ' '
            1>&2 printf "$progress_message"
            1>&2 printf '\n'
            ;;
        text)
            1>&2 printf "$progress_message"
            1>&2 printf '\n'
            ;;
        json)
            cat << EOF
{ "progress_total": $progress_total, "progress_done": $progress_done, "progress_remaining": $progress_remaining, "message": "$progress_message" }
EOF
            ;;
    esac
}

function progress_update {
    case "${output_format}" in
        pb)
            clear_lines $pb_lines # message line, progress bar line, extra line
            ;;
        text)
            ;;
        json)
            ;;
    esac

    progress_write "$@"
}

output_format=pb
rm -f "${dest}"
time (informational_idempotent_cp "${src}" "${dest}")
echo '---'
time (informational_idempotent_cp "${src}" "${dest}")

output_format=text
rm -f "${dest}"
time (informational_idempotent_cp "${src}" "${dest}")
echo '---'
time (informational_idempotent_cp "${src}" "${dest}")

output_format=json
rm -f "${dest}"
time (informational_idempotent_cp "${src}" "${dest}")
echo '---'
time (informational_idempotent_cp "${src}" "${dest}")

informational_idempotent_cp "${src}" "${dest}" | jq

informational_idempotent_cp "${src}" "${dest}" | jq '.progress_remaining'

Error Handling

This section uses the download example.

Concept 10: Accumulate And Summarize

What: When an error happens, save it, then display it at the very end.

Value: User doesn't have to spend time and effort investigating.

Example

Don't Do This:

# Per subprocess
log_info "${id}: Start."
log_info "${id}: Processing."

download_file
download_result=$?

if [[ "$download_result" -eq 0 ]]
then
    log_info "${id}: Successful."
    log_info "${id}: Notifying service."
    return 0
else
    log_error "${id}: Download failed: ${download_result}"
    return 1
fi

Info : Starting process.
Info : b1: Start.
Info : b2: Start.
Info : b3: Start.
Info : main: Waiting for results.
Info : b1: Processing.
Info : b2: Processing.
Info : main: Waiting for results.
Info : b3: Processing.
Error: b2: Download failed: 12
Info : b1: Processing complete.
Info : main: Waiting for results.
Info : b3: Processing complete.
Info : b1: Successful.
Info : main: Waiting for results.
Info : b3: Successful.
Info : b3: Notifying service.
Info : main: Waiting for results.
Info : b1: Notifying service.
Info : main: Collected results.
Info : main: Analyzing.
Error: Process failed: b2.

Do This:

# Per subprocess
download_file
download_result=$?

if [[ "$download_result" -eq 0 ]]
then
    printf "{ \"id\": ${id} \"success\": true }"
    return 0
else
    printf "{ \"id\": ${id} \"success\": false, \"error_code\": 12 }"
    return 1
fi

Info : Starting process.
Info : b1: Start.
Info : b2: Start.
Info : b3: Start.
Info : main: Waiting for results.
Info : b1: Processing.
Info : b2: Processing.
Info : main: Waiting for results.
Info : b3: Processing.
Error: b2: Download failed: 12
Info : b1: Processing complete.
Info : main: Waiting for results.
Info : b3: Processing complete.
Info : b1: Successful.
Info : main: Waiting for results.
Info : b3: Successful.
Info : b3: Notifying service.
Info : main: Waiting for results.
Info : b1: Notifying service.
Info : main: Collected results.
Info : main: Analyzing.
Error: Process failed: b2.

Error: b2 failed with error code: 12

So that:

Info : Starting process.
Info : b1: Start.
Info : b2: Start.
Info : b3: Start.
Info : main: Waiting for results.
Info : b1: Processing.
Info : b2: Processing.
Info : main: Waiting for results.
Info : b3: Processing.
Error: b2: Download failed: 12
Info : b1: Processing complete.
Info : main: Waiting for results.
Error: b3: Download failed: 13
Info : b1: Successful.
Info : main: Waiting for results.
Info : main: Waiting for results.
Info : b1: Notifying service.
Info : main: Collected results.
Info : main: Analyzing.
Error: Process failed: b2, b3.

Error:
  b2 failed with error code: 12
  b3 failed with error code: 13

Concept 11: Understandable Error Messages

What: Translate the technical terms into spoken language terms.

Value: User can understand the message and take action.

Example

download -v init http://non_existent_domain file.txt

Error: peace_item_file_download::src_get

  × Failed to download file.
  ├─▶ error sending request for url (http://non_existent_domain/): error
  │   trying to connect: dns error: failed to lookup address information:
  │   Temporary failure in name resolution
  ├─▶ error trying to connect: dns error: failed to lookup address
  │   information: Temporary failure in name resolution
  ├─▶ dns error: failed to lookup address information: Temporary failure in
  │   name resolution
  ╰─▶ failed to lookup address information: Temporary failure in name
      resolution

Error: peace_item_file_download::src_get

  × Failed to download file.
   ╭────
 1 │ download init http://non_existent_domain/ file.txt
   ·               ─────────────┬─────────────
   ·                            ╰── defined here
   ╰────
  help: Check that the URL is reachable: `curl http://non_existent_domain/`
        Are you connected to the internet?

Concept 12: Capture the Source of Information

What: Show where the information came from.

Value: User doesn't have to spend time and effort investigating.

Example

download init http://localhost:3000/ peace_book.html

Stop the server, then:

download ensure

Error: peace_item_file_download::src_get

  × Failed to download file.
   ╭────
 1 │ download init http://localhost:3000/ peace_book.html
   ·               ───────────┬──────────
   ·                          ╰── defined here
   ╰────
  help: Check that the URL is reachable: `curl http://localhost:3000/`
        Are you connected to the internet?

Workflow Concepts

Concept 13: Clean Up

What: Leave a place in a state, no worse than when you found it.

Value: Don't waste resources.

Example

ls

download clean

Bash

function cp_flow {
    sub_cmd=$1

    case ${sub_cmd} in
        ensure)
            informational_idempotent_cp "$2" "$3"
            ;;
        clean)
            if   test -f "$2"
            then rm -f "$2"
            fi
            ;;
    esac ;
}

rm -f "${dest}"
time (cp_flow ensure "${src}" "${dest}")
time (cp_flow ensure "${src}" "${dest}")

time (cp_flow clean "${dest}")

User Facing Automation - Part 2

Workflow Concepts

Concept 14: Storing Parameters

What: Take in parameters, recall them automatically.

Value: Don't require users to pass in parameters repeatedly.

Example

download init https://ifconfig.me ip.json

download ensure
download clean

Concept 15: Current State, Goal State, Difference

What: Show users what is, and what will be, before doing anything.

Value: Give users clarity on what would happen, before doing it.

Example

download clean
download init https://ifconfig.me ip.json

download status
download goal
download diff

download ensure
download diff

download clean
download diff

Concept 16: Profiles

What: Use the same workspace input for logically separate environments.

Value: Ease the creation of cloned environments.

Concept 17: Flows

What: Perform different workflows within an environment.

Value: Workflows automatically share parameters with each other.

Concept 18: Parameter Limits

What: Guard automation from executing with unusual values.

Value: Don't accidentally incur large costs.

Efficiency Concepts (Again!)

Concept 19: Do It Before It's Asked

If a process cannot take less than 10 minutes,
then to do it in 5,
you must begin in the past.

What: Execute the automation in the background, show where it is when it is asked to be executed.

Value: Reduce waiting time.

Cost: Background work consumes resources, and may be redundant.

Concept 20: Reverse Execution

At the beginning of a process, what's the fastest way to get to step 9?

Go through steps 1 through 9.

What if you're on step 10?

Transition from step 10 to 9.

What: The beginning and destination state are not necessarily the start and end of a process.

Value: Allows inspection / mutation of state at a particular point, or re-testing of automation steps after that point.

Empathetic Code Design

Communicating Clearly In Code

1. Today's talk is about empathetic code design.

2. The project I will use to demonstrate this is Peace, a framework to create empathetic and forgiving software automation.

3. Peace provides constraints over logic and data types, and common functionality.

4. By using the framework, logic and data types will be shaped to produce an automation tool that is clear in its communication.

5. A user uses that tool to get their work done, with good user experience.

6. Here's a 3 step process to upload a file to the internet.

7. Download the file, create an S3 bucket, upload the file to that bucket.

8. We can use a shell script to execute this process. (live demo)

9. We can also use a tool, created using the Peace framework, to do the same thing.

10. envman is a tool I created earlier to do this process. (live demo)

11. This is a short demonstration of why one would use this framework.

12. When changing any environment, first we want to understand where we are.

13. Second, we also want to know, where we are headed.

14. Third, we want to know the cost of getting there.

15. Finally, when we understand enough about the change, we can decide to do it.

16. Now, when I say clear communication, it should be quite easy to see that A B C here maps to 1 2 3 there.

17. By presenting information in the right form, at the right level of detail, it's much easier to map between the concept and what is seen.

18. This is how Peace takes care of the user.

19. Today, we are going to look at how Peace takes care of the automation tool developer.

20. How clearly can we communicate between the mental model of a process, and code.

21. When designing a flow, we need to define the items in the flow, and the ordering between those items.

22. The reason these are called "items" instead of "steps", is because in Peace, the emphasis is on the outcome, rather than the actions.

23. Said another way, the purpose of technological processes is the end state, not the work undertaken to get there.

24. That statement does not apply to other kinds of processes, where the work is meaningful.

25. In code, adding items and ordering looks like this.

26. This shows linear ordering; it is possible to have work done in parallel when there is no dependency between items.

27. So here we can adjust the ordering such that A and B are both predecessors of C. A comes before C, and B comes before C.

28. So A and B can both begin at the same time, reducing the overall duration.

29. We'll adjust this in the example, I want you to see that both the file download and the s3 bucket turn blue, and when both are done, then the upload begins. We'll slow down our connection so it is easier to see. (live demo)

30. Try doing that in a shell script.

31. After a developer defines the items in a flow, they also need to define how data passes through these items.

32. If you've written automation using bash scripts or YAML, usually data is just strings.

33. For the non-devs among us, using "strings" everywhere is like saying, "I took stuff and did stuff, and here is stuff".

34. It would be much better to say, "I grab my camera, and took a shot, and here's a picture."

35. To better communicate with developers, Peace uses type safety.

36. Because Peace is a framework, each item's automation logic should be shareable.

37. This means, the input to each item, and its output, are API.

38. In Peace, we call the input "parameters", and we use call the output "state".

39. For a file download item, the parameters are the source URL to download from and the path to write to.

40. and the state, can be either "the file's not there", or "it's there, and here is its content hash".

41. Given each item defines its parameters and state, we need a way for developers to pass values into the parameters.

42. Usually, at the start of a process, we know the values of some parameters, and we can pass them straight in.

43. However, sometimes we need part of the process to be completed, to have the information to pass to a subsequent item.

44. For example, if we had a server instead of an S3 bucket, we cannot know what its IP address will be until we the server is created.

45. But we still want to encode into the flow, that the state from an earlier item, is needed to determine the parameter value for a subsequent item.

46. For the simple case, where we know the parameter values up front, we can just plug them in.

47. But we want to be able to express, "get the state from step two, and pass it into step three".

48. So I came up with this, for every item's parameters, the framework will generate a specification type that records how to figure out what values to use.

49. and you can pass it direct values, or you can tell it "here's what states to read, and here's how to extract the value to use for this field".

50. Here's how it looks like in code.

51. Yes, we can read states from multiple items, and use them to compute a value.

52. Best of all, it's type safe -- so within the code you can ask "what parameters do I need to pass in, and get them right".

53. If you make a mistake, you have a really big safety net called the compiler, that will save you.

54. The error message needs improvement, though I haven't yet figured that out.

55. If the parameter type changes, such as when you upgrade your item version, then you get a compilation error, which means you don't have to work to discover where to change -- the compiler tells you.

56. Now that we have our logic and our data, we need a way to interact with it.

57. Peace is designed to separate the automation logic and data, from how it is interacted with and presented.

58. Whether it is interactive shell, uninteractive shell, web API, or web page, there is no rendering code in any item.

59. This allows automation developers to use different front ends to the same automation backend.

60. In code, the developer can choose which output to use.

61. Currently the only user facing output implementation is the CliOutput, which you've seen.

62. There are a number of other output implementations used in tests,

63. such as the NoOpOutput which discards all information, and the FnTrackerOutput which stores what output methods were called, with what parameters.

64. Developers can choose to write their own output implementation as well, but

65. It is within the scope of Peace provide implementations for common use cases.

66. So far, we've seen Items, Parameters, Specifications, and Output.

67. As a developer, this should be sufficient to run some commands.

68. First we group all of the above into a CmdCtx, then we call the command we want to run for the flow.

69. StatesCurrentReadCmd and EnsureCmd are provided by the framework.

70. The real code still contains code to format the output, but this is something that the framework can provide.

71. For any automation software, the user should only need to pass in item parameters once.

72. Commands run using that flow should be able to recall parameters for the user.

73. How can we make this easy for the automation software developer?

74. The easiest thing, is for them to do nothing.

75. So whenever a command context is built, the following happens.

76. Previous parameter specifications will attempt to be loaded from disk.

77. The provided parameter specification will be merged with those.

78. Validation happens, to make sure there is enough information to resolve parameters for all items.

79. The merged parameter specifications will be serialized to disk.

80. If the parameter specifications don't pass validation, we should get a useful error message.

81. When a command is run, Peace stores information about that execution.

82. It stores this in a .peace directory within a workspace directory.

83. The automation tool developer needs to tell Peace how to determine the workspace directory.

84. Because if the tool is run in a sub directory, information should still be stored in the workspace directory.

85. If you think about git, if you run git status in any subdirectory of a repository, it will still find the .git directory somewhere up the chain.

86. One undeniable way to tell if software works, is to run it.

87. And to tell if it will work in production, we usually want to run it in a replica environment first.

88. Out of the box, Peace provides support for logically separate environments, through profiles.

89. So you can have multiple environments under different names, just like git branching.

90. Let's see a demo.

91. How do we communicate clearly in code?

92. We make sure the API matches the mental model.

93. We make sure we have good error messages.

94. When we require something of the developer, make it easy for them to provide it.

95. and when we adhere to these constraints, it will improve the automation development experience.

Background

Peace

Constraints and
Common functionality

+

💻

Code

Logic and data types

=

⚙️

Nice Tool

Automation software

⚙️

Nice Tool

Automation software

+

🔡

Data

Values for parameters

=

🚀

Work Done

App built and published

Environment deployed

Scenario

Shell Script

bucket_name='azriel-peace-demo-bash'
file_path='web_app.tar'
object_key='web_app.tar'

curl --fail \
  -o "${file_path}" \
  --location \
  https://github.com/azriel91/web_app/releases/download/0.1.1/web_app.tar

aws s3api create-bucket \
  --bucket "${bucket_name}" \
  --acl private \
  --create-bucket-configuration LocationConstraint=ap-southeast-2 |
  bat -l json

aws s3api put-object \
  --bucket "${bucket_name}" \
  --key "${object_key}" \
  --body "${file_path}" |
  bat -l json

aws s3api delete-object \
  --bucket "${bucket_name}" \
  --key "${object_key}" |
  bat -l json

aws s3api delete-bucket --bucket "${bucket_name}" | bat -l json

rm -f "${file_path}"

What does automation look like, from the perspective of an automation tool developer, or a workflow designer.

• Clarity between concept and code.
• Easy to write.
• Fast feedback when developing automation.
• Provide good UX without needing to write UI code.

---

Command Context

At the bottom of every page, we will be building up a cmd_context, which holds the information about a workflow. Different commands can be invoked with the cmd_context to gather information about the workflow.

let mut cmd_context = CmdContext::builder()
    /* ?? */
    .build();

StatusCmd(&mut cmd_context).await?;
DeployCmd(&mut cmd_context).await?;

Flow Definition

• Items: Steps of a process.
• Ordering: Sequence between steps.

// examples/envman/src/flows/app_upload_flow.rs
let flow = {
    let graph = {
        let mut graph_builder = ItemGraphBuilder::<EnvManError>::new();

        let [a, b, c] = graph_builder.add_fns([
            FileDownloadItem::<WebApp>::new(item_id!("app_download")).into(),
            S3BucketItem::<WebApp>::new(item_id!("s3_bucket")).into(),
            S3ObjectItem::<WebApp>::new(item_id!("s3_object")).into(),
        ]);

        graph_builder.add_logic_edges([(a, b), (b, c)])?;
        graph_builder.build()
    };

    Flow::new(flow_id!("app_upload"), graph)
};

Non-linear Ordering

 graph_builder.add_logic_edges([
-    (a, b),
+    (a, c),
     (b, c),
 ])?;

Inputs and Outputs

For items to be reusable, its inputs and outputs are API.

Item API

impl<Id> Item for FileDownloadItem<Id>
{
    type Params<'exec> = FileDownloadParams<Id>;
    type State = FileDownloadState;
    // ..
}

Input:

pub struct FileDownloadParams<Id> {
    src: Url,
    dest: PathBuf,
    marker: PhantomData<Id>,
}

Output:

pub enum FileDownloadState {
    None,
    Some {
        path: PathBuf,
        md5: Md5Sum,
    },
}

Parameters Specification

Specify where to get the value for each item's input.

The value may not necessarily exist until the flow is executed.

let app_download_params = FileDownloadParams::<WebApp> {
    src: Url::parse("https://example.com/web_app.tar")?,
    dest: PathBuf::from("/tmp/path/to/web_app.tar"),
    marker: PhantomData,
};

let s3_bucket_params = S3BucketParams::<WebApp>::new(bucket_name);

let s3_object_params = S3ObjectParams::<WebApp> {
    file_path: PathBuf::from("/tmp/path/to/web_app.tar"),
    object_key: String::from("web_app.tar"),
    bucket_name: !?, /* Somehow get the bucket name from `b` */
};

Deferred Values

// examples/envman/src/flows/app_upload_flow.rs
let s3_object_params_spec = S3ObjectParams::<WebApp>::field_wise_spec()
    .with_file_path(PathBuf::from("/tmp/path/to/web_app.tar"))
    .with_object_key(String::from("web_app.tar"))
    .with_bucket_name_from_map(|s3_bucket_state: &S3BucketState| {
        match s3_bucket_state {
            S3BucketState::None => None,
            S3BucketState::Some {
                name,
                creation_date: _,
            } => Some(name.clone()), // type safe!
        }
    })
    .build();

let file_download_params_spec = file_download_params.into();
let server_instance_params_spec = server_instance_params.into();

Output and Presentation

When the workflow is defined, we want to interact with it in different ways.

Presentation

Command Line Interface

> ./envman deploy
Using profile ⦗demo⦘ -- type `development`

✅ 1. app_download ▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰ done!
⏳ 2. s3_bucket    ▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱ 0/1 creating bucket (el: 6s, eta: 0s)
⏳ 3. s3_object    ▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱ (el: 0s, eta: 0s)

Continuous Integration

2023-05-07T01:30:31.135Z | app_download | downloaded at /tmp/.a9efb2/web_app.tar
2023-05-07T01:30:31.754Z | s3_bucket    | creating bucket
2023-05-07T01:30:32.687Z | s3_bucket    | bucket created

Web API

// GET /api/v1/demo/env_deploy/progress
{
    "item_id": "s3_bucket",
    "progress_update": { "Delta": "Tick" },
    "msg_update": { "Set": "creating bucket" }
    // ..
}

Web page

Current
State

Deploy
(preview)

Clean
(preview)

▶

// outputs to CLI
let mut output = CliOutput::new();

// === Used in tests === //
// discards all output
let mut output = NoOpOutput;

// records which output functions were called
// and what parameters they were called with
let mut output = FnTrackerOutput::new();

// === Write your own === //
// writes JSON to HTTP response
let mut output = JsonResponseOutput::new();

Command Invocation

// examples/envman/src/cmds/profile_init_cmd.rs
// fn app_upload_flow_init
let cmd_ctx = CmdCtx::builder_single_profile_single_flow
    ::<EnvManError, _>(output, workspace)
    .with_profile_from_workspace_param(profile_key)
    .with_flow(flow)
    .with_item_params::<FileDownloadItem<WebApp>>(
        item_id!("app_download"),
        app_download_params_spec,
    )
    .with_item_params::<S3BucketItem<WebApp>>(
        item_id!("s3_bucket"),
        s3_bucket_params_spec,
    )
    .with_item_params::<S3ObjectItem<WebApp>>(
        item_id!("s3_object"),
        s3_object_params_spec,
    )
    .await?;

// examples/envman/src/cmds/env_status_cmd.rs
// envman status
StatesCurrentReadCmd::exec(&mut cmd_ctx).await?;

// examples/envman/src/cmds/env_deploy_cmd.rs
// envman deploy
EnsureCmd::exec(&mut cmd_ctx).await?;

Parameters Recall

./envman init demo --type development azriel91/web_app 0.1.1

# For good user experience, parameters
# should not have to be passed in on
# every command invocation

./envman status  # demo --type development azriel91/web_app 0.1.1
./envman deploy  # demo --type development azriel91/web_app 0.1.1

We need to store and load the parameters passed in previously.

First command invocation:

// examples/envman/src/cmds/profile_init_cmd.rs
// fn app_upload_flow_init
cmd_ctx_builder
    .with_profile_from_workspace_param(profile_key)
    .with_flow(flow)
    .with_item_params::<FileDownloadItem<WebApp>>(
        item_id!("app_download"),
        app_download_params_spec,
    )
    .with_item_params::<S3BucketItem<WebApp>>(
        item_id!("s3_bucket"),
        s3_bucket_params_spec,
    )
    .with_item_params::<S3ObjectItem<WebApp>>(
        item_id!("s3_object"),
        s3_object_params_spec,
    )
    .await?

Subsequent command invocations:

// examples/envman/src/cmds/app_upload_cmd.rs
cmd_ctx_builder
    .with_profile_from_workspace_param(&profile_key)
    .with_flow(&flow)
    // * file_download params spec not specified
    // * s3_bucket params spec not specified
    .with_item_params::<S3ObjectItem<WebApp>>(
        item_id!("s3_object"),
        s3_object_params_spec,
    )
    .await?

let s3_object_params_spec = S3ObjectParams::<WebApp>::field_wise_spec()
    // Note:
    //
    // * file_path not specified
    // * object key not specified
    // * Function logic cannot be deserialized,
    //   so needs to be provided
    .with_bucket_name_from_map(|s3_bucket_state: &S3BucketState| {
        match s3_bucket_state {
            S3BucketState::None => None,
            S3BucketState::Some {
                name,
                creation_date: _,
            } => Some(name.clone()),
        }
    })
    .build();

Workspace

When a command is run, Peace stores information about that execution.

It stores this in a .peace directory within a workspace directory. The automation tool developer needs to tell Peace how to determine the workspace directory.

let cmd_ctx = CmdCtx::builder_single_profile_single_flow
    ::<EnvManError, _>(output, workspace)
    // ..
    .await?;

pub enum WorkspaceSpec {
    /// Use the exe working directory as the workspace directory.
    ///
    /// The working directory is the directory that the user ran the program in.
    WorkingDir,
    /// Use a specified path.
    Path(PathBuf),
    /// Traverse up from the working directory until the given file is found.
    ///
    /// The workspace directory is the parent directory that contains a file or
    /// directory with the provided name.
    FirstDirWithFile(OsString),
}

Profiles

Profiles are a way of logically separating environments, aka namespacing.

First execution / init:

let cmd_ctx_builder = CmdCtx::builder_single_profile_no_flow
    ::<EnvManError, _>(output, &workspace)
    .with_profile(profile!("demo"))

    // for recall
    .with_workspace_param_value(
        WorkspaceParamsKey::Profile,
        Some(profile!("demo")),
    );

Subsequent executions:

let cmd_ctx_builder = CmdCtx::builder_single_profile_single_flow
    ::<EnvManError, _>(output, workspace)
    .with_profile_from_workspace_param(WorkspaceParamsKey::Profile)
    .with_flow(flow);

Summary

How do we communicate clearly in code?

• Consistency between the conceptual model and the written code.
• Make use of type safety with strong types.
• When we require something of the developer, make it easy to be provided.
• Prefer errors at compile time than runtime.
• Error messages should show what is wrong and how to fix it.

Q & A

Interruptible Software

▶️

⏸️

⏹️

Processes

Build Process

Deployment Process

Execution

Manual

Automatic

Bus Analogy

🚏
🚌🚌

Designing Interruptibility

🛑 Stop Button

🚏 Stopping Point

🔀 Parallelism

📃 Reporting

🗺️ Strategy

🛑 Stop Button

Command Line

Web

Desktop

> deploy_

🗄️🗄️🌐

x

go

❌ Don't

let outcome = tokio::select! {
    value = process.next()          => Outcome::Done(value),
        _ = tokio::signal::ctrl_c() => Outcome::Interrupted,
};

✅ Do

struct InterruptSignal;
let (interrupt_tx, interrupt_rx) = tokio::sync::mpsc::<InterruptSignal>(8);

let outcome = tokio::select! {
    value = process.next()      => Outcome::Done(value),
        _ = interrupt_rx.recv() => Outcome::Interrupted,
};

and:

// CLI
tokio::spawn(async move {
    tokio::signal::ctrl_c().await.unwrap();
    let _ = interrupt_tx.send(InterruptSignal).await;
});

// Web / Desktop
async fn handle_request(request: Request) -> Response {
    match request {
        Request::Interrupt { execution_id } => {
            let executions = executions.lock().await;
            let execution = executions.get(execution_id);
            let interrupt_tx = execution.interrupt_tx();
            let _ = interrupt_tx.send(InterruptSignal).await;
        }
        // ..
    }
}

🚏 Stopping Point

🛑 Interrupt

🚏 Stop

No Interruptibility

fn execute(
    params: Params,
) -> Outcome {
    let output_1 = step_1(params);
    let output_2 = step_2(output_1);
    let outcome = step_3(output_2);

    return outcome;
}

Basic

fn execute(
    interrupt_rx: mut Receiver<InterruptSignal>,
    params: Params,
) -> ControlFlow<InterruptSignal, Outcome> {
    let () = interruptibility_check(&mut interrupt_rx)?;
    let output_1 = step_1(params);

    let () = interruptibility_check(&mut interrupt_rx)?;
    let output_2 = step_2(output_1);

    let () = interruptibility_check(&mut interrupt_rx)?;
    let outcome = step_3(output_2);

    ControlFlow::Continue(outcome)
}

fn interruptibility_check(receiver: &mut Receiver<InterruptSignal>)
-> ControlFlow<InterruptSignal, ()> {
    if let Ok(interrupt_signal) = interrupt_rx.try_recv() {
        ControlFlow::Continue(())
    } else {
        ControlFlow::Break(interrupt_signal)
    }
}

Fine Grained Interruptibility

fn execute(
    interrupt_rx: mut Receiver<InterruptSignal>,
    params: Params,
) -> ControlFlow<InterruptSignal, Outcome> {
    let () = interruptibility_check(&mut interrupt_rx)?;
    let output_1 = step_1(&mut interrupt_rx, params);

    // ..
}

fn step_1(
    interrupt_rx: mut Receiver<InterruptSignal>,
    params: Params,
) -> ControlFlow<Output1, Outcome> {
    let mut output_1 = Output1::new();
    for i in 0..1_000_000 {
        if i % 1_000 == 0 {
            let () = interruptibility_check(&mut interrupt_rx)?;
        }
        do_something(output_1, params, i);
    }

    ControlFlow::Continue(output_1)
}

🔀 Parallelism

Parallelism and concurrency

🛑 Interrupt

🛑 Interrupt

🛑 Interrupt

🚏 Stop

🚏 Stop

🚏 Stop

Safe Interruption Rules

1. 🔵 Finish everything in progress.
2. ⚫ Don't start anything new.

See fn_graph on Github.

• Queuer:
  • Sends IDs of steps that can be executed.
  • Receives IDs of steps that are complete.
  • Checks for interruption.
• Scheduler
  • Receives IDs of steps that can be executed.
  • Sends IDs of steps that are complete.

📃 Reporting

Non-Interruptible Software

fn execute(params: Params) -> Value {
    // ..
}

Interruptible Software

fn execute(params: Params) -> Outcome {
    // ..
}

Outcome

Basic

enum Outcome {
    /// Execution completed, here is the return value.
    Complete(Value),
    /// Execution was interrupted.
    Interrupted,
}

Step Values

enum Outcome {
    /// Execution completed, here is the return value.
    Complete(Value),
    /// Execution was interrupted.
    ///
    /// Here's the information we collected so far.
    Interrupted {
        step_1_value: Option<Step1Value>,
        step_2_value: Option<Step2Value>,
        step_3_value: Option<Step3Value>,
    },
}

Step Values and Execution Info

enum Outcome {
    /// Execution completed, here is the return value.
    Complete(Value),
    /// Execution was interrupted.
    ///
    /// Here's the information we collected so far.
    Interrupted {
        step_1_value: Option<Step1Value>,
        step_2_value: Option<Step2Value>,
        step_3_value: Option<Step3Value>,
        steps_processed: Vec<StepId>,
        steps_not_processed: Vec<StepId>,
    },
}

🗺️ Strategy

🛑 Interrupt

🚏 Stop

🚏 Stop

🚏 Stop

/// How to poll an underlying stream when an interruption is received.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum InterruptStrategy {
    /// On interrupt, keep going.
    IgnoreInterruptions,
    /// On interrupt, wait for the current future's to complete and yield its
    /// output, but do not poll the underlying stream for any more futures.
    FinishCurrent,
    /// On interrupt, continue polling the stream for the next `n` futures.
    ///
    /// `n` is an upper bound, so fewer than `n` futures may be yielded if the
    /// underlying stream ends early.
    PollNextN(u64),
}

Interruptible Implementation

No Interruptibility

fn execute(
    params: Params,
) -> Outcome {
    let output_1 = step_1(params);
    let output_2 = step_2(output_1);
    let outcome = step_3(output_2);

    return outcome;
}

Basic

fn execute(
    interrupt_rx: mut Receiver<InterruptSignal>,
    params: Params,
) -> ControlFlow<InterruptSignal, Outcome> {
    let () = interruptibility_check(&mut interrupt_rx)?;
    let output_1 = step_1(params);

    let () = interruptibility_check(&mut interrupt_rx)?;
    let output_2 = step_2(output_1);

    let () = interruptibility_check(&mut interrupt_rx)?;
    let outcome = step_3(output_2);

    ControlFlow::Continue(outcome)
}

fn interruptibility_check(receiver: &mut Receiver<InterruptSignal>)
-> ControlFlow<InterruptSignal, ()> {
    if let Ok(interrupt_signal) = interrupt_rx.try_recv() {
        ControlFlow::Continue(())
    } else {
        ControlFlow::Break(interrupt_signal)
    }
}

"Better"

fn execute<T>(
    interrupt_rx: mut Receiver<InterruptSignal>,
    params: Params,
) -> Result<T, InterruptSignal> {
    [step_1, step_2, step_3]
        .into_iter()
        .try_fold(params, |(mut last_param, step)| {
            interruptibility_check(&mut interrupt_rx)?;
            step(last_param)
        })
}

Real Code

Playground

use std::any::Any;
use std::iter::IntoIterator;
use std::marker::PhantomData;

fn main() {
    match execute::<String, _>(Receiver::new(3), true) {
        Ok(value) => println!("main: {}", value),
        Err(InterruptSignal) => println!("main: interrupted!"),
    }
}

fn execute<T, I>(
    mut interrupt_rx: Receiver<InterruptSignal>,
    params: I,
) -> Result<T, InterruptSignal>
where
    T: 'static,
    I: 'static,
{
    [step(step_1), step(step_2), step(step_3)]
        .into_iter()
        .try_fold(Box::new(params) as Box<dyn Any>, |last_param, step| {
            interruptibility_check(&mut interrupt_rx)?;
            Ok(step.exec(last_param))
        })
        .map(|boxed_any| *boxed_any.downcast().unwrap())
}

fn step_1(takes_bool: bool) -> u32 {
    let number = 123;
    println!("step_1 ({takes_bool}: bool) -> {number}");
    number
}

fn step_2(takes_u32: u32) -> f64 {
    let float = 1.5;
    println!("step_2 ({takes_u32}: u32) -> {float}");
    float
}

fn step_3(takes_f64: f64) -> String {
    let string = String::from("magic");
    println!("step_3 ({takes_f64}: f64) -> {string}");
    string
}

fn step<F, I, O>(f: F) -> Box<dyn StepErased>
where
    F: Fn(I) -> O + 'static,
    I: 'static,
    O: 'static,
{
    Box::new(StepWrapper::new(f))
}

trait Step {
    type Input: Any;
    type Output: Any;
    fn exec(&self, input: Self::Input) -> Self::Output;
}

impl<F, I, O> Step for StepWrapper<F, I, O>
where
    F: Fn(I) -> O,
    I: 'static,
    O: 'static,
{
    type Input = I;
    type Output = O;

    fn exec(&self, input: Self::Input) -> Self::Output {
        (self.0)(input)
    }
}

trait StepErased {
    fn exec(&self, input: Box<dyn Any>) -> Box<dyn Any>;
}

struct StepWrapper<F, I, O>(F, PhantomData<(I, O)>);
impl<F, I, O> StepWrapper<F, I, O> {
    fn new(f: F) -> Self {
        Self(f, PhantomData)
    }
}

impl<F, I, O> StepErased for StepWrapper<F, I, O>
where
    StepWrapper<F, I, O>: Step<Input = I, Output = O>,
    I: 'static,
    O: 'static,
{
    fn exec(&self, input: Box<dyn Any>) -> Box<dyn Any> {
        let input = *input.downcast::<I>().unwrap();
        let output = Step::exec(self, input);
        Box::new(output)
    }
}

struct Receiver<T>(u32, PhantomData<T>);
impl<T> Receiver<T> {
    pub fn new(n: u32) -> Self {
        Self(n, PhantomData)
    }
}
#[derive(Debug)]
struct InterruptSignal;
fn interruptibility_check(rx: &mut Receiver<InterruptSignal>) -> Result<(), InterruptSignal> {
    if (rx.0) == 0 {
        Err(InterruptSignal)
    } else {
        rx.0 -= 1;
        Result::Ok(())
    }
}

Resumability

🟢 Start 1

🚏 Stop

🟢 Start 2

Designing Resumability

🕸️ Executor Resumability

💮 Stepwise Resumability

🕸️ Executor Resumability

 fn execute<T>(
    execution_state: ExecutionState, // Add this
    interrupt_rx: mut Receiver<InterruptSignal>,
//    params: Params,                // Remove this
) -> Result<T, InterruptSignal> {
    let mut steps = [step_1, step_2, step_3];

    let (steps, params) = match &execution_state {
        ExecutionState::CleanSlate { params } => (steps, params),
        ExecutionState::PreviousState { step_ids_not_done, step_values, .. } => {
            steps.retain(|step| step_ids_not_done.contains(step.id()));
            let params = step_values.last().unwrap();

            (steps, params)
        }
    };

    steps
        .into_iter()
        .try_fold(params, |(mut last_param, step)| {
            interruptibility_check(&mut interrupt_rx)?;
            step(last_param)
        })
}

#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)]
pub enum ExecutionState {
    CleanSlate {
        params: Params,
    },
    PreviousState {
        step_ids_done: Vec<StepId>,
        step_ids_not_done: Vec<StepId>,
        step_values: Vec<Value>,
    },
}

Pros / Cons

• Simpler API: Step logic does not need to know about interruptibility concepts.
• Stale Reasoning: Saved state may be stale, and the executor does not have the means to check for staleness.

💮 Stepwise Resumability

 fn execute<T>(
    execution_state: ExecutionState, // Add this
    interrupt_rx: mut Receiver<InterruptSignal>,
) -> Result<T, InterruptSignal> {
    [step_1, step_2, step_3]
        .into_iter()
        .try_fold(params, |(mut last_param, step)| {
            interruptibility_check(&mut interrupt_rx)?;

            let step_state = execution_state.get(step.id());
            step(step_state, last_param)
        })
}

#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)]
pub struct ExecutionState { .. }

impl ExecutionState {
    pub fn get<T>(step_id: StepId) -> T { .. }
}

Pros / Cons

• Heavier API: Step logic needs to cater for initial state.
• Usage Safety: Step can determine if the saved initial state is in-sync with actual state, and warn user.

Resumable Implementation

Playground

use std::any::Any;
use std::fmt::Debug;
use std::iter::IntoIterator;
use std::marker::PhantomData;

fn main() {
    let _execution_state = ExecutionState::clean_slate(true);
    let execution_state = ExecutionState::PreviousState {
        step_ids_done: vec![1, 2],
        step_ids_not_done: vec![3],
        step_values: vec![Box::new(234u32), Box::new(3.5f64)],
    };

    match execute::<String>(execution_state, Receiver::new(3)) {
        Ok(value) => println!("main: {}", value),
        Err(InterruptSignal) => println!("main: interrupted!"),
    }
}

fn execute<T>(
    execution_state: ExecutionState,
    mut interrupt_rx: Receiver<InterruptSignal>,
) -> Result<T, InterruptSignal>
where
    T: 'static,
{
    let steps = vec![step(1, step_1), step(2, step_2), step(3, step_3)];
    let (steps_to_execute, params) = filter_what_to_do(execution_state, steps);

    steps_to_execute
        .into_iter()
        .try_fold(params, |last_param, step| {
            interruptibility_check(&mut interrupt_rx)?;
            Ok(step.exec(last_param))
        })
        .map(|boxed_any| *boxed_any.downcast().unwrap())
}

fn filter_what_to_do(
    execution_state: ExecutionState,
    mut steps: Vec<Box<dyn StepErased>>,
) -> (Vec<Box<dyn StepErased>>, Box<dyn Any>) {
    match execution_state {
        ExecutionState::CleanSlate { params } => (steps, params),
        ExecutionState::PreviousState {
            step_ids_not_done,
            mut step_values,
            ..
        } => {
            steps.retain(|step| step_ids_not_done.contains(&step.id()));
            let params = step_values.pop().unwrap();

            (steps, params)
        }
    }
}

fn step_1(takes_bool: bool) -> u32 {
    let number = 123;
    println!("step_1 ({takes_bool}: bool) -> {number}");
    number
}

fn step_2(takes_u32: u32) -> f64 {
    let float = 1.5;
    println!("step_2 ({takes_u32}: u32) -> {float}");
    float
}

fn step_3(takes_f64: f64) -> String {
    let string = String::from("magic");
    println!("step_3 ({takes_f64}: f64) -> {string}");
    string
}

fn step<F, I, O>(id: u32, f: F) -> Box<dyn StepErased>
where
    F: Fn(I) -> O + 'static,
    I: 'static,
    O: 'static,
{
    Box::new(StepWrapper::new(id, f))
}

trait Step {
    type Input: Any;
    type Output: Any;
    fn id(&self) -> u32;
    fn exec(&self, input: Self::Input) -> Self::Output;
}

impl<F, I, O> Step for StepWrapper<F, I, O>
where
    F: Fn(I) -> O,
    I: 'static,
    O: 'static,
{
    type Input = I;
    type Output = O;

    fn id(&self) -> u32 {
        self.0
    }

    fn exec(&self, input: Self::Input) -> Self::Output {
        (self.1)(input)
    }
}

trait StepErased {
    fn id(&self) -> u32;
    fn exec(&self, input: Box<dyn Any>) -> Box<dyn Any>;
}

struct StepWrapper<F, I, O>(u32, F, PhantomData<(I, O)>);
impl<F, I, O> StepWrapper<F, I, O> {
    fn new(id: u32, f: F) -> Self {
        Self(id, f, PhantomData)
    }
}

impl<F, I, O> StepErased for StepWrapper<F, I, O>
where
    StepWrapper<F, I, O>: Step<Input = I, Output = O>,
    I: 'static,
    O: 'static,
{
    fn id(&self) -> u32 {
        self.0
    }

    fn exec(&self, input: Box<dyn Any>) -> Box<dyn Any> {
        let input = *input.downcast::<I>().unwrap();
        let output = Step::exec(self, input);
        Box::new(output)
    }
}

struct Receiver<T>(u32, PhantomData<T>);
impl<T> Receiver<T> {
    pub fn new(n: u32) -> Self {
        Self(n, PhantomData)
    }
}
#[derive(Debug)]
struct InterruptSignal;
fn interruptibility_check(rx: &mut Receiver<InterruptSignal>) -> Result<(), InterruptSignal> {
    if (rx.0) == 0 {
        Err(InterruptSignal)
    } else {
        rx.0 -= 1;
        Result::Ok(())
    }
}

#[allow(dead_code)]
#[derive(Debug)]
enum ExecutionState {
    CleanSlate {
        params: Box<dyn Any>,
    },
    PreviousState {
        step_ids_done: Vec<u32>,
        step_ids_not_done: Vec<u32>,
        step_values: Vec<Box<dyn Any>>,
    },
}

#[allow(dead_code)]
impl ExecutionState {
    fn clean_slate<I>(i: I) -> Self
    where
        I: 'static,
    {
        ExecutionState::CleanSlate {
            params: Box::new(i),
        }
    }
}

Real Crates

📦🦀

• 🗂️ interruptible: Stops a stream from producing values when interrupted.
• 🧬 fn_graph: Dynamically managed function graph execution. Integrates with interruptible.
• 🕊️ peace: Framework to build empathetic and forgiving software automation.

Summary

Q & A

🦂🦞🦀⁉️

source

Dot IX: Diagram Generator

Picture Your Understanding

https://azriel.im/dot_ix

Azriel Hoh

August 2024

1. Problem: Diverge

   1. Diagram types:
      1. Process diagrams
      2. Deployment diagrams (outcome)
      3. Class Diagrams
      4. Gantt charts
   2. Clarity / Appearance / Styling
      1. Nodes, Edges, Text
      2. Animations
      3. Lossless Resolution
      4. Interactive
      5. Level of detail
   3. Input
      1. Labels for nodes, edges
      2. Styles
      3. Common input
      4. Software library
   4. Where the diagrams will be displayed
      1. Web app
      2. Github comments
      3. Any documentation software

2. Problem: Converge

   1. Diagram types:
      1. Process diagrams
      2. Deployment diagrams (outcome)
      3. Class Diagrams
      4. Gantt charts
   2. Clarity / Appearance / Styling
      1. Nodes, Edges, Text
      2. Animations
      3. Lossless Resolution
      4. Interactive
      5. Level of detail
   3. Input
      1. Labels for nodes, edges
      2. Styles
      3. Common input
      4. Software library
   4. Where the diagrams will be displayed
      1. Web app
      2. Github comments
      3. Any documentation software

3. Solution: Diverge

   • What solutions can we do?
   • How does each solution address the concerns in the problem space we have chosen?

   Diagrams tend to be an abstraction over something actual.

   1. Output format:
      1. SVG
      2. HTML elements
      3. HTML canvas
      4. Image (pixels)
   2. Input format:
      1. Visual drawing tool
      2. Structured input, e.g. YAML, JSON
      3. Software library

4. Solution: Converge

   1. Output format:
      1. SVG: without foreignObject
      2. HTML elements
      3. HTML canvas
      4. Image (pixels)
   2. Input format:
      1. Visual drawing tool
      2. Structured input, e.g. YAML, pass through graphviz for layout
      3. Software library

Everyday Diagrams

Process Diagrams

source

Deployment Diagrams

source

🛑 Problems

Appearance

Annoying To Change Many Objects

source

1. Changing colours for the same type of object
2. Positioning objects "nicely"

Accuracy

Drift

The diagrams are out of date

source

1. Input is inherently manual.
2. Code tends to evolve separately to documentation.

Abstraction

Diagrams tend to be an abstraction over the detail. That's okay!

Portability

How To Save Diagrams

source

• Store output diagram:

  • 🟢 Don't need to regenerate.
  • 🔴 Cannot edit later.

• Store input:

  • 🟢 Can edit / update diagram.
  • 🔴 Must have tooling to regenerate diagram.

Where Diagrams Are Uploaded / Viewed

• Internal documentation site
• Github / blogs
• Instant messaging

Dynamics

Static

Example 1

source

Example 2

source

Example 3

1. 🟢 Good for showing a snapshot in time.
2. 🟢 Good for printing.
3. 🔴 Messy when showing a lot of overlapping information.

Animated

Example 1

source

1. 🟢 Able to show the sequence of things on screen.
2. 🔴 Still susceptible to information overload.

Interactive

Example 1

source

1. Highlight / Focus on the part of the diagram that is relevant.
2. Hide / dim parts of the diagram that are not relevant.

💡 Ideation

Output Format

	🖼️ PNG	🌐 HTML	🎨 SVG
Externally Uploadable	🟢	🔴	🟡
Styling	🟢	🟢	🟢
Lossless Resolution	🔴	🟢	🟢
Interactive	🔴	🟢	🟡
Dynamic Level of Detail	🔴	🟢	🔴

Input Format

Problem	Solution
Easy multi-object edits	📝 Text format, good text editor
Compact	📄 YAML for referenceable values
Out-of-the-box styling	🖼️ Default theme
Customizable Styling	🎨 Styling by ID with TailwindCSS

Logistics

Problem	Solution
Free Hosting	🐙 Github Pages
No Web Server	💻 Client side logic
Easy diagram sharing	🔖 Diagram stored in URL 📝 Copy SVG to clipboard
No intellectual property transmission	🔖 Diagram stored in URL fragment

🎨 Demo

🧪 Solution

Supported Usages

source

Generation Process

source

• Example Diagram
• GraphViz Cluster
• GraphViz SVG

Challenges

Security Constraints

1. No inline javascript.
2. Self-contained / no external resources.

Adhering to these constraints without subtracting from:

1. 🎨 Appearance
2. 🧸 Interactivity

Animation

Diagram

source

source

source

1. Supported visual representations -- what representations provide the best clarity.
2. Encoding that into the diagram:
   1. SVG fill="url(#gradient)" doesn't work across all browsers.
   2. Effort to write it can be high.
3. Provide an abstraction in the data model to reduce effort.

What's Next

1. 🖼️ Support images properly.

2. 🖌️ Use encre -- Rust version of Tailwind.

3. 🔏 Escape values.

4. 🏛️ Stabilize API.

Q & A

🦀🎨🖼️

Peace

Zero Stress Automation

https://peace.mk/

Azriel Hoh

October 2024

Notes

1. Heya everyone, my name is Azriel, and today I'll be showing you my automation side project, called Peace.

2. Peace is a framework to create user friendly automation.

3. It provides a set of constraints and common functionality, that when you write code according to these constraints, and plug them into the common functionality, you get a nice tool.

4. As with every side project, there is an origin story.

5. In my first job, a team of us were given the task of fully automating our solution deployment.

6. So this was the deployment process, it's all manual.

7. and, obviously this is the solution. Genius.

8. We wanted this process: click, wait, done.

9. And through engineering eyes, we aimed for:. (linking: Dimensions)

10. End-to-end automation.

11. Repeatable correctness.

12. and Performance.

13. and we delivered!

14. If you measure success using these metrics, it was undeniable.

15. We reduced the deployment duration down from 2 weeks of manual steps, to 30 minutes.

16. However, our users said "we hate this!". Azriel, this doesn't enhance our lives.

17. What we really did when we introduced automation, was this.

18. When switching from manual steps to automation, the work changes from doing each step at your own pace, to setting up the parameters for all of the steps, pressing go, and waiting.

19. When it's done, you check if your parameters were correct.

20. If they're correct, that's fine.

21. If they weren't, then you had to understand the error, figure out which step it came from, and which parameters feed into that step.

22. Then when the parameter is fixed, which may take 30 seconds, they still had to wait 30 minutes to confirm if their fix worked,

23. and this delayed feedback loop was frustrating.

24. For new users, it was especially painful:

25. We're telling them to fill in parameters that they don't understand,

26. to feed into a process that they cannot see,

27. to create an environment, that they cannot visualize.

28. So they may not have understood what they were doing, but it was certainly our fault.

29. We created pain.

30. We had engineering eyes, but not human eyes:

31. We took away understandability,

32. We took away control.

33. And when you take away understanding and control, you inadvertently also take away morale. What little they had.

34. Ideally we should have built something that provides the benefits of automation,

35. while retaining the benefits of manual execution.

36. This is what the Peace framework aims to do.

37. And today I'd like to show you how it does this, through a tool built using the Peace framework.

38. It's called envman, short for environment manager.

39. envman automates the download of a web application from github, creates some resources in Amazon, and uploads the web application.

40. Notably there's a missing step to launch a server that runs the web application, but I'm all out of AWS credits. So if you have spare, I'd gladly take them.

41. The first thing we took away was understandability, so let's put that back.

42. There are two ways we tend to write automation:

43. Either we produce too little information, and we can't tell what's going on,

44. or, we produce too much information, and we still can't tell what's going on.

45. For understandability, we need to have something in between.

46. Let's take a look.

47. This is what it looks like when you have too little information:

48. clear, ./envman deploy --format none, clean.

49. Something's going on, I promise!

50. This is what it looks like when you have too much information:

51. ./envman deploy --format json, clean.

52. And finally, something in between.

53. See if you can see how many steps there are in this process, and whether they complete successfully:

54. clear, ./envman deploy.

55. How many steps are there? gesture

56. Did every step complete successfully?

57. "Green means good", so I believe so.

58. What resources were created? gesture

59. And if we clean up the environment, you'll see a similar interface, so you can tell that each resource is deleted: ./envman clean.

60. That's all good when things go well, but what happens in a failure? Can we understand it?

61. First we'll limit the connection speed of the tool to 40 kilobits per second:

    New-NetQosPolicy `
      -Name "envman" `
      -AppPathNameMatchCondition "envman.exe" `
      -PolicyStore ActiveStore `
      -ThrottleRateActionBitsPerSecond 40KB
    

62. and run the deployment again: clear, ./envman deploy.

63. You can see that our download from github has slowed,

64. and in a little while we should see an error happen.

65. Here we go.

66. With fresh eyes, can you see which step went wrong?

67. Red means bad, so it should be apparent.

68. In detail, what went wrong, why it went wrong, and how to recover, are all shown.

69. We failed to upload the object. Why? The upload timed out, and make sure you are connected to the internet and try again.

70. We're also shown which resources exist and which don't, so we don't have to guess.

71. If we fix our connection, and re-run the automation:

    Remove-NetQosPolicy `
      -Name "envman" `
      -PolicyStore ActiveStore `
      -Confirm:$false
    

72. You'll see that it picks up where it left off, and completes the process.

73. That is, what you think it should do, it does. No surprises.

74. So in summary, with information, the goldilocks principle applies:

75. Too much information is overwhelming, too little is not useful, and there's some middle ground which is just right.

76. The Peace framework generally tries to fit the most relevant information on one screen.

77. The second thing we took away, was control.

78. Most automation tools give you one button -- start -- and that's it.

79. Start the creation, or update, and start the deletion.

80. While pressing start is not difficult, knowing whether the automation will do what we think it will, before we press start, is difficult.

81. What we should understand before starting anything, is:

82. Where we are -- our current state,

83. Where we want to go -- our goal state, and

84. The distance between the two.

85. Because if we start with nothing, and end up with something, the distance is something.

86. And if we start with something, and our goal state is something, the distance is nothing.

87. And if we start with something, and our goal state is something else, the distance is that else

88. When we understand these three things, then we can make an informed decision if we should press go.

89. Now, if we press start, and change our mind, can we stop the process?

90. Without automation, we can.

91. Like, if someone said, "Azriel! Stop work."

92. I'd say, "Gladly." I can stop where I am.

93. With automation, you need to intentionally build interruptibility into the process.

94. And while pressing Ctrl C on a command line tool is one form of interruption,

95. what we really care about, is safe interruption.

96. i.e. Stop what you're doing when it is safe to do so.

97. Maybe we're at step 5 of a 10 step process, and we want to adjust the parameter for step 7.

98. If we can interrupt the process, adjust the parameters, press go, and have the automation pick up where it left off, that would be great.

99. As in, don't undo all of the work you've already done to get to this point.

100. I just want to fix the parameter for the later step, and continue.

101. Let's see all of this control, in action.

102. Before we run our deployment, what is our environment's current state.

103. Just like we can run git status, we can also run ./envman status.

104. What state will the automation bring our environment to, when we run it? ./envman goal

105. What's the difference? ./envman diff

106. The commands are intentionally similar to git commands so we make use of familiar names.

107. And for interruptibility, when we deploy, we'll stop the process halfway.

108. ./envman deploy, ctrl c.

109. Here you can see steps 1 through 3, and step 5 were complete,

110. and step 4 and 6 were not started due to the interruption.

111. If we look at the diff: ./envman diff,

112. you can see that steps 1, 2, 3, and 5 are done, steps 4 and 6 haven't been executed.

113. If we change our parameters, to using version 0.1.2 instead of 0.1.1 of our web application,

114. the diff will now show that step 1 will change.

115. And if we run deploy again, that is exactly what happens.

116. When cleaning up, we can also interrupt the process.

117. Steps 1, 4, 5, and 6 were cleaned, and 2 and 3 were not.

118. And we can choose to either deploy the environment again, or clean up fully.

119. Let's deploy it to completion. deploy, clean.

120. What's the use of this?

121. Well there was once we were told,

122. "hey this customer doesn't need their environment anymore, you can delete it."

123. "You sure?"

124. "Yes."

125. So we started the deletion process, and we got this "Hey stop. Stop what you're doing."

126. "We can't. It's all just going to go."

127. And that was the beginning of a very exciting day.

128. So, build a stop button into your automation people.

129. If you use Peace, it is built in for you.

130. We've given back to the user some control, but there are other things still to be implemented like running a subset of the process.

131. Not too hard to implement, just needs time.

132. Morale.

133. Not everyone who uses automation tools has a software background, and not everyone uses the command line all the time.

134. So why not create something that caters for these situations as well?

135. Back to understandability, normally when explaining what automation does,

136. we tend to draw a diagram on the whiteboard,

137. or create a diagram in an internal documentation site.

138. However, it's never really accurate, and it's usually a tangle of overlapping boxes and lines,

139. so it is hard to understand, because the information isn't clear.

140. So here's a web interface. ./envman web

141. Based on the code written for your automation, two diagrams are generated:

142. The one on the left is called the Progress diagram, which shows the steps in your process,

143. and the one on the right is the Outcome diagram, which shows what the deployed environment looks like, before you deploy it.

144. By clicking on these steps on the right, we get to see what is happening in that step.

145. For example the first step is to download a file from Github, it shows you the request to github and where it saves the file on the file system.

146. Then it creates the IAM policy, role, and instance profile, and S3 bucket,

147. then uploads the web application to that bucket.

148. All of this is generated from your automation code. Magic.

149. This is what you can use to teach someone, or self learn, what the automation process is, and what the environment looks like.

150. And you don't have to keep erasing and redrawing lines on the whiteboard.

151. Which step was unclear? This one? Let's go through that again.

152. Now, this is great, but I like this one.

153. The diagram you saw is the example environment, but what does the actual environment look like?

154. We can discover it.

155. The diagram on the right has faded boxes for each resource, indicating that it doesn't exist.

156. When I click deploy, you can watch the progress diagram on the left, which will show you which steps are being executed,

157. or you can watch the outcome diagram on the right, which will show you the interactions between hosts, that are happening in real time.

158. All of the steps completed successfully, that why they're green,

159. and the resources have been created, so they are now visible.

160. We can do the same for clean up, and it will delete all of the resources from Amazon, as well as on disk.

161. And if we were to have an error, as we did before, we should see it clearly.

162. slow down internet, click deploy

163. Let's take a moment to admire this diagram.

164. Ooh look it's gone red.

165. So very quickly, from the user interface, you can tell which step the error came from,

166. as well as which resources it involves.

167. And we can surface the timeout message on the web interface, I just haven't coded that part yet.

168. Cool.

169. So for morale, a lot of effort has been put into aesthetics.

170. For seeing the state of the system, showing one line for each resource, with a link to the full detail, is deliberate.

171. If you've ever been on-call and gotten a call out in the middle of the night, it's very annoying to have to go and find each resource that is part of the system you are investigating.

172. If I can think it, take me there.

173. For progress, we present the information at a level of detail that is digestable,

174. and for errors, instead of panicking, which is visually equivalent of printing a stack trace,

175. we take that error, refine it, and make it beautiful.

176. Always include what went wrong, the reason, and how to recover,

177. because when help people recover from a bad situation,

178. you recover their morale.

179. With all of these aesthetic refinements, that box, is no longer opaque.

180. It is completely, clear.

181. You can see inside it, you can understand it, and you can control it.

182. How does all of this work?

183. Magic.

184. Architecture, how does it fit together?

185. The Peace framework is categorised into two main parts.

186. The item definition, which is the common shape of logic and data, for anything that is managed by automation, and

187. Common functionality, which works with those shapes to provide command execution and a user interface.

188. Item crates contain the logic and data to automate one thing, and

189. the tool crate connects different items together, and passes them to the common functionality from the Peace framework, to provide automation.

190. These groupings are deliberate, so that you can share and reuse common item logic from the standard package registry,

191. while keeping proprietary values and workflows within your tool.

192. Let's go deeper.

193. Starting with Item.

194. If you think of one step in a process, normally we would write code to do the step.

195. But instead of only writing code that does the work of that step,

196. an Item is a collection of functions that interact with the thing that is being automated.

197. What is the current state of the thing I'm managing?

198. What will it be, after the automation logic is executed?

199. What's the difference between these states?

200. What does it look like if it's not there?

201. The actual work logic, and

202. interactions -- what are the hosts, and paths that are involved in this automation.

203. Is it a request to fetch data back in, or is it a push to push data out.

204. This information is used to generate the diagram you saw earlier.

205. An example implementation of this, the File Download.

206. The current state function returns the state of the file on disk -- whether or not it exists.

207. And if it does exist, it also returns the MD5 hash.

208. The goal state function returns the state of the file from the server, because the state of the file on the server, will become the state of the file on disk, when the download is executed.

209. So this would fetch the content-length and etag from the server, as a way to compare with what is on disk locally.

210. Many servers use the MD5 hash of a file as its etag.

211. state_diff returns whether the local file has the same hash as the remote file.

212. If it's got a different hash, then we assume we need to download it.

213. state_clean returns "the file does not exist".

214. apply downloads the file.

215. and interactions says I'm pulling data from this host, and writing to this path on localhost.

216. A collection of functions is called an Item.

217. And a collection of items, is called a Flow.

218. And a flow also contains the dependency ordering between items.

219. And in Rust, since we cannot store different concrete types in a collection, we have to put them on the heap and store their addresses.

220. Then this flow is what is passed to Peace's common functionality to use in execution or display.

221. Commands. Commands are one of the common functionality that Peace provides.

222. Given a flow and parameters, it invokes different functions within each item.

223. For example, the Discover command.

224. What is the current state of each item? What is the goal state of each item?

225. The discover command will run these functions, store the state, and display it to the user.

226. The Diff command will compute and show the difference between the current and goal states of each item.

227. The Ensure command will turn the current state of each item, into its goal state, through the apply function.

228. The Clean command is similar, where it turns the current state into the clean state, also through the apply function.

229. So Peace provides common logic to iterate through the items, and call the appropriate functions.

230. and it will also pass the appropriate values between each item.

231. That, is magic.

232. Going back to the Item definition, besides the functions to read from or write to the item, implementors also have to specify these data types.

233. Input, which we call parameters, and

234. Output, which we call State.

235. The parameters tell the item where to fetch data from, and where to write to, as well as any other information needed to access the item.

236. The state indicates whether or not the item exists, where it lives, and a summary of its contents.

237. This is the type that is returned from the current, goal, and clean state functions.

238. Putting it all together:

239. We combine the items into a flow,

240. We specify the parameters for each item,

241. Pick an output -- the command line, or web, or both,

242. and these three things together is called a command context.

243. Essentially "all the things you need to run a command".

244. Surface the commands to the user with appropriate names,

245. and this is your tool.

246. So Peace is a side project, and

247. there are side-side projects that were built in the making of Peace.

248. The first noticeable one is Interruptible, which adds the ability to interrupt a stream.

249. If you think about playing music, we are streaming bytes to a speaker, and out comes some audio.

250. When we pause the music, the bytes that were buffered still play, but any bytes that were not the audio buffer will not be played.

251. In automation, instead of streaming bytes to a speaker, we are streaming logic, to an executor.

252. When we pause, any logic that was already queued and is executing, will continue to run to completion.

253. Any logic that hasn't been queued, will not be started.

254. So we fully execute what is in progress to completion, and safely stop between steps.

255. And that's how we get safe interruptibility.

256. The second noticeable project is Dot Interactive.

257. This generates diagrams from structured input.

258. So it takes a data model with the nodes and edges, generates a diagram using GraphViz dot, and adds styles using Tailwind.

259. And that's what I've used for most of the diagrams you've seen today.

260. Now rounding off, what's the status of Peace? Is it ready to be used?

261. For development workflows, or short lived environments, where the environment does not live longer than one version of a tool,

262. I'd say it is ready.

263. But for production workflows, or environments that need to be stable, then Peace is not ready.

264. Don't use it, you will not have Peace.

265. In the table below, you can see the command execution and CLI functionality is stable,

266. The web interface is definitely not stable -- it was hacked together last month for this demo, and

267. the most important one for readiness is API and data stability, which may take me a year to complete.

268. Links to the project:

269. peace.mk for the project website

270. Slides are on peace.mk/book.

271. github.com/azriel91/peace for the repository.

272. To wrap up, I'd like to end with this note:

273. To engineer with empathy,

274. whether it is verbal, visual, or vocal,

275. refine your voice, connect,

276. and communicate with clarity.

277. Thank you for listening, and I'm happy to take questions.

🕊️ Peace Framework

Peace

Constraints and
Common functionality

+

💻

Code

Logic and data types

=

⚙️

Nice Tool

Automation software

⚙️

Nice Tool

Automation software

+

🔡

Data

Values for parameters

=

🚀

Work Done

App built and published

Environment deployed

🐉 Origin Story

Manual Deployment

source

Automation

source

Engineering Eyes

source

1. 🔁 End-to-end automation

2. ✅ Repeatable correctness

3. ⚡ Performance: Time / Cost efficiency

Engineering Success

source

1. 🔁 End-to-end automation

2. ✅ Repeatable correctness

3. ⚡ Performance: Time / Cost efficiency

⌚

2 weeks → 30 minutes

Automation (For Real)

source

Human Eyes

source

1. 🔁 End-to-end automation

2. ✅ Repeatable correctness

3. ⚡ Performance: Time / Cost efficiency

4. 💡 Understandability

5. 🎮 Control

6. 💟 Morale

🏗️ envman (example tool)

Peace

Constraints and
common functionality

+

💻

Code

Logic and data types

=

⚙️

envman

Automation software

What It Does

1. 📥 Downloads a file

2. ☁️ Creates AWS resources

3. 📤 Uploads the file to AWS

source

💡 Understandability

🎨 Demo

Goldilocks Principle

🐻🐻‍❄️🐼

1. Too much information is overwhelming.

2. Too little information is not useful.

3. There is a level specific to each individual that is "just right".

🎮 Control

Minimal Control

🏁🏁🏁

🟢 Start

Discovery

🗺️

1. 📍 Current State

2. 🎯 Goal State

3. 🚗 Difference

Interruption

☝️

Why:

1. 🧠 Remembered something
2. 🔀 Changed our mind

What we want:

1. 🛡️ Safe interruption
2. 🚏 Don't undo what's done

🎨 Demo

💟 Morale

Aesthetics: Outcome

CLI

❯ ./envman status
# States Current

1. `app_download`    : `azriel91\web_app\0.1.1\web_app.tar` containing 6492160 bytes, ETag: "0x8DAEA1931AF89EE"
2. `iam_policy`      : exists at https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::140353555758:policy/demo_1
3. `iam_role`        : exists at https://console.aws.amazon.com/iamv2/home#/roles/details/demo_1 with policy attached
4. `instance_profile`: exists at https://console.aws.amazon.com/iamv2/home#/roles/details/demo_1 associated with same named role
5. `s3_bucket`       : exists at https://s3.console.aws.amazon.com/s3/buckets/azrielh-peace-envman-demo-1
6. `s3_object`       : uploaded at https://s3.console.aws.amazon.com/s3/object/azrielh-peace-envman-demo-1?prefix=web_app.tar (MD5: 66e1cfaf498426ad444257d765c0766a)

Web

source

Aesthetics: Progress

CLI

{"item_id":"app_download","progress_update":"ResetToPending","msg_update":"Clear"}
{"item_id":"iam_policy","progress_update":"ResetToPending","msg_update":"Clear"}
{"item_id":"iam_role","progress_update":"ResetToPending","msg_update":"Clear"}
{"item_id":"instance_profile","progress_update":"ResetToPending","msg_update":"Clear"}
{"item_id":"s3_bucket","progress_update":"ResetToPending","msg_update":"Clear"}
{"item_id":"s3_object","progress_update":"ResetToPending","msg_update":"Clear"}
{"item_id":"s3_bucket","progress_update":{"Delta":"Tick"},"msg_update":{"Set":"listing buckets"}}
{"item_id":"iam_policy","progress_update":{"Delta":"Tick"},"msg_update":{"Set":"listing policies"}}
{"item_id":"app_download","progress_update":{"Delta":"Tick"},"msg_update":"Clear"}

❯ ./envman clean
🔵 1. app_download     ▰▰▰▰▱▱▱▱▱▱▱▱▱▱ 955.00 KiB/6.19 MiB
                                                  (el: 19s, eta: 5s)
✅ 2. iam_policy       ▰▰▰▰▰▰▰▰▰▰▰▰▰▰ done!
✅ 3. iam_role         ▰▰▰▰▰▰▰▰▰▰▰▰▰▰ done!
✅ 4. instance_profile ▰▰▰▰▰▰▰▰▰▰▰▰▰▰ done!
✅ 5. s3_bucket        ▰▰▰▰▰▰▰▰▰▰▰▰▰▰ done!
⚪ 6. s3_object        ▱▱▱▱▱▱▱▱▱▱▱▱▱▱

Web

source

Aesthetics: Errors

thread 'main' panicked at C:\Users\azrielh\work\github\azriel91\peace\examples\envman\src\items\peace_aws_s3_object\s3_object_apply_fns.rs:204:30:
called `Result::unwrap()` on an `Err` value: S3ObjectUploadError { bucket_name: "azrielh-peace-envman-demo-1", object_key: "web_app.tar", aws_desc: "timed out", aws_desc_span: SourceSpan { offset: SourceOffset(0), length: 9 }, error: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 3.1s }), connection: Unknown } }) }
stack backtrace:
   0: <unknown>
   ..
  73: <unknown>
  74: BaseThreadInitThunk
  75: RtlUserThreadStart

❯ ./envman clean
..
✅ 5. s3_bucket        ▰▰▰▰▰▰▰▰▰▰▰▰▰▰ done!
❌ 6. s3_object        ▱▱▱▱▱▱▱▱▱▱▱▱▱▱ 0/1
                        Failed to upload S3 object: `web_app.tar`.
# Errors
`s3_object`:
  × Failed to upload S3 object: `web_app.tar`.
   ╭────
 1 │ timed out
   · ─────────
   ╰────
  help: Make sure you are connected to the internet and try again.

Automation Clarified

source

source