504 lines
15 KiB
Plaintext
504 lines
15 KiB
Plaintext
// Moved from docs/src/architecture-internals/MESSAGING.adoc to docs/working/scratch/src/architecture-internals/MESSAGING.adoc on 2026-06-16 (docs reorg Phase 2)
|
||
== 1. Overview
|
||
:toc: left
|
||
:toc-title: Contents
|
||
:toclevels: 3
|
||
xref:../README.adoc[← Back to Documentation Index]
|
||
|
||
|
||
|
||
This document defines the *thermodynamic model for the StarshipOS
|
||
Messaging Field*, a formal framework describing the behavior of message
|
||
propagation and processing as a physical system.
|
||
|
||
The messaging subsystem is not treated as a passive bus. Instead, it is
|
||
conceptualized as a *field* in which discrete *messages* behave
|
||
analogously to quanta (photons), and *words* (the fundamental executable
|
||
units of the StarshipOS runtime) behave as *particles* or loci of
|
||
interaction.
|
||
|
||
This model provides a unified basis for:
|
||
|
||
* *Prioritization & Scheduling*
|
||
* *Credit Allocation and Flow Control*
|
||
* *Thermal Regulation of System Load*
|
||
* *Entropy-Driven Policy and ML Feedback*
|
||
* *Instrumentation of Message Dynamics*
|
||
|
||
'''''
|
||
|
||
== 2. Conceptual Model
|
||
|
||
=== 2.1 Message Field
|
||
|
||
The *Message Field* is the logical substrate mediating all communication
|
||
within StarshipOS. It consists of:
|
||
|
||
* *Endpoints*: Boundaries between word spaces and the field.
|
||
* *Channels*: Directed pathways through which messages propagate
|
||
(virtual channels, pub/sub topics, RPC routes).
|
||
* *Field Properties*: Dynamic parameters such as message density,
|
||
propagation latency, and entropy.
|
||
|
||
The field is *not centralized*. It emerges from the coordinated behavior
|
||
of `+sf_msg-srv+` (broker/router), `+sf_msg-drv+` (ABI provider), and
|
||
`+libsfmsg+` (participant library), operating over shared memory rings
|
||
and control IPC.
|
||
|
||
'''''
|
||
|
||
=== 2.2 Messages
|
||
|
||
Messages are modeled as *energy quanta* with the following core
|
||
properties:
|
||
|
||
[cols=",",options="header",]
|
||
|===
|
||
|Property |Description
|
||
|Energy |Encodes priority, TTL, QoS, and thermodynamic state
|
||
|Momentum |Routing vector through the field
|
||
|Cross Section |Probability of absorption by a word
|
||
|Entropy |Thermodynamic measure of the flow’s state space
|
||
|===
|
||
|
||
Messages can be *absorbed*, *emitted*, or *scattered* by words and
|
||
services, analogous to photon–atom interactions.
|
||
|
||
'''''
|
||
|
||
=== 2.3 Words as Particles
|
||
|
||
Each *word* is treated as a *particle* in the field — a locus of
|
||
interaction rather than an isolated routine. Words have intrinsic
|
||
properties:
|
||
|
||
[cols=",",options="header",]
|
||
|===
|
||
|Property |Meaning
|
||
|Mass |Computational cost or complexity
|
||
|Charge |Degree of side-effect or state mutation
|
||
|Cross-Section |Message types and classes to which the word responds
|
||
|===
|
||
|
||
When a message reaches a word, the interaction may *excite* the word
|
||
(trigger execution), *alter its state*, or __ __pass through
|
||
unabsorbed**. Executed words may **emit new messages**, either
|
||
deterministically or in a burst (stimulated emission).
|
||
|
||
'''''
|
||
|
||
== 3. Mathematical Formulation
|
||
|
||
=== 3.1 Maxwellian Entropy
|
||
|
||
The system adopts a *Maxwell–Boltzmann entropy model*, treating each
|
||
endpoint/channel as a thermodynamic micro-system.
|
||
|
||
For a given flow ( f ):
|
||
|
||
{empty}[ S_f(t) = k ( 1 + )]
|
||
|
||
Where:
|
||
|
||
* ( ^2_\{t}(f) ): Variance of inter-arrival intervals
|
||
* ( Q_\{}(f) ): Normalized queue occupancy (0–1)
|
||
* ( U_\{}(f) ): Ratio of credits consumed to granted
|
||
* ( R_\{}(f) ): Number of delivery paths (fan-out)
|
||
* ( T_\{}(f) ): Normalized TTL of messages in the flow
|
||
* ( k ): Scaling constant (typically 1)
|
||
|
||
( S_f ) represents the *thermodynamic entropy* of the flow. High entropy
|
||
indicates unpredictable, high-energy flows approaching saturation. Low
|
||
entropy indicates stable, structured behavior.
|
||
|
||
=== 3.2 System Temperature and Pressure
|
||
|
||
* *Temperature (Θ)* is the average energy of messages within the field,
|
||
weighted by density.
|
||
* *Pressure (Π)* represents backlog intensity: [ = _f Q_\{}(f) U_\{}(f)]
|
||
* *Entropy (Σ)* is the aggregate system entropy: [ = _f S_f]
|
||
|
||
These quantities provide real-time metrics for system health and are
|
||
suitable as inputs for control algorithms.
|
||
|
||
'''''
|
||
|
||
== 4. Header and Data Structure Extensions
|
||
|
||
Messages carry *thermodynamic metadata* in addition to conventional
|
||
routing and control information.
|
||
|
||
=== 4.1 Thermodynamic Header Extension
|
||
|
||
[source,c]
|
||
----
|
||
struct sfm_hdr_thermo {
|
||
uint16_t energy_q8; // Encoded priority/TTL/entropy composite (fixed point)
|
||
uint16_t entropy_mx; // Maxwellian entropy (Q8.8 fixed-point)
|
||
uint16_t temp_hint; // Optional producer-side temperature hint
|
||
uint16_t reserved;
|
||
};
|
||
----
|
||
|
||
* `+energy_q8+` is a composite scalar derived from QoS class, TTL, and
|
||
policy weighting.
|
||
* `+entropy_mx+` is computed at the sender or by the broker to
|
||
characterize flow state.
|
||
* `+temp_hint+` allows producers to indicate expected burstiness or
|
||
instability.
|
||
|
||
=== 4.2 Endpoint State
|
||
|
||
Endpoints maintain thermodynamic state variables:
|
||
|
||
[source,c]
|
||
----
|
||
struct sfm_endpoint_state {
|
||
float entropy_current;
|
||
float temperature;
|
||
float pressure;
|
||
float credit_utilization;
|
||
uint64_t last_refresh_tsc;
|
||
};
|
||
----
|
||
|
||
This state is updated continuously during message transfer and
|
||
scheduling.
|
||
|
||
'''''
|
||
|
||
== 5. Scheduler and Credit System Integration
|
||
|
||
=== 5.1 Credits as Energy Quanta
|
||
|
||
Credits represent the *available energy budget* for message emission
|
||
along a channel.
|
||
|
||
* *High-entropy flows* receive smaller, more frequent credit refreshes
|
||
(tight regulation).
|
||
* *Low-entropy flows* receive larger, batched credits (looser
|
||
regulation).
|
||
|
||
Credit recycling policies (immediate, piggyback, or batched) are
|
||
selected dynamically based on the flow’s entropy and temperature.
|
||
|
||
'''''
|
||
|
||
=== 5.2 Thermodynamic Scheduling
|
||
|
||
Scheduling priority is a function of QoS class, deadline, and entropy:
|
||
|
||
{empty}[ P_\{sched} = f(, , S_f)]
|
||
|
||
* Low-entropy, real-time flows are scheduled deterministically.
|
||
* High-entropy, bulk flows are throttled or coalesced.
|
||
* Aging and anti-starvation mechanisms apply within entropy bands.
|
||
|
||
This approach prevents high-entropy flows from destabilizing the system,
|
||
while ensuring low-entropy flows achieve predictable latency.
|
||
|
||
'''''
|
||
|
||
== 6. Thermodynamic Monitoring and Control Loop
|
||
|
||
A *control loop* runs in `+sf_msg-srv+`, maintaining field stability:
|
||
|
||
[arabic]
|
||
. *Sampling* — Entropy, temperature, and pressure are sampled
|
||
periodically from endpoints and channels.
|
||
. *Aggregation* — System-wide Σ, Θ, Π are computed.
|
||
. *Policy Evaluation* — Control laws or ML bandits adjust credit
|
||
windows, scheduling weights, and routing based on sampled state.
|
||
. *Actuation* — The scheduler and credit allocator apply the updated
|
||
parameters.
|
||
|
||
This loop can operate using fixed thresholds or adaptive policies. ML
|
||
components treat entropy as the *order parameter * for optimization.
|
||
|
||
'''''
|
||
|
||
== 7. Implications for Forth and StarshipOS Runtime
|
||
|
||
In StarshipOS, *words are the fundamental execution loci*. By modeling
|
||
them as particles interacting via message photons, the runtime gains:
|
||
|
||
* *A unified abstraction for computation and communication*
|
||
* *Fine-grained control over execution dynamics through entropy*
|
||
* *A natural basis for prioritization without ad-hoc heuristics*
|
||
* *Thermodynamic instrumentation for debugging and analysis*
|
||
|
||
This model integrates cleanly with the existing Forth execution model,
|
||
since words are already explicit entities with well-defined entry
|
||
points.
|
||
|
||
'''''
|
||
|
||
== 8. Future Work
|
||
|
||
[arabic]
|
||
. *Formal Policy Language* Define a compact language for specifying
|
||
entropy-based routing and scheduling policies.
|
||
. *Distributed Thermodynamic Fields* Extend the model to multi-node
|
||
topologies, where message photons propagate across network transports.
|
||
. *Entropy-Driven GC & Memory Tiering* Couple message entropy metrics
|
||
with VM memory placement decisions.
|
||
. *Visualization* Develop tooling to visualize message flow as a dynamic
|
||
field, aiding in debugging and optimization.
|
||
|
||
'''''
|
||
|
||
== 9. APPEXDIX
|
||
|
||
....
|
||
/*
|
||
* StarshipOS - Messaging Thermodynamics
|
||
* File: include/sfm_thermo.h
|
||
* Status: Draft / Reference Only
|
||
*
|
||
* This header defines the public structures and APIs for the
|
||
* Maxwellian entropy model applied to StarshipOS messaging flows.
|
||
* It is intended as a design reference, not production code.
|
||
*
|
||
* Language: C99
|
||
* License: CC0 1.0 / Public Domain
|
||
*/
|
||
|
||
#ifndef SFM_THERMO_H
|
||
#define SFM_THERMO_H
|
||
|
||
#include <stdint.h>
|
||
#include <stddef.h>
|
||
|
||
#ifdef __cplusplus
|
||
extern "C" {
|
||
#endif
|
||
|
||
/* =============================================================
|
||
* Fixed-point helpers (Q8.8 and Q16.16 formats)
|
||
* =========================================================== */
|
||
|
||
/** Convert float → Q8.8 fixed point. */
|
||
static inline uint16_t sfm_fp_to_q8_8(float x) {
|
||
if (x <= 0.0f) return 0u;
|
||
float v = x * 256.0f;
|
||
if (v > 65535.0f) v = 65535.0f;
|
||
return (uint16_t)(v + 0.5f);
|
||
}
|
||
|
||
/** Convert Q8.8 fixed point → float. */
|
||
static inline float sfm_q8_8_to_fp(uint16_t q) {
|
||
return ((float)q) / 256.0f;
|
||
}
|
||
|
||
/** Convert float → Q16.16 fixed point. */
|
||
static inline uint32_t sfm_fp_to_q16_16(float x) {
|
||
if (x <= 0.0f) return 0u;
|
||
double v = (double)x * 65536.0;
|
||
if (v > 4294967295.0) v = 4294967295.0;
|
||
return (uint32_t)(v + 0.5);
|
||
}
|
||
|
||
/** Convert Q16.16 fixed point → float. */
|
||
static inline float sfm_q16_16_to_fp(uint32_t q) {
|
||
return ((float)q) / 65536.0f;
|
||
}
|
||
|
||
/* =============================================================
|
||
* Thermodynamic Header Extension
|
||
* =========================================================== */
|
||
|
||
/**
|
||
* Additional header fields attached to each message.
|
||
* These encode thermodynamic properties derived from the
|
||
* flow's state and scheduling context.
|
||
*
|
||
* All fields are fixed-point Q8.8 unless otherwise stated.
|
||
*/
|
||
struct sfm_hdr_thermo {
|
||
uint16_t energy_q8; /**< Composite of priority, TTL, entropy. */
|
||
uint16_t entropy_mx; /**< Maxwellian entropy estimate (per-flow). */
|
||
uint16_t temp_hint; /**< Optional producer hint for burstiness. */
|
||
uint16_t reserved; /**< Align to 8 bytes. */
|
||
};
|
||
|
||
/* =============================================================
|
||
* Per-flow Thermodynamic State
|
||
* =========================================================== */
|
||
|
||
/**
|
||
* Credit refresh policy hints returned to allocator.
|
||
*/
|
||
typedef enum {
|
||
SFM_CREDIT_IMMEDIATE = 0, /**< Refresh credits on every recycle. */
|
||
SFM_CREDIT_PIGGYBACK = 1, /**< Return credits with next outbound. */
|
||
SFM_CREDIT_BATCHED = 2 /**< Batch refresh until watermark. */
|
||
} sfm_credit_mode_t;
|
||
|
||
/**
|
||
* Per-flow thermodynamic state, maintained by sf_msg-srv.
|
||
* All quantities are continuously updated during operation.
|
||
*/
|
||
typedef struct {
|
||
/* --- Inter-arrival statistics (Welford) --- */
|
||
uint64_t last_arrival_us;
|
||
double iat_mean;
|
||
double iat_M2;
|
||
uint64_t iat_count;
|
||
|
||
/* --- Queue occupancy EWMA --- */
|
||
double q_occ_ewma;
|
||
double q_alpha;
|
||
|
||
/* --- Credit utilization EWMA --- */
|
||
uint64_t credits_granted_total;
|
||
uint64_t credits_consumed_total;
|
||
double credit_util_ewma;
|
||
double credit_alpha;
|
||
|
||
/* --- Fanout & TTL tracking --- */
|
||
double fanout_ewma;
|
||
double ttl_ms_ewma;
|
||
|
||
/* --- Derived thermodynamic quantities --- */
|
||
double entropy_mx; /**< Maxwellian entropy S_f */
|
||
double temperature; /**< Average energy proxy Θ */
|
||
double pressure; /**< Backlog pressure proxy Π */
|
||
|
||
/* --- Policy thresholds --- */
|
||
double wm_low;
|
||
double wm_high;
|
||
double entropy_cool;
|
||
double entropy_hot;
|
||
|
||
/* --- Last update timestamp --- */
|
||
uint64_t last_update_us;
|
||
|
||
} sfm_flow_state_t;
|
||
|
||
/**
|
||
* Snapshot of field-wide thermodynamic state.
|
||
* Useful for monitoring and control loops.
|
||
*/
|
||
typedef struct {
|
||
double system_entropy_sum;
|
||
double system_temperature_sum;
|
||
double system_pressure_sum;
|
||
uint64_t flows_count;
|
||
} sfm_field_snapshot_t;
|
||
|
||
/**
|
||
* Credit allocation decision returned to the scheduler.
|
||
*/
|
||
typedef struct {
|
||
sfm_credit_mode_t mode; /**< Suggested refresh mode. */
|
||
uint32_t grant_quanta; /**< Number of descriptors to grant. */
|
||
uint8_t qos_hint; /**< Optional QoS class hint (255 = no change). */
|
||
} sfm_thermo_decision_t;
|
||
|
||
/* =============================================================
|
||
* API Prototypes
|
||
* =========================================================== */
|
||
|
||
/**
|
||
* Initialize a flow state structure with sane defaults.
|
||
*
|
||
* @param st Pointer to the state structure.
|
||
* @param q_alpha EWMA alpha for queue occupancy.
|
||
* @param credit_alpha EWMA alpha for credit utilization.
|
||
* @param wm_low Low watermark for batching (0–1).
|
||
* @param wm_high High watermark for backpressure (0–1).
|
||
* @param entropy_cool Entropy threshold for loosening control.
|
||
* @param entropy_hot Entropy threshold for tightening control.
|
||
*/
|
||
void sfm_thermo_init_flow(sfm_flow_state_t *st,
|
||
double q_alpha,
|
||
double credit_alpha,
|
||
double wm_low,
|
||
double wm_high,
|
||
double entropy_cool,
|
||
double entropy_hot);
|
||
|
||
/**
|
||
* Update flow state when a message is enqueued or processed.
|
||
*
|
||
* @param st Flow state.
|
||
* @param now_us Current monotonic timestamp in microseconds.
|
||
* @param q_norm Current normalized queue occupancy [0..1].
|
||
* @param credits_g Delta of credits granted since last update.
|
||
* @param credits_c Delta of credits consumed since last update.
|
||
* @param fanout Observed fanout for this message.
|
||
* @param ttl_ms Message TTL in milliseconds.
|
||
*/
|
||
void sfm_thermo_on_enq(sfm_flow_state_t *st,
|
||
uint64_t now_us,
|
||
double q_norm,
|
||
uint32_t credits_g,
|
||
uint32_t credits_c,
|
||
double fanout,
|
||
double ttl_ms);
|
||
|
||
/**
|
||
* Sample the current state and compute a scheduling / credit
|
||
* allocation decision. This is invoked periodically or on events.
|
||
*
|
||
* @param st Flow state.
|
||
* @param batch_watermark Recycle count watermark for batching.
|
||
* @param max_grant Safety cap on credit grants.
|
||
* @return Suggested decision structure.
|
||
*/
|
||
sfm_thermo_decision_t
|
||
sfm_thermo_sample_and_decide(sfm_flow_state_t *st,
|
||
uint32_t batch_watermark,
|
||
uint32_t max_grant);
|
||
|
||
/**
|
||
* Aggregate a set of flow states into a single field snapshot.
|
||
*
|
||
* @param flows Array of flow states.
|
||
* @param n Number of elements in `flows`.
|
||
* @param out Output snapshot structure.
|
||
*/
|
||
void sfm_thermo_aggregate(const sfm_flow_state_t *flows,
|
||
size_t n,
|
||
sfm_field_snapshot_t *out);
|
||
|
||
/**
|
||
* Populate a thermodynamic header extension for a message.
|
||
*
|
||
* @param st Flow state.
|
||
* @param h Pointer to header extension struct.
|
||
* @param energy_q8 Precomputed composite energy (Q8.8).
|
||
*/
|
||
static inline void
|
||
sfm_thermo_fill_hdr(const sfm_flow_state_t *st,
|
||
struct sfm_hdr_thermo *h,
|
||
uint16_t energy_q8)
|
||
{
|
||
if (!st || !h) return;
|
||
h->energy_q8 = energy_q8;
|
||
h->entropy_mx = sfm_fp_to_q8_8((float)st->entropy_mx);
|
||
h->temp_hint = sfm_fp_to_q8_8((float)st->temperature);
|
||
h->reserved = 0;
|
||
}
|
||
|
||
#ifdef __cplusplus
|
||
}
|
||
#endif
|
||
|
||
#endif /* SFM_THERMO_H */
|
||
....
|
||
|
||
== 10. References
|
||
|
||
* Maxwell, J.C., _Illustrations of the Dynamical Theory of Gases_, Phil.
|
||
Mag. 1860.
|
||
* Boltzmann, L., _Weitere Studien über das Wärmegleichgewicht unter
|
||
Gasmolekülen_, 1872.
|
||
* StarshipOS Internal Messaging Architecture Specifications.
|
||
|
||
'''''
|
||
|
||
*End of Document*
|
||
|
||
'''''
|