Testing

Purpose

The testing model is intentionally small at the source level. A useful unit test should normally need only a *_test.sec file, one or more test declarations and the compiler-known testing namespace.

The same source model is designed to work for hosted applications, simulators, RTOS targets and bare-metal firmware. Test execution may differ by target, but the meaning of a test declaration does not.

Sec 0.1 testing does not intrinsically require:

• a third-party runner;
• a user-written test main;
• runtime reflection or dynamic registration;
• a heap, filesystem, operating system or threads;
• a testing runtime linked into production artifacts.

A First Test

A top-level test has a compile-time string name and a normal Sec block body.

// math_test.sec
module math

test "Add returns the sum" {
    testing.ExpectEqual(4, Add(2, 2))
}

Reaching the end of the body without a recorded failure passes the test. An explicit testing.Pass() is normally unnecessary.

Test Files

The canonical test-source filename form is:

*_test.sec

Examples include:

request_test.sec
parser_test.sec
regression_test.sec

A test declaration is legal only in a *_test.sec file. Production builds exclude these files completely. Test-only declarations, dependencies and generated harness code must not leak into ordinary production compilation or emitted artifacts.

// request.sec -- invalid

test "Request parses GET" {
}

Expected diagnostic:

test declarations are only allowed in *_test.sec files

Test Declarations

The canonical source form is:

test "Request parses GET" {
    ...
}

A top-level test declaration:

• is a module-level declaration;
• has a non-empty compile-time string literal name;
• has no source-visible parameters;
• has no source-visible return type;
• is not an ordinary callable value;
• cannot be invoked directly by user source.

Tests are not declared inside functions, impl blocks, properties, lambdas or other tests. Nested test execution uses testing.Run.

Test Identity

A top-level test identity combines its test module identity with the human-readable test name. The same test name may therefore exist in different modules without creating one global name collision.

Two tests in the same test module may not have the same test name. Subtests extend the parent identity hierarchically rather than being flattened into one source-level string.

module :: top-level test :: subtest :: nested subtest

Stable structured identity is used by the runner, LSP, editor integrations and test result reporting.

White-Box Tests

A *_test.sec file placed beside production source participates in the same module during test compilation.

net/http/request.sec
net/http/request_test.sec

The test file declares the same module name and may use declarations visible to that module under normal visibility rules. Testing does not create a friend mechanism.

White-box test files may contain helper functions, fixture types, constants and other ordinary declarations needed only during test compilation.

Integration and Black-Box Tests

Project-level integration tests live below the project-root tests/ source tree. They belong to test-only modules and access production modules through normal imports and normal external visibility rules.

tests/
└── http/
    └── request_test.sec

import "net/http"

test "public request API parses GET" {
    let request := try http.ParseRequest("GET / HTTP/1.1")
    testing.ExpectEqual(http.Method.GET, request.Method)
}

Black-box testing means testing through the external Sec API. It does not imply a separate operating-system process, container, network deployment or fresh machine.

Test Body Semantics

A test body contains ordinary Sec statements and expressions. Normal control flow, ownership, borrowing, defer, collections, concurrency, hardware access and function calls remain governed by their ordinary rules.

A bare return may terminate the current test invocation normally. Returning a value is invalid because a test has no source-visible return type.

test "optional setup" {
    if !SetupNeeded() {
        return
    }

    RunSetup()
}

Failure state is monotonic. Once an invocation has failed, later normal completion, testing.Pass() or a later skip request cannot erase that failure.

try at the Test Boundary

A test invocation is a compiler-known error-propagation boundary. This makes ordinary fallible Sec code convenient to test without declaring a fake Result return type on the test itself.

test "valid request parses" {
    let request := try ParseRequest("GET / HTTP/1.1")
    testing.ExpectEqual(Method.GET, request.Method)
}

If an unhandled Err(errorValue) reaches the test boundary, the current invocation terminates after ordinary cleanup and becomes Failed. The concrete error identity should be retained in the diagnostic.

Expected errors are tested with ordinary try handlers or match. Sec 0.1 deliberately does not add a parallel testing.ExpectError API.

test "invalid header returns InvalidHeader" {
    try ParseHeader(input) {
        Err(ParseError.InvalidHeader) => {
            testing.Pass()
        }
    }

    testing.Fail("expected ParseError.InvalidHeader")
}

The testing Namespace

During test compilation the compiler provides the compiler-known testing namespace. No import is required, and it is not a hidden standard library import.

The Sec 0.1 source surface is:

testing.Pass()

testing.Fail(message: string)
testing.Skip(message: string)
testing.Log(message: string)

testing.Expect(condition: bool)
testing.Expect(condition: bool, message: string)

testing.Require(condition: bool)
testing.Require(condition: bool, message: string)

testing.ExpectEqual(expected, actual)
testing.ExpectEqual(expected, actual, message: string)

testing.RequireEqual(expected, actual)
testing.RequireEqual(expected, actual, message: string)

testing.Run(name, body)

testing.TestResult

Production code must not depend on this namespace. Its availability is part of the test compilation context only.

Pass, Fail and Skip

testing.Pass() terminates the current invocation successfully unless the invocation has already recorded a failure.

testing.Fail(message) records a failure and terminates the current invocation after ordinary cleanup. The message is mandatory.

testing.Skip(message) records a skipped outcome and terminates the current invocation when it has not already failed. A runtime skip is different from a test that was never selected or compiled.

if !device.IsPresent {
    testing.Skip("device is not present")
}

testing.Log(message) records structured diagnostic information without changing the invocation outcome or control flow. It is not defined as ordinary stdout or stderr.

Expect and Require

testing.Expect is non-fatal. A false condition records a failure and the current invocation continues.

testing.Expect(response.Valid)
testing.Expect(response.Valid, "response should be valid")

testing.Require is fatal for the current invocation. A false condition records a failure, performs ordinary cleanup and terminates that invocation.

testing.Require(response != None, "response must exist")

Use Expect when later checks remain useful and Require when continuing would make subsequent diagnostics meaningless or unsafe.

Equality Expectations

Equality helpers use the argument order expected, actual.

testing.ExpectEqual(Method.GET, request.Method)
testing.RequireEqual(3, items.Length)

ExpectEqual records failure and continues. RequireEqual records failure and terminates the current invocation.

These operations use ordinary Sec equality. Testing does not add implicit conversions, deep-comparison magic, approximate floating-point equality or special pointer comparison rules.

When possible, failures report structured expected and actual values, type information, source location and test identity. A lack of pretty rendering does not make an otherwise legal equality comparison invalid.

assert vs testing.Expect

The language-level assert keeps its ordinary runtime meaning inside test code.

assert condition
assert condition, "message"

It is not rewritten into testing.Expect. Use assert when the program invariant itself is being asserted. Use testing.Expect or testing.Require when reporting test outcomes to the test harness.

Cleanup

Controlled test termination participates in ordinary Sec cleanup. This includes:

• normal completion;
• bare return;
• unexpected propagated Err;
• testing.Pass;
• testing.Fail;
• testing.Require and RequireEqual failure;
• testing.Skip.

Owned values are destroyed normally and registered defer blocks run under the ordinary cleanup ordering rules. Testing does not invent a hidden teardown phase that restores arbitrary files, databases, global state or hardware registers.

Subtests

Subtests are nested test invocations created with testing.Run(name, body). The body uses Sec's ordinary callable/lambda syntax; testing does not add another closure syntax.

Unlike top-level test names, subtest names may be runtime strings. This supports table-driven tests.

testing.Run is synchronous. Before it returns, the child body has finished, its outcome is known and child cleanup has completed.

Its result is:

testing.TestResult

Passed
Failed
Skipped

A failed child marks the parent invocation failed, but the parent body may continue. A skipped child does not automatically skip or fail the parent.

Sibling subtest names must be unique within the parent invocation. Because names may be computed at runtime, duplicate detection may occur in the generated harness.

Ordering and Parallelism

Independent top-level tests must not depend on source-file order or filesystem enumeration order. Sec 0.1 permits an implementation to run top-level tests sequentially for simplicity, but this is not a semantic ordering guarantee.

Future runners may schedule independent top-level tests concurrently without changing source semantics. Sec 0.1 does not define testing.Parallel.

Ordinary Sec concurrency remains usable inside a test where the selected target supports it. Testing does not provide a second race, ownership or synchronization model.

Isolation

Every invocation has its own ordinary local scope, owned locals, borrow scopes and cleanup boundary. Sec testing does not, however, guarantee a new process, fresh machine or reset global state for each test.

Testing does not automatically reset:

• mutable module state;
• filesystem or environment state;
• network services or databases;
• hardware or remote resources.

An execution provider may offer stronger isolation such as a new process, target reset, fresh simulator or emulator. That is provider capability, not portable source semantics.

Test-Only Dependencies

Production source must remain valid independently of test source. The permitted dependency direction is:

Production -> Production
Test       -> Production
Test       -> Test

This direction is forbidden:

Production -> Test

Imports or dependencies used only by test source participate only in the test compilation and must not become production dependencies.

TestCompilationPlan

sec test does not mean “perform a production build and then hand the result to an unrelated external runner”. The compiler constructs a first-class TestCompilationPlan whose compilation mode is Test.

The test graph may include:

• selected production source;
• same-module *_test.sec source;
• test modules below tests/;
• test-only dependencies;
• compiler-known testing support;
• generated harness code.

Target, variant, ABI, profile and platform resolution are the same canonical models used for normal compilation. A cross-compiled test must not silently adopt the host computer's semantics.

Discovery

The compiler workspace is the semantic source of truth for test discovery. Top-level tests are discovered from selected *_test.sec files as real test declarations, not by naming convention on ordinary functions.

Discovery does not require:

• runtime reflection;
• constructor registration;
• a global runtime registry;
• linker-section scanning as source semantics.

The compiler may generate a static harness containing only the selected test set. Dynamically named subtests are discovered when testing.Run executes.

sec test

The canonical user command is:

sec test

From the active project, the default test scope contains same-module white-box tests and integration tests below the project-root tests/ tree.

Tests belonging to imported dependency modules are not automatically executed merely because those dependencies are built. Likewise, importing a standard-library module does not cause that module's internal tests to run.

The CLI must support scoped selection and deterministic filtering by test identity or name. The precise selector and pattern syntax belongs to tooling/CLI policy rather than Sec 0.1 source semantics.

Where practical, filtering should happen before final linking so constrained targets can build a test artifact containing only the selected test set and required support.

Runner Behavior

A controlled test failure does not stop the complete suite by default. The runner continues scheduling other selected top-level tests while the execution environment remains usable.

A fail-fast runner policy may stop scheduling new top-level tests after a failure, but it must not rewrite source-level control flow inside an already executing invocation.

Presentation should remain deterministic for equivalent results even if a future runner executes tests concurrently.

Discovering or selecting zero tests is not itself a source-language test failure, although the CLI must report that condition clearly.

Results and Exit Status

A completed test invocation has one primary outcome:

Passed
Failed
Skipped

Selection and infrastructure states are separate. Conditions such as CompilationFailed, ExecutionUnavailable, timeout or target reset must not be disguised as Skipped.

The command succeeds when every completed selected test is Passed or Skipped and no compilation or execution infrastructure failure occurred. A failed test or failed test run produces a non-zero command status.

Internally the harness and runner use structured result data. Human-readable console output is presentation, not the sole semantic representation. This permits CI, editor and hardware-farm integrations without redefining testing semantics.

Generated Harness

The programmer does not write a test main. The compiler generates the harness required for the selected test set.

A harness may use a statically generated table or equivalent compiler representation. Runtime test registration is not required, and ordinary production programs do not carry test registries, result collectors, formatting support or result transports.

Hosted and Embedded Targets

Testing is not host-only. The same source test may execute as a native host binary, simulator image, emulator image, RTOS image or bare-metal firmware when the selected target and execution provider support it.

test "UART initializes" {
    ...
}

Cross-compiled tests retain the selected target's ABI and semantic assumptions. They are not silently recompiled for the host merely because sec test was invoked on a desktop system.

The test artifact is separate from the production artifact, so test-only code does not increase the production binary.

Execution Providers

Compiled test artifacts execute through an execution-provider abstraction. A provider may be a native host runner, simulator, emulator, debug-probe integration, hardware farm or target-specific launcher.

A provider may support operations such as:

• deploying and starting a test artifact;
• resetting the target;
• collecting structured test results;
• interrupting execution;
• debugging;
• reporting processor or target faults.

Build support and execution support are separate. A target may successfully compile a test artifact even when no local execution provider is configured.

The provider owns result transport. Debug channels, semihosting, shared memory, mailboxes or an explicitly allocated UART may all be valid provider choices. Testing must not silently consume the same peripheral the test is trying to validate.

Faults, Reset and Timeout

A controlled test failure is different from an execution fault. Expectation failures, testing.Fail and unexpected Err are ordinary structured test failures. Processor faults, unhandled panic, process crashes, target reset or lost provider connections are execution failures.

When possible, an execution provider should attribute a fault or reset to the active test identity. It must not fabricate an ordinary assertion failure merely to fit the event into a Passed/Failed/Skipped model.

Sec 0.1 defines no universal source-level per-test timeout. Timeout policy belongs to runner, project configuration or execution provider policy, and a timeout is an execution failure category rather than Skipped.

Environment and Resources

Tests may use normal platform facilities such as filesystem, network, clock, environment variables or hardware devices when the active target provides them. Testing itself does not guarantee that any of these exist.

The current working directory is not portable test semantics. Likewise, Sec 0.1 has no automatic testdata discovery, automatic resource embedding or magic testing environment variables.

Arguments passed to the sec test command configure the test runner unless a separate contract explicitly says otherwise. They do not automatically become program arguments visible to every test invocation.

LSP and Editor Integration

The compiler workspace owns test discovery. The LSP consumes the compiler's test identities and source ranges rather than maintaining a second parser that scans for *_test.sec or test "..." patterns.

The workspace must support separate production and test compilation views so test-only declarations, imports and testing members are understood without contaminating ordinary production analysis.

Where the editor supports it, tooling should expose actions such as:

• Run Test;
• Debug Test;
• Run Tests in File;
• Run Tests in Module;
• Run Project Tests.

Editor execution uses the same TestCompilationPlan and result semantics as sec test. The editor does not define a second test runner with different language behavior.

Diagnostics

Compile-time diagnostics include errors such as:

• a test declaration outside a *_test.sec file;
• a nested test declaration;
• an empty or duplicate top-level test name;
• test parameters or a source-visible return type;
• returning a value from a test;
• production code depending on a test-only declaration;
• using testing outside test context;
• invalid Expect/Require condition types;
• an illegal equality passed to equality helpers;
• invalid subtest identity.

Runtime test failures are not compiler diagnostics. A program may compile successfully and later produce structured test failure records. Tooling must preserve that distinction.

Not in Sec 0.1

The testing rulebook deliberately leaves several areas for later design:

• benchmark declaration syntax and sec benchmark semantics;
• fuzz declaration syntax, generation, shrinking and coverage guidance;
• snapshot and golden-file testing DSLs;
• general system/end-to-end orchestration;
• universal portable test-resource embedding;
• source-level per-test timeout syntax;
• testing.Parallel;
• executable documentation examples.

Current Implementation Status

The Sec 0.1 testing rulebook is normative, but the implementation ledger currently marks the language-testing integration as planned.

The intended model is therefore specified, but the compiler has not yet integrated the complete testing vertical slice. Remaining implementation work includes the top-level test declaration, test-mode source selection, TestCompilationPlan, compiler-known testing operations, test-boundary error propagation, generated harnesses, subtests, sec test, structured results, execution providers and compiler-owned LSP/editor discovery.

Best Practices

• Keep source tests in *_test.sec files.
• Use same-module files for white-box testing and tests/ modules for external API testing.
• Use testing.Expect when later checks remain useful.
• Use testing.Require when continuing would produce misleading failures.
• Use ordinary try and match to test expected errors.
• Use defer and normal ownership for cleanup rather than assuming hidden test teardown.
• Do not make top-level tests depend on execution order.
• Do not assume each test receives a fresh process or reset hardware state.
• Keep test-only dependency direction one-way: tests may depend on production, never the reverse.
• Remember that target tests execute with target semantics, not host semantics.
• Use compiler-owned test discovery through sec test and editor integration rather than custom naming conventions.
• Distinguish a failed test from a compiler failure, target fault, timeout or unavailable execution provider.

Summary

• Sec source tests live in *_test.sec files.
• Top-level tests use test "name" { ... }.
• Production builds exclude test files and test-only dependencies.
• Same-directory test files participate in the production module for white-box testing.
• Project-root tests/ contains test-only modules for black-box and integration testing.
• Tests use ordinary Sec statements, ownership, errors, cleanup and concurrency rules.
• A test invocation is a compiler-known try propagation boundary.
• Unhandled Err at that boundary becomes a structured test failure after cleanup.
• The compiler-known testing namespace needs no import.
• Expect is non-fatal; Require is fatal to the current invocation.
• Equality helpers use normal Sec equality and evaluate arguments exactly once.
• assert retains ordinary runtime assertion semantics.
• testing.Run creates synchronous hierarchical subtests.
• Top-level tests must not rely on a defined execution order.
• Testing guarantees local scope and cleanup, not process or global-state isolation.
• sec test constructs a first-class TestCompilationPlan.
• The compiler workspace owns static test discovery and generates the test harness.
• Results are structured as Passed, Failed or Skipped; infrastructure failures remain separate.
• The same source testing model applies to hosted, simulated, RTOS and bare-metal targets.
• LSP/editor actions use the same compiler-owned discovery and execution semantics as sec test.
• The testing specification is normative, while the current implementation ledger remains planned.