Use the JIT task pool

Example source: loom/binding/c/example/jit_task_pool.c

LoomC separates a queueable unit of work from the scheduler that executes it. An embedding can submit the same loomc_task_t records to its own event loop, deterministic executor, or an optional standard loomc_task_queue_t. The core compiler API does not link the task runtime. The standard implementation splits its shared worker population (loomc_task_pool_t) from each independent task domain (loomc_task_queue_t), and neither object owns compiler policy, caches, workspaces, requests, products, or completion state.

This separation keeps cache hits on the shortest path. A host first derives the identity of the requested product and checks its process-local cache. Only a miss allocates a task and crosses a scheduler boundary:

request -> classify -> cache lookup -> hit  -> reuse product
                                    -> miss -> submit task -> compile -> publish

The checked example loads eight immutable typed configuration modules, then compiles eight independently configured program modules through the public LoomC API. Configuration parsing stays outside the worker callback; each task borrows its prepared configuration module for the compile invocation. The example is intentionally target-independent; a native JIT uses the same task shape with its prepared target pipeline and emitter.

Pipeline independent work domains

One process-wide pool can drive compiler queues, HAL materializers, and application processes without forcing their work through one FIFO:

                             +-> compile queue --------+
request -> cache miss -------+                         +-> publish product
                             +-> command queue --------+
                                                       |
shared worker pool <------ HAL executable loader <-----+
                   <------ command-buffer recorder <---+
                   <------ application processes

A queue owns readiness and scheduling only within its domain; callbacks from one queue may execute concurrently and complete in any order. A compile callback that publishes an executable-load task therefore makes that task immediately eligible on the shared workers instead of placing it behind the remaining compilation roots. The worker population remains bounded while compilation, executable loading, and command-buffer recording pipeline naturally.

Applications using the IREE task runtime include loomc/iree/task_pool.h to allocate a pool from an existing iree_task_executor_t or borrow the executor created by a LoomC pool. HAL integrations attach their own cooperative processes to that executor and do not depend on LoomC task queues. Applications using another scheduler continue to implement loomc_task_sink_t directly.

Split process, worker, and request state

The scheduler exposes a dense, mutually exclusive worker ordinal on every task callback. That ordinal indexes caller-owned mutable scratch directly, without a workspace lease, hash lookup, or lock:

Lifetime	Example state
Process	Context, source catalog, frozen link indexes, compiler, pass programs, cache, shared task pool, and independent queues.
Worker	One loomc_workspace_t and any target-specific scratch indexed by worker ordinal.
Request	Immutable request/configuration, task record, terminal result, and application completion state.

The example prepares the shared compiler state, requests a four-worker pool, attaches one compilation queue, and allocates exactly one workspace per actual worker:

static loomc_status_t configure_jit_service(jit_service_t* service) {
  loomc_status_t status =
      loomc_context_create(NULL, service->allocator, &service->context);

  const loomc_source_options_t source_options = {
      .type = LOOMC_STRUCTURE_TYPE_SOURCE_OPTIONS,
      .structure_size = sizeof(source_options),
      .format = LOOMC_SOURCE_FORMAT_TEXT,
      .identifier = loomc_make_cstring_view("jit_task_pool.loom"),
      .contents = loomc_make_byte_span(kSourceText, sizeof(kSourceText) - 1),
      .storage = LOOMC_SOURCE_STORAGE_BORROWED,
  };
  if (loomc_status_is_ok(status)) {
    status = loomc_source_create(&source_options, service->allocator,
                                 &service->source);
  }
  if (loomc_status_is_ok(status)) {
    status = loomc_compiler_create(service->context, NULL, service->allocator,
                                   &service->compiler);
  }

  loomc_result_t* pass_result = NULL;
  if (loomc_status_is_ok(status)) {
    status = loomc_pass_program_create_from_pipeline_text(
        service->context, loomc_make_cstring_view("canonicalize,cse,dce"), NULL,
        service->allocator, &service->pass_program, &pass_result);
  }
  if (loomc_status_is_ok(status)) {
    status =
        require_successful_result(pass_result, "pass program creation failed");
  }
  loomc_result_release(pass_result);

  const loomc_task_pool_options_t pool_options = {
      .type = LOOMC_STRUCTURE_TYPE_TASK_POOL_OPTIONS,
      .structure_size = sizeof(pool_options),
      .max_worker_count = 4,
  };
  if (loomc_status_is_ok(status)) {
    status = loomc_task_pool_allocate(&pool_options, service->allocator,
                                      &service->task_pool);
  }
  if (loomc_status_is_ok(status)) {
    status = loomc_task_queue_allocate(service->task_pool, service->allocator,
                                       &service->compile_queue);
  }
  if (loomc_status_is_ok(status)) {
    const loomc_host_size_t worker_count =
        loomc_task_pool_worker_count(service->task_pool);
    status = loomc_allocator_malloc(service->allocator,
                                    worker_count * sizeof(*service->workspaces),
                                    (void**)&service->workspaces);
    for (loomc_host_size_t i = 0;
         loomc_status_is_ok(status) && i < worker_count; ++i) {
      status = loomc_workspace_create(NULL, service->allocator,
                                      &service->workspaces[i]);
      if (loomc_status_is_ok(status)) ++service->workspace_count;
    }
  }
  for (loomc_host_size_t i = 0;
       loomc_status_is_ok(status) && i < JIT_VARIANT_COUNT; ++i) {
    const char* config_text = kVariantConfigTexts[i];
    const loomc_source_options_t config_source_options = {
        .type = LOOMC_STRUCTURE_TYPE_SOURCE_OPTIONS,
        .structure_size = sizeof(config_source_options),
        .format = LOOMC_SOURCE_FORMAT_TEXT,
        .identifier = loomc_make_cstring_view("jit_task_pool_config.loom"),
        .contents = loomc_make_byte_span(config_text, strlen(config_text)),
        .storage = LOOMC_SOURCE_STORAGE_BORROWED,
    };
    loomc_source_t* config_source = NULL;
    loomc_result_t* config_result = NULL;
    status = loomc_source_create(&config_source_options, service->allocator,
                                 &config_source);
    if (loomc_status_is_ok(status)) {
      status = loomc_module_deserialize_text_from_source(
          service->context, service->workspaces[0], config_source, NULL,
          service->allocator, &service->config_modules[i], &config_result);
    }
    if (loomc_status_is_ok(status)) {
      status = require_successful_result(config_result,
                                         "config deserialization failed");
    }
    loomc_result_release(config_result);
    loomc_source_release(config_source);
  }
  if (service->workspace_count != 0) {
    loomc_workspace_trim(service->workspaces[0]);
  }
  return status;
}

Four workers are the default and suit common latency-oriented JIT workloads. Eight is another normal application choice when independent compilations have enough work to amortize scheduling. Search and autotuning services can request wider pools when many parameter variants produce independent reports. The actual worker count may be lower when the process affinity or container CPU set exposes fewer physical cores.

Keep the task record application-owned

A concrete task embeds loomc_task_t at offset zero and carries only the state needed to terminalize that operation:

typedef struct jit_task_t {
  // Generic task base owned by the accepting sink.
  loomc_task_t base;

  // Shared immutable service state and worker-local workspace table.
  jit_service_t* service;

  // Immutable typed configuration module for this compilation request.
  const loomc_module_t* config_module;

  // Caller-owned output slot that outlives task execution.
  jit_output_t* output;
} jit_task_t;

Successful submission transfers the task reference to the sink. The sink runs the callback exactly once and then invokes its destructor exactly once. A rejected submission leaves ownership with the caller, which may retry another sink or destroy the task immediately.

The callback selects worker-local scratch by ordinal and otherwise uses shared immutable compiler state:

static void execute_jit_task(loomc_task_t* base_task,
                             loomc_host_size_t worker_ordinal) {
  jit_task_t* task = (jit_task_t*)base_task;
  jit_service_t* service = task->service;
  loomc_workspace_t* workspace = service->workspaces[worker_ordinal];

  loomc_module_t* module = NULL;
  loomc_result_t* result = NULL;
  loomc_status_t status = loomc_module_deserialize_text_from_source(
      service->context, workspace, service->source, NULL, service->allocator,
      &module, &result);

  if (loomc_status_is_ok(status) && loomc_result_succeeded(result)) {
    loomc_result_release(result);
    result = NULL;

    const loomc_compile_options_t options = {
        .type = LOOMC_STRUCTURE_TYPE_COMPILE_OPTIONS,
        .structure_size = sizeof(options),
        .artifact_flags = LOOMC_COMPILE_ARTIFACT_FLAG_MODULE_BYTECODE,
        .config_flags = LOOMC_CONFIG_POLICY_FLAG_REQUIRE_RESOLVED,
        .config_module = task->config_module,
    };
    status = loomc_compile_module(service->compiler, workspace,
                                  service->pass_program, module, &options,
                                  service->allocator, &result);
  }

  task->output->status = status;
  task->output->result = result;
  loomc_module_release(module);
  loomc_workspace_trim(workspace);
}

Task execution has no scheduler-facing status return. The concrete task writes its compiler result or infrastructure status into caller-owned completion state before returning. This prevents an application compilation failure from poisoning unrelated tasks or turning the scheduler into a second compiler API.

Compose completion separately from scheduling

The compact example submits one finite batch and uses queue shutdown as its join:

static loomc_status_t submit_jit_tasks(jit_service_t* service,
                                       jit_output_t* outputs,
                                       loomc_host_size_t output_count) {
  loomc_task_sink_t sink = loomc_task_queue_sink(service->compile_queue);
  loomc_status_t status = loomc_ok_status();
  for (loomc_host_size_t i = 0; loomc_status_is_ok(status) && i < output_count;
       ++i) {
    loomc_task_t* task = NULL;
    status = allocate_jit_task(service, service->config_modules[i], &outputs[i],
                               &task);
    if (loomc_status_is_ok(status)) {
      status = loomc_task_sink_submit(sink, task);
    }
    if (!loomc_status_is_ok(status) && task != NULL) {
      // Rejection preserves the ownership offered to the sink.
      loomc_task_destroy(task);
    }
  }

  // This short-lived example uses shutdown as its batch join. A persistent
  // compiler service tracks request completion separately and shuts its
  // compiler queue down only during service teardown. Other queues and native
  // processes attached to the same pool remain independently runnable.
  if (loomc_status_is_ok(status)) {
    status = loomc_task_queue_shutdown(service->compile_queue);
  }
  if (loomc_status_is_ok(status)) {
    status = loomc_task_queue_await_shutdown(service->compile_queue);
  }
  if (!loomc_status_is_ok(status)) {
    // Drain accepted work before caller-owned output storage is inspected or
    // released. The void teardown path preserves the earlier operation error.
    loomc_task_queue_free(service->compile_queue);
    service->compile_queue = NULL;
  }
  return status;
}

A long-lived compiler service instead allocates one pool and its attached queues for the service lifetime, then uses request completion objects, callbacks, futures, or event-loop messages. Neither a queue nor the pool exposes a global idle wait: another producer may publish work at any time, and observing a transiently empty domain does not mean an application request is complete. The request owner knows exactly which products it is waiting for and terminalizes its completion object when all of them have resolved.

Queue shutdown is drain-only. It stops that domain from accepting work, executes every accepted task, and releases its cooperative process without affecting other domains attached to the pool. Work that recursively publishes child tasks therefore completes its request before its queue begins teardown; publication attempted after shutdown is correctly rejected. Attached queues retain the shared executor, so they may outlive the convenience pool handle and the final owner joins the worker threads.

Run the checked example

python dev.py bazel run //loom/binding/c/example:jit_task_pool

The output confirms that every scheduled variant produced its requested bytecode artifact:

compiled=8

Applications that already own a scheduler include loomc/task.h and implement loomc_task_sink_t::submit. Applications wanting the standard implementation link //loom/binding/c:task_pool and //loom/binding/c:task_queue, then include loomc/task_pool.h and loomc/task_queue.h. Both paths use the same concrete task records and preserve the same ownership contract.