Chiltepin Documentation

Chiltepin is a Python library for exploring federated agentic workflow capabilities using Parsl, Globus Compute, and Academy Agents. It provides tools and demonstrations for implementing distributed, agentic, exascale scientific workflows on HPC systems.

Warning

This project is for research and exploration purposes only. It is not intended for use in operational production environments.

Platform Support

Chiltepin is developed and tested on Linux:

• ✅ Linux: Full support for all features

• 🍎 macOS: Task submission and data transfer supported (endpoint management not available)

• ❌ Windows: Not supported natively due to reliance on POSIX fork semantics (use Docker or WSL2)

• 🐳 Docker: Full support available via container on all platforms

Note

Windows users can use Chiltepin via Docker container or WSL2 with a Linux distribution. Chiltepin and its use of Parsl both require fork-based multiprocessing which is not available on native Windows.

Overview

This repository is a collection of tools and demonstrations used for implementing distributed, agentic, exascale scientific workflows. The project focuses on:

• Workflow management using Parsl

• Federated distributed computing with Globus Compute

• Autonomous workflow agents with Academy Agents integration

• HPC integration of multiple on-prem and/or cloud-based systems

• Container-based testing with Docker and Slurm

Key Features

• Configuration-based resource management for both HPC platforms and laptops

• Support for both MPI (HPC) and non-MPI (HTC) applications

• Agentic workflows with Academy Agents integration

• Task decorators for seamless integration of Parsl and Globus Compute

• Globus Compute endpoint management utilities

• Dynamic distributed task execution across heterogeneous resources

• Docker container environment for development and testing

• Comprehensive test suite with high coverage for core modules

Getting Started

• Installation
  • Prerequisites
  • Installing from PyPI
  • Development Installation
  • Activating the Environment
  • Dependencies
• Quick Start
  • Overview
  • Prerequisites
  • Complete Workflow Example
  • Workflow Usage Patterns
  • Local-Only Quickstart
  • Working with MPI Tasks
  • Key Concepts
  • Directory Structure
  • Troubleshooting
  • Next Steps
• Tasks
  • Overview
  • Python Tasks
  • Bash Tasks
  • MPI Tasks and Task Geometry
  • Join Tasks
  • Tasks as Class Methods
  • Specifying Resources
  • Futures and Results
  • File and Data Handling
  • Advanced Topics
  • Best Practices
  • Common Patterns
  • See Also
• Agents
  • Overview
  • When to Use Agents
  • Behaviors vs Agents
  • Basic Usage
  • Runtime Configuration
  • Action Decorators
  • Loop Decorators
  • AgentSystem Helper
  • ChiltepinManager
  • Agent Composition
  • Best Practices
  • Complete Example
  • Troubleshooting
  • See Also
• Data Transfer and Management
  • Overview
  • Data Transfer Task
  • Data Deletion Task
  • Workflow Integration
  • Authentication
  • Setting Up Data Endpoints
  • Best Practices
  • Troubleshooting
  • See Also
• Configuration
  • Overview
  • Basic Structure
  • Understanding Resource Configuration
  • Resource Types
  • Configuration Options
  • Example Configurations
  • Loading Configurations
  • Configuration Best Practices
  • See Also
• Endpoint Management
  • Overview
  • The chiltepin Command
  • Authentication
  • Endpoint Commands
  • Endpoint Lifecycle
  • User Endpoint Configuration
  • Troubleshooting
  • Python API
  • See Also
• Testing
  • Prerequisites
  • Globus Authentication
  • Platform-Specific Testing
  • Running Tests
  • Coverage Reports
  • Test Organization
  • Docker Container Testing
• Docker Container
  • Prerequisites
  • Building the Container
  • Accessing the Container
  • Installing Chiltepin in the Container
  • Running Tests in the Container
  • Adjusting Core Count
  • Container Architecture
  • Stopping the Container
  • Troubleshooting

API Reference

• API Reference
  • Configuration Module
  • Workflow Module
  • Endpoint Management Module
  • Tasks Module
  • Agents Module
  • Data Module

Disclaimer

This repository is a scientific product and is not official communication of the National Oceanic and Atmospheric Administration, or the United States Department of Commerce. All NOAA GitHub project code is provided on an ‘as is’ basis and the user assumes responsibility for its use. Any claims against the Department of Commerce or Department of Commerce bureaus stemming from the use of this GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.