Skip to content

Security: PipeOpsHQ/Rexec

docs/SECURITY.md

Rexec Security Hardening

This document outlines the security measures implemented in Rexec containers.

Container Security Configuration

Capability Management

Containers run with minimal Linux capabilities:

CapDrop: ALL
CapAdd:
  - CHOWN           # Change file ownership
  - DAC_OVERRIDE    # Bypass file permission checks (needed for sudo)
  - FOWNER          # Bypass permission checks on file owner
  - SETGID          # Set group ID
  - SETUID          # Set user ID (needed for su/sudo)
  - KILL            # Send signals
  - NET_BIND_SERVICE # Bind to ports < 1024

Privilege Escalation Prevention

  • no-new-privileges:true - Prevents processes from gaining additional privileges
  • Privileged: false - Container cannot access host devices
  • Default Docker seccomp profile blocks dangerous syscalls

Filesystem Security

Read-only Root Filesystem:

  • ReadonlyRootfs: true - Root filesystem is mounted read-only
  • Writable directories use tmpfs mounts with size limits

Tmpfs Mounts:

Path Options Size Limit
/tmp rw,noexec,nosuid 100MB
/var/tmp rw,noexec,nosuid 50MB
/run rw,noexec,nosuid 50MB
/var/run rw,noexec,nosuid 50MB
/root rw,nosuid 50MB

User Data:

  • /home/user uses a persistent Docker volume (not tmpfs)
  • Volume is isolated per-container

Masked Paths

Sensitive host information is hidden from containers:

  • /proc/acpi - ACPI information
  • /proc/asound - Sound devices
  • /proc/kcore - Kernel memory
  • /proc/keys - Kernel keyrings
  • /proc/latency_stats - Latency statistics
  • /proc/timer_list - Timer information
  • /proc/timer_stats - Timer statistics
  • /proc/sched_debug - Scheduler debug info
  • /proc/scsi - SCSI devices
  • /sys/firmware - Firmware information
  • /sys/devices/virtual/powercap - Power capping
  • /sys/kernel - Kernel parameters

Read-only Paths

Critical paths are mounted read-only:

  • /proc/bus - Bus information
  • /proc/fs - Filesystem info
  • /proc/irq - Interrupt info
  • /proc/sys - Kernel parameters
  • /proc/sysrq-trigger - Magic SysRq

Resource Limits

  • Memory: Configurable per-tier (512MB - 4GB)
  • CPU: Quota-based limiting
  • PIDs: Maximum 256 processes (fork bomb protection)
  • Disk: Quota support on XFS with pquota

Network Isolation

  • Containers run in isolated Docker network (rexec-isolated)
  • Inter-container communication is restricted
  • Controlled sysctl: net.ipv4.ip_unprivileged_port_start=0
  • default and restricted sandboxes cannot reach cloud instance metadata (169.254.169.254 and the related link-local addresses) or the Azure wireserver (168.63.129.16). The host installs those drops in the DOCKER-USER chain via scripts/block-sandbox-metadata.sh. Traffic the host itself generates still reaches IMDS, so the host agent can read its region.
  • network_mode=none has no network interface
  • network_mode=restricted is an HTTP(S) allowlist through the egress proxy. It is not the metadata control. A direct connection ignores HTTP_PROXY, which is why the firewall drops metadata addresses for every sandbox that has a network interface

Do not put API tokens in Azure VM custom data or userData. The instance metadata service returns that field to any process on the VM that can reach it. Agent credentials belong in the host systemd unit, and an agent token is scoped to agent so it cannot call user or admin APIs.

Security Verification

Run these commands inside a container to verify security settings:

# Check capabilities
cat /proc/self/status | grep Cap

# Check user context
id
whoami

# Check mount restrictions
mount | grep -E "(proc|sys|dev)"

# Check network isolation
ip addr show

# Check resource limits
ulimit -a

Security Scoring

Configuration Score Description
Basic Docker 6/10 Default Docker isolation
Current Rexec 8/10 Enhanced container security
With gVisor 9/10 Kernel-level isolation

Terminal MFA Lock

Rexec supports per-terminal MFA (Multi-Factor Authentication) protection. When enabled, accessing a terminal requires entering a valid TOTP code from your authenticator app.

Prerequisites

  • MFA must be enabled on your account first (Account Settings → Security → Enable MFA)
  • Set up your authenticator app (Google Authenticator, Authy, 1Password, etc.)

How It Works

  1. Lock a Terminal: Click "Lock with MFA" on any running terminal in your dashboard
  2. Access Protection: When trying to connect to an MFA-locked terminal, you'll be prompted for your 6-digit authenticator code
  3. Unlock a Terminal: Enter your MFA code to remove the lock and allow normal access

Use Cases

  • Sensitive Workloads: Protect terminals with production access or secrets
  • Shared Accounts: Add extra protection even if someone has your session
  • Compliance: Meet security requirements for privileged access

API Endpoints

Endpoint Method Description
/api/security/terminal/:id/mfa-status GET Check MFA lock status
/api/security/terminal/:id/mfa-lock POST Enable MFA protection
/api/security/terminal/:id/mfa-unlock POST Remove MFA protection (requires code)
/api/security/terminal/:id/mfa-verify POST Verify access without unlocking

Audit Logging

All MFA lock/unlock actions are logged to the audit trail:

  • terminal_mfa_locked - Terminal was protected with MFA
  • terminal_mfa_unlocked - MFA protection was removed
  • terminal_mfa_access_verified - User verified MFA to access terminal

Future Improvements

  1. User Namespace Remapping - Map container root to unprivileged host user
  2. Custom Seccomp Profile - More restrictive syscall filtering
  3. AppArmor/SELinux - Mandatory access control profiles
  4. Network Policies - Fine-grained egress controls

References

There aren't any published security advisories