The challenge
A native command-line tool has to run on machines that share almost nothing: different operating systems, different package managers, and different C toolchains. Compiling each binary by hand is slow, and a release should never hinge on someone remembering the right command on the right host in the right order.
The controller also lives on a private network with no inbound access, so the usual webhook-driven trigger is off the table — the pipeline has to discover new work on its own.
Our approach
We run a controller-and-agent Jenkins topology: one Jenkins LTS controller orchestrates ten build agents spanning five OS families. A multibranch pipeline indexes the Git repository, so every branch and pull request gets its own job and each build runs the Jenkinsfile committed at that exact revision — the build definition can never drift from the code it builds.
Because the fleet has no inbound webhooks, the controller polls the repository on a short interval and picks up moved head commits within minutes. A branch guard keeps pull requests honest: they compile and archive artifacts for inspection but never touch a production binary. Only a merge to main installs, and every install is atomic — copy then move into place — so a running binary is never overwritten mid-write.
Technical specifics
- One Jenkins LTS controller (Ubuntu, Java 21) driving 10 agents across Ubuntu, Fedora, FreeBSD, Solaris, and OpenBSD; the two FreeBSD agents run as vnet jails.
- A private-LAN fleet with no inbound webhooks, so builds trigger on a 5-minute repository poll — the worst-case push-to-build latency.
- A single build matrix fans compilation across every node in parallel, using one identical make invocation with per-OS flags kept in the repository Makefile.
- A least-privilege connection model: the general-purpose agents connect as a sudo-capable user, while the two FreeBSD jails connect as root for jail-local installs.
- Per-node artifact names and toolchain-grouped checksums that prove each host received its own correct native build.
- Releases cut with the GitHub CLI (gh release create) as a single pipeline step, straight from a green build.
The fleet is Infrastructure as Code, too
The pipeline is only half the story: the machines that run it are provisioned the same way the software is built — declaratively and repeatably. Three idempotent Ansible playbooks stand up the entire fabric, so a rebuilt or newly added agent is one command away from being production-ready.
- install-jenkins-controller — Java 21 and Jenkins LTS on the controller host.
- install-jenkins-client — the JDK, git, and OS-correct packages every agent needs to connect and check out code.
- install-build-tools — a uniform toolchain across the fleet: C/C++, CMake, make / gmake, and C# via .NET and Mono, with per-host package managers (apt / dnf / pkg) handled transparently.
Example configuration
The Jenkinsfile fans the build across agents by label and only installs on main, with an atomic copy-then-move so a running binary is never overwritten in place. The tools each agent needs are declared once, in Ansible.
pipeline {
agent none
triggers { pollSCM('H/5 * * * *') } // private LAN: poll, don't wait for webhooks
stages {
stage('build') {
matrix {
axes { axis { name 'NODE'; values 'ubuntu','fedora','freebsd','solaris','openbsd' } }
stages {
stage('compile') {
agent { label "${NODE}" }
steps {
sh 'gmake -C src' // one command; per-OS flags live in the Makefile
archiveArtifacts "src/ascii-monitor-${NODE}"
}
}
stage('install') {
when { branch 'main' } // PRs build & archive; only main installs
agent { label "${NODE}" }
steps {
sh 'cp src/ascii-monitor /usr/local/bin/.am.new && \
mv -f /usr/local/bin/.am.new /usr/local/bin/ascii-monitor'
}
}
}
}
}
stage('release') {
when { branch 'main' }
agent { label 'ubuntu' }
steps { sh 'gh release create "v${BUILD_NUMBER}" src/ascii-monitor-*' }
}
}
}- name: Uniform build toolchain across the fleet
hosts: build_agents
become: true
tasks:
- name: Debian / Ubuntu toolchain
ansible.builtin.apt:
name: [build-essential, cmake, git, mono-complete]
state: present
update_cache: true
when: ansible_os_family == 'Debian'
- name: Fedora toolchain
ansible.builtin.dnf:
name: [gcc, gcc-c++, cmake, make, git, mono-devel]
state: present
when: ansible_os_family == 'RedHat'
- name: FreeBSD toolchain
community.general.pkgng:
name: [cmake, gmake, git]
state: present
when: ansible_os_family == 'FreeBSD'Outcome
One push to main produces one native binary per operating system and a tagged GitHub release in minutes, with no manual steps from commit to release. Atomic installs prevent corruption, toolchain-grouped checksums prove correctness on every host, and because both the pipeline and the fleet are code, the whole system is reproducible from scratch.