Building in forgejo #3
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Hypothetically, if I had our agent army build an app here in forgejo, would they be able to use the build system we have for docker containers to also build a project?
SysAdmin Pete has been assigned to this task.
SysAdmin Pete has been assigned to this task.
Short answer: yes, absolutely. The build infrastructure is already set up for exactly this kind of thing.
Here's how it maps out:
What we have:
git.lab.lostsource.netdocker/build-push-action, same secrets managementWhat an agent-built app would need:
Dockerfilein the repo (multi-stage build preferred).forgejo/workflows/build.ymlwith the standard build-push workflowThe agents already push code via git to Forgejo repos in the
agentsorg. Adding a CI workflow file is just another committed file. When they push a tag or merge to main, the runner picks it up, builds the image, and pushes it to the registry. Standard stuff.Considerations:
npm install+ webpack/vite step) need 8-12GB. Simple Go/Python/Rust apps are fine.The plumbing is all there. The gap is just making the agents aware that CI exists and giving them the patterns to use it.
You reference building docker apps. Can it build, for example, c++ or rust apps and spit out a binary?
Yes, but with a nuance on how.
The runner itself is a Debian 13 VM. It doesn't have gcc, rustc, or any native toolchains installed on the host -- it was purpose-built for Docker image builds. But that's actually fine, because CI workflows run inside containers.
Two approaches:
1. Build inside a Docker container, extract the binary (most practical)
The workflow spins up a container with the right toolchain image (
rust:latest,gcc:latest, etc.), compiles inside it, and uploads the binary as a workflow artifact or copies it out viadocker cp. No Dockerfile needed in the repo -- just a workflow that does:The artifact gets attached to the workflow run and is downloadable from the Forgejo UI or API.
2. Multi-stage Dockerfile that exports just the binary
Write a Dockerfile that builds in a compiler stage and copies only the binary to a
scratchordistrolessfinal image. This is what we already do for Go-based projects like cadvisor. The result is a tiny container image with just the binary in it. You pull the image and extract the binary, or just run the container directly.3. Install toolchains on the runner host (not recommended)
We could install gcc/rustc/etc directly on the Forgejo VM and run builds natively without Docker. But that defeats the whole point of reproducible containerized builds and would bloat the VM. Don't love this option.
Practical considerations:
--platformand buildx, which the runner supports.Short version: the runner doesn't compile natively, but it doesn't need to. Docker containers give you any toolchain you want, and artifacts come out the other end as downloadable binaries or minimal container images. The plumbing supports it today.
Having containers do the building does sound easier. No dependencies and can be easily built on other systems using the same container pipeline.
Exactly right. That's the preem part of the container build approach -- the Dockerfile is the dependency manifest. Anyone (or any agent) can reproduce the exact same build on any system with a Docker daemon. No "works on my machine" nonsense.
It also means the agents don't need to care what's installed on the runner. They write a Dockerfile, reference whatever base image has their toolchain, and the runner just executes it. Rust today, C++ tomorrow, some cursed Fortran project next week -- same pipeline, same workflow file structure, just a different
FROMline.If you want to move forward with this, the next step would be having HAL scaffold a Dockerfile + workflow template as part of its design phase when an issue calls for a compiled binary. That way the agents get the build pipeline for free without needing to figure it out from scratch each time.
So the building of the docker image would have it output the binary itself then the agent could stick that binary in the repo as an artifact?
Not quite -- let me clarify the flow because there are two different meanings of "artifact" here.
Workflow artifacts (what CI produces):
The build runs inside a Docker container. When it finishes compiling, the workflow uses
upload-artifactto attach the binary to that workflow run. It shows up in the Forgejo UI under the Actions tab for that run -- anyone can download it from there, or grab it via the API. This is the standard approach and what I'd recommend.Git repo artifacts (committing binaries to the repo):
You could have the workflow commit the binary back into the repo, but I'd strongly advise against it. Binaries in git repos bloat the history permanently, make clones slow, and git isn't designed for large binary files. Every version of the binary lives in the git history forever. Bad practice.
Better options for distributing the binary:
Workflow artifacts -- attached to each CI run, downloadable from the UI or API. Simple, built-in, no extra setup. These are ephemeral though -- Forgejo cleans them up after a configurable retention period.
Forgejo Releases -- the workflow creates a tagged release and attaches the binary as a release asset. This is the proper way to distribute versioned binaries. Think of it like GitHub Releases. Permanent, versioned, downloadable via URL. The workflow would use the Forgejo API or a release action to create the release and upload the binary.
Push a minimal container image -- if the binary is meant to run as a service, wrap it in a
scratchordistrolesscontainer image and push it to the registry. Then deploy it like any other container.Option 2 (Releases) is probably what you're actually after. The agent builds the app, tags a version, CI compiles the binary, and the workflow creates a release with the binary attached. Clean URL, permanent, versioned. The workflow for this is straightforward -- maybe 10 extra lines on top of the build step.
Ok, so what is ultimately output is a file, not a docker image with the binary stuck inside it, right?
It can be either -- you choose based on the workflow.
Option A: Just the binary (standalone file)
The workflow spins up a compiler container, builds the code, and the binary comes out as a workflow artifact -- a plain file you download. No Docker image involved in the final output. The container is just the build environment, it gets thrown away after compilation. You end up with a raw ELF binary (or whatever the target platform needs) sitting in the Forgejo Actions artifacts tab.
Option B: A minimal Docker image containing the binary
Multi-stage Dockerfile: first stage compiles, second stage is
scratchordistroless(basically an empty filesystem) with just the binary copied in. The output is a container image, but it's essentially just a thin wrapper around the binary. This is what you'd use if the binary is meant to run as a service on the infrastructure.They're not mutually exclusive. A single workflow can do both -- compile, upload the raw binary as an artifact, and package it into a minimal container image. But if all you want is the compiled binary as a downloadable file, Option A is all you need. No Docker image in the output at all -- Docker is just the build tool, not the delivery format.
Option a is what I'd prefer for building a binary.
SysAdmin Pete has been assigned to this task.
Short answer: yes, absolutely. The Forgejo runner is GitHub Actions-compatible and it's not limited to Docker image builds.
Here's what we're working with on the Forgejo VM (forgejo-vm, 100.65.188.81):
Right now the 27 workflows in the
buildsorg are all focused on building Docker container images (15 from source, 12 mirrored). But the runner itself is just executing workflow YAML -- it can run whatever you throw at it.npm build,cargo build,go build, Python packaging, static site generation, whatever. If the toolchain can be installed in a workflow step or a container action, it'll work.A few things to keep in mind:
The runner uses container actions by default -- each job runs in an ephemeral Docker container (usually
node:ltsor whatever you specify inruns-on/container:). So build deps need to be installed per-run or baked into a custom runner image.12GB RAM is the ceiling. Grafana's frontend build already pushes that limit. If the agents are building something with a heavy frontend (big Node/webpack builds), that could be a constraint.
Artifacts and deployment -- building is one thing, deploying is another. The existing workflows push to the Forgejo container registry. For a non-Docker project you'd need to define where artifacts go -- Forgejo has a generic package registry, or you could use release attachments, or push to a deploy target via SSH/SCP from the workflow.
The
customorg already exists for exactly this kind of thing -- personal/custom projects separate from the mirror and build orgs.So if the agent army is building an app in this repo and you want CI for it -- tests, builds, packaging, whatever -- the infrastructure's already there. Just needs a
.forgejo/workflows/directory with the right YAML. The hard part you already solved was getting the runner stable in host mode.Makes sense. Clean and simple -- Docker is the build environment, not the delivery format. The binary comes out the other end as a plain downloadable artifact.
So to lock this down, the pattern for agent-built compiled projects would be:
.forgejo/workflows/build.ymlrust:latest,gcc:latest, etc.) to compileupload-artifactNo Docker image in the output. No toolchain bloat on the runner. The agents just need to know the workflow template pattern and which base image to use for their language.
If you want, I can sketch out the exact workflow YAML template that HAL would scaffold for compiled binary projects -- Rust and C++ variants. That way when you're ready to wire it into the agent design phase, the pattern is already documented.
do that and save it as a memory for building binaries in forgejo for future use
Done. Saved four memory entries under
forgejo/binary_build_*-- an overview plus one template per language. Here are the workflow templates HAL can scaffold:Rust Binary Build
C/C++ Binary Build
Adjust the cmake/make commands for the project's build system -- autotools (
./configure && make), meson (meson setup build && ninja -C build), or plain Makefiles. Add project-specific libs to theapt-get installstep as needed.Go Binary Build
CGO_ENABLED=0gives a fully static binary.-ldflags='-s -w'strips debug info for smaller output.The pattern across all three:
v*) or manual dispatchupload-artifact, downloadable from Actions tabhttps://git.lab.lostsource.net/ORG/REPO/releases/download/TAG/BINARY_NAMEGITHUB_TOKENis auto-provided by ForgejoMemory keys saved:
mem get forgejo binary_build_overview-- concept + constraints + agent integration notesmem get forgejo binary_build_rust_workflow-- Rust templatemem get forgejo binary_build_cpp_workflow-- C/C++ templatemem get forgejo binary_build_go_workflow-- Go templateWhen HAL designs a compiled binary project, it can pull the right template from memory and drop it into
.forgejo/workflows/build.yml.