Overview
- Background and Motivation
- Anatomy of a bulk build
- pbulk components
- Future plans
Some background
- Fixing issues in FreeBSD ports since end of 2003
- Moved to pkgsrc in summer 2005
- Regular bulk builds since October 2005
- Rule: A problem fixed in the bulk build is a problem no user can complain about.
- Rule: A problem in a bulk build is easier to reproduce than a problem from a user.
- Binary packages are a by-product
Anatomy of a bulk build
- Setup of machine and bulk build
- Prepare the pkgsrc tree
- Find out what to build
- Build packages
- Summarize results
- Upload results
- Upload binary packages
Specific problems
- Stray directories in /usr/pkg
- Stray dependencies in /usr/pkg (gmake, pkg-config)
- .if exists(${X11BBASE}/include/X11/Xdefs.h) ... .endif
- math/p5-Math-Pari
- Writing to /etc
- Writing to /home
- Scalability
pbulk components (I)
- pbulk-scan: Schedule scanning of pkgsrc parts
- pbulk-resolve: Compute dependency tree
- pbulk-build: Schedule build of packages
- Small glue in mk/pbulk for scan target
pbulk components (II)
- Schedulers in C: network access, memory usage
- Resolver in C: code reuse for Dewey, speed
- Upload logic somewhat overlapped with pbulk-build
- Report logic completely in awk
The scan component
- Take list of PKGPATH or extract subdirectories using make
- Pull/push model for distributed scans
- Call "make pbulk-index" for every PKGPATH
- make is the only way to get reliable results
- Collect and reorder list, drop duplicate entries
The resolver component
- Implement a mix of pkg_add and pkgsrc algorithm to find dependencies
- If in doubt, prefer pkgsrc view, but warn
- {gs,gs-esp}-[0-9]*:../../print/ghostscript requested, gs-7.0 and gs-esp-7.0nb1 exist — what to pick?
- Complain or list what couldn't be found
The build component
- Pull/push model for distributed scans
- Find package with all dependencies done and highest priority
- Sort order: number of packages depending on this
Future plans
- Allow to easier restrict multi-version logic
- Allow using incremental mode + post-filtering when missing dependencies
- Move restricted subset logic for upload into upload itself
- Tag bootstrap packages and don't install them
- Remove package if install/deinstall test fails
- Implement poll loop directly to get rid of libevent
- Split into base part and distinguish official bulk build and simple user case