Name: test Version: 3.4.0 Release: 1%{?dist} Summary: Build-service wrapper that builds Apache Bigtop Hadoop RPMs via Gradle License: Apache-2.0 URL: https://bigtop.apache.org %global debug_package %{nil} # --- Bigtop source ------------------------------------------------------------ %global bigtop_ref branch-3.4 %global bigtop_archive_name bigtop-%{bigtop_ref} # Relative path (within the Hadoop source tree) of the YARN Application # Catalog webapp module. Excluded from the Maven reactor in %prep — see the # do-component-build patch below for why. %global catalog_webapp_path hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp Source0: https://github.com/apache/bigtop/archive/%{bigtop_ref}.tar.gz # --- Build arch --------------------------------------------------------------- BuildArch: noarch # --- Build requirements ------------------------------------------------------- # Core toolchain BuildRequires: java-1.8.0-openjdk-devel BuildRequires: java-11-openjdk-devel # Build tools BuildRequires: autoconf BuildRequires: automake BuildRequires: bash BuildRequires: cmake BuildRequires: curl BuildRequires: coreutils BuildRequires: diffutils BuildRequires: findutils BuildRequires: gawk BuildRequires: gcc BuildRequires: gcc-c++ BuildRequires: git BuildRequires: grep BuildRequires: gzip BuildRequires: hostname BuildRequires: libtool BuildRequires: make BuildRequires: nasm BuildRequires: patch BuildRequires: rsync BuildRequires: sed BuildRequires: tar BuildRequires: unzip BuildRequires: wget BuildRequires: which BuildRequires: yasm # Documentation tools BuildRequires: asciidoc BuildRequires: xmlto # Compression / archive libraries BuildRequires: bzip2-devel BuildRequires: libzstd-devel BuildRequires: lzo-devel BuildRequires: snappy-devel BuildRequires: zlib-devel BuildRequires: zstd # Crypto / auth BuildRequires: cyrus-sasl-devel BuildRequires: krb5-devel BuildRequires: openldap-devel BuildRequires: openssl-devel # Network / RPC BuildRequires: libcurl-devel BuildRequires: libtirpc-devel # FUSE BuildRequires: fuse BuildRequires: fuse-devel BuildRequires: fuse-libs BuildRequires: fuse3 BuildRequires: fuse3-devel BuildRequires: fuse3-libs # ISA-L (Intel Storage Acceleration) — required for Hadoop erasure coding # isal-devel provides the prebuilt headers/lib; if absent the build # bootstraps from source below, but prefer the system package when available. BuildRequires: isa-l-devel # Miscellaneous devel BuildRequires: boost-devel BuildRequires: bison BuildRequires: cppunit-devel BuildRequires: flex BuildRequires: gmp-devel BuildRequires: libffi-devel BuildRequires: libevent-devel BuildRequires: libuuid-devel BuildRequires: libxml2-devel BuildRequires: libxslt-devel BuildRequires: libyaml-devel BuildRequires: mariadb-connector-c-devel BuildRequires: ncurses-devel BuildRequires: readline-devel BuildRequires: sqlite-devel BuildRequires: xfsprogs-devel # Python BuildRequires: python3 BuildRequires: python3-devel BuildRequires: python3-flake8 BuildRequires: python3-setuptools BuildRequires: python3-wheel # Perl helpers BuildRequires: perl-Digest-SHA BuildRequires: perl-FindBin # RPM / repo tooling BuildRequires: createrepo BuildRequires: net-tools BuildRequires: redhat-rpm-config BuildRequires: rpm-build %description This package is a COPR build wrapper to run Apache Bigtop's Gradle packaging entry point for Hadoop (hadoop-pkg) and export generated RPMs. It emulates the Bigtop Puppet toolchain bootstrap without root or container access, as required by Fedora COPR's rootless build environment. The spec follows the parameter convention from the Ambari Bigtop Stack build guide: ./gradlew hadoop-pkg -PparentDir=/usr/bigtop -PpkgSuffix -PbuildThreads=2C # ============================================================================= %prep # ============================================================================= %setup -q -n %{bigtop_archive_name} # --------------------------------------------------------------------------- # Patch do-component-build: exclude the YARN Application Catalog webapp from # both Maven invocations. # # hadoop-yarn-applications-catalog-webapp fails because its # frontend-maven-plugin pins Node.js 12.22.1, while the transitive npm # dependency color@5.0.3 requires Node>=18. It's an optional self-service UI # for deploying YARN Services, not required for core HDFS/YARN/MapReduce or # Ambari integration. # # The exclusion is baked directly into do-component-build's two "mvn" lines # (install, and site site:stage — javadoc aggregation in the latter also # walks into the catalog webapp module) via -pl !. We deliberately do # NOT use a global EXTRA_GOALS environment variable here: Gradle's exec # inherits the calling environment, and EXTRA_GOALS is a generic passthrough # consumed by many other components' do-component-build scripts (e.g. Hive). # A global export would leak this hadoop-specific exclusion into those # unrelated builds and break them with "Could not find the selected project # in the reactor". Keeping the patch local to hadoop's own do-component-build # avoids that entirely. # --------------------------------------------------------------------------- sed -i \ -e 's|install ${EXTRA_GOALS} "$@"|install -pl !%{catalog_webapp_path} ${EXTRA_GOALS} "$@"|' \ -e 's|site:stage ${MAVEN_OPTS} $@|site:stage ${MAVEN_OPTS} -pl !%{catalog_webapp_path} $@|' \ bigtop-packages/src/common/hadoop/do-component-build # Sanity check: fail fast in %prep if either pattern wasn't found (e.g. # upstream do-component-build changed) rather than silently building an # unpatched copy. Use fixed-string grep to avoid regex/anchor pitfalls # with the '$' characters in these substrings. grep -qF -- 'install -pl !%{catalog_webapp_path} ${EXTRA_GOALS} "$@"' \ bigtop-packages/src/common/hadoop/do-component-build grep -qF -- 'site:stage ${MAVEN_OPTS} -pl !%{catalog_webapp_path} $@' \ bigtop-packages/src/common/hadoop/do-component-build # ============================================================================= %build # ============================================================================= # --------------------------------------------------------------------------- # Locale / umask # --------------------------------------------------------------------------- export LANG=C export LC_ALL=C umask 022 # --------------------------------------------------------------------------- # Writable home directories for tools that expect $HOME to be writable # (Gradle, Maven, Ivy all write caches here; COPR's build user is not root) # --------------------------------------------------------------------------- export HOME=%{_builddir}/home mkdir -p "$HOME" \ %{_builddir}/.gradle \ %{_builddir}/.m2 \ %{_builddir}/.ivy2 # --------------------------------------------------------------------------- # Java selection # # Hadoop's *native* (JNI/C) components must be compiled with JDK 8. # The JVM bytecode target is controlled separately by the Gradle build scripts. # We pick the JDK 8 path explicitly; the readlink dance is intentionally # avoided to keep the logic transparent and easy to debug in COPR logs. # --------------------------------------------------------------------------- if [ -d /usr/lib/jvm/java-1.8.0-openjdk ]; then export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk elif [ -d /usr/lib/jvm/java-8-openjdk ]; then export JAVA_HOME=/usr/lib/jvm/java-8-openjdk elif [ -d /usr/lib/jvm/java-1.8.0 ]; then export JAVA_HOME=/usr/lib/jvm/java-1.8.0 else echo "ERROR: JDK 8 not found — required for Hadoop native build" >&2 exit 1 fi export PATH="$JAVA_HOME/bin:$PATH" export BIGTOP_JDK=8 export BIGTOP_PREFERRED_JAVA_VERSION=8 # --------------------------------------------------------------------------- # Toolchain bootstrap # # Puppet manifests (gradle.pp, groovy.pp, maven.pp, ant.pp, isal.pp) normally # run as root and install under /usr/local. COPR forbids root, so we install # everything under $TOOLCHAIN_HOME inside the build directory instead and # adjust PATH / environment variables accordingly. # --------------------------------------------------------------------------- export TOOLCHAIN_HOME=%{_builddir}/toolchain export MAVEN_HOME="$TOOLCHAIN_HOME/maven" export ANT_HOME="$TOOLCHAIN_HOME/ant" export GROOVY_HOME="$TOOLCHAIN_HOME/groovy" export PROTOBUF_HOME="$TOOLCHAIN_HOME/protobuf" export ISAL_HOME="$TOOLCHAIN_HOME/isal" mkdir -p "$TOOLCHAIN_HOME" # Versions — keep in sync with bigtop_toolchain/manifests/*.pp # NOTE: Gradle version is NOT here — gradlew owns that; see gradle-wrapper.properties. MAVEN_VERSION=3.9.11 ANT_VERSION=1.9.16 GROOVY_VERSION=4.0.24 PROTOBUF_VERSION=3.7.1 # -- Maven (maven.pp) ---------------------------------------------------------- if [ ! -x "$MAVEN_HOME/bin/mvn" ]; then echo "--- Bootstrapping Maven ${MAVEN_VERSION} ---" curl -fL "https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz" \ -o "%{_builddir}/apache-maven-${MAVEN_VERSION}-bin.tar.gz" mkdir -p "$MAVEN_HOME" tar -xzf "%{_builddir}/apache-maven-${MAVEN_VERSION}-bin.tar.gz" \ -C "$MAVEN_HOME" --strip-components=1 fi # -- Ant (ant.pp) -------------------------------------------------------------- if [ ! -x "$ANT_HOME/bin/ant" ]; then echo "--- Bootstrapping Ant ${ANT_VERSION} ---" curl -fL "https://archive.apache.org/dist/ant/binaries/apache-ant-${ANT_VERSION}-bin.tar.gz" \ -o "%{_builddir}/apache-ant-${ANT_VERSION}-bin.tar.gz" mkdir -p "$ANT_HOME" tar -xzf "%{_builddir}/apache-ant-${ANT_VERSION}-bin.tar.gz" \ -C "$ANT_HOME" --strip-components=1 fi # -- Groovy (groovy.pp) -------------------------------------------------------- if [ ! -x "$GROOVY_HOME/bin/groovy" ]; then echo "--- Bootstrapping Groovy ${GROOVY_VERSION} ---" curl -fL "https://archive.apache.org/dist/groovy/${GROOVY_VERSION}/distribution/apache-groovy-binary-${GROOVY_VERSION}.zip" \ -o "%{_builddir}/apache-groovy-binary-${GROOVY_VERSION}.zip" mkdir -p "$GROOVY_HOME" unzip -q "%{_builddir}/apache-groovy-binary-${GROOVY_VERSION}.zip" \ -d "%{_builddir}/groovy-tmp" mv "%{_builddir}/groovy-tmp"/groovy-"${GROOVY_VERSION}"/* "$GROOVY_HOME/" rm -rf "%{_builddir}/groovy-tmp" fi # -- Protobuf (protobuf.pp) ---------------------------------------------------- if [ ! -x "$PROTOBUF_HOME/bin/protoc" ]; then echo "--- Bootstrapping Protobuf ${PROTOBUF_VERSION} ---" curl -fL "https://github.com/protocolbuffers/protobuf/archive/refs/tags/v${PROTOBUF_VERSION}.tar.gz" \ -o "%{_builddir}/protobuf-${PROTOBUF_VERSION}.tar.gz" mkdir -p "%{_builddir}/protobuf-src" tar -xzf "%{_builddir}/protobuf-${PROTOBUF_VERSION}.tar.gz" \ -C "%{_builddir}/protobuf-src" --strip-components=1 pushd "%{_builddir}/protobuf-src" ./autogen.sh ./configure --prefix="$PROTOBUF_HOME" --disable-shared --with-pic make -j"$(nproc)" make install popd fi # -- ISA-L (isal.pp) ----------------------------------------------------------- # Required for Hadoop native erasure-coding support. # Use the system package (isa-l-devel, BuildRequires above) when available; # fall back to a source build only if the header is absent. if pkg-config --exists libisal 2>/dev/null || [ -f /usr/include/isa-l.h ]; then echo "--- ISA-L: using system package ---" export ISAL_HOME=/usr else echo "--- Bootstrapping ISA-L from source ---" ISAL_VERSION=2.31.0 curl -fL "https://github.com/intel/isa-l/archive/refs/tags/v${ISAL_VERSION}.tar.gz" \ -o "%{_builddir}/isa-l-${ISAL_VERSION}.tar.gz" mkdir -p "%{_builddir}/isal-src" tar -xzf "%{_builddir}/isa-l-${ISAL_VERSION}.tar.gz" \ -C "%{_builddir}/isal-src" --strip-components=1 pushd "%{_builddir}/isal-src" ./autogen.sh ./configure --prefix="$ISAL_HOME" make -j"$(nproc)" make install popd export PKG_CONFIG_PATH="$ISAL_HOME/lib/pkgconfig:${PKG_CONFIG_PATH:-}" fi # --------------------------------------------------------------------------- # PATH — expose all bootstrapped tools; keep /usr/bin for system tools # --------------------------------------------------------------------------- export PATH="%{_builddir}/bin:$GROOVY_HOME/bin:$MAVEN_HOME/bin:$ANT_HOME/bin:$PROTOBUF_HOME/bin:$PATH" # --------------------------------------------------------------------------- # Fake sudo — Bigtop's Gradle configuration phase calls sudo in a few places # even for non-root tasks. Redirect it to /usr/bin/true so those calls # silently succeed without actually elevating privileges. # --------------------------------------------------------------------------- mkdir -p %{_builddir}/bin ln -sf /usr/bin/true %{_builddir}/bin/sudo # --------------------------------------------------------------------------- # Cache directories # --------------------------------------------------------------------------- export GRADLE_USER_HOME=%{_builddir}/.gradle export MAVEN_OPTS="-Dmaven.repo.local=%{_builddir}/.m2 -Xmx2g -XX:ReservedCodeCacheSize=512m" export IVY_HOME=%{_builddir}/.ivy2 # Suppress the Bigtop build timestamp to keep package names deterministic. export BIGTOP_BUILD_STAMP=1 # NOTE: the YARN Application Catalog webapp exclusion is baked directly into # hadoop's do-component-build in %prep (search for "catalog-webapp" below). # We deliberately do NOT export EXTRA_GOALS here: it is a generic passthrough # variable consumed by many Bigtop do-component-build scripts, and Gradle's # exec inherits the calling environment — so a global export here would leak # our hadoop-specific "-pl !...catalog-webapp" exclusion into unrelated # component builds (e.g. hive-rpm), which then fail with "Could not find the # selected project in the reactor" because no such module exists there. # --------------------------------------------------------------------------- # Installation prefix # # -PparentDir matches the Ambari Bigtop Stack build guide convention. # /usr/bigtop does not exist in the COPR chroot, so we redirect to a writable # path under _builddir and copy the RPM outputs out afterwards. # --------------------------------------------------------------------------- export BIGTOP_OUTPUT=%{_builddir}/bigtop-output mkdir -p "$BIGTOP_OUTPUT" # --------------------------------------------------------------------------- # Sanity-check: print versions to the COPR log for easy debugging # --------------------------------------------------------------------------- echo "=== Build environment ===" java -version "$MAVEN_HOME/bin/mvn" --version "$ANT_HOME/bin/ant" -version "$GROOVY_HOME/bin/groovy" --version "$PROTOBUF_HOME/bin/protoc" --version # gradlew is the authoritative Gradle entry point; its version is set by # gradle/wrapper/gradle-wrapper.properties (currently 5.6.4). ./gradlew --version echo "=========================" # --------------------------------------------------------------------------- # Build # # Parameters follow the Ambari Bigtop Stack documentation: # -PparentDir installation prefix (Ambari convention: /usr/bigtop) # -PpkgSuffix appends the Bigtop version to package names # (e.g. hadoop_3_3_0) to satisfy Ambari service specs; # the flag is intentionally passed without a value — Bigtop # derives the suffix from its own version metadata # -PbuildThreads 2C = 2 × CPU cores # # allclean is omitted: COPR always starts from a clean chroot. # --------------------------------------------------------------------------- ./gradlew --no-daemon --stacktrace \ -Dbuildwithdeps=true \ -PparentDir=/usr/bigtop \ -PpkgSuffix \ -PbuildThreads=2C \ hadoop-pkg # --------------------------------------------------------------------------- # Export RPMs for COPR pickup # Bigtop writes output under output//; search broadly for safety. # --------------------------------------------------------------------------- find . -type f -name "*.rpm" -path "*/output/*" \ -exec cp -v -- {} %{_rpmdir}/ \; # ============================================================================= %changelog # ============================================================================= * Fri Jun 12 2026 Detlef Eppers - 3.4.0-6 - Stop exporting EXTRA_GOALS globally in %build: Gradle's exec inherits the calling environment, so the hadoop-only "-pl !...catalog-webapp" exclusion was leaking into other components' do-component-build scripts (observed breaking hive-rpm with "Could not find the selected project in the reactor"). Bake the -pl exclusion directly into hadoop's do-component-build for both mvn invocations instead — self-contained, no cross-component side effects. * Fri Jun 12 2026 Detlef Eppers - 3.4.0-5 - Patch do-component-build in %prep: append ${EXTRA_GOALS} to the "mvn site site:stage" call too. javadoc aggregation in that step still walks into hadoop-yarn-applications-catalog-webapp and re-triggers the same Node 12 / color@5.0.3 yarn install failure that -pl already fixed for the main "mvn ... install" invocation. * Fri Jun 12 2026 Detlef Eppers - 3.4.0-4 - Fix EXTRA_GOALS: drop literal single quotes around -pl argument. do-component-build expands EXTRA_GOALS unquoted as a plain bash string, so quote characters were passed to mvn literally and broke project lookup. * Fri Jun 12 2026 Detlef Eppers - 3.4.0-3 - Skip catalog webapp via EXTRA_GOALS: color@5.0.3 requires Node>=18 but frontend-maven-plugin pins Node 12.22.1; optional UI not needed for Ambari - Fix changelog day-of-week (Jun 12 2026 is Friday, not Thursday) * Fri Jun 12 2026 Detlef Eppers - 3.4.0-2 - Revert Source1/file:// Gradle pre-seed; all toolchain downloads are uniform at build time — consistent with Maven, Ant, Groovy, Protobuf, ISA-L * Fri Jun 12 2026 Detlef Eppers - 3.4.0-1 - Incorporate rootless toolchain bootstrap for COPR (no Puppet/sudo/root) - Add Groovy bootstrap (groovy.pp equivalent) required by Hadoop build scripts - Add ISA-L bootstrap (isal.pp equivalent) for Hadoop erasure-coding native libs - Switch JAVA_HOME to JDK 8 (required for Hadoop native/JNI compilation) - Remove BIGTOP_JDK=11 / BIGTOP_PREFERRED_JAVA_VERSION=11 (wrong for Hadoop) - Redirect -PparentDir to writable _builddir path; /usr/bigtop absent in chroot - Drop allclean task (redundant in COPR clean chroot; saves one Gradle config run) - Document -PpkgSuffix and -PbuildThreads per Ambari Bigtop Stack build guide - Add fake-sudo shim earlier in PATH to silence Gradle config-phase sudo calls