Running from source

The following steps will help you get Puppet Server up and running from source.

# clone git repository and initialize submodules
$ git clone --recursive git://github.com/puppetlabs/puppetserver
$ cd puppetserver

# Remove any old config if you want to make sure you're using the latest
# defaults
$ rm -rf ~/.puppetserver

# Run the `dev-setup` script to initialize all required configuration
$ ./dev-setup

# Launch the clojure REPL
$ lein repl
# Run Puppet Server
dev-tools=> (go)
dev-tools=> (help)

You should now have a running server. All relevant paths ($confdir, $codedir, etc.) are configured by default to point to directories underneath ~/.puppetlabs. These should all align with the default values that puppet uses (for non-root users).

You can find the specific paths in the dev/puppetserver.conf file.

In another shell, you can run the agent:

# Go to the directory where you checked out puppetserver
$ cd puppetserver
# Set ruby and bin paths
$ export RUBYLIB=./ruby/puppet/lib:./ruby/facter/lib
$ export PATH=./ruby/puppet/bin:./ruby/facter/bin:$PATH
# Run the agent
$ puppet agent -t

To make an API request to the running server, run a command like this:

$ curl \
    --cert ~/.puppetlabs/etc/puppet/ssl/certs/localhost.pem \
    --key ~/.puppetlabs/etc/puppet/ssl/private_keys/localhost.pem \
    --cacert ~/.puppetlabs/etc/puppet/ssl/ca/ca_crt.pem \
    https://localhost:8140/puppet/v3/file_content/modules/mymodule/script.sh?environment=production

More detailed instructions follow.

Use your system's package tools to ensure that the following prerequisites are installed:

• JDK 1.8 or Java 11

• Leiningen 2.9.1 (latest)

• Git (for checking out the source code)

$ git clone --recursive git://github.com/puppetlabs/puppetserver
$ cd puppetserver

The easiest way to do this is to just run:

$ ./dev-setup

This will set up all of the necessary configuration files and directories inside of your ~/.puppetlabs directory. If you are interested in seeing what all of the default file paths are, you can find them in ./dev/puppetserver.conf.

The default paths should all align with the default values that are used by puppet (for non-root users).

If you'd like to customize your environment, here are a few things you can do:

• Before running ./dev-setup, set an environment variable called MASTERHOST. If this variable is found during dev-setup, it will configure your puppet.conf file to use this value for your certname (both for Puppet Server and for puppet) and for the server configuration (so that your agent runs will automatically use this hostname as their Puppet primary server).

• Create a file called dev/user.clj. This file will be automatically loaded when you run Puppet Server from the REPL. In it, you can define a function called get-config, and use it to override the default values of various settings from dev/puppetserver.conf. For an example of what this file should look like, see ./dev/user.clj.sample.

You don't need to create a user.clj in most cases; the settings that I change the most frequently that would warrant the creation of this file, though, are:

• jruby-puppet.max-active-instances: the number of JRuby instances to put into the pool. This can usually be set to 1 for dev purposes, unless you're working on something that involves concurrency.

• jruby-puppet.splay-instance-flush: Do not attempt to splay JRuby flushing, set when testing if using multiple JRuby instances and you need to control when they are flushed from the pool

• jruby-puppet.master-conf-dir: the puppet primary server confdir (where puppet.conf, modules, manifests, etc. should be located).

• jruby-puppet.master-code-dir: the puppet primary server codedir

• jruby-puppet.master-var-dir: the puppet primary server vardir

• jruby-puppet.master-run-dir: the puppet primary server rundir

• jruby-puppet.master-log-dir: the puppet primary server logdir

The preferred way of running the server for development purposes is to run it from inside the clojure REPL. The git repo includes some files in the /dev directory that are intended to make this process easier.

When running a clojure REPL via the lein repl command-line command, lein will load the dev/dev-tools.clj namespace by default.

Running the server inside of the clojure REPL allows you to make changes to the source code and reload the server without having to restart the entire JVM. It can be much faster than running from the command line, when you are doing iterative development. We are also starting to build up a library of utility functions that can be used to inspect and modify the state of the running server; see dev/dev-tools.clj for more info.

(NOTE: many of the developers of this project are using a more full-featured IDE called Cursive Clojure, built on the IntelliJ IDEA platform, for our daily development. It contains an integrated REPL that can be used in place of the lein repl command-line command, and works great with all of the functions described in this document.)

To start the server from the REPL, run the following:

$ lein repl
nREPL server started on port 47631 on host 127.0.0.1
dev-tools=> (go)
dev-tools=> (help)

Then, if you make changes to the source code, all you need to do in order to restart the server with the latest changes is:

dev-tools=> (reset)

Restarting the server this way should be significantly faster than restarting the entire JVM process.

You can also run the utility functions to inspect the state of the server, e.g.:

dev-tools=> (print-puppet-environment-states)

Have a look at dev-tools.clj if you're interested in seeing what other utility functions are available.

If you prefer not to run the server interactively in the REPL, you can launch it as a normal process. To start the Puppet Server when running from source, simply run the following:

$ lein run -c /path/to/puppetserver.conf

If you get an error like the following:

Execution error (LoadError) at org.jruby.RubyKernel/require
(org/jruby/RubyKernel.java:970).
(LoadError) no such file to load -- puppet

Then you've probably forgotten to fetch the git submodules.

If you change the :webserver :ssl-port config option from the default value of 8140, tests will fail with errors like the following:

lein test :only puppetlabs.general-puppet.general-puppet-int-test/test-external-command-execution

ERROR in (test-external-command-execution) (SocketChannelImpl.java:-2)
Uncaught exception, not in assertion.
expected: nil
2019-02-06 14:58:50,541 WARN  [async-dispatch-18] [o.e.j.s.h.ContextHandler] Empty contextPath
  actual: java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.checkConnect (SocketChannelImpl.java:-2)
    sun.nio.ch.SocketChannelImpl.finishConnect (SocketChannelImpl.java:717)
    org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent (DefaultConnectingIOReactor.java:171)
    org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents (DefaultConnectingIOReactor.java:145)
    org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute (AbstractMultiworkerIOReactor.java:348)
    org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute (PoolingNHttpClientConnectionManager.java:192)
    org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run (CloseableHttpAsyncClientBase.java:64)
    java.lang.Thread.run (Thread.java:844)

Changing the ssl-port variable back to 8140 makes the tests run properly.

Use a command like the one below to run an agent against your running puppetserver:

    puppet agent --confdir ~/.puppetlabs/etc/puppet \
                 --debug -t

Note that a system installed Puppet Agent is ok for use with source-based PuppetDB and Puppet Server. The --confdir above specifies the same confdir that Puppet Server is using. Because the Puppet Agent and Puppet Server instances are both using the same confdir, they're both using the same certificates as well. This alleviates the need to sign certificates as a separate step.

You can easily run a Puppet Agent inside a Docker container, either by using the host network profile or by accessing the Puppetserver service using the Docker host IP:

docker run -ti                    \
       --name agent1              \
       puppet/puppet-agent-ubuntu \
       agent -t --server 172.17.0.1

docker run -ti                     \
       --name agent2               \
       --network host              \
       --add-host puppet:127.0.0.1 \
       puppet/puppet-agent-ubuntu

To start another Puppet Agent run in a previous container you can use the docker start command:

docker start -a agent1

• lein test to run the clojure test suite

• rake spec to run the jruby test suite

The Clojure test suite can consume a lot of transient memory. Using a larger JVM heap size when running tests can significantly improve test run time. The default heap size is somewhat conservative: 1 GB for the minimum heap (much lower than that as a maximum can lead to Java OutOfMemory errors during the test run) and 2 GB for the maximum heap. While the heap size can be configured via the -Xms and -Xmx arguments for the :jvm-opts defproject key within the project.clj file, it can also be customized for an individual user environment via either of the following methods:

1. An environment variable named PUPPETSERVER_HEAP_SIZE. For example, to use a heap size of 5 GiB for a lein test run, you could run the following:

   $ PUPPETSERVER_HEAP_SIZE=6G lein test

2. A lein profiles.clj setting in the :user profile under the :puppetserver-heap-size key. For example, to use a heap size of 5 GiB, you could add the following key to your ~/.lein/profiles.clj file:

{:user {:puppetserver-heap-size "5G"
        ...}}

With the :puppetserver-heap-size key defined in the profiles.clj file, any subsequent lein test run would utilize the associated value for the key. If both the environment variable and the profiles.clj key are defined, the value from the environment variable takes precedence. When either of these settings is defined, the value is used as both the minimum and maximum JVM heap size.

The gems that are vendored with the puppetserver OS packages will be automatically installed into your dev environment by the ./dev-setup script. If you wish to install additional gems, please see the Gems document for detailed information.

For more information about debugging both Clojure and JRuby code, please see Puppet Server: Debugging documentation.

To run a source PuppetDB with Puppet Server, Puppet Server needs standard PuppetDB configuration and how to find the PuppetDB terminus. First copy the dev/puppetserver.conf file to another directory. In your copy of the config, append a new entry to the ruby-load-path list: <PDB source path>/puppet/lib. This tells PuppetServer to load the PuppetDB terminus from the specified directory.

From here, the instructions are similar to installing PuppetDB manually via packages. The PuppetServer instance needs configuration for connecting to PuppetDB. Instructions on this configuration are below, but the official docs for this can be found here.

Update ~/.puppetlabs/etc/puppet/puppet.conf to include:

[server]
storeconfigs = true
storeconfigs_backend = puppetdb
reports = store,puppetdb

Create a new puppetdb config file ~/.puppetlabs/etc/puppet/puppetdb.conf that contains

[main]
server_urls = https://<SERVERHOST>:8081

Then create a new routes file at ~/.puppetlabs/etc/puppet/routes.yaml that contains

---
server:
  facts:
    terminus: puppetdb
    cache: yaml

Assuming you have a PuppetDB instance up and running, start your Puppet Server instance with the new puppetserver.conf file that you changed:

lein run -c ~/<YOUR CONFIG DIR>/puppetserver.conf

Depending on your PuppetDB configuration, you might need to change some SSL config. PuppetDB requires that the same CA that signs it's certificate, also has signed Puppet Server's certificate. The easiest way to do this is to point PuppetDB at the same configuration directory that Puppet Server and Puppet Agent are pointing to. Typically this setting is specified in the jetty.ini file in the PuppetDB conf.d directory. The update would look like:

[jetty]

#...
ssl-cert = <home dir>/.puppetlabs/etc/puppet/ssl/certs/<SERVERHOST>.pem
ssl-key = <home dir>/.puppetlabs/etc/puppet/ssl/private_keys/<SERVERHOST>.pem
ssl-ca-cert = <home dir>/.puppetlabs/etc/puppet/ssl/certs/ca.pem

After the SSL config is in place, start (or restart) PuppetDB:

lein run services -c <path to PDB config>/conf.d

Then run the Puppet Agent and you should see activity in PuppetDB and Puppet Server.