Don't forget to feed the APIs!

Install Splunk in CentOS 7

$ sudo rpm -Uvh splunk-6.6.2-4b804538c686.x86_64

Start Splunk

$ cd /opt/splunk/bin
$ sudo ./splunk start --accept

Browse to the Splunk web interface at http://127.0.0.1:8000 with the default credentials of

username admin
password changeme

Enable splunk to start up at boot

$ sudo su
$ ./splunk enable boot-start -user pentestlabs

To install a Splunk Add-On:

1. Download the add-on
2. Browse to the Splunk web interface and click on the gear icon next to the Apps menu on the left.
3. Click on Install app from file button
4. Navigate to the add-on and click on Upload
5. Restart Splunk

To check if Splunk is running:

$ sudo service splunk status
$ netstat -an | grep 8000

Side note to myself!!! I had to modify the firewall and chmod 755 on the /opt/splunk folder -

# Web
sudo firewall-cmd --zone=public --add-port=8000/tcp --permanent
# Management / REST API
sudo firewall-cmd --zone=public --add-port=8089/tcp --permanent
# KV store
sudo firewall-cmd --zone=public --add-port=8191/tcp --permanent
# Data
sudo firewall-cmd --zone=public --add-port=9997/tcp --permanent
# Replication
sudo firewall-cmd --zone=public --add-port=8080/tcp --permanent
# Restart firewall 
sudo firewall-cmd --reload

$ sudo chmod -R 755 /opt/splunk
$ cd /opt/splunk/bin
$ sudo ./splunk restart
$ sudo su
$ ./splunk enable boot-start -user pentestlabs 
# I had to run the command above because I had uninstalled and reinstalled Splunk as a troubleshooting step.

# Init script installed at /etc/init.d/splunk.
# Init script is configured to run at boot.

But Splunk would still fail to load after a restart so I YOLO'd it and changed the folder permissions.

$ sudo ./splunk restart

$splunkd is not running [FAILED]
$ERROR - Failed opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied

$ ls -l /opt/splunk

drwxr-xr-x.  4 splunk splunk    4096 Jul 26 09:02 bin
-rwxr-xr-x.  1 splunk splunk      57 Jun 21 15:52 copyright.txt
drwxr-xr-x. 16 splunk splunk    4096 Jul 26 09:19 etc
drwxr-xr-x.  3 splunk splunk      44 Jul 26 09:02 include
drwxr-xr-x.  6 splunk splunk    4096 Jul 26 09:02 lib
-rwxr-xr-x.  1 splunk splunk   66575 Jun 21 15:52 license-eula.txt
drwxr-xr-x.  3 splunk splunk      58 Jul 26 09:02 openssl
-rwxr-xr-x.  1 splunk splunk     841 Jun 21 15:55 README-splunk.txt
drwxr-xr-x.  3 splunk splunk      86 Jul 26 09:02 share
-rwxr-xr-x.  1 splunk splunk 2228103 Jun 21 16:57 splunk-6.6.2-4b804538c686-linux-2.6-x86_64-manifest
drwxr-xr-x.  6 root   root        52 Jul 26 09:03 var

$ sudo chmod -R 777 /opt/splunk
$ sudo ./splunk restart

Starting splunk server daemon (splunkd)...  
Done
                                                           [  OK  ]

Waiting for web server at http://127.0.0.1:8000 to be available... Done

After successfully restarting Splunk I received 4 error messages

1. Failed to start KV Store process. See mongod.log and splunkd.log for details.
2. KV Store changed status to failed. KVStore process terminated
3. KV Store process terminated abnormally (exit code 1, status exited with code 1). See mongod.log and splunkd.log for details.
4. Unable to initialize modular input 'api' defined inside the app 'Splunk_TA': Introspecting scheme=api: script running failed (exited with code 1).

Steps I took to troubleshoot this

$ sudo chmod -R 400 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key
$ sudo chown -R splunk:splunk /opt/splunk/var/lib/splunk/kvstore/mongo/
$ ls -l /opt/splunk/var/lib/splunk/kvstore/mongo/

The issue with the KVStore is still not resolved.