FOSSology Installation Documentation
Introduction
============
This document is designed to help you get FOSSology installed and ready
to use. It's intended audience is the system administrator who wants to
quickly get a local FOSSology instance up and running,or a distribution
developer looking to create packages.
For extended discussion on how to use or tune the software,please see
the User Documentation,available at
http://www.fossology.org/projects/fossology/wiki/User_Documentation.
This document is arranged as follows:
Section 1 - Short Installation Instructions
Section 2 - Full Installation Instructions
2.1 - Preparing your system
2.1.1 Disk space
2.1.2 Dependencies
2.1.3 Adjusting the Kernel
2.1.4 Preparing Postgresql
2.1.5 Configuring PHP
2.1.6 Configuring Apache
2.2 - Building and Installing FOSSology
2.2.1 Build and Install Instructions
2.2.2 Automated Post-Installation
2.2.3 Detailed Post-Install Instructions
2.2.4 Checking the Default Configuration Files
2.2.5 Start the Scheduler
Section 3 - After Installation
Section 1 - Short Installation Instructions
===========================================
(In examples,# is as root,$ is mortal)
1. If you have any old FOSSology installs you want to remove from the
system you can run the fo-cleanold script
# utils/fo-cleanold
If you have an existing install that you want to transition,be sure
back up your repository,database,and config files before proceeding.
2. install dependencies
# utils/fo-installdeps
3. Adjust the kernel shmmax (described in detail in section 2.1.3)
# echo 512000000 > /proc/sys/kernel/shmmax
# echo "kernel.shmmax=512000000" >> /etc/sysctl.conf
4. Adjust postgresql config (described in detail in section 2.1.4)
5. Adjust PHP config (described in detail in section 2.1.5)
6. Add apache config for FOSSology (described in detail in section 2.1.6)
There is an example config provided in install/fo-apache-example.conf.
After adding the config restart apache.
7. Build FOSSology
$ make
8. Install FOSSology
# make install
9. Run the postinstall script
# /usr/local/lib/fossology/fo-postinstall
10. Test that things are installed correctly
# /usr/local/etc/fossology/mods-enabled/scheduler/agent/fo_scheduler -t
11. Start the fossology scheduler daemon
# /etc/init.d/fossology start
You're done,point your web browser at the new install
(http://yourhostname/repo/) and start using FOSSology!
Section 2 - Full Installation Instructions
==========================================
The rest of this file is the much more verbose instructions. Consult
them if you have questions or need more explanation. If you are doing
a multi-system install please also see the INSTALL.multi document.
Section 2.1 - Preparing your system
***********************************
2.1.1 - Disk Space
------------------
FOSSology stores uploaded data in a filesystem repository. As you
upload and analyze packages via FOSSology,the repository can grow very
large. The default location for a single system repository is
/srv/fossology/repository/ however this can be adjusted by the system
administrator to another location if desired.
It is recommended that the area you choose to keep the repository in,be a separate mount point with at least 4x the size of the unpacked
data you intend to scan. For a small system intended to just scan a
few small personal projects this might mean gigabytes,but for systems
intended for scanning large collections of software including Linux
distributions,this probably means hundreds of gigabytes to terabytes.
If you are using multiple hosts to store the repository,it is best to
spread the repository data evenly across the hosts. See the User Guide
for more information about using multiple hosts.
2.1.2 - Dependencies
--------------------
FOSSology uses lots of different existing tools and software and
expects to find it on the system.
FOSSology depends on recent versions of the following software,dependencies needed when building fossology are:
Debian or Ubuntu or LinuxMint:
libxml2-dev libextractor-dev build-essential
postgresql-server-dev
libtext-template-perl subversion rpm
libpcre3-dev libmagic-dev librpm-dev libglib2.0-dev
Fedora:
"Development Tools" perl-Text-Template subversion
postgresql-devel file-devel libxml2 libextractor-devel
pcre-devel rpm-devel glib2-devel
RedHatEnterprise or CentOS:
postgresql-devel gcc make file libxml2 perl-Text-Template
subversion libextractor-devel
pcre-devel rpm-devel glib2-devel
dependencies needed at FOSSology runtime for the varIoUs distros are:
Debian or Ubuntu or LinuxMint:
apache2 libapache2-mod-PHP5 PHP5 PHP5-pgsql PHP-pear PHP5-cli
libxml2 libextractor1c2a binutils PHP-gettext
libextractor-plugins postgresql
bzip2 cabextract cpio genisoimage p7zip poppler-utils rpm
tar upx unzip gzip sleuthkit p7zip-full unrar-free libpcre3
libmagic1 libglib2.0-0 wget
RedHatEnterprise or CentOS:
postgresql-server httpd postgresql PHP PHP-pear PHP-pgsql
smtpdaemon file libxml2 binutils
bzip2 cpio mkisofs p8zip poppler-utils rpm tar file-libs
unzip gzip p7zip-plugins sleuthkit unrar pcre glib2 wget
Fedora:
postgresql-server httpd postgresql PHP PHP-pear PHP-pgsql
smtpdaemon libxml2 libextractor libextractor-plugins binutils
bzip2 cabextract cpio genisoimage p7zip p7zip-plugins file-libs
poppler-utils rpm tar upx unzip gzip sleuthkit unrar glib2 wget
Where we know them,commands are listed for installing on varIoUs
distributions.
* For Debian nearly all packages can be found in main,unless you want
the non-free version of unrar
* For Ubuntu you will need a universe apt source setup
* For RHEL you can find needed packages from EPEL
(http://fedoraproject.org/wiki/EPEL) or rpm.pbone.net.
We provide a script for helping install dependencies:
utils/fo-installdeps
To install just the build dependencies use the -b flag.
To install just the runtime dependencies use the -r flag.
To install just the agent runtime dependencies use the -a flag
(and see INSTALL.multi)
For more information run it with the --help flag.
If this utility doesn't support the distro you are running (or fails
to install everything that's needed) please report a bug.
The fo-installdeps command provides output that can be helpful in
figuring out what dependent packages you will have to install either
from your distro or some other location.
2.1.2.1 - Optional Email Notifications
---------------------------------------
Fossology can send email to users when their jobs finish. To enable this
feature sendmail and a mail transport agent (MTA) must be installed. The script
fo-installdeps does NOT install a MTA as there is no easy way for fossology to
determine which MTA your site uses. All mail transport agents(MTA) (like
postfix,exim,sendmail,etc.) provide a sendmail command and you probably
already have it on your system,but you may need to configure the MTA to be
able to send the mail where you want it to go.
2.1.3 Adjusting the Kernel
--------------------------
On modern large memory systems(>4gb),the linux kernel needs to be
adjusted to give postgresql more SysV shared memory.
To set on a running system:
# echo 512000000 > /proc/sys/kernel/shmmax
To make sure it gets set on boot
# echo "kernel.shmmax=512000000" >> /etc/sysctl.conf
This number is the number of pages (usually 4k each),and is based on a
fairly complicated formula,please see the postgresql tuning part of the
user guide.
2.1.4 Preparing Postgresql
--------------------------
Your postgresql install should be configured and running. If you need
help doing that consult the user documentation at
http://www.postgresql.org/docs/manuals/. If you are using SSL in
particular see the section Secure TCP/IP Connections with SSL to set it up.
Edit /etc/postgresql/<version>/main/postgresql.conf:
The tuning and preferences in the config file will vary depending on your
installation. We don't provide an automated way to do this because it
is complicated and specific to your particular install goals. A good
discussion of configuration settings can be found at
http://wiki.postgresql.org/wiki/Tuning_Your_Postgresql_Server. Here are a
few of the more common settings you should consider changing. The integer
values below are based on hypothetical system with 4GB of RAM.
#hba_file = 'ConfigDir/pg_hba.conf' # host-based authentication file
listen_addresses = '*' # listen to connections from all IPs
max_connections = 50 # maximum number of client connections
# allowed
shared_buffers = 1GB # If you have a system with 1GB or more
# of RAM,a reasonable starting value
# for shared_buffers is 1/4 of the memory
# in your system.
effective_cache_size = 2GB # how much memory is available for disk
# caching; 1/2-3/4 of memory
work_mem = 128MB # work_mem parameter allows Postgresql
# to do larger in-memory sorts
maintenance_work_mem = 200MB # roughly 50MB/GB of ram
fsync = on # turns forced synchronization on or off
full_page_writes = off # recover from partial page writes
log_line_prefix = '%t %h %c' # prepend a time stamp to all log entries
standard_conforming_strings = on
autovacuum = on # Enable autovacuum subprocess? 'on'
2.1.5 Configuring PHP
---------------------
Some PHP config variables may need to be adjusted for FOSSology. We don't
provide an automated way to do this because it can be specific to your
particular install if you are using PHP for other things. Edit your
PHP.ini file for apache (location dependent on your install,but
probably something like /etc/PHP5/apache2/PHP.ini) and make adjustments
that will work for your system and usage. Here are some things to
consider:
max_execution_time = 90
This controls how long,in seconds,a PHP process is allowed to run.
For "one shot" license analysis,particularly large jobs,or if your
system is slow you may need to increase this (the default is 30).
memory_limit = 702M
post_max_size = 701M
upload_max_filesize = 700M
These control the size of file you will be able to upload via the UI,"Upload from File". For very large uploads (for example DVD images) we
recommend using the command line upload method or "Upload from URL",but you might want to increase these to handle up to CDROM sized
uploads. (the defaults are 128M,8M,and 2M)
2.1.6 Configuring Apache
------------------------
1) You need to add something like the following to the apache config,and this will depend on
A) How you have apache configured. You might be creating a new site
config using apache's "sites-available"/a2ensite(8) mechanism or
editing and existing config you have setup. For example,on a
Debian apache2 install you would have site config files in
/etc/apache2/sites-available/ and you might be editing the default
one or creating a new one.
B) The path you want the FOSSology UI to appear on the server,this
example uses "/repo/"
C) Where your FOSSology is installed,this example assumes
/usr/local/share/fossology/www/ui
D) What other things you might be using apache for on the system.
for these reasons we can't provide an automated way of doing this.
========================================================================
# FOSSology example apache config
Alias /repo/ /usr/local/share/fossology/www/ui
<Directory "/usr/local/share/fossology/www/ui">
AllowOverride None
Options FollowSymLinks MultiViews
Order allow,deny
Allow from all
# uncomment to turn on PHP error reporting
#PHP_flag display_errors on
#PHP_value error_reporting 2039
</Directory>
========================================================================
NOTE: included in the above example are some commented lines used for
enabling PHP error reporting. If you are having problems you might
choose to enable these to help determine the problem. Normally you
probably want them turned off so they don't report confusing error
messages to your end users or reveal information about your system
configuration to potential attackers.
2) Because this software dynamically generates web pages based on the
database,you may want to tell web robots not to index pages. You
can do this with a robots.txt file in your DocumentRoot. Here is
a sample that tells all agents to ignore your /repo urls:
========================================================================
User-agent: *
Disallow: /repo
========================================================================
Once you have installed the configuration you can test it by running
(as root):
apache2ctl configtest
and if it tests ok,then you can restart the server with the new config
by running (as root):
apache2ctl graceful
Note: the site won't work yet until we install FOSSology below.
Your system should now be ready for installing FOSSology!
Section 2.2 - Building and Installing FOSSology
***********************************************
In the following steps,you will need to use files included in the
fossology source. If you haven't done so already,untar the
fossology-<version>.tar.gz tarball into a staging directory.
NOTE: FOSSology uses a system of variables to control where the build
will install things. The defaults are all set for a standard local
sysadmin install,but if you prefer other locations or are building
distribution or 3rd party packages you can adjust to meet your needs.
See the HACKING file for more info.
NOTE: some commands are required to be run as the root user and are
indicated so. You should be able to use sudo,"su -",a normal root
login,or your favorite root-obtaining utility.
2.2.1 Build and Install Instructions
--------------------------------------
1) If you have any old fossology installs you want to remove from the
system you can run the fo-cleanold script
# utils/fo-cleanold
If you have an existing install that you want to transition,and config files before proceeding.
For more information on what this utility can do use the --help flag.
2) cd into the directory created when you unpacked the tarball
(fossology-<version>/).
3) To build the software run "make".
If you are using an existing SVN tree that you have updated,you
might want to run a "make clean" first to get to a known good state.
The Makefiles are pretty good about declaring dependencies that
will make sure things get rebuilt,but to be sure use "make clean".
4) To install the software run "make install". You will need the proper
permissions to install to where the files will be install,which in
most cases means this needs to be run by root.
2.2.2 Automated Post-Installation
-----------------------------------
As part of the install a script was placed in
/usr/local/lib/fossology/fo-postinstall
(by default,or similar if you configured a different install location).
Running this script to help finish the install is highly recommended,as
it performs tasks that go beyond merely installing files on the filesystem.
(For example,it will add the user "fossy" and group "fossy". See next section
for more details.) The script must be run as root. By default the script
does everything you will need for a single machine install. If you are
doing a multiple machine install,please see the INSTALL.multi document.
Run the postinstall script
# /usr/local/lib/fossology/fo-postinstall
2.2.3 Detailed Post-Install Instructions
------------------------------------------
It is highly recommended that you run the fo-postinstall script. However,you may want to know the details of what the script is doing. This may be out
of academic curIoUsity,or more likely because you may have to troubleshoot
something gone wrong.
If you are confident that fo-postinstall has done all the right things,you may skip to step 2.2.4. Otherwise,continue reading...
1) Setting up Users and Groups
You are expected to already have a "postgres" user as part of the
system postgresql install,and a "www-data" user as part of the
apache2 install. Note that some versions of Apache (like the one
shipped with RHEL) use the user apache,not www-data. Whichever user
your Apache uses,make sure that the user (www-data or apache) gets
added to the "fossy" group.
FOSSology requires a system user and a system group both named
"fossy".
The /etc/passwd entries for these user should look
something like (Note: your uid & gid values may be different):
fossy:x:1001:1001:FOSSology:/srv/fossology:/bin/bash
and the /etc/group entry
fossy:x:1001:fossy
On a system with the useradd and groupadd commands (all LSB systems
including Debian,Fedora,etc) you can create the above system user
and group with the following commands as root:
groupadd fossy
useradd -c FOSSology -d /srv/fossology -g fossy -s /bin/false fossy
Alternatively,you can use the adduser command:
adduser --gecos "FOSSology" --home /srv/fossology --system --group fossy
2) Setting up the database
A sample database schema file can be found either on the installed system
at /usr/local/lib/fossology/fossologyinit.sql or in the tarball at
fossology-<version>/install/db/fossologyinit.sql. This will create the
database when it is read in. It will also create an owner for the
database "fossy" and owner's password "fossy". For security reasons,please change this password,after the database is created,using
the alter command. Here are the steps:
Check to see the database server is running
A) Ensure that postgresql is running using this command:
/etc/init.d/postgresql-8.4 status
Version Cluster Port Status Owner Data directory Log file
8.4 main 5432 online postgres /var/lib/postgresql/8.4/main \
/var/log/postgresql/postgresql-8.4-main.log
If the status reported is "online",you can proceed to the next step.
If the status is "down",use this command to start postgres:
/etc/init.d/postgresql-8.4 start
You should see the following message:
Starting Postgresql 8.4 database server: main.
B) The default database and owner is created by reading in the
fossologyinit.sql file of sql statements to define the database schema.
You must be logged in as user postgres to create the database schema
and define the owner. As root,you can su to user postgres (su
postgres) and then run the psql command:
psql < fossologyinit.sql
C) If any steps fail,check the postgres log file for errors:
/var/log/postgresql/postgresql-8.4-main.log
D) Make sure /etc/postgresql/8.4/main/pg_hba.conf is configured
correctly to allow your connection. This file controls: which hosts
are allowed to connect,how clients are authenticated,which
Postgresql user names they can use,which databases they can access.
As a starting point,you will need something like the following for
local connections:
# local DATABASE USER METHOD [OPTION]
local all all md5
See http://www.postgresql.org/docs/current/static/client-authentication.html
for detailed information.
If you do need to edit it then restart postgresql so the changes take
effect:
/etc/init.d/postgresql-<version> restart
E) Once the database is defined,verify connection with
psql -d fossology -U fossy
use the default password "fossy". You should connect and see the
following:
Welcome to psql 8.4.11,the Postgresql interactive terminal.
Type: \copyright for distribution terms
\h for help with sql commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit
fossology=>
If so then you successfully connected. Type \q to quit.
F) If any steps fail,check the postgres log file for errors:
/var/log/postgresql/postgresql-8.4-main.log
2.2.4 Checking the Default Configuration Files
------------------------------------------------
On a fresh install you start with default versions of configuration
files that contain reasonable defaults (where possible),or some files
are generated via the post install script. You need to review and edit
where needed.
A) /usr/local/etc/fossology/fossology.conf
This file contains FOSSology configuration information that is not
agent specific. Application variables and settings prevIoUsly defined
in separate config files (Depth.conf,RepPath.conf,Proxy.conf,Hosts.conf) are now combined into one simple key-value configuration
file. Several project pathname variables are also defined in
fossology.conf. For a detailed description of the new scheduler and
config file,please see
http://www.fossology.org/projects/fossology/wiki/Sched-20_
B) Individual module (agent) configuration files:
With the introduction of FOSSology 2.0 and the new scheduler,each
agent will come with its own configuration file.
/usr/local/etc/fossology/mods-enabled/adj2nest/adj2nest.conf
/usr/local/etc/fossology/mods-enabled/buckets/buckets.conf
/usr/local/etc/fossology/mods-enabled/copyright/copyright.conf
/usr/local/etc/fossology/mods-enabled/delagent/delagent.conf
/usr/local/etc/fossology/mods-enabled/mimetype/mimetype.conf
/usr/local/etc/fossology/mods-enabled/nomos/nomos.conf
/usr/local/etc/fossology/mods-enabled/pkgagent/pkgagent.conf
/usr/local/etc/fossology/mods-enabled/ununpack/ununpack.conf
/usr/local/etc/fossology/mods-enabled/wget_agent/wget_agent.conf
Each configuration file provides the following information to the
scheduler:
* command: the command line that will be used when starting the agent
* max: the maximum number of instances of this agent running
concurrently,-1 for no max.
* special: list of anything special about the agent.
See the "Agent.conf's" section at
http://www.fossology.org/projects/fossology/wiki/Sched-20_ for more
info.
2.2.5 Start the Scheduler
---------------------------
As root,start the scheduler with:
/etc/init.d/fossology start
The scheduler logs to /var/log/fossology/fossology.log by default,you
can look there for scheduler activity or errors.
Section 3 - After Installation
==============================
Congratulations,FOSSology is now installed!
You can now point your web browser at the new install
(http://yourhostname/repo/) and start using FOSSology!
You should login to FOSSology with the default fossy/fossy user and
password and:
1) create yourself a user with administrative privileges
2) change the default password for user fossy
3) customize (if desired) your FOSSology website using the directions at
http://www.fossology.org/projects/fossology/wiki/Config-How_to_Configure_the_FOSSology_site
For extended discussion on how to use or tune the software,available on the website at
http://fossology.org/user_documentation
