This guide has been developed and tested on Ubuntu Server 12.04 LTS. Since Linux distrutions manage software packages and file placement in a variety of ways, parts of the guide may need modification to suit other distributions.

 Skip to:

The server

Assume the host has a IP address and registered domain as follows:

Domain: new-ark.org.au
IP address: 115.146.80.50

Create aliases for ark-database, ark-ldap and ark-smtp in /etc/hosts:

127.0.0.1       localhost
115.146.80.50   new-ark.org.au   new-ark   ark-database   ark-ldap   ark-smtp

Server firewall

Use Ubuntu's "Uncomplicated firewall" tool ufw to set basic rules:

 Allow ssh, http, and https connections from anywhere:

sudo ufw allow ssh
sudo ufw allow www
sudo ufw allow https

Allow external MySQL connections only from a work laptop with IP address 128.250.94.2:

sudo ufw allow proto tcp from 128.250.94.2 to any port 3306

Set the default rules to deny incoming connections and allow outgoing connections:

sudo ufw default deny incoming
sudo ufw default allow outgoing

Enable the firewall and check its status:

sudo ufw enable
sudo ufw status verbose

SMTP Mail

Enable "universe" repositories in /etc/apt/sources.list

Install the mail software:

sudo apt-get install exim4-daemon-light mailutils

Configure the mail server:

sudo dpkg-reconfigure exim4-config

  1. Choose the option for "internet site" and select "Ok" to continue.
  2. Enter the fully qualified domain name new-ark.org.au in the "mail name" configuration screen.
  3. Enter "127.0.0.1" when asked which IP address to listen on for SMTP connections.
  4. Enter new-ark.org.au when asked which destinations mail should be accepted for.
  5. Leave the relay domains and relay machines fields blank.
  6. Select "No" when asked whether to keep DNS queries to a minimum.
  7. Select "Maildir" when asked about the delivery method used for incoming mail.
  8. Accept the default "non-split" option for your mail configuration file.        

Test the SMTP server by mailing to a valid external address:

echo "This is a test." | mail -s Testing This email address is being protected from spambots. You need JavaScript enabled to view it.

MySQL database

Install:

sudo apt-get install mysql-client mysql-server

Root user:

Username: 'root'@'localhost'
Password: myDbRootPass


Stop the MySQL server and edit /etc/mysql/my.cnf so that the service binds to the host's IP address:

bind-address = 115.146.80.50

Set "max_allowed_packet = 256M" and "key_buffer = 32M", and also set these configuration parameters under [mysqld]:

lower_case_table_names = 1
skip-host-cache
skip-name-resolve
innodb = FORCE
innodb_file_per_table
innodb_buffer_pool_size=2G
innodb_additional_mem_pool_size=256M
innodb_log_file_size=256M
innodb_log_buffer_size=64M

Users of MySQL 5.7 (e.g. Ubuntu 16.04LTS) should also include this line for compatibility with The Ark:

sql-mode=""

See this link for information on resizing log files http://www.dbasquare.com/2012/04/04/how-to-resize-innodb-logs/

To setup i8n at the database server level, add the following entries to /etc/mysql/my.cnf under the appropriate section headings (based on http://stackoverflow.com/questions/3513773/change-mysql-default-character-set-to-utf-8-in-my-cnf):

[client]
default-character-set=utf8

[mysql]
default-character-set=utf8

[mysqld]
collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8

Create an Ark database user for localhost:

CREATE USER 'arkadmin'@'localhost' IDENTIFIED BY 'arkadminPasswordHere';
GRANT ALL PRIVILEGES ON *.* TO 'arkadmin'@'localhost';

Create an Ark database user for the host's IP address

CREATE USER 'arkadmin'@'115.146.80.50' IDENTIFIED BY 'arkadminPasswordHere';
GRANT ALL PRIVILEGES ON *.* TO 'arkadmin'@'115.146.80.50';

Create an Ark database user so that the database can be access from a work computer with IP address 128.250.94.2. This is useful for debugging and examining database contents directly (via MySQL Workbench).

CREATE USER 'arkadmin'@'128.250.94.2' IDENTIFIED BY 'arkadminPasswordHere';
GRANT ALL PRIVILEGES ON *.* TO 'arkadmin'@'128.250.94.2';

Import the Ark's base schemas and data from the ark-db repository; databases need not be created manually - they are created by the script.

mysql -h localhost -u root -p < runThisSqlToCreateANew111eDatabase.sql

Applying i8n patches

The following instructions will allow The Ark to store non-Latin characters in textual data fields. The instructions assume that the following scripts have been checked out of the SVN branch for the Ark version being installed:

  • changeEverySchemaSlashDatabaseToUTF8CharSetAndCollation.sql
  • generate-changeCharsetCollationOfAllTables.sql
  • generate-changeCharsetCollationOfAllFields.sql

Set charset/collation for Ark databases, then login and check the result:   

mysql -h localhost -u root -p < changeEverySchemaSlashDatabaseToUTF8CharSetAndCollation.sql
SELECT * FROM information_schema.SCHEMATA;

Login to the database via MySQL Workbench, run generate-changeCharsetCollationOfAllTables.sql, export the results as a tab-delimited file changeCharsetCollationOfAllTables-new-Ark.sql, then remove the enclosing quotes:

cat changeCharsetCollationOfAllTables-new-Ark.sql | tr -d '\"' > changeCharsetCollationOfAllTables.sql

Set charset/collation for tables in Ark databases:

mysql -h localhost -u root -p < changeCharsetCollationOfAllTables.sql

Within MySQL Workbench, run generate-changeCharsetCollationOfAllFields.sql, export the results as a tab-delimited file changeCharsetCollationOfAllFields-new-Ark.sql, then remove the enclosing quotes:

cat changeCharsetCollationOfAllFields-new-ARK.sql | tr -d '\"' > changeCharsetCollationOfAllFields.sql

Note: in Ark version 1.1.1e, the VARCHAR field study.consent_section.NAME is too large to be a key with UTF-8 encoding. Therefore, login to the database and issue:

ALTER TABLE `study`.`consent_section` MODIFY `NAME` VARCHAR(255);

 ...then edit line 284 of changeCharsetCollationOfAllFields.sql to match VARCHAR(255)

Set charset/collation for fields within Ark tables:

mysql -h localhost -u root -p < changeCharsetCollationOfAllFields.sql

Ensure that the Ark .war is built with an applicationContext database entry:

LDAP

Install:

sudo apt-get install slapd ldap-utils

Install Apache Directory Studio:

   http://directory.apache.org/studio/downloads.html

Stop the LDAP service:

sudo service slapd stop

Generate a password for the config file:

slappasswd
myLdapPassword

Copy the generated hash into a temporary file slapd-password.txt

By default, the slapd service stores its configuration in internal structures; we wish to use a slapd.conf file instead. As root, edit /etc/default/slapd and set the variable SLAPD_CONF=/etc/ldap/slapd.conf

Create the file /etc/ldap/slapd.conf with the following contents, remembering to set the rootpw to the hash stored in slapd-password.txt.

#
# See slapd.conf(5) for details on configuration options.
# This file should NOT be world readable.
#
include        /etc/ldap/schema/core.schema
include         /etc/ldap/schema/cosine.schema
include         /etc/ldap/schema/inetorgperson.schema
include         /etc/ldap/schema/nis.schema

# Define global ACLs to disable default read access.

# Do not enable referrals until AFTER you have a working directory
# service AND an understanding of referrals.
#referral    ldap://root.openldap.org

pidfile        /var/run/slapd/slapd.pid
argsfile    /var/run/slapd/slapd.args

# Allow bind
allow bind_v2


# Load dynamic backend modules:
modulepath    /usr/lib/ldap
moduleload    back_bdb.la
moduleload    syncprov.la
# moduleload    back_hdb.la
# moduleload    back_ldap.la

# Sample security restrictions
#    Require integrity protection (prevent hijacking)
#    Require 112-bit (3DES or better) encryption for updates
#    Require 63-bit encryption for simple bind
# security ssf=1 update_ssf=112 simple_bind=64

# Sample access control policy:
#    Root DSE: allow anyone to read it
#    Subschema (sub)entry DSE: allow anyone to read it
#    Other DSEs:
#        Allow self write access
#        Allow authenticated users read access
#        Allow anonymous users to authenticate
#    Directives needed to implement policy:
# access to dn.base="" by * read
# access to dn.base="cn=Subschema" by * read
# access to *
#    by self write
#    by users read
#    by anonymous auth
#
# if no access controls are present, the default policy
# allows anyone and everyone to read anything but restricts
# updates to rootdn.  (e.g., "access to * by * read")
#
# rootdn can always read and write EVERYTHING!

#######################################################################
# BDB database definitions
#######################################################################

database    bdb
suffix        "dc=the-ark,dc=org,dc=au"
rootdn        "cn=admin,dc=the-ark,dc=org,dc=au"
# Cleartext passwords, especially for the rootdn, should
# be avoid.  See slappasswd(8) and slapd.conf(5) for details.
# Use of strong authentication encouraged.
rootpw        {SSHA}1sc2UoHOMrlZFxxvOUBPKQ/UFZAmEI53
# The database directory MUST exist prior to running slapd AND
# should only be accessible by the slapd and slap tools.
# Mode 700 recommended.
directory    /var/lib/ldap
# Indices to maintain
index    objectClass    eq,pres
index    ou,cn,mail,surname,givenname    eq,pres,sub
index member,labeledURI            pres,eq

# Replicas

# Entrey cache-size
cachesize    10000

checkpoint 128 15

# Server is a sync provider
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100

 

Create a new file bareBase.ldif with contents:

dn: dc=the-ark,dc=org,dc=au
objectClass: dcObject
objectClass: organization
dc: the-ark
o: LDAP Ark

As root, copy the default LDAP database configuration into place:

sudo cp /usr/share/slapd/DB_CONFIG /var/lib/ldap
sudo chown openldap:openldap /var/lib/ldap/DB_CONFIG

Restart the LDAP server:

sudo service slapd restart

Initialise the LDAP database using the baseBase.ldif created earlier:

ldapadd -x -D "cn=admin,dc=the-ark,dc=org,dc=au" -W -f bareBase.ldif
Password: myLdapPassword

With the new configuration in place, restart the LDAP service:

sudo service slapd restart
sudo service slapd status

View the LDAP contents using Apache Directory Studio (version 1.5.3, installed on the host):
    
LDAP -> New connection
Name: The Ark
Host: localhost     
Bind DN or user: cn=admin,dc=the-ark,dc=org,dc=au
Password: myLdapPassword (i.e., the password used (in slappasswd) to create the hash earlier)

On the host, create a file initGroup.ldif with the contents:

dn: ou=arkUsers,dc=the-ark,dc=org,dc=au
objectClass: top
objectClass: organizationalUnit
ou: arkUsers
description: All people in the directory go in this OU.

Import initGroup.ldif to create the arkUsers group:

Select "Root DN"
File -> Import -> LDIF into LDAP

In the project ark-user-account, edit the file src/main/resources/applicationContext.xml. Set the LDAP source and arkUserDao to:

<bean id="ldapDataContextSource" class="org.springframework.ldap.core.support.LdapContextSource">    
<property name="url"  value="ldap://ark-ldap:389"/>
<property name="base" value="dc=the-ark,dc=org,dc=au"/>
<property name="userDn" value="cn=admin,dc=the-ark,dc=org,dc=au" />
<property name="password" value="myLdapPassword" />
</bean>
                                
<bean id="arkUserDao" class="au.org.theark.dao.ArkUserDao">
<property name="ldapTemplate" ref="ldapTemplate" />
<property name="basePeopleDn" value="ou=arkUsers" />
<property name="baseDC" value="dc=the-ark,dc=org,dc=au" />
</bean>

Build ark-user-account by navigating to its root directory and issuing:

mvn assembly:assembly

Copy the built .jar file over to the Ark host. Rename the jar to createuser.jar. Run createuser.jar to create the entry for This email address is being protected from spambots. You need JavaScript enabled to view it. in LDAP:

java -jar createuser.jar This email address is being protected from spambots. You need JavaScript enabled to view it. superUserPassword MyFirstName MyLastName

Configure and build The Ark

Visit https://www.google.com/recaptcha/admin/create and create a public-private key pair for new-ark.org.au.  Save the resultant web page containing the reCaptcha public and private keys,  e.g.

reCaptchaPublicKey: 6LcPqswSAAAAABTwe7ikosifme3YpXC1Cy9Sj6Vs
reCaptchaPrivateKey: 6LcPqswSAAApe8fNcXciYDgeNEQ2y54Sy1BrRrlX

In ark-container, edit src/main/resources/applicationContext.xml to reflect the instance's configuration settings:

<bean id="reCaptchaContextSource" class="au.org.theark.core.dao.ReCaptchaContextSource">
<property name="reCaptchaPublicKey" value="6LcPqswSAAAAABTwe7ikoEUbWB3YpXC1Cy9Sj6Vs" />
<property name="reCaptchaPrivateKey" value="6LcPqswSAAAAADVTcXciYDgeNEQ2y54Sy1BrRrlX" />
</bean>

<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url" value="jdbc:mysql://ark-database:3306?characterEncoding=utf8" />
<property name="username" value="arkadmin" />
<property name="password" value="arkadminPasswordHere" />
</bean>

<bean id="ldapDataContextSource" class="au.org.theark.core.dao.ArkLdapContextSource">
<property name="url" value="ldap://ark-ldap:389" />
<property name="basePeopleDn" value="ou=arkUsers" />
<property name="base" value="dc=the-ark,dc=org,dc=au" />
<property name="userDn" value="cn=admin,dc=the-ark,dc=org,dc=au" />
<property name="password" value="myLdapPassword" />
</bean>

Build The Ark webapp from the command-line:

cd ~/workspace/ark-1.0.1/the-ark
./ark-build-pwd.sh

Apache Tomcat

Install:

sudo apt-get install tomcat6 tomcat6-admin tomcat6-docs tomcat6-examples tomcat6-user authbind

Stop service:

sudo service tomcat6 stop

Enable authbind:

sudo vi /etc/default/tomcat6
AUTHBIND=yes

Configure Tomcat connectors for port 80 and 443 (for SSL), comment all other connectors:

sudo vi /etc/tomcat6/server.xml

<Connector port="80" protocol="HTTP/1.1"
connectionTimeout="20000"
URIEncoding="UTF-8"
redirectPort="443" />

<Connector port="443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS" />

 

To optionally install a digital certificate for the web server service, check the alias of the keystore (the .jks created during CSR generation):

keytool -list -v -keystore new-ark_org_au.jks
Password: keyStorePassword
Alias should be "server"

Install the SSL certificate received from the Certificate Authority into the keystore:

keytool -import -trustcacerts -alias server -file new-ark_org_au.p7b -keystore new-ark_org_au.jks
Password: keyStorePassword

Configure Tomcat to use the keystore:

sudo cp new-ark_org_au.jks /etc/tomcat6
sudo chmod 644 /etc/tomcat6/new-ark_org_au.jks
sudo chown tomcat6 /etc/tomcat6/new-ark_org_au.jks

sudo vi /etc/tomcat6/server.xml
        
<Connector port="443"
maxHttpHeaderSize="8192"
maxThreads="150"
minSpareThreads="25"
maxSpareThreads="75"
enableLookups="false"
disableUploadTimeout="true"
acceptCount="100"
scheme="https"
secure="true"
SSLEnabled="true"
clientAuth="false"
sslProtocol="TLS"
keyAlias="server"
keystoreFile="/etc/tomcat6/new-ark_org_au.jks"
keypass="keyStorePassword" />

See https://www.digicert.com/ssl-certificate-installation-tomcat.htm for more information.

Start service:

sudo service tomcat6 start

Check service status:

sudo service tomcat6 status

Notes:

 "...the tomcat6 package creates a small file in /etc/authbind/byuid named with the UID of the tomcat6 user, and containing the line:
0.0.0.0/32:1,1023
This allows the tomcat6 user to bind to any IP address with any low-numbered port, TCP or UDP."

See http://thelowedown.wordpress.com/2010/08/17/tomcat-6-binding-to-a-privileged-port-on-debianubuntu/

Install the libmadeline shared library

The following steps are required to compile and install the libmadeline native library for pedigree visualisation within The Ark. The guide assumes that you have checked out the Madeline native code that is bundled with Ark releases 1.1.1e onwards.

Install compilers and other essential build tools:

sudo apt-get install build-essential

Install the runtime and development packages for libxml2, libcurl and libbz2.

sudo apt-get install libxml2 libxml2-dev
sudo apt-get install libcurl3 libcurl4-gnutls-dev
sudo apt-get install zlib1g zlib1g-dev
sudo apt-get install libbz2-1.0 libbz2-dev
sudo apt-get install libssl libssl-dev

Edit the Makefile as necessary; this usually means changing the paths to your JDK and headers. See comments in the Makefile for a guide; the default Makefile is configured to run on an Ubuntu 12.04LTS (64-bit) system using OpenJDK 6 (amd64 build).

Remove any previous compilation output and build a new libmadeline:

make distclean
make

Copy libmadeline.so to the Tomcat6 shared library directory:

sudo service tomcat6 stop
    
sudo cp libmadeline.so /usr/share/tomcat6/lib
sudo chown root:root /usr/share/tomcat6/lib/libmadeline.so
sudo chmod 644 /usr/share/tomcat6/lib/libmadeline.so

Edit /etc/default/tomcat6 to set the library path as part of JAVA_OPTS:

JAVA_OPTS="-Djava.awt.headless=true -Djava.library.path=/usr/share/tomcat6/lib -Xms1024m -Xmx2048m -XX:+UseConcMarkSweepGC"

Restart the Tomcat6 service:

sudo service tomcat6 restart

Note: the JAVA_OPTS example above sets the starting Java heap size to 1GB (1024m) and the maximum heap size to 2GB (2048m). Adjust these values to suit your server's memory resources.

Deploy the Ark WAR file

From the development machine, copy the webapp to the host:

scp ~/ark/workspace/ark-container/target/ark.war This email address is being protected from spambots. You need JavaScript enabled to view it.:~/'

On the host, stop Tomcat:

sudo service tomcat6 stop

On the host, remove the existing deployment and logfiles:

sudo rm -rf /var/lib/tomcat6/webapps/ark.war /var/lib/tomcat6/webapps/ark /var/log/tomcat6/*.*

On the host, copy the WAR into the deployment directory and set permissions:

sudo cp ~/ark.war /var/lib/tomcat6/webapps
sudo chmod 644 /var/lib/tomcat6/webapps/ark.war
sudo chown tomcat6 /var/lib/tomcat6/webapps/ark.war

On the host, start Tomcat:

sudo service tomcat6 start

On the host, check that The Ark webapp started successfully:

sudo less /var/log/tomcat6/catalina.out

On any computer, test the new Ark installation further by launching a web browser and going to: https://new-ark.org.au/ark/