install

To completely install the website and software some effort has been put in creating provisioning. This chapter will attempt to describe that.

new version

This install is done by hand, and while doing that a script was built to automate the steps.

Note that we chose to use debian as a base because of maintainability. For instance we could choose for arch linux so we could roll the distribution without ever installing a new os again, but every update 'pacman -Suy' would potentially break the running code.

Debian stable would start with a stable and working system, we then would only perform security updates, which might break code but i don't recall that ever happening. Also a major release happens roughly every two years, and the old version stays in maintenance for a long time, for instance version 8 "Jessie" is still under LTS (Long Time Support) and current stable is 10 "Buster" (LTS 2022) So we have plenty of time for a transition and even skipping a release could be considered !!

In any case we would perform an update on a test system first and run a system test before doing the rest of the systems. If we would use arch-linux this would have to be done daily !! If using debian we would have to do this every time a security update is needed.

This is much less frequent and much less dangerous.

In principle :

• All systems use the same base : debian stable (buster in 2020)
• All systems use a base set of packages. Cheatsheet is so much different that it does not fall under this installation policy (doc could).
• All software (bag,backend,frontend) should be startable on all servers, so each server will be identical, and identically installed.

Cheatsheet is a completely separate installation, however it does make sense to install doc on the cheatsheet VM because these are purely web sites.

history

apt-get install aptitude
apt-get install openssh-server
apt-get install aide
apt-get install mkisofs xorrisso
apt-get install vim tree watch

To save you some headache, edit /etc/bash.bashrc :

export VISUAL=vi # for edit sessions of sudo, git etc

password handling

We want to perform this as much automated as possible. We start from a safe machine so there must be a way to setup a safe path using ssh certificates in which no passwords have to be given.

There are two reasons why i opted to go back to ansible here :

• Provisioning multiple machines.
• The vault.

These would be two things i have to implement and they would not be tested in full.

vagrant

The provisioning part is in the deploy repository, so just start with checking that out.

git clone git@bitbucket.org:keesklopt/deploy.git
cd deploy
vagrant up
vagrant ssh
sudo su 
# install some usefull stuff :
apt-get install vim tree screen git python-pip aptitude
# and disable the horrible default settings
touch ~/.vimrc
vim /etc/ssh/sshd_config
# set PasswordAuthentication yes
service sshd restart
adduser kees
addgroup kees sudo
ctrl-D # 
ssh-copy-id kees@192.168.44.200
ssh kees@192.168.44.200

This will instantiate a buster VM at the time of writing, Vagrant boxes do not ship as 'stable' so this will have to be altered by hand if debian skips to another stable. Of course if you want to install everything as vagrant you can skip some steps. Actually the end user in production is bound te be 'klopt'.

provisioning

We could use ansible for this, but i don't like the myriad of files that it needs to operate. Ansible works by invoking a local python script on the remote machine and we can also do this and keep everything down to one file.

However it is only possible to run one file :

ssh 192.168.44.200 python < onescript.py

There is no easy way to let onescript.py import anything else than system modules, since these scripts are not available on the remote system.

# this won't work :
ssh 192.168.44.200 python < onescript.py deploy.py
bash: deploy.py: command not found

# this would work but it will be concatenated,
# and onescript imports deploy, so that will also fail
cat onescript.py deploy.py | ssh 192.168.44.200 python
ImportError: No module named deploy

Python has a way to combine a script into one file by renaming the main script to main.py and then zipping all together.

Though interesting.. this solution does not work over ssh !!

This example uses python3. Most linux versions still have python2.7 as default because scripts will break otherwise. But python3 is installed as python3 on all new distros so you can just use it :

mv klopt.py __main__.py
zip klopt.zip __main__.py deploy.py
python klopt.zip
# works !
# and even shorter, note you need to use python3 here :
echo '#!/usr/bin/env python3' | cat - klopt.zip > klopt
chmod +x klopt
./klopt
# also works

So this can be used to maintain a separate module for deployment and installer however it still fails to install over ssh and complains about

Non-UTF-8 code starting with 'xf8'

So we are back at single script versions again. For now i opt for a base script containing all functionality in which you should override the main function. The script is now in the deploy repository with the main function implementing the klopt install.

older version

Next section to the end is obsolete but may contain interesting parts, so it will not be deleted yet.

You can always do this on a vagrant VM, though it should also be ready for a production install. To instantiate a jessie VM, run vagrant in the provision directory:

git clone https://keesklopt@bitbucket.org/keesklopt/provision
cd provision
vagrant up 

This should give you a runnable jessie VM. Then you need to prepare the .deb packages and install fpm. If you use the standard "debian/jessie64" VM as mentioned on the hashicorp site : visit then login in with a password is disabled. You can only login with vagrant ssh :

vagrant ssh
sudo su 
vi /etc/ssh/sshd_config

Alter this setting to yes, it is the last line in this box :

PasswordAuthentication yes

service sshd restart

fpm

effing package manager. There is a guide on installation here : visit

To go short.. it is a ruby program so you need gem/ruby.

apt-get install ruby ruby-dev rubygems build-essential

And then install fpm with gem :

gem install --no-ri --no-rdoc fpm

Also to accommodate the packages you are going to create with this :

mkdir -p ~/projects/packages

registratie webpage

The registratie webpage has been moved to the inactive repository , so check it out from :

git clone git@bitbucket.org:keesklopt/inactive
cd inactive/registrate/web
make package
make provision

Similar steps have to be taken for the other packages

::: {.note} ::: {.title} Note :::

These packages have to be prepared before you run the complete playbook because they get installed during the 'repo' role. :::

git clone git@bitbucket.org:keesklopt/klopt.web
cd ~/projects/klopt.web
make package
make provision

cd ~/home
make package
make provision

cd ~/projects
git clone git@bitbucket.org:keesklopt/3pty
cd 3pty
apt install libbz2-dev
./compile.sh

network

This is a little different, and i think it is only needed for test installs with a little network. If you want to do a complete Europe don't bother packaging it.

But till that time just go with a small one (maybe Netherlands ?). As taken from the makefile :

any and all networks you put in the network directory will be added !

So let's put Netherlands in there see if it performs.

cd ~/projects/osrm-backend/
wget http://download.geofabrik.de/europe/netherlands-latest.osm.pbf
./build/osrm-extract netherlands-latest.osm.pbf
./build/osrm-prepare netherlands-latest.osrm
mkdir -p ~/projects/grid/src/networks
cp netherlands-latest.osrm.* ~/grid/src/networks/

klopt packages

This needs to be done after all other steps.

cd ~/projects
git clone git@bitbucket.org:keesklopt/grid
cd grid
apt install libicu-dev libyaml-dev libjsoncpp-dev 
apt install libboost-all-dev libtbb-dev libstxxl-dev
apt install libssl-dev libcrypto++-dev
apt install libbz2-dev liblua5.2-dev lua5.2
make package
make provision

ansible provisioning

For this to work you apparently need the sshpass program :

apt install sshpass
cd provisining
annsible-playbook -i hosts playbook.yml -k 

This will attempt to install a complete working server. Leave the custom and zfs part commented out since these are not needed and don't work.

testing

Now the install should be reachable on :

visit

And possibly even :

visit