OpenSIPS/Kamailio High Availability Clustering - 2

High Availability Setup

This post was due for a very very long time since I posted about the general design of a HA-SIP-Proxy in one of my old blogpost.

Now lets start working on this. Using this setup we can cluster two or more machine by using one single Public IP on their WAN interface one-at-a-time. To ensure service availability in case primary (current master) server crashes the application heartbeat resource is configured to monitor the service's status every after 30 seconds. If the Service is found to be stopped it tries start it on the same server for couple of times and then migrate the group of services to the other node. 

This tutorial is equally good for OpenSIPS or Kamailio or any other services.

Active/Passive design diagram

Pre-Requisites:

- Atleast two servers with WAN interfaces empty but cable connected, such that if Public IP is assigned it readily gets accessible from Internet.

- LAN interfaces on both servers should be on the same subnet and should have static Private IP configured.

- The WAN and LAN interfaces on both servers should have similar names i,e eth0=WAN, eth1=LAN

- There should be NO default route inserted to these servers.

Installing Packages:

- Insert a default g/w for LAN interface on both machines temporarily

- Install OpenSIPS or Kamailio or any other tools as per requirement.

- Install heartbeat and sipsak 

Linux-console:~# apt-get install heartbeat sipsak

SIPSAK can be used in the opensips LSB init.d script to send an SIP OPTIONS packet to OpenSIPS port and on the server's reply it'll announce the service is running. This is optional and I suggest users to try this on their own. I recommend looking into the sample asterisk lsb-script provided by heartbeat, there they use sipsak to monitor asterisk's sip port and decided if service is up or not.

Configuring Files for Heartbeat

NOTE: All the files we are going to edit here should be copied to the second server as well.

1- Edit the /etc/hosts file to add hostname for the two servers

192.168.100.148 SIP-SERVER_HA1

192.168.100.62 SIP-SERVER_HA2

2- Edit the /etc/heartbeat/ha.cf file and insert the following.

# enable pacemaker, without stonith

crm             yes

# log where ?

logfacility     local0

# warning of soon be dead

warntime        10

# declare a host (the other node) dead after:

deadtime        20

# dead time on boot (could take some time until net is up)

initdead        120

# time between heartbeats

keepalive       2

# the nodes

node            SIP-SERVER_HA2

node            SIP-SERVER_HA1

# heartbeats, over dedicated replication interface!

ucast           eth1 192.168.100.148

# ignored by node1 (owner of ip)

ucast           eth1 192.168.100.62 # ignored by node2 (owner of ip)

# ping the switch to assure we are online

ping            192.168.100.100

3- Edit the /etc/heartbeat/authkeys file and insert the following:

auth 1
1 sha1 S3cr3tP@ssw0rd

- Assign permissions to the above mentioned file:

Linux-console:~# chmod 0600 /etc/heartbeat/authkeys

Files editing is done here. Copy the files to other server(s).

- Start the heartbeat service on both servers:

Linux-console:~# /etc/init.d/heartbeat start

- Wait for at least 30 seconds and then check the status of the cluster by issuing the following command on both servers.

At the end there will be displayed the cluster online nodes.

Linux-console:~# crm status
============
Last updated: Tue Jan 22 08:02:17 2013
Stack: Heartbeat
Current DC: SIP-SERVER_ha2 (8b5cf63e-4f77-448c-9a75-6a91d4a00cb7) - partition with quorum
Version: 1.0.9-74392a28b7f31d7ddc86689598bd23114f58978b
2 Nodes configured, unknown expected votes
0 Resources configured.
============
Node SIP-SERVER_ha1 (fe3e635f-0d4e-4d8c-99e1-195d1952ac53): UNCLEAN (offline)
Online: [ SIP-SERVER_ha2 SIP-SERVER_ha1]

Note that in the very last line above we've a list of nodes which have joined this Heartbeat group.

Configuring Heartbeat

Goto one of the active nodes in the cluster and on that server’s console issue the following commands sequentially. Once these commands are executed on one server they will be replicated to other serves in the cluster automatically so don't need to copy these commands to other servers.

Linux-console:~# crm configure property stonith-enabled=false
Linux-console:~# crm configure primitive FAILOVER-IP ocf:heartbeat:IPaddr2 params ip="11.22.33.44" nic="eth0" cidr_netmask="255.255.255.240" op monitor interval="10s"
Linux-console:~# crm configure primitive OSIPS lsb:opensips op monitor interval="30s"
Linux-console:~# crm configure primitive SETRoute ocf:heartbeat:Route params destination="default" device="eth0" gateway="11.22.33.1" op monitor interval="10s"
Linux-console:~# crm configure group PIP-OSIP-ROUTE FAILOVER-IP SETRoute OSIPS
Linux-console:~# crm configure colocation OSIPS-WITH-PIP-ROUTE inf: FAILOVER-IP SETRoute OSIPS
Linux-console:~# crm configure order IP-ROUTE-OSIPS inf: FAILOVER-IP SETRoute OSIPS

The very first line is important to disable the Shoot The Other Nood In The Head

In second line we're configuring a resource for the Public IP that will be assigned to the Interface eth1 and named it FAILOVER-IP.

In third line we configure the resource for OpenSIPS LSB (/etc/init.d/opensips start/stop/status) script and named it OSIPS

In fourth line we configured the resource for the Linux default route to access to Internet and named it SETRoute.

So now  in fifth line we created a group of these above mentioned resources.

In next line we bound them to move together whenever they're shifted from one machine to another.

In the very last line we arranged the services to be started in such an order that FAILOVER-IP is assigned to Interface first, then the SETRoute resource is executed to put in the default route to reach to Internet, and then OSIP resource is called to start OpenSIPS.

References and Useful Links:

http://zeldor.biz/2010/12/activepassive-cluster-with-pacemaker-corosync/

http://clusterlabs.org/wiki/Debian_Lenny_HowTo#Resource_operations

http://foaa.de/old-blog/2010/10/intro-to-pacemaker-on-heartbeat/trackback/index.html

http://blog.foaa.de/2010/10/intro-to-pacemaker-part-2-advanced-topics/#use-case-2-node-proftpd-cluster

OpenSIPS and Cassandra Integrated

Finally back to the blog after a long busy schedule, well its still the same scheule but I've managed some time to take a look at setting up OpenSIPS with Cassandra as it's cache store. In one of my blog post I showed how to setup opensips and redis (which is also now updated for new Hiredis installation github path).

The advanatges can be numerious while we are using Cassandra and Redis individually or together from one opensips which are outlined in a very detailed article by Vlad Paiu - One of the developers and contributors of OpenSIPS project.
Possible use cases from the opensips documentation page are; Password caching for DB authentication
or Simple but distributed billing. With recent new developments going on specially on cacheDB are its now possible to have a distirubuted OpenSIPS cluster with a shared CacheDB backbone and all iternal data is available across all the opensips node for example dialog profiles of callers, loadbalanced resources statuses.
Over all these CacheDB modules enable multiple OpenSIPS instances to share internal data like number of calls, registration, call counters, billing data, IP balcklists etc. Refer to this presentation at Cluecon-2012.
So lets get started to integrate OpenSIPS 1.8 with Cassandra. Installation process for OpenSIPS 1.8 has been changed a little bit. Install all the libraries and packages as mentioned in this post.

OPENSIPS 1.8 INSTALLATION

Once after the repositiores are installed we need to install thrift-0.6.1 which is the Cassandra's interfcing client used by OpenSIPS.

root@OpenSIPS-CAS:~# wget http://archive.apache.org/dist/thrift/0.6.1/thrift-0.6.1.tar.gz
root@OpenSIPS-CAS:~# tar zxvf thrift-0.6.1.tar.gz
root@OpenSIPS-CAS:~# cd thrift-0.6.1/
root@OpenSIPS-CAS:~# ./configure
root@OpenSIPS-CAS:~# make
root@OpenSIPS-CAS:~# make install

UPDATE[28-July-2013]: As commented by a respected viewer; we need to install C++ boost libraries for OpenSIPS 1.9 on Debian 6. See comment by RQ in the comments section for further details.

I tried installing the newest version of thrift but OpenSIPS won't compile and give errors. So next, download and extract the 1.8 version of OpenSIPS and execute the 'make menuconfig' command.

root@OpenSIPS-CAS:~#wget http://opensips.org/pub/opensips/1.8.2/src/opensips-1.8.2-svn9601_src.tar.gz
root@OpenSIPS-CAS:~#tar zxvf opensips-1.8.2-svn9601_src.tar.gz
root@OpenSIPS-CAS:~#cd opensips-1.8.2-tls/
root@OpenSIPS-CAS:~#make menuconfig

A menu will open up like this.

Press Right arrow key on the first option and goto the following sub menu.

Now pressing right arrow key will take us to the module selection menu where we have to select which modules we need to install. Make sure to select the Cassandra module.

Once the modules' selection is made press left arrow key to go back to the figure 2. Select "Save Changes" and press enter. A list of suggested libraries and packages will show up at the bottom of the screen for help.

Do change installation paths and other flags according to your requirements.

Press left arrow key again to go back to main menu as in figure 1. Select "Compile And Install OpenSIPS" and press Enter. If everything goes well our opensips will be installed successfully. Follow the steps as mentioned in this post to setup the opensips LSB script (/etc/init.d/opensips); default opensips file; and creating the opensips DB. 

CASSANDRA INSTALLATION

The OpenSIPS installation process is complete till here, now take a look at the cassandra installation.

root@CAS-N1:~# vim /etc/apt/sources.list

Add the follwoing lines at the bottom of the apt/sources.list file.

deb http://www.apache.org/dist/cassandra/debian 10x main 

deb-src http://www.apache.org/dist/cassandra/debian 10x main

Save and exit the file. and run apt-get update.

root@CAS-N1:~# apt-get update

If any keys error is encountered at the update then we need to add the keys using following command. (Make sure to change the '4BD736A82B5C1B00' string with the one you're seeing on your screen)

root@CAS-N1:~#gpg --keyserver wwwkeys.pgp.net --recv-keys 4BD736A82B5C1B00
root@CAS-N1:~#gpg --export --armor 4BD736A82B5C1B00 | sudo apt-key add -

Once the key is successfully added execute the apt-get update once again and after that install cassandra.

root@CAS-N1:~#apt-get install cassandra

I highly recommend to go through the following links to get quick know how about cassandra working and its files.

Planning a Cassandra Cluster Deployment

Cassandra’s data model cheat sheet

Node and Cluster Configuration (cassandra.yaml)

Configuring and Starting a Cassandra Cluster

Edit the /etc/init.d/cassandra.yaml file and change the cluster name and any other property as needed. Start cassandra and verify that it is running in processes.

I edited the following:

listen_address: 192.168.56.3

broadcast_address: 192.168.56.255

rpc_address: 0.0.0.0

rpc_port: 9160

root@CAS-N1:~#vim /etc/cassandra/cassandra.yaml

root@CAS-N1:~#/etc/init.d/cassandra start
root@CAS-N1:~#ps -ef | grep cass

If cassandra has started successfully then connect to cassandra-cli and creake a keyspace for opensips to use.

root@CAS-N1:~# cassandra-cli --host 192.168.56.3 --port 9160
Connected to: "Test Cluster" on 192.168.56.3/9160
Welcome to Cassandra CLI version 1.0.12

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.
[default@unknown]CREATE KEYSPACE opensips WITH placement_strategy = 'SimpleStrategy' AND strategy_options = {replication_factor:2}; 
a43901e0-6263-11e2-0000-af8ddedd2ed7
Waiting for schema agreement...
... schemas agree across the cluster
[default@unknown]quit;

OPENSIPS CONFIGURATIONS FOR CASSANDRA
Now, everything is ready. Time to open up the opensips configurations file and load the module for cassandra.

root@OpenSIPS-CAS:~# vim /etc/opensips/opensips.cfg

loadmodule "cachedb_cassandra.so"
modparam("cachedb_cassandra", "cachedb_url","cassandra:group1://192.168.56.3:9160/opensips_cdrs")
modparam("cachedb_cassandra", "connection_timeout",1000)
modparam("cachedb_cassandra", "send_timeout",1000)
modparam("cachedb_cassandra", "receive_timeout",1000)
modparam("cachedb_cassandra", "wr_consistency_level",1)
modparam("cachedb_cassandra", "rd_consistency_level",1)
Go throught the OpenSIPS Cassandra module documentation; we can start using Cassandra as our KVP backend by using the following commands in our opensips.cfg main route;
cache_store("cassandra:group1","key","$ru value")
cache_fetch("cassandra:cluster1","key",$avp(10))
cache_remove("cassandra:cluster1","key");

That is all , now its upto users to take advantage from this according to their needs.
Thanks for reading.

Reference URLS:

http://wiki.apache.org/cassandra/GettingStarted
http://www.opensips.org/html/docs/modules/1.8.x/cachedb_cassandra.html
http://wiki.apache.org/cassandra/Operations
http://www.sinbadsoft.com/blog/cassandra-data-model-cheat-sheet/
http://news.ycombinator.com/item?id=1244914
http://www.datastax.com/docs/0.8/configuration/node_configuration#init-properties
http://www.datastax.com/docs/0.8/install/cluster_init
http://www.datastax.com/docs/0.8/cluster_architecture/cluster_planning#node-init-config
http://www.datastax.com/docs/1.0/ddl/keyspaces
http://wiki.apache.org/cassandra/FAQ#clustername_mismatch
http://lzyeval.blogspot.com/2012/02/install-cassandra-in-debian-ubuntu.html

Call Center Solution : Techincal Review

Technical Review
Hosted Call Center Solution
(by Tech Bridge Consultancy)

This is a technical review of a Hosted Call center solution which is an all in one stop for all the call center components required in any such setups. This is developed by my senior friends in my country and I think this product needs appreciation since this is one of a kind solution which is developed in my region for International users.

This hosted call center server is Linux based with Asterisk as its telephony back end and uses Microsoft Silverlight for its User Interfaces. 

The most appealing features which I found are its sleek design and performance compared to other available products. It has a very rich feature set in both telephony as well as in CRM areas. 

Focusing the VoIP component first, it has the ability to run multiple clients with multiple campaigns in parallel and each campaign can have as many agents and super agents as needed.  The agents need not to be using SIP phones, the calls can be routed to agents using cellular or regular land lines via client specified gateway. 

Another big feature is its Predictive Autodialer which calculates the campaign agents available and trigger outbound calls. We can set outbound Caller ID for campaigns too. What I liked most about it is the availability of designing a custom IVR tree such that as soon as any autodialed caller answers it can get landed on the IVR and then can proceed by going into any category i.e press 1 for maintenance, 2 for operations..etc. This can be pretty handy to sort down the responses and areas where most of the campaign callers went.

Other features include Voicemails and its beautiful interface to listen to the recorded messages, Live calls super agent panel, live call monitoring, and Real Time Call Data Records. 

The other major component which can not be ignored here is its GUI, where we can manage our everything, collect reports, schedule meetings, manage leads and so on.

From Call Center point of view we can have super agents/supervisors and regular call center agents. Agents can be assigned priority according to their skill. There is ofcourse an in build Softphone in the GUI and a notification pop ups for each incoming call. Yes, only privileged agents can make outbound calls and regular agents can only receive calls. 

Agents can drag and drop schedule meetups for the mature leads. There is a very powerful utility available for managers to collect all campaigns statistics for analysis. 

Like any other competing solution they've focused necessarily on reporting and a very strong reporting is made available for administrators and super agents. 

The above screenshot explains the reporting features. 

From my personal experience this application is quite wonderful and works perfectly for any required customer services solution. Inbuilt Predictive Dialer, custome IVR building, Agent management, Agent performance monitoring, campaign reporting is all that one can wish for in this category. More than anything with simple addition of powerful proxy like OpenSIPS or Kamailio it can be scaled horizontally and handle as many concurrent calls as needed.

I wish my friend best of luck for this product and hope they keep up doing this good professional work in the world of VoIP.