FreeSWITCH mod_xml_curl With mod_callcenter

FreeSWITCH Loading Queues from Database

This has been a long due post waiting in my drafts for above two years now. This was done right after when I finished loading SIP users from Database in this blog post.

I always found myself at trouble when FreeSWITCH's MOD_XML_CURL was named together with loading configurations but as soon as I gave it a shot it became very easy and interesting. 

Here is how to get started with loading queue from Database in just few minutes. 

Here is what we need to get started:

1 - Database table containing Queue's parameters.

2 - A WebServer containing our DB-to-XML converter code

3 - FreeSWITCH with mod_xml_curl installed and configured.

Creating Table for Queue Parameters 

So, first things first lets create a simple Database table:

root@DBSERVER:# su - postgres
postgres@DBSERVER:~$ psql
psql (9.4.6)
Type "help" for help.

postgres=# \c freeswitch_configs
You are now connected to database "freeswitch_configs" as user "postgres".
freeswitch_configs=#

CREATE TABLE queues (
queue_id  serial primary key,
name VARCHAR(255) not null,
strategy VARCHAR(100) default 'longest-idle-agent',
moh_sound VARCHAR(255) default 'local_stream://moh',
announce_sound VARCHAR(255) default NULL,
announce_frequency VARCHAR(5) default NULL,
time_base_score VARCHAR(255) DEFAULT 'queue' ,
tier_rules_apply VARCHAR(25) DEFAULT 'false',
tier_rule_wait_second VARCHAR(4) DEFAULT '300',
tier_rule_wait_multiply_level VARCHAR(5) DEFAULT 'true',
tier_rule_no_agent_no_wait VARCHAR(255) DEFAULT 'false',
discard_abandoned_after VARCHAR(5) DEFAULT '60',
abandoned_resume_allowed VARCHAR(5) DEFAULT 'false',
max_wait_time VARCHAR(25) DEFAULT '0',
max_wait_time_with_no_agent VARCHAR(10) DEFAULT '0',
max_wait_time_with_no_agent_time_reached VARCHAR(10) DEFAULT '0',
record_template VARCHAR(255) DEFAULT NULL
);

Once Table is created you can go ahead and create some queues in it via INSERT statements.

Setting up WebServer

Alright, we've some queues defined into our database table. Now, is the time to create a web service which will query the Database and pull required configurations. Once configurations are found for a queue we will put them nicely into the required XML format and reply back.

Take the PHP sample code from my github repo:

https://github.com/goharahmed/saevolgo/blob/master/getconfig.php

Configure the Database IP, Port, User, and Password as required and make sure there is no error in it.

Once this is ready and running we can query to this web service and see that it returns an Error XML if no queue name is passed to it.

Test it:
http://ip.of.webserver:port/freeswitch/getconfig.php?CC-Queue=MyQueueName

If everything is configured properly and Queue: MyQueueName exists then it should print a fine XML - if that doesn't happen then check your PHP code, PHP PDO extension for PGSQL and verify that port is accessible and webserver is running. 

Configuring MOD_XML_CURL

Assuming mod_xml_curl is installed on your FreeSwitch server now its time to configure its module xml_curl to pull configurations from webserver instead of local file.

root@FREESWITCH1:# cd /usr/local/freeswitch/conf
root@FREESWITCH1:conf# cd autoload_configs/
root@FREESWITCH1:autoload_configs# vim xml_curl.conf.xml

<configuration name="xml_curl.conf" description="cURL XML Gateway">
  <bindings>
    <binding name="configuration">         
      <param name="gateway-url" value="http://WEBSERVER/freeswitch/getconfig.php" bindings="configuration"/>
    </binding>
  </bindings>
</configuration>

Ensure that there are no other configuration binding to conflict here.

Save and exit. Go inside freeswitch console and reload mod_xml_curl

root@FREESWITCH1:# fs_cli
fs_cli>reload mod_xml_curl 
+OK Reloading XML
+OK module unloaded
+OK module loaded

All set, now we are ready to load Queues from Database directly. To test things out issue this command

freeswitch@internal> callcenter_config queue load MynewQueue
+OK
freeswitch@internal> callcenter_config queue list
name|strategy|moh_sound|time_base_score|tier_rules_apply|tier_rule_wait_second|tier_rule_wait_multiply_level|tier_rule_no_agent_no_wait|discard_abandoned_after|abandoned_resume_allowed|max_wait_time|max_wait_time_with_no_agent|max_wait_time_with_no_agent_time_reached|record_template
support@default|longest-idle-agent|local_stream://moh|system|false|300|true|false|60|false|0|0|5|
MynewQueue|longest-idle-agent||queue|false|0|false|true|60|false|0|0|5|
+OK

To debug XML Curl

freeswitch@internal> xml_curl debug_on
OK

freeswitch@internal> callcenter_config queue load NOTMYQUEUE
-ERR Invalid Queue not found!

2016-03-28 19:32:21.128783 [CONSOLE] mod_xml_curl.c:323 XML response is in /tmp/3337e053-077c-4f39-9c3f-0805c4896851.tmp.xml
freeswitch@internal

root@FREESWITCH1:# cat /tmp/3337e053-077c-4f39-9c3f-0805c4896851.tmp.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<document type="freeswitch/xml">
<section name="result"><result status="not found" />
</document>

Conclusion

So, now we see how simple it is to pull configurations from Database - for a practice try this all with Conferences

Hosted PBX using FreeSWITCH

"Reinventing the Wheel"  

Only according to my requirements

Though I've made almost two Hosted PBXs and couple of projects with Hosted IVR but those were golden Asterisk times and I feel like I haven't done anything big yet. This time I am fully occupied in my official activities and yet to get the most out of me I'm starting to convert my Hosted PBX solution from Asterisk to FreeSWITCH. Not to mention that the Autodialer using FreeSWITCH I made recently was a big accomplishment and worked flawlessly which gave me motivation for this. 

That Autodialer consisted of  FreeSWITCH, Lua, Perl, MySQL, and Redis. It inspired me to do more but I was somewhat less busy to think of something like this current Project.

Starting with a quick overview. The ideal Hosted PBX should have these characteristics from developers point of view:

1- Scaleable [Extend, Expand, Integrate easily and quickly]

            The structure should be easy to expand as in hardware and extend as in software. Both ways should be easy to integrate with the ongoing structure. 

In HostedPBX Context: I should be able to add as many clients and resellers as I want, add more servers and new nodes into the system and increase my capacity. Any client should be able to use system provided ITSPs or add their own ITSP for their usage. Personally I Like the idea of pooling the services so at any given time adding anything to the pools add overall to the infrastructure. 

2- Performance [Effective code and virtually least possible bottlenecks]

           Use as simple and as intelligently-lazy code as possible, find quick and easy ways to make a module for a particular feature but at the same time keeping in mind that it doesn't closes its doors for extension or integration with other modules/layers.

In HostedPBX Context: Call processing should be as speedy as possible, nothing should keep the caller waiting - Anything thats behaviour of application is excluded from this i.e Queues, Conferences, PDD from Carriers. But there should be no PDD coming from withing the PBX - example a curl function that dumps/retrieves values from web server blocking the call from proceeding further.

3- Accessibility [ Ease of use for everyone]

           Everyone means everyone connected to it which includes developers, administrators, resellers, clients.Any functions/APIs or GUIs should be as easy to work with as possible.

In HostedPBX Context: Mostly clients and their usability should be given priority( GUI ), That doesn't mean that anything else should be ignored totally or given less preference. Internal code and infrastructure should be as much easily accessible to authenticated and authorized Engineers and admins.

4- Reliable and Secure [Fail & Fool proof structure]

            Failure to any one or at max up to three nodes, services, providers, or links should not effect overall system. This reliability should be increased with introduction of more Hardware/Software code keeping point-1 and point-2 in mind. At any given time at any layer should be any single point of failure be it a DNS provider, power services provider, internet services provider, or any software application/tool.

In HostedPBX Context: Everything should've atleast one backup node, every layer should has its own security mechanism. An IVR with infinite input timeout is as dangerous to service as a less secure SIP extensions password. In any case its a loss to revenue for all the parties involved with the service. Failure to SIP Inbound SBC or REGISTRAR server should be invisible to users.

5- Customizable [Add/remove modules(features) with less pain]

           Keeping point-3 in mind the structure and the programming used should've always interaction points to be used by any other module in the structure. If Point-3 is not ignored then the customization will come easy as cake. Point-1 Provides basic rule of thumb to design the project as to facilitate the extension in code or feature set.

In HostedPBX Context: Adding new features or altering GUI menus must be able to fit easily into the structure. Collect values from anywhere from the system and use it accordingly.

Expected Tools

Keeping my own expertise in mind, what I can tell at the moment which I can do on my own is use the following tools to design the PBX

1- OpenSIPS                                   1- Kamailio

2- FreeRADIUS                               3- FreeSWITCH + SEMS + SylkServer

4- Languages: Perl, Lua, PHP         5- Redis or Cassandra

6- MySQL                                       7- Vyatta

8- DNS(BIND) Server                     9- MediaProxy + RTPproxy

Since I'm strictly not a Web devel so :( unfortunately can't tell how and when it can be developed. Since its a personal project so I dont know if this area will be constructed or not. Wish I'd done some web development courses while I was doing summer semesters in university :D

Initially Expected Features

Most of these are almost the same features I've created before,so this time its just conversion from Asteriks code to FreeSWITCH code but obviously keeping in mind the new development style.

Multi-Domain Clients provisioning             DID provisioning

ACL Manager (for Any Secret IVR Menu/App)            Click to Dial

Call Limits(Minutes/Concurrent Calls)       AutoDialer

Billing for Outbound Calls                            CDRs

Multiple IVR creation                                  Incoming Call Settings

Call Recordings                                            DISA service

Custom SIP Trunk                                        Features Permissions

IM/Chat and Presence

N x N level IVR tree creation [ Using combination of the following applications ]

Time Conditions                                                            Announcements/Play Sounds

Queues                                                                         Follow Me / Call Forwarding

Outbound Caller ID Set[Client's DID or Custom ]         Voicemails

ACL                                                                                Conferences 

Ultimate Ideal to Follow

Though there are so many Hosted/Virtual PBX out on internet, but one of my ideal is one by 2600 Hz - though too much to commit but the feature list should be some what a subset  of this feature list. Also I think this new Kazoo project by 2600 Hz is brilliant and amazing to learn from. On SIP is another beautiful and inspiring source for me.

I can't obviously compare individual myself to the experts and senior team behind the 2600 Hz project or On SIP but atleast I can follow how their wheel looks like and give mine a shape.


Thats all I had in mind for my little home project. Hope to keep up my timelines and come up with something good.

Disclaimer: The primary purpose of this post was to get ideas and suggestions as to what and how should the Hosted/Virtual PBX based on FreeSWITCH should look like. This is my personal attempt and so far no other developer or resource has been added to the pool yet ;)

FreeSWITCH with SIP Users in MySQL [Mod XML_CURL]

FreeSWITCH is one of the biggest attraction for me since my introduction to VoIP. Since I like it too much and work with it most of the time but couldn't post anything good related to Freeswitch so far. Luckily I got a link from one of my friend's blog Touchkanalogy about Freeswitch module mod_xml_curl sort-of saying "FreeSWITCH SIP Realtime". Like Asterisk's SIP real-time where SIP users are inserted into database and asterisk authenticates incoming SIP user REGISTRATIONs from database settings.

FreeSWITCH don't exactly work like asterisk. FreeSWITCH needs XML configurations for users' settings. Following my friend's blog and the source document for the freeswitch XML_CURL  module. I was able to successfully insert users in Database and incoming SIP user's authentication requests were processed according to the data in DB.

Increasing VoIP services capacity

Using Kamailio & RTPproxy

This is a very advanced VoIP topic, but I'm sharing it here before I forget the hurdles which I faced while setting it up.

Introduction:
Setting up just one or two Media-Servers(Asterisk/FreeSWITCH/SEMS) isn't much of a headache but just one server is not enough once we start having more and more incoming call volume. Specially in larger deployments just one heavy duty server with multiple asterisks and freeSWITCH instances is never enough to handle everything efficiently and in fail-safe mode.

Current Media-Servers and their call capacity:
Remember a single instance of Asterisk on one regular server can handle 150~180 calls without any QoS issue. Running two or more instances on such a regular server may increase this capacity to around 300~380 concurrent calls but the fear of Single Point Of Failure is there all the time.

FreeSWITCH on the other hand claims to handle around 700~1200 concurrent calls on the same spec machines(rough figure to be on safe side, they claim more than this :P), but FreeSWITCH is a fairly complicated application to master for most of us.

SEMS claims to handle around 1800~2300 concurrent calls BUT..BUT SEMS is one of the hardest tool to configure and manage. Not to mention that it isn't promising any applications or services as Asterisk or FreeSWITCH are offering i.e Queues or Conferences, one may need to setup asterisk or FreeSWITCH behind SEMS. SEMS is an excellent choice for IVRs and announcements.

Problem at hand:
No matter how good you are in setting up Linux-HA, the biggest challenge of increasing VoIP media services capacity remains a big ? another big hurdle is assignment of Public IPs for each individual cluster of HA-Media Servers. Even if we manage somehow to handle capacity without thinking outside the Traditional-Asterisk-Solutions, reserving Public IPs still remains annoying problem.

Consider you've to buy Public IPs each time you need to increase VoIP services capacity along with headache of communicating with your services provider to add new IP into their call distribution list or provision traffic for new server.

Solution: 
So the solution lies in setting up a SIP-Proxy which acts as gateway distributor for all the incoming calls. This SIP-Proxy encapsulates a huge farm of Media-Servers. World only knows to communicate with you using just one Public IP belonging to the SIP-proxy.
All the Media-Servers are on Private LAN. Without any struggle anyone can put in 253 Media-Servers of all types in the Media-Servers zone. Simple and Easy ! isn't it.

RTP & SIP Handling on SBC

Using SIP-proxy as Load-distributor enables us to group all the media-servers in different resource types i.e 10 servers only handling IVRs, 13 serving Voicemails, 60 for Handling conferences only. 9 Servers as PSTN gateways etc.

SIP-proxy load-balancer will make sure each of these resources only get the requests for their service types only and all the load is distributed evenly(could be any other algorithm as well)

SIP-proxy will be sending Keep-Alive to all the Media-Servers and if any servers stops responding it marks it as Probing and continues sending KAs but not routing any calls to it until it becomes active again.

Whats NEXT:
In next post I'll try to show where to start from for setting up such an environment, what will be required, what problems I faced and their solution.

Special Thanks:
Following GURUs of VoIP from user's mailing list helped me alot in understanding and overcoming the problems I faced while setting this up.

Alex Balashov
davy van de moere
Klaus Darilion
Muhammad Danish Moosa
Muhammad Shahzad


Community list are definitely the best place to ask for minor help and understanding the problems.