Knowledgebase: FeedFetcher
How to download and install FeedFetcherDeluxe
Posted by Casey Trauer on 16 July 2013 03:30 PM

Please read the system requirements for FeedFetcher.

Here are the general installation steps:

  1. Select the correct download for your system.
  2. Install the Perl libraries.
  3. Set up your SportsDB database.
  4. Configure FeedFetcherDeluxe.
  5. Test FeedFetcherDeluxe.
  6. Optional packages to install.
  7. Monitor your log files.

 

General notes

Every install and upgrade of FeedFetcher should be handled carefully by clients. Each install has its own notes and SQL schema changes and notes. Please read these carefully. Please do not upgrade a production FeedFetcher install without first testing out the upgrade on a separate database. 

Release notes for releases of FeedFetcher can be found here

Downloading the application

Please note that you may be prompted for a user name and password to access these files. You will need the client name and password used to access the feed rather than your personal email user name that can be used to access the support center.

Linux Distribution

  • G'zipped Tar File. (Save As... to your hard drive. Should have a .tar.gz extension.)

Windows Distribution

  • Self-extracting executable. (Save As... to your hard drive). The installer contains the perl modules for running using an installed Perl, a 32-bit compiled executable (ff-2016-04-19.exe, you can rename it to ff.exe), and a 32-bit Windows Service.

 

Installing the Perl libraries

Linux

You will want to ensure your distribution contains all the necessary Perl packages to run FeedFetcher. Please keep these tips in mind when installing the modules:

  • Package names may vary between Linux distributions and versions; where possible, use your package manager to search for and install these packages and their required dependencies. We list here the module names by their Perl naming convention, then the Redhat Package Manager name in brackets.
  • XML Team's experience has been that all of these packages are now included in the distribution CDs for Debian and RedHat machines, and probably the others as well, although they may not be installed by default (especially if you have selected a 'server' or 'bare' style installation). So you can confirm first whether the latest package is installed. For example, if rpm is your package manager of choice, then run rpm -Uvh *.rpm as root for each rpm that you need.
  • If you are using a Debian based system such as Ubuntu all necessary packages should be available from the default remote package repository. Versions and default package installations change over time and by vendor release, so we cannot indicate the exact package names to install. However, you should be able to install needed packages in a short time by using two login sessions, one with the aptitude package installer program running and one at FeedFetcher's perl directory. Invoking FeedFetcher (for example, using perl Feedfetcher/ff.pl -pitwalk), will cause the program to display any package declared as missing. Use the package installer to update the package and then repeat this until FeedFetcher runs without error.

 

Install (in order) the following Perl modules via your Linux Perl Package Manager of choice (aptitude, up2date, etc.):

  1. Compress::Zlib (perl-Compress-Zlib)
  2. File::Slurp (perl-File)
  3. Date::Pcalc (perl-Date-Pcalc)
  4. XML::LibXML (perl-XML-LibXML)
    • libxml2-python
    • libxml2
    • perl-XML-LibXML-Common
    • perl-XML-SAX
    • perl-XML-LibXML
    • perl-XML-NamespaceSupport
  5. XML::LibXSLT (perl-XML-LibXSLT)
    • libxslt-devel
    • libxslt-python
    • libxslt
    • G'Zipped Tar of Perl LibXSLT
      • perl Makefile.PL
      • make
      • make test
      • make install
  6. DBI
    • perl-DBI
  7. DBD::MYSQL
    • perl-DBD-MySQL
    • This module must be manually created for MySQL 4
      • libmysqlclient.so.10 is required by the package, but is not found in MySQL 4
      • See this thread for details
  8. URI
    • perl-URI
  9. LWP
    • perl-libwww-perl
    • Requires a "force install from the shell"
  10. HTML::TokeParser
    • perl-HTML-Parser
    • Requires a "force install from the shell"

Windows

If you are using the compiled version of Windows, this step is not necessary.

If you wish to use a non-compiled FeedFetcher you can do so by installing Strawberry Perl. Strawberry Perl is a mature and well maintained release of Perl for Windows and comes complete with all modules required to run FeedFetcher, including ODBC and XML modules. Please let us know if you have any difficulties using it to run FeedFetcher.

Please note that ActiveState has stopped providing free downloads of older releases of its ActivePerl product. Newer releases do not work with the third-party modules such a those providing support for XML parsing (XML::LibML, XML::LibXSLT). 


Setting up your SportsDB database

The /dbSetup/ folder in the FeedFetcherDeluxe application folder will contain build scripts, transition scripts (if you are updating from a previous version of FFD) and some index building scripts to set up the SportsDB database schema. To create these databases, here are one possible set of steps.

Linux

For MySQL users

Create the "sportsdb" database and import our table structure and defaults for reference tables (adjust pathnames as needed):

/mysql/bin> mysql -u root
mysql> create database sportsdb;
mysql> \q
/mysql/bin> mysql -D sportsdb < "/FeedFetcherDeluxe/dbSetup/sportsdb-[LATEST_NUMBER_GOES_HERE]-mysql-3.sql"

For PostgreSQL users

Create the "sportsdb" database and import our table structure and defaults for reference tables (adjust pathnames as needed):

/postgres/bin> psql
psql> create database sportsdb;
psql> \q
/psql/bin> psql -U[postgresql username] -dsportsdb -f "/FeedFetcherDeluxe/dbSetup/sportsdb-[LATEST_NUMBER_GOES_HERE]-postgressql-8.sql"

Windows

For MySQL users

Create the "sportsdb" database and import our table structure and defaults for reference tables (adjust pathnames as needed):

c:\mysql\bin> mysql 
mysql> create database sportsdb; 
mysql> \q 
c:\mysql\bin> mysql -D sportsdb < "C:\FeedFetcherDeluxe\dbSetup\sportsdb-[LATEST_NUMBER_GOES_HERE]-mysql-3.sql"

For PostgreSQL users

Create the "sportsdb" database and import our table structure and defaults for reference tables (adjust pathnames as needed):

c:\postgres\bin> psql
psql> create database sportsdb;
psql> \q
c:\psql\bin> psql -d sportsdb -f "C:\FeedFetcherDeluxe\dbSetup\sportsdb-[LATEST_NUMBER_GOES_HERE]-postgresql-8.sql"
For SQL Server users

Create the "sportsdb" database and import our table structure (adjust pathnames as needed):

  • Open SQL Server Enterprise Manager
    • Right-click on Databases
    • Select "New Database..."
    • Name it "sportsdb" and hit OK
    • Expand the "sportsdb" database
    • Right Mouse-Click on the Users icon and select "New Database User"
    • To create a brand-new user, select "<new>" from the Login dropdown
      • Type in a username
      • Select the "SQL Server Authentication" option
      • Type in a password
      • Select "sportsdb" as the default database
      • Select "Server Roles" tab
        • Check off "System Administrators" or other appropriate permission(s).
      • Click the "Database Access" tab
        • Check off the "sportsdb" database
        • Check off the "db_datareader" and "db_datawriter" permissions
    • Hit OK and Confirm your password
    • Otherwise, select a current user, and check off the "db_datareader" and "db_datawriter" permissions
    • Check that your server supports "SQL Server Authentication"
      • Right mouse click on the server (i.e., "(local) (Windows NT)")
      • Select Properties
      • Select the Security tab
      • Make sure that "SQL Server and Windows" is checked off, under Authentication
      • Hit OK (Allow SQL Server to be restarted, if necessary)
  • Open SQL Query Analyzer
    • Query... Change Database... Choose "sportsdb"
    • Open sportsdb-[LATEST_NUMBER_GOES_HERE]-sql-server-2k.sql and then Execute Query (F5)
  • Adjust the Data Source permissions
    • Go to Control Panel... Administrative Tools... Data Sources (ODBC)
    • Choose the "System DSN" tab
    • Highlight LocalServer (or use another current or new source, if you'd like)
    • If you are adding a new System DSN and you are given the choice between "SQL Native Client" and "SQL Server", choose "SQL Server"
    • If you are installing FeedFetcher on a 64-bit Windows Server system, please see specific ODBC installation notes here.
      The critical part of that page involves creating a 32-bit ODBC connection definition, not 64, which is the default on 64-bit windows.
      To create 32-bit ODBC connections, you have to run %WINDIR%\SysWOW64\odbcad32.exe.  
      Adding an ODBC connection through this application will create the ODBC connection in the 
      HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ODBC\ODBC.INI key, and allow your 32-bit applications 
      to see and use the ODBC connection.
    • If you are modifying an existing System DSN, click Configure...
    • Click Next
    • Select "With SQL Server authentication using a login ID and password entered by the user."
    • Click Next twice, and then click Finish


Configuring FeedFetcherDeluxe

The conf/feedfetcher.cfg file dictates how FeedFetcher works, including how it interacts with the server, where to store the data and many other functions. We will focus here on the basic configuration required to ensure FeedFetcher works correctly.

# Login Parameters for XML Team server

XT_User=
XT_Pass=

These should contain the user name and password you were given to access our feed servers.

# Pull Connection Parameters

# Primary Servers:
XT_Server_1=feed.xmlteam.com

This should be already filled out. If you have a value of feed6.xmlteam.com, that old value will still work but we recommend changing it to feed.xmlteam.com. The # Secondary Servers parameter (XT_Server_2) is currently not necessary because of new redundant systems on our feed servers.

# Database Parameters

Use_database=True

DBtype=
DBhost (or DBname, for SQL Server)=
DBdatabase=
DBuser=
DBpass=

Fill these parameters out as appropriate for your database type.

 

Testing FeedFetcherDeluxe

On the command line, cd to the application's /perl/ folder. Execute the following command:

perl Feedfetcher/ff.pl -pwaitS 10

(for Windows compiled users: Feedfetcher/ff.exe -pwaitS 10)

This should pull the files published in the last 10 minutes, including at least a heartbeat file. If there is an issue, an ERR message will be written.

 

Installing optional packages

SportsCaster Webservice API

The SportsCaster Webservice is an API that retrieves data from SportsDB and XML files from the archive. It returns that data in SportsML and can optionally use XSLT to display it in another format, such as HTML or JSON.

For a basic primer on this API, see the SportsCaster section of the Knowledgebase.

XSLT package

XML Team offers an XSLT library called SportsMLT for converting key files to HTML. It is considered a starter set, to be customized by the client. It can be used by FeedFetcherDeluxe to create new files after processing the incoming XML feed, or it can be used by the SportsCaster webservice to create new documents on the fly.

Read how to configure FeedFetcher to make these transformations.

The xsl package should be saved in the /xsl/ directory of the application folder.

Windows service

Download the readme file and the Windows Service into your FeedFetcherDeluxe\perl\WindowsService directory, and follow instructions in the readme.

Sample Linux startup script

Here is a sample linux startup script. It is provided just for reference and is not supported.

This script, if you choose to use it and once verified to work on your system, can be put in /etc/init.d and managed on Redhat systems with a call like this:  

chkconfig feedfetcher on

for the script to execute on reboot

and on debian based systems with a call like this:

update-rc.d feedfetcher defaults

NOTE: It is your responsibility to ensure that feedfetcher is run appropriately and with appropriate privilege restrictions (i.e. not as root) on your server.

League directory files

Typically, the feed data does not populate the database with information such as team abbreviations and team affiliations (what conference/division it is in). XML Team provides league directory files that populate this information that can be found at:

http://private.xmlteam.com/league-directory/

These files are publisher and league specific, so make sure you download the file(s) appropriate to your publisher/league permissions. You can save them in the /local-source/ folder of the FeedFetcherDeluxe application, and process them into the database using the options -pitwalk on the command line.

Here is a sample query from the SportsDB query library that would display the information that these files provide:

http://www.sportsstandards.org/sd/querylib/viewquery/74

Other queries -- specifically, standings queries -- may require this data.

Monitoring log files

As with any production-grade feed-processing software, you should be monitoring your logfiles.

If, say, you're FeedFetcher is configured to check for new content every five minutes, then your logfile will be written to at least once every five minutes.

Set up a monitor (using whatever tools you like to use for log monitoring -- there are tons of free and commercial packages that do this, for every platform) that checks to see whether your log file has changed at least every 15 minutes, for example.

The default FeedFetcher logfile is logs/current-log-{name of config file used}.txt

Many monitoring platforms allow you to send out email alerts if some problem is encountered.

You may also want to consider rotating your log files, so it's easy to see smaller log files for particular days.

(1 vote(s))
This article was helpful
This article was not helpful

Comments (0)
Post a new comment
 
 
Full Name:
Email:
Comments:
Help Desk Software by Kayako Resolve