Best practices when using our feed services
Posted by Casey Trauer on 03 April 2013 08:50 PM
XML Team is constantly working to improve the speed, flexibility and utility of our feed. Here's a primer on some basic procedures you should follow to get the best out of the XML Team feed.
Download once, store forever.
As a content distributor, XML Team wants to ensure you get timely delivery of your data as soon as it is published by the vendor. Although we have some archival services, clients should try to retrieve their licensed content as soon as it is published. Once downloaded, content should not have to be retrieved again by clients.
Data stays on in our live repository for three days. Beyond that you may need special permissions to access older content. We also have some archival repositories to help you catch up in case you had an emergency outage or are joining us in the middle of a season.
Web service clients should not use the API to serve content dynamically to their end users. Our servers are not set up to handle direct requests from end users, which might result in a lot of hits to a database that supports many clients. If your application requires this functionality, we can provide software for setting up an internal web service on your servers.
Keep your connection constant.
You should be contacting our server regularly to check for new documents.
For delivered feed clients using FeedFetcher or their own software, this means using a getFeed query regularly — at least every 15 minutes — to retrieve new documents. Push functionality — we send the doc to you as soon as we get it — is also available.
For web service clients, this means querying our content database at regular intervals for newly published documents. All queries must have a time scope, and this scope should be limited for fastest results.
Older content is available, but...
It is not always easy to retrieve past files. Archival data older than three days is organized based on when the content was sent; finding that content can mean downloading many files to get the one you want or a web service query with a large time scope that takes forever to execute. This is because you don't always know when a doc you are looking for is sent.
XML Team is happy to assist you with finding the best way to retrieve this older content.
Remember this: Recent content stays on the live server for three days. After that, it must be retrieved from the archive server. Clients do not have universal access to archival content, but XML Team can arrange access on a per-request basis, and especially for new clients who need to get up to speed.
Monitor your log files.
For delivered feed clients, these are critical resources for alerting you to problems or troubleshooting feed issues. FeedFetcher keeps daily files in its logs folder. Any custom software you may have to retrieve data should have logs that are monitored for uptime or errors.
While XML Team does not provide software for analyzing log files — each client's needs are different — we strongly recommend clients set up systems for monitoring feed logs.
Web Service and Flexsport On Demand users: Keep queries sane.
Essentially, this boils down to making queries that are specific in scope, both in terms of content and time. Good querying practices will keep the service responsive and working well for all clients.
For example (in English):
Good query: Return all NFL news stories for the past hour. Bad query: Return all NFL news stories for the past month. Really bad query: Every 10 minutes, return all NFL news stories for the past month.
Avoid repeating queries that are bound to return the same documents over and over again.
Help with FeedFetcher issues.
If you are having problems with FeedFetcher or are not seeing data you expect, it always helps to attach your configuration file and any appropriate log files so that we can diagnose the issue correctly.