API Download Scripts

API Download Scripts

Do Not Download Listings & Store Them Locally

The Oodle API is designed to be a realtime service. Do not use scripts to download listings and store them locally. Here's why.

Downloading and storing results in your own database is a bad idea for your API application. Things may start well, but you'll soon run into many bugs and other issues that cause a poor user experience for your users. A good percentage of API developers who report issues to us are the ones who use scripts or programs to "download all of the Oodle listings".

It's no surprise why these scripts start to fail: Oodle classifieds data is very dynamic. A listing that is live on Oodle right now may be gone entirely in an hour. If you download Oodle listings, store them locally, and then show them to your users from that local database, it doesn't take long before you're showing your users all sorts of bad information.

Specific Problems With This Practice

Below, we outline reasons why you should call the API in realtime and not download Oodle API listings and store them locally:
  1. Old Listings
    Classifieds transactions themselves are inherently time-sensitive. When someone is selling an item on Oodle, chances are good that -- once sold -- the user will come back to Oodle and deactivate the listing. As an API developer, if you downloaded that listing yesterday through a script, and you're still storing it and showing it to your users, you're now showing an item that has already sold and no longer exists on Oodle. So when the user clicks the listing on your site to see the item, they'll get taken to a page at Oodle that says the ad has been removed. That's a terrible user experience for your users.
  2. Spam & Fraud
    At Oodle, we take spam, trust, safety very seriously. Like many classifieds sites, we always encounter some users that try to use the service to start fraudulent schemes against potential buyers. But unlike many other classifieds sites, we have some very advanced controls in place to find those fraudulent listings and eliminate them en masse.

    Not only do we have human staffers scanning for spam and frauds, we also have automated processes that are constantly analyzing listings, using heuristics to identify spam and fraud. And Oodle users themselves can report suspicious listings on the site.

    In all cases, when we identify listings that are fraudulent, we remove them from Oodle right away. It's a *very* fluid data set. So similar to #1 above, if you download Oodle listings today, they may include spam and frauds that we've since removed. But you're still showing those spammy and fraudulent listings to your users.

  3. Incomplete or Incorrect Data
    We get listings from a variety of sources. In addition to the general crawlers that we have running to pull data from various sites, we also have folks that are sending us data directly in feeds. If the price of a car changes from $15000 to $12000 at a car dealership that's in our index, we'll see that update and the API will reflect that immediately. But if you downloaded the listing yesterday, or even 15 minutes ago, you're still showing the old price.
  4. Broken URLs
    We occasionally change the structure of our URL's -- to allow us to do better reporting, to change the way we encode data on the listing URL, etc. We strive to maintain backwards compatibility with our links, but we do not guarantee it. If you downloaded a bunch of listings yesterday, your links may not work today due to any changes in the format of our URLs. Again, our API is built to be a realtime service and people who use it as such will never have a problem with this.
  5. Bugs In Your Application
    One bug we see repeatedly in our API support queue is from people who store Oodle listings locally in their own database, but then do so incorrectly, resulting in broken functionality for their users. Specifically, our redirect URLs can be up to 300 or 400 characters long. But many API developers who download and store Oodle results locally end up storing the URL in a field that is only 255 characters long. Each of those URLs are now broken for their users, because the URL is truncated and does not work correctly. Even if you avoid this particular issue, if you you download and store results locally, you may very well run into other issues based on incorrect assumptions about how to store listings data.

Avoid This Entirely - Use the Oodle API in Realtime!

If you call the API in realtime, you'll avoid all of these issues entirely. If you're displaying classifieds to your users, call us when the page is rendered. 90% of the calls to version 2 of the API return in less than 1 second. 60% of the calls it complete in less than 500ms. The new Oodle API is fast, it's reliable, and you can count on it to give you the data you need quickly.

And most importantly, the data you're getting back will be the latest, freshest, Oodle classifieds data. You won't run into any of the problems described above. And, in turn, your users will have a great classifieds experience.

Rate Limiting

Most web service APIs have some sort of rate limiting. That is to say, they'll allow a certain number of requests per second or a certain number of total requests per day.

At Oodle, we've previously been the exception to this rule. We've always wanted to give you -- the Oodle API developer -- the opportunity to develop a classifieds application that takes off in popularity and grows rapidly, without bounds.

As traffic to the Oodle API continues to grow, however, we need to maintain the ability to manage incoming traffic. It should be obvious that a handful of users with download scripts should not degrade the service for the hundreds of other API applications that are using it as a realtime service and are calling it as such.

In the future, we may decide to join the crowd and install rate limiting on the Oodle API. If so, any API download scripts that call the Oodle API in a rapid fire method are likely to be affected. Those scripts may stop working altogether.

Thanks!