This documentation assumes you are working towards launching an bot using your own infrastructure. Our API is very simple and straightforward, but if you have any questions, please contact us.

Before doing any actual work, do not forget to contact us first. We have a lot of bots running in invisible test mode and - though unlikely - it could be that your idea was already implemented by someone else.


Pick any solution you like: Heroku, Amazon, Digital Ocean or whatever.

Your endpoint

In order for us to send anything to you, we need to know your endpoint. It cannot be dynamic. Every bot of yours can have its own endpoint.


Before we start exchanging data with each other, we will provide you with a secret handshake code, which we will need to use to identify each other properly.

What sends your way

When a user launches your bot on our site you will receive the following 'application/json' POST request:

job_request_idA numerical identificator of a Job Request. You need to store this number and send it back to us later.
bot_idThe ID of your bot as it is stored on your server. Something like parse_nodes or 112233.
data_file_urlTemporary link with user submitted data for you to download. We support JSON ('application/json' MIME) and CSV ('text/comma-separated-values' MIME) file types.
app_secretA secure string. We give it to you.

Additionally, if your bot supports custom options, we will send them to you as a hash:

options : {option_one: 1, option_two: 'whatever'} etc.

For standartization, all data is being sent to you in as string values only - does not matter if it is a boolean or an integer.

Upon successful submission your server will need to reply with a 201 success code and provide an optional message:

message: 'started'

If your server is not repying, we will repeat the request four more times for the next several hours before canceling it.

{  "job_request_id": "4242",  "data_file_url": "",  "bot_id": "your_bot_id_or_name",  "app_secret": "abc123ABC789=",    "options": {      "allow_subdomains": "true",      "max_nodes": "1000"     }}

What we're expecting back

Upon successful job completion your server will need to send us a POST request with a JSON file location (type 'application/json') to the following endpoint:

with the following data:

job_request_idThe identificator of the job request you were working on (it's the same ID you got from our POST request before.)
payloadLink to file with the output data. This field can be blank in case your bot failed to do the job properly at some point. You will need to provide additional details in that case (see below).
app_secretYour secret identificator, the same you used above.

If the bot failed at some point you should additionally provide the following:

errorcodeThis is a numerical value, currently we are only using 1 to distinct between failed and successful jobs. We are considering including more error codes in the future. If the job was completed successfully you may skip this field or send us a 0.
messageProvide a meaningful error message. This will help with debugging later on.

These two parameters are only required if the bot failed to deliver the payload.

{  "job_request_id": "4242",  "app_secret": "abc123ABC789=",  "payload": ""  "message": "Success!"  "errorcode": "1" # only send '1' if processing unsuccessful and payload is empty. Provide a message. Ignore if operation successful.}

JSON format

To be compatible with our system, your JSON file has to have the following schema:

[ {"key": "value 1"}, {"key": "value 2"} ]


To take off the complexities off your plate and allow you to focus on the bot effectiveness, we try to perform all the necessary validations on our servers. However, you can absolutely implement all sorts of validations on your side too.

Making it available to the public

Providing your details

After the server-side of your bot is working, we need to add it to our front end. We will need the following information from you:

  • Your bot's name.
  • Short description. We will write a longer version of it.
  • Additional options, if any, and their formats (string, integer, etc.).
  • bot ID as you store it on your server.

Providing Custom options

As mentioned before, you can allow users to provide additional settings by using the options hash. Additionally, you can also specify defaults and options that cannot be modified by users but are stored on as hidden defaults that could be adjusted later. At the current moment these two types of options need to be set manually.