Data Sources API

The Data Sources API provides read and write access to sources, datasources and scans.

Source-specific notes

There are several types of sources. Each exposes the same set of core fields, but some fields are only exposed or accepted for specific types of sources. Everything in options is optional.

Info sources

Info sources merely provide a way to point humans to some data. They are never scanned by Koordinates; they don’t have datasources; and you can’t create layers from them.

CIFS, URL, ArcGIS, and WFS source options

All these source types can accept credentials in an options field:

options:
- username : username to access this server.
- password : password for this server.
- ssl_verify : A boolean specifying whether Koordinates should verify the SSL certificate of the host when scanning this source. Defaults to true.

Username and password are write-only. They are never returned to clients via the API.

ArcGIS Server

url_remote : The URL of your ArcGIS server, usually ending with/rest/api/services/. Must not contain credentials.
options : In addition to the options listed above, ArcGIS also accepts:
- folders : A JSON array of folders to scan. If not given, the whole server will be scanned.

CIFS

url_remote : A URL of the form "cifs://host/sharename", optionally with extra path segments. Must not contain credentials.

PostgreSQL

url_remote : A URL of the form "postgresql://host:port/database". Must not contain credentials.
options :
- username : username to access the PostgreSQL database.
- password : postgres database password.
- schemas : A JSON array containing schema names, e.g.: "schemas": ["public", "other_schema"]

URL

A URL source defines an FTP, HTTP or HTTPS url to an archive Koordinates can scan for data.

Currently accepts these archives:

.zip
.tar.gz, .tgz
.7z

URL sources accept the following extra fields:

url_remote : The URL to download the archive. Must not contain credentials.

WFS Server

url_remote : The URL to the WFS server endpoint. Must not contain credentials.

List and Create Sources

GET /sources/

List Sources

Returns a list of all the sources available to you.

Response 200

Headers

Content-Type: application/json

Body

[
    {
        "id": 1, 
        "url": "https://{domain}/services/api/v1/sources/1/", 
        "name": "harbour-images.zip", 
        "type": "upload", 
        "description": "", 
        "description_html": "", 
        "categories": [], 
        "user": {
            "id": 3, 
            "name": "Bob Jones"
        }, 
        "last_scanned_at": null, 
        "scan_schedule": null, 
        "options": {}, 
        "metadata": null
    }
]

POST /sources/

Create a Source

To create a source, post to this view.

Parameters

type: string (required). The type of source to create. Example: info. Choices: info upload cifs arcgis wfs postgres
name: string (required). The name of the new source.
group: string|id (optional)
user: string|id (optional). The group or user (at least one required) that owns the source, by ID or API URL.
url_remote: string (optional). The remote URL to the source, not required for info or uploadsources
options: object (optional). Refer to the source specific notes above.
scan_schedule: string (optional). A Cron expression specifying how frequently the source should be scanned. All normal cron values are supported but the ‘minutes’ value is currently ignored. Predefined scheduling definitions are supported except for "@reboot".

Request

Headers

Content-Type: application/json

Body

{
    "name": "Bob's ArcGIS 10 Server",
    "type": "arcgis",
    "description": "All of Bob's data", 
    "categories": [], 
    "user": 3,
    "options": {
        "username": "bob",
        "password": "sekret_p@55w0rd"
    },
    "url_remote": "https://bob.example.com/ArcGIS/rest/services",
    "scan_schedule": "@daily"
}

Response 201

Headers

Content-Type: application/json

Body

{
    "id": 2, 
    "url": "https://{domain}/services/api/v1/sources/2/", 
    "name": "Bob's ArcGIS 10 Server",
    "type": "arcgis", 
    "description": "All of Bob's data", 
    "description_html": "<p>All of Bob's data</p>", 
    "categories": [], 
    "user": {
        "id": 3, 
        "name": "Bob Jones"
    },  
    "last_scanned_at": null,
    "scan_schedule": "@daily",
    "options": {}, 
    "metadata": null, 
    "url_remote": "https://bob.example.com/ArcGIS/rest/services"
}

POST /sources/

Create an upload source

To upload files to the API, use the multipart/form-data content-type instead of JSON.

Your multipart request should include one part with name="source", which contains the JSON object with the required fields, as per above. Other parts should include their filename in the Content-Disposition header.

Each file you upload can either be a normal file, or an archive containing more files. We currently support these archive types:

.zip
.tar.gz, .tgz
.7z

Request

Headers

Content-Type: multipart/form-data; boundary=b0017bc3bab14fed95910bfcc4a1ddd8

Body

--b0017bc3bab14fed95910bfcc4a1ddd8
Content-Disposition: form-data; name="abc"; filename="points-of-interest.zip"
Content-Type: application/x-zip

(binary data not shown)
--b0017bc3bab14fed95910bfcc4a1ddd8
Content-Disposition: form-data; name="source"
Content-Type: application/json

{
    "name": "points-of-interest", 
    "type": "upload", 
    "user": 3
}
--b0017bc3bab14fed95910bfcc4a1ddd8--

Response 201

Headers

Content-Type: application/json

Body

{
    "id": 3, 
    "url": "https://{domain}/services/api/v1/sources/3/", 
    "name": "points-of-interest", 
    "type": "upload", 
    "description": "", 
    "description_html": "", 
    "categories": [], 
    "user": {
        "id": 3, 
        "name": "Bob Jones"
    }, 
    "last_scanned_at": null, 
    "scan_schedule": null, 
    "options": {}, 
    "metadata": null
}

Source detail

GET /sources/{id}/

Get source detail

Response 200

Body

{
    "id": 1, 
    "url": "https://{domain}/services/api/v1/sources/1/", 
    "name": "harbour-images.zip", 
    "type": "upload", 
    "description": "", 
    "description_html": "", 
    "categories": [], 
    "user": {
        "id": 3, 
        "name": "Bob Jones"
    }, 
    "last_scanned_at": null, 
    "scan_schedule": null, 
    "options": {}, 
    "metadata": null
}

PUT /sources/ {id}/

Edit a source

Note that:

You can’t change the type of a source.
You can’t upload more files into an upload source. If you need to upload more files you should create a new source.

The following fields are editable (other fields will be ignored):

name
description
categories
user / group
scan_schedule
options (note: existing options will be preserved if they’re not specified here)

Request

Headers

Content-Type: application/json

Body

{
    "description": "Harbour Imagery (4m/px)"
}

Response 200

Headers

Content-Type: application/json

Body

{
    "id": 1, 
    "url": "https://{domain}/services/api/v1/sources/1/", 
    "name": "harbour-images.zip", 
    "type": "upload", 
    "description": "Harbour Imagery (4m/px)",
    "description_html": "<p>Harbour Imagery (4m/px)</p>",
    "categories": [], 
    "user": {
        "id": 3, 
        "name": "Bob Jones"
    }, 
    "last_scanned_at": null, 
    "scan_schedule": null, 
    "options": {}, 
    "metadata": null
}

DELETE /sources/{id}/

Delete a source

A ‘scan’ occurs when we inspect a source to find content that can be used to create layers, tables, and documents.

GET /sources/{id}/scans/

List scans

Returns a list of scans for a source.

Notable fields:

status : The current status of this scan. Valid values are:
- running
- completed
- cancelled
- error
change_counts : counts of datasources deleted, updated and added in this scan. This is only guaranteed to be present in completed scans. For other scans it may be null or the counts may all be 0.

Response 200

Headers

Content-Type: application/json

Body

[
    {
        "id": 4, 
        "url": "https://{domain}/services/api/v1/sources/1/scans/4/",
        "status": "completed", 
        "started_at": "2013-10-13T02:53:28Z", 
        "completed_at": "2012-10-13T02:53:47Z", 
        "change_counts": {
            "deleted": 0, 
            "updated": 3, 
            "added": 0
        }, 
        "log": "https://{domain}/services/api/v1/sources/1/scans/4/log/",
    }
]

POST /sources/{id}/scans/

Start scan

Starts a scan for this source. No post data is required.

Response 201

Headers

Content-Type: application/json

Body

{
    "id": 4, 
    "url": "https://{domain}/services/api/v1/sources/1/scans/4/",
    "status": "completed", 
    "started_at": "2013-10-13T02:53:28Z", 
    "completed_at": "2012-10-13T02:53:47Z", 
    "change_counts": {
        "deleted": 0, 
        "updated": 3, 
        "added": 0
    }, 
    "log": "https://{domain}/services/api/v1/sources/1/scans/4/log/"
}

Scan detail

GET sources/{id}/scans/{scan}

Get scan detail

Information about a particular scan. This is the same as an item in the list view above, except it also includes a progress indicator, which is a float between 0 and 1. Progress is approximate.

Response 200

Headers

Content-Type: application/json

Body

{
    "id": 4, 
    "url": "https://{domain}/services/api/v1/sources/1/scans/4/",
    "status": "completed", 
    "started_at": "2013-10-13T02:53:28Z", 
    "completed_at": "2012-10-13T02:53:47Z", 
    "change_counts": {
        "deleted": 0, 
        "updated": 3, 
        "added": 0
    }, 
    "log": "https://{domain}/services/api/v1/sources/1/scans/4/log/",
    "progress": 1.0
}

DELETE /sources/{id}/scans/{scan}

Cancel or delete scan

Cancels an in-progress scan.

Response 204

Scan log

GET /sources/{id}/scans/{scan}/log/

Get scan log

Gets a log for a scan. By default this is returned as text/plain, but you can request JSON via the Accept header.

Response 200

Headers

Content-Type: text/plain

Body

2012-11-14 15:53:28: Scraping ArcGIS server at https://bob.example.com/ArcGIS/rest/services
2012-11-14 15:53:29: Looking for layers in folder: Bob/MapServer
2012-11-14 15:53:29: Requesting access to https://bob.example.com/ArcGIS/rest/services/
2012-11-14 15:53:29:     get: https://bob.example.com/ArcGIS/rest/services/Bob/MapServer?f=json
2012-11-14 15:53:29: crawled in 0.445s
2012-11-14 15:53:29:   Looking in folder: Bob/MapServer
2012-11-14 15:53:29:   Found layer: 0 - 'Null Island Polygons'
2012-11-14 15:53:29:     get: https://bob.example.com/ArcGIS/rest/services/Bob/MapServer/0?f=json
2012-11-14 15:53:30: crawled in 0.447s
2012-11-14 15:53:30:     get: https://bob.example.com/ArcGIS/rest/services/Bob/MapServer/0/query?returnGeometry=false&outFields=%2A&returnIdsOnly=true&where=1%3D1&f=json
2012-11-14 15:53:31: crawled in 0.601s
2012-11-14 15:53:47: Finishing up
2012-11-14 15:53:47: Updating 1 new or changed datasets
2012-11-14 15:53:47: Updating user interface
2012-11-14 15:53:47: Finished!

Data sources

The API for datasources is read-only. Datasources are updated by scanning your source. If you don’t see what you expect here, check that your source has been scanned recently.

GET /sources/{id}/datasource/

List datasources

Lists datasources for this source.

Response 200

Headers

Content-Type: application/json

Body

[
    {
        "id": 1, 
        "url": "https://{domain}/services/api/v1/sources/1/datasources/1/", 
        "title": "", 
        "data_type": "vector", 
        "geometry_type": "polygon", 
        "updated_at": "2013-10-13T02:53:46Z", 
        "arcgis_folder_service_path": "Bob/MapServer", 
        "arcgis_geometry_type": "esriGeometryPolygon", 
        "arcgis_layer_name": "Null Island", 
        "arcgis_type": "Feature Layer"
    }
]

Datasource detail

GET /sources/{id}/datasources/{datasource}/

Get datasource detail

Detailed information for a given datasource. Includes all the info shown in the list above, as well as much more information describing the datasource.

Field notes

type is the source type, not to be confused with data_type, format or geometry_type
data_type refers to the broad type of data. Different types of data require significantly different handling. Current data types are:
- table
- vector
- raster
- grid :
- attribute-grid
format is a string describing the media type of this datasource.
geometry_type describes the type of geometry in vector datasources. It is null for non-vector datasources. Possible values are:
- point
- linestring
- polygon
- multipoint
- multilinestring
- multipolygon
Spatial datasources may or may not have a known Coordinate Reference System (crs).
extent is a GeoJSON geometry. It may be null for aspatial datasources. The coordinate system for this geometry is the same as that in the crs field, which may not be known.
For vector and table datasources, fields is a list of JSON objects. It will be null for other datasources.

Response 200

Headers

Content-Type: application/json

Body

{
    "id": 1, 
    "url": "https://cds.dev.kx.gd/services/api/v1/sources/36/datasources/1/", 
    "title": "Null Island Polygons", 
    "data_type": "vector", 
    "geometry_type": "polygon", 
    "updated_at": "2012-11-14T02:53:46Z", 
    "format": "application/json", 
    "crs": "EPSG:4326", 
    "fields": [
        {
            "type": "integer", 
            "name": "OBJECTID"
        }, 
        {
            "type": "unicode", 
            "name": "name"
        }, 
        {
            "type": "geometry", 
            "name": "SHAPE"
        }
    ], 
    "num_dims": 2, 
    "num_features": 1500, 
    "num_bands": null, 
    "extent": {
        "type": "Polygon",
        "coordinates": [
            [
                [-1.0, -1.0], 
                [-1.0, 1.0], 
                [1.0, 1.0], 
                [1.0, -1.0], 
                [-1.0, -1.0]
            ]
        ]
    }, 
    "metadata": null, 
    "arcgis_folder_service_path": "Bob/MapServer", 
    "arcgis_geometry_type": "esriGeometryPolygon", 
    "arcgis_layer_name": "Geology", 
    "arcgis_type": "Feature Layer"
}

Datasource metadata

GET /sources/{id}/datasources/{datasource}/metadata/

Get datasource metadata

Some datasources can have attached XML metadata. This is attached by the CIFS and Upload scanners, if a datasource has an .xml file with the same basename. For example:

my-points-with-metadata.geojson
my-points-with-metadata.xml

As with sources, datasource XML must be in one of these three formats:

ISO 19115/19139
FGDC CSDGM
Dublin Core

Response 200

Body

<dc xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/">
    <title></title>
    <creator>Bob Jones and Associates</creator>
    <subject>hydro</subject>
    <description>Some nice colourful aerial photos of the Auckland Harbour.</description>
    <date>2013-01-01</date>
    <type>raster</type>
    <language>eng</language>
</dc>

Site Admin APIs

Data Sources API

Source-specific notes

Info sources

CIFS, URL, ArcGIS, and WFS source options

ArcGIS Server

CIFS

PostgreSQL

URL

WFS Server

List and Create Sources

Parameters

Source detail

Scan detail

Scan log

Data sources

Datasource detail

Datasource metadata