API¶
The BuildHub API is just a Kinto instance with a collection of records, coupled with ElasticSearch. A series of jobs is charge of keeping those records up to date when new releases are published.
Servers¶
- Production: https://buildhub.prod.mozaws.net/v1/ (catalog-prod)
- Stage: https://buildhub.stage.mozaws.net/v1/ (catalog-stage)
- Development: https://kinto-ota.dev.mozaws.net/v1/ (catalog-dev)
Clients¶
Since it is an HTTP API and the records are public, you can simply use any HTTP client, like curl or HTTPie.
But for more convenience, especially regarding pagination and error handling, some dedicated libraries are also available:
- kinto-http.js (JavaScript)
- kinto-http.py (Python)
- kinto-http.rs (Rust)
- kinto-http.java (Java)
Data¶
A single record has the following fields (see below for more details):
{
"id": "firefox_beta_50-0b11_macosx_el",
"schema": 1497453926485,
"last_modified": 1498140377629,
"target": {
"platform": "macosx",
"os": "mac",
"version": "50.0b11",
"channel": "beta",
"locale": "el"
},
"build": {
"id": "20161027110534",
"date": "2016-10-27T11:05:34Z",
"number": 2,
"as": "$(CC)",
"ld": "ldd",
"cc": "/usr/bin/ccache /home/worker/workspace/build/src/gcc/bin/gcc -std=gnu99",
"cxx": "/usr/bin/ccache /home/worker/workspace/build/src/gcc/bin/g++ -std=gnu++11",
"host": "x86_64-pc-linux-gnu",
"target": "x86_64-pc-linux-gnu",
},
"source": {
"tree": "releases/mozilla-beta",
"product": "firefox",
"revision": "829a3f99f2606759305e3db204185242566a4ca6",
"repository": "https://hg.mozilla.org/releases/mozilla-beta"
},
"download": {
"mimetype": "application/x-apple-diskimage",
"date": "2016-10-28T00:56:42Z",
"size": 86180614,
"url": "https://archive.mozilla.org/pub/firefox/releases/50.0b11/mac/el/Firefox 50.0b11.dmg"
}
}
Listing records¶
Elasticsearch API¶
An ElasticSearch endpoint is available for faster and more powerful queries. It powers the online catalog.
Simple queries can be done with the QueryString:
Or via a POST
request:
echo '{
"query": {
"bool": {
"filter": [{
"term": {
"source.product": "devedition"
}
}]
}
},
"size": 100
}' | http POST $SERVER/buckets/build-hub/collections/releases/search
Note
For aggregations (ie. distinct values) there is no need to retrieve the whole set of results. For example:
echo '{
"aggs": {
"platforms": {
"terms": {
"field": "target.platform",
"size": 100
}
}
},
"size": 0
}' | http POST $SERVER/buckets/build-hub/collections/releases/search
Basic Kinto search API¶
A set of filters and pagination options can be used to query the list. The most notable features are:
- querystring filters (with
?field=value
or dedicated operators like?min_field=value
or?has_field=true
) - paginated list of records (follow the URL in the
Next-Page
response header) - fields selection (with
?_fields=
) - polling for changes (with
?_since=timestamp
filter or ETags in request headers)
Example queries¶
Is this an official build id?¶
In order to check that a build id exists, we’ll just check that it is mentioned in at least one record.
curl -s $SERVER/buckets/build-hub/collections/releases/search?q=build.id=20170713200529 | \
jq -r '.hits.total'
Or using the Kinto records endpoint, with the JavaScript client:
import KintoClient from "kinto-http";
const client = new KintoClient(SERVER);
const collection = client.bucket("build-hub").collection("releases");
records = await collection.listRecords({limit: 1, filters: {"build.id": "20110110192031"}});
console.log(records.length >= 1);
Or the Python client:
import kinto_http
client = kinto_http.Client("https://buildhub.prod.mozaws.net/v1")
records = client.get_records(**{"build.id": "20110110192031", "_limit": 1, "pages": 1},
bucket="build-hub", collection="releases")
print(len(records) >= 1)
What is the Mercurial commit ID of a build ID?¶
client = kinto_http.Client("https://buildhub.prod.mozaws.net/v1")
records = client.get_records(**{"build.id": "20110110192031", "_limit": 1, "pages": 1},
bucket="build-hub", collection="releases")
try:
revision = records[0]["source"]["revision"]
except IndexError:
raise ValueError("Unknown build id")
except KeyError:
raise ValueError("Unknown revision")
What locales are available for a certain version?¶
Using the ElasticSearch endpoint, with HTTPie and jq:
$ echo '{
"aggs": {
"locales": {
"terms": {
"field": "target.locale",
"size": 1000,
"order": {
"_term": "asc"
}
}
}
},
"query": {
"bool": {
"filter": [{
"term": {
"target.version": "57.0b9"
}
}, {
"term": {
"source.product": "firefox"
}
}]
}
},
"size": 0
}' | http POST $SERVER/buckets/build-hub/collections/releases/search | \
jq -r '.aggregations.locales.buckets[] | .key'
ach
af
an
ar
bn-BD
bn-IN
...
Using the Kinto records endpoint, with the Kinto JavaScript client:
import KintoClient from "kinto-http";
const client = new KintoClient("https://buildhub.prod.mozaws.net/v1");
const collection = client.bucket("build-hub").collection("releases");
const records = await collection.listRecords({filters: {"target.version": "53.0b9"}});
const locales = new Set(records.map(r => r.target.locale));
What are the available build ids of a specific version?¶
Using the ElasticSearch endpoint, with Python aiohttp:
async def fetch_build_ids(session, product, version):
query = {
"aggs": {
"build_ids": {
"terms": {
"field": "build.id",
"size": 100000,
"order": {
"_term": "desc"
}
}
}
},
"query": {
"bool": {
"filter": [{
"term": {
"target.version": version
}
}, {
"term": {
"source.product": product
}
}]
}
},
"size": 0,
}
async with session.post(SERVER_URL, data=json.dumps(query)) as response:
data = await response.json()
aggs = data['aggregations']['build_ids']['buckets']
buildids = [r['key'] for r in aggs]
return buildids
Using the Kinto records endpoint, with curl and jq:
$ curl -s "${SERVER}/buckets/build-hub/collections/releases/records?target.version=56.0b12" | \
jq -r '.data[] | .build.id' | \
sort -u
20170914024831
More about the data schema¶
Field | Description |
id |
A unique ID for a build (see details). |
schema |
The schema version when the record was added to the database. |
last_modified |
The timestamp incremented when the record was created/modified. |
source |
Information about the source code version used to build the release. |
source.product |
One of firefox , thunderbird , fennec or devedition |
source.revision |
Optional Mercurial changeset |
source.repository |
Optional Mercurial repository |
source.tree |
Optional Mercurial tree |
target |
Major information about the release. |
target.version |
Public version number |
target.locale |
Locale name |
target.channel |
AUS update channel name |
target.os |
Operating system |
target.platform |
OS and CPU architecture |
build |
Information about the build itself. |
build.id |
Optional Build identifier. |
build.date |
Optional Build date time. |
build.number |
Optional Release candidate number. |
build.as |
Optional Assembler executable |
build.ld |
Optional Linker executable |
build.cc |
Optional C compiler command |
build.cxx |
Optional C++ compiler command |
build.host |
Optional Compiler host alias (cpu)-(vendor)-(os) |
build.target |
Optional Target host alias (cpu)-(vendor)-(os) |
download |
Information about the resulting downloadable archive. |
download.url |
Public archive URL |
download.size |
In Bytes |
download.mimetype |
File type |
download.date |
Publication date |
The complete JSON schema is available in the collection metadata:
The records added to the collection will be validated against that schema.
More about the release record ID¶
If you have some information about a release, you might want to guess its ID directly in order to fetch the individual record directly.
The unique ID of a release is the following:
{PRODUCT_NAME}_{CHANNEL}_{VERSION}_{PLATFORM}_{LOCALE}
{PRODUCT_NAME}
: It can be eitherfirefox
,fennec
orthunderbird
{CHANNEL}
: It can be eitheraurora
,beta
,nightly
,nightly-old-id
The channel is not part of the ID forrelease
andesr
builds{VERSION}
: It is the full version of the build. Dots are replaced by-
i.e55-0-1
,55-1b2
For nightly we use the date and time of the build as a version prefix. i.e:2017-06-01-10-02-05_55-0a1
{PLATFORM}
: It is the target platform. i.e:macosx
,android-arm
,android-api-15
,win32
,win64
,linux-i386
, etc.{LOCALE}
: It is the locale code. i.efr-fr
,en-us
All dots are replaced with dashes and all string are in lowercase.
Here are some example of release ID:
firefox_nightly_2017-05-03-03-02-12_55-0a1_win64_en-us
thunderbird_52-0-1_linux-x86_64_en-us
firefox_aurora_54-0a2_macosx_en-us
firefox_beta_52-0b6_linux-x86_64_en-us
firefox_50-0rc1_linux-x86_64_fr
firefox_52-0esr_linux-x86_64_en-us
fennec_nightly-old-id_2017-05-30-10-01-27_55-0a1_android-api-15_multi