Bulk Records Broker (BRB)

Note This information is only intended for partnering organizations who have entered into record sharing agreements with FamilySearch.

The Bulk Records Broker (BRB) delivers genealogical data in GEDCOM X record set files. A customer reads from an Atom feed to get a list of URLs of the available files, then fetches and processes the files. The records delivered are divided into collections. Each collection is delivered with one metadata file (a collection description document) and any number of record/image item files. Each record file contains any number of records which will consist of data such as persons, relationships and fields. Each image item file contains a list of simple records that consist of a list of fields that describe attributes of an image that could be used for image browsing. Each record has its own source description that contains the description of the record and a reference to its collection.

Reading from BRB

Obtain an access token by authenticating to FamilySearch and process the Atom feed at https://beta.familysearch.org/brb/feed. You must be authenticated and have a valid access token in order to access the feed. The feed will return an Atom entry for each collection. Each entry describes a collection and has a link to the metadata associated with the collection. Iterate through each entry. The records link takes you to a file with a list of chunked records.

Two-level Feed

The BRB Atom feed has two levels: one listing the collections that have changed since the last request, along with the current time of the request, and one that provides a feed of record chunk files and a feed of image item chunk files.

Collection List Feed

The collection list feed provides one entry for each collection that has had any changes since the last request. After all collection entries have been processed successfully, the feed provides a link element, with the link relation of "next" (<link rel="next" href="https://beta.familysearch.org/brb/feed?since46"/>), that allows you to read the next set of changes.

When there are no more changes to send and you are "caught up", the server will return a 204 (No Content) response indicating there are no new changes (entries) to process. You can continue to check for new content at the same URL until there is content.

There may be times when FamilySearch needs to unpublish a collection or "unshare" it with its partners. When reading the collection list feed, if there is a "delete" element set to "true" within a collection entry, you are obligated to delete your copy of this collection, if you have it.

Each collection entry in the feed will have a link with "rel" of "alternate" which contains the URI of the collection. It may also contain one or more of the following elements:

Collection Description URL
- Find the <link rel="collectionDescription" link and use it to update the collection's titles and descriptions.
- Example: <link rel="collectionDescription" href="https"//familysearch.org/brb/collection/1"/>
Record Chunk Feed URL

Find the <link rel="records" link and use it to get the record chunk feed. The feed will have entries that link to the record set files.
Example: <link rel="records" href="https"//familysearch.org/brb/feed?collectionID=1&entryType=Record&since=0&until=49"/>

Image Item Chunk Feed URL

Find the <link rel="imageItems" link and use it to get the image chunk feed. The feed will have entries that link to the image items.
Example: <link rel="imageItems" href="https"//familysearch.org/brb/feed?collectionID=1&entryType=ImageItems&since=0&until=49"/>

Delete Collection Element

Delete the entire collection, including records and image items.
Example: <fs:delete>true</fs:delete>

Record Chunk File "catch-up" URL

If you already have this collection in your data set, use the normal record chunk feed URL above. Otherwise, use this one instead. This allows you to catch up on data you may have missed.
Example: <link rel="recordsCatchUp" href="https"//familysearch.org/brb/feed?collectionID=1&entryType=RecordChunk&until=49"/>

Image Item Chunk File "catch-up" URL

If you already have this collection in your data set, use the normal image item chunk feed URL above. Otherwise, use this one instead. This allows you to catch up on data you may have missed.
Example: <link rel="imageItemsCatchUp" href="https"//familysearch.org/brb/feed?collectionID=1&entryType=ImageItemChunk&until=49"/>

Catch-up feeds

Sometimes the permissions on a collection will change so that it can now be shared with additional consumers. When this happens, a series of record or image item chunk files may exist that a consumer may have missed out on. This is because the files were made available before the current timestamp when the consumer did not have access to that collection. BRB knows when permissions have changed and knows whether each consumer now has access to that collection, but doesn't know if they previously had access to that collection or not. Therefore, whenever permissions on a collection change, a "catch-up" record/image item feed link is provided.

Example Collection List Feed

The following is an example of a level 1 feed with several kinds of updates:

Collection 1 is being deleted.
Collection 6 has had collection description and record changes.
Collection 5 has had a permission change so in addition to an updated collection description and updated records, there is a catch-up feed for both records and image items. (Apparently, there were no image item updates in the current round of changes, but there were some previously, which is why there is an image item catch-up feed but no normal image item feed.)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:fs="http://familysearch.org/">
  <author>
    <name>FamilySearch</name>
  </author>
  <id>https://beta.familysearch.org/brb/feed</id>
  <link rel="status" href="https://beta.familysearch.org/brb/feed"/>
  <link rel="self" href="https://beta.familysearch.org/brb/feed?since=46"/>
  <title>BRB Collection List Feed</title>
  <updated>2014-05-19T19:07:21.305Z</updated>
  <entry>
    <fs:delete>true</fs:delete>
    <id>brb:1</id>
    <link rel="alternate" href="https://beta.familysearch.org/brb/collection/1"/>
    <title>Changes for Collection 1</title>
    <updated>2014-05-19T19:07:21.305Z</updated>
  </entry>
  <entry>
    <id>brb:6</id>
    <link rel="alternate" href="https://beta.familysearch.org/brb/collection/6"/>
    <link rel="collectionDescription" href="https://beta.familysearch.org/brb/collection/6"/>
    <link rel="records" href="https://beta.familysearch.org/brb/feed?collectionId=6&type=RecordChunk&until=46"/>
    <title>Changes for Collection 6</title>
    <updated>2014-05-19T16:07:21.305Z</updated>
  </entry>
  <entry>
    <id>brb:5</id>
    <link rel="alternate" href="https://beta.familysearch.org/brb/collection/5"/>
    <link rel="collectionDescription" href="https://beta.familysearch.org/brb/collection/5"/>
    <link rel="records" href="https://beta.familysearch.org/brb/feed?collectionId=5&type=RecordChunk&until=46"/>
    <link rel="recordsCatchUp" href="https://beta.familysearch.org/brb/feed?collectionId=5&type=RecordChunk&until=46"/>
    <link rel="imageItemsCatchUp" href="https://beta.familysearch.org/brb/feed?collectionId=5&type=ImageItemChunk&until=46"/>
    <title>Changes for Collection 5</title>
    <updated>2014-05-19T19:07:21.305Z</updated>
  </entry>
  </feed>

Record/Image Item Chunk Feed

Records and image items are delivered in GEDCOM X Record Set (or "chunk") files. Each file contains a list of record elements, each of which represents a historical record or image item.

The BRB record/image item chunk feed provides URLs to record chunk and image item chunk files, respectively. The URLs expire after some time period (currently 24 hours). The files in the record/image item chunk feed should be processed sequentially in order to avoid overwriting a record or image item.

Example Record Chunk Feed

The following is an example of a record chunk feed containing three entries that each point to a record chunk file:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:fs="http://familysearch.org/">
  <author>
    <name>FamilySearch</name>
  </author>
  <id>fsRecordChunk: 0</id>
  <link rel="status" href="https://beta.familysearch.org/brb/feed"/>
  <link rel="self" href="https://beta.familysearch.org/brb/feed?collectionId=6&type=RecordChunk&since=0&until=46"/>
  <title>BRB Record Chunk Feed for Collection 6</title>
  <updated>2014-05-19T19:11:14.894Z</updated>
  <entry>
    <id>fs:10</id>
    <link rel="alternate" href="https://brb-dev.s3.amazonaws.com/6/RecordChunk/1.gz?Expires=1400613074&AWSAccessKeyId=AKIAITQXJ6FCVP4UANKQ&Signature=GCUypds4/3hlNoUBoJAxXTGOUWM%3D"/>
    <title>Chunk 1</title>
    <updated>2014-04-10T15:09:59Z</updated>
  </entry>
  <entry>
    <id>fs:25</id>
    <link rel="alternate" href="https://brb-dev.s3.amazonaws.com/6/RecordChunk/2.gz?Expires=1400613074&AWSAccessKeyId=AKIAITQXJ6FCVP4UANKQ&Signature=BgZ8QGn7u17E1itHTmixfSKoDAs%3D"/>
    <title>Chunk 2</title>
    <updated>2014-04-30T16:45:03Z</updated>
  </entry>
  <entry>
    <id>fs:44</id>
    <link rel="alternate" href="https://brb-dev.s3.amazonaws.com/6/RecordChunk/3.gz?Expires=1400613074&AWSAccessKeyId=AKIAITQXJ6FCVP4UANKQ&Signature=jfO%2Brmrad/%2BXfuE0WPgZE/OgC4Q%3D"/>
    <title>Chunk 3</title>
    <updated>2014-04-08T14:55:46Z</updated>
  </entry>
  </feed>

Processing the Feeds

Each collection list feed entry represents all the changes in all the collections over a given time range. Each record/image item chunk feed URL is created so that it ends at the same time as the collection list feed. This ensures that the collection list feed and the record/image item chunk feeds have a synchronized starting and ending point. Catch-up feeds are the exception since they start at the beginning. For best results, process the files in the following order:

Read the collection list feed, either from the beginning or using the "next" element from the previous read.

If a 204 "No Content" response is returned, wait and repeat step 1 with the same URL until there is a 200 response with content.

For each collection entry:

Process the collection in one of three ways:

If the entry has a <fs:delete>true</fs:delete>, delete the collection, if you have it.
Otherwise, if there are catch-up links and you don't already have the collection, use the catch-up link to:

Read the catch-up image item chunk feed. For each entry, read the image item chunk file and process it.
Read the catch-up record chunk feed. For each entry, read the record chunk file and process it.

Otherwise, if there are record or image items feeds:

Read the image items chunk feed. For each entry, read the image item chunk file and process it.
Read the record chunk feed. For each entry, read the record chunk file and process it.

Wait until all collections have been completely processed, including record/image chunk feeds or catch-up record/image feeds.
Use the "next" URL and repeat at step 1.

For information on processing a GEDCOM X record set, click here.