Data Pipelining

From OpenSocial

Jump to: navigation, search

Contents

Summary

Data Pipelining is a declarative syntax for defining the data you want the container to provide to you for processing. Examples would be <os:PersonRequest id="@viewer"/> or <os:DataRequest href="http://www.developer.com/api">.

The data that is retrieved will be available in three contexts:

  1. Proxied content requests. This data will be POSTed to the developer party server with requests for proxied content.
  2. OpenSocial Templates. The data will be available as named variables in OpenSocial templates
  3. JavaScript. This data will be available in the JavaScript API

Spec Updated

Detailed Proposal

Developers will insert the following <os:*> namespaced tags within a <Content> block to specify what data should be provided to the gadget.

  • RESTful call equivalents:
  1. <os:PeopleRequest> - request to get profile information for a group or list of people
  2. <os:ViewerRequest>, <os:OwnerRequest> - request to get the viewer or owner's profile
  3. <os:ActivitiesRequest> - request to get activities
  4. <os:PersonAppDataRequest> - not needed due to Radically simplify the Persistence/AppData API
  • <os:DataRequest>, equivalent to gadgets.io.makeRequest
    • This is a call to an arbitrary URL 
<Content type="html">
  <script type="text/os-data">
    <os:ViewerRequest key="vwr" fields="name,birthday"/>
    <os:DataRequest key="mydata" href="http://developer.com/api"/>
  </script>
  ... HTML content here ...
</Content>

Note that we only have equivalents for REST calls that get data - calls that update data cannot be safely used in Data Pipelining.

Common request processing

All XML tags MUST have an @key string attribute. This is used to identify the data in the response back.

Calls with corresponding REST APIs have a 1:1 mapping between the XML attributes and the RESTful API parameters. See http://www.opensocial.org/Technical-Resources/opensocial-spec-v081/rpc-protocol#TOC-8.-Services for an overview of the REST calls.

Tag: <os:PeopleRequest>

Request to get profile information for a group or list of people

Attributes

  • @key {string}
  • @userId {list<string>} Comma delimited IDs of the users to use with "@groupId". Each item can be one of "@me", "@viewer", "@owner", or a user ID
  • @groupId {string} The group of users to get. Defaults to "@self", which returns the actual users listed in @userId. Other values are "@friends" to get friends, or any other string to get an arbitrary group
  • @fields {list<string>} Comma-delimited list of strings of OpenSocial profile fields to return. Optional.
  • @startPage {int} Current page to return results from. Optional.
  • @startIndex {int: Starting index to return results from. Optional. Behavior is undefined if you send both startPage and startIndex
  • @count {int} Number of people to return. Optional.
  • @sortBy {string} Specifies the field name whose value is be used to order the returned people. The sort order is determine by the sortOrder parameter. Optional.
  • @sortOrder {string} The order in which the sortBy parameter is applied. Can either be "ascending" or "descending", defaults to ascending. Optional.
  • @filterBy, @filterOp, @filterValue all {string} Filter to apply over users to retrieve, matches REST specification. Practically no container has this feature implemented.

Returns

  • An array of OpenSocial person JSON objects

Example: 

<os:PeopleRequest key="PagedFriends" userId="@owner" groupId="@friends" startPage="2" count="20"/>

Tags: <os:ViewerRequest>, <os:OwnerRequest>

Requests to get the viewer's or owner's profile.

Attributes:

  • @key {string}
  • @fields {List<String>}. Comma-delimited list of strings of OpenSocial profile fields to return. Optional.

Returns

  • OpenSocial person JSON object

Example: 

<os:ViewerRequest key="Viewer" fields="name, birthday"/>

Tag: <os:ActivitiesRequest>

Request to get activities

Fields

  • @key {string}
  • @userId {list<string>} Comma delimited IDs of the users to use with "@groupId". Each item can be one of "@me", "@viewer", "@owner", or a user ID. Required.
  • @groupId {string} The group of users to get activities for. Defaults to "@self", which returns the actual users listed in @userid. Other values are "@friends" to get friends, or any other string to get an arbitrary group. Optional.
  • @activityIds {list<String>} Comma delimited list of strings of activity ids to retreive. If set, @userid and @groupId will be ignored. Optional.
  • @appId {String} ID of app to get information for. Defaults to current app, and containers are not required to support other values. Optional.
  • @fields {list<string>} Comma-delimited list of strings of OpenSocial activity fields to return. Optional.
  • @startPage {int} Current page to return results from. Optional.
  • @startIndex {int: Starting index to return results from. Optional. Behavior is undefined if you send both startPage and startIndex
  • @count {int} Number of activities to return. Optional.

Returns

  • An array of OpenSocial activity JSON objects

Example: 

<os:ActivitiesRequests key="ViewerActivities" userid="@viewer" startIndex="40" count="20"/>

Tag: <os:DataRequest>

Request to get OpenSocial data, including person data, activity data, and any container-specific endpoints.

Fields

  • @key {string}
  • @method {string} The name of the REST endpoint and operation called. Implementations MUST support "people.get" and "activities.get", and MUST NOT support ".update", ".create", or ".delete" operations.

All other attributes will be passed as string parameters to the endpoint. See the OpenSocial RESTful protocol for information on the parameters available for each endpoint.

Returns

  • JSON object (typically, an array) representing the results of invoking the endpoint.

Example: 

<os:DataRequest key="PagedFriends" method="people.get" userId="@owner" groupId="@friends" startIndex="40" count="20"/>

Tag: <os:HttpRequest>

Request for arbitrary URL data, equivalent to gadgets.io.makeRequest().

This tag has additional attributes to match the functionality of gadgets.io.makeRequest (see http://code.google.com/apis/opensocial/docs/0.8/reference/gadgets/#gadgets.io.makeRequest and http://code.google.com/apis/opensocial/docs/0.8/reference/gadgets/#gadgets.io.RequestParameters)

Fields

  • @key {string}
  • @href {string} The URL to send data to. Optional if not set the method contains be a full RPC method (see @method)
  • @params {string} Additional URL parameters, of format "a=b&c=d". Optional.
  • @method {string} HTTP method to use, one of "get" or "post". Defaults to "get". Optional.
    • If method is "post", than the "@params" parameters are sent as POST content. If method is "get", then the params are appended to the URL.
    • Note that POSTing arbitrary content is not supported.
  • @format {string} Format data is returned in for processing, values are "json" or "text". Defaults to "json". Optional.
    • There is no ability to set arbitrary POST content, this just determined whether the key/value pair parameters are sent as a POST body.
    • Containers MAY optionally support full [RPC] methods, such as "people.get", or extensions, such as "email.get"
  • @refreshInterval {int} Period of time for which the cotnainer can cache the data. If this parameter is set, it overrides the HTTP response header sent back from the makeRequest(). Optional.
  • @authz. String, one of "none", "signed", "oauth", defaults to "none". This tells the gadget what authorization method to use when sending data to the remote server.
  • @sign_viewer {boolean} Sign the request and include the current viewer id. Defaults to true
  • @sign_owner {boolean} Sign the request and include the current owner id. Defaults to true
  • @oauth_service_name {string} Identifies the service element in the gadget spec to use for this request. Default is ""
  • @oauth_token_name {string} Identifies the OAuth token used to make a request to the service provider. Default is ""
  • @oauth_request_token {string}. Identify a token that is pre-approved by the provider for access to the resource.
  • @oauth_request_token_secret {string}. Secret associated with the pre-approved request token
  • @oauth_use_token {string}. Control whether an OAuth token should be used with the request. Allowed values are: "always" | "if_available" | "never"

Example: 

<os:HttpRequest key="Pets" href="http://pets.com/api" authz="signed"/>

Embedding in @type="html" content

When embedding data pipelining a HTML content (either in CDATA or content returned with Content-Type:text/html from a proxied request), data pipelining tags are enclosed in a <script type="text/os-data"> block.

Example: 

<Content type="html"><![CDATA[ 
  <script type="text/os-data">
    <os:PeopleRequest key="vf" userid="@viewer" groupid="@friends"/>
  </script>
  <script type="text/os-template">
    <div repeat="${vf}">${Name} is a friend.</div>
  </script>
]]>
</Content>

The first <script> block of type text/os-data must be processed by the data pipeline processor. Behavior is undefined with multiple script blocks.

There is a general goal to be able to remove <script> blocks, but there are concerns in not having this <script> marker for processing, both in server and client-side implementations. We will experiment in the initial implementation and propose a streamlined syntax for the 1.0 OpenSocial release if possible.

Proxied content

Data pipelining tags can be top level elements in the <Content> section of a gadget using Proxied Content.

Example: 

<Content href="http://developer.com/canvas">
  <os:PeopleRequest key="vf" userid="@viewer" groupid="@friends"/>
</Content>

The results of the data pipelining requests will be sent to the href of the <Content> section as POSTed JSON data. The structure will match the [JSON-RPC] format.

Elements in a data pipeline referencing the viewer (via @viewer, @me) will return a 403 (Forbidden) error response if the container is unable to also send the opensocial_viewer URL parameter to the 3rd party server. The same holds true for @owner and opensocial_owner URL parameter.

JavaScript API

There will be new JavaScript APIs created to access the pipelined data on the client, all on the opensocial.data.DataContext object.

There will be APIs to get the data returned from the data pipeline, set additional data, and listen to changes in data of a given key:

  • opensocial.data.getContext() - Gets the DataContext associated with the gadget.
  • opensocial.data.DataContext.getDataSet(string key) - Retrieves the object data that is currently mapped to the specified key.
  • opensocial.data.DataContext.putDataSet(stringkey, json|object value) - Puts additional data into the data context keyed by the key.
  • opensocial.data.DataContext.registerListener (array<String>|string keys, Function(array<String> keys) callback) - Listen for any updates to data keyed by one of "keys". You can also pass in the wildcard key "*" to listen to all keys.
    • The callback function is returned with an array of the keys that have changed.

Examples: 

var viewer = opensocial.data.getContext().getDataSet('Viewer');
alert('Hello ' + viewer.name);
os.data.getDataContext().putDataSet('Params', {"Page": 2, "PageSize": 10});
opensocial.data.getDataContext().registerListener('Friends', function(key) {
   var el = document.getElementById('friend-details');
   el.style.display = 'show';
});

Handling of unrecognized tags

Containers should ignore unrecognized tags or return error code 404 (Not Found). If a tag is ignored, the JSON structure can either include a null value for the request key or just not include the request key.

Namespace

XML namespace for all of the Data Pipelining tags is http://ns.opensocial.org/2008/markup.

Dynamic parameters

Data requests often need to change based on input parameters. Two key use cases:

  • Passing gadgets view parameters to a request, for example the current page in a paginated list
  • Passing OpenSocial data to a following <os:DataRequest>, specifically IDs of (viewer, owner, viewer friends, owner friends)

Data pipelining will support a limited subset of the expression language used in OpenSocial templates (based on JSP EL):

  • Expressions of format ${A.B} will be supported, where "A" is the key and "B" is the property.
  • The key "ViewParams" is reserved and refers to the parameters passed into the gadget rendering, equivalent to [gadgets.views.getParams()]
  • The key "UserPrefs" is reserved and refers to gadget user preferences - ${UserPrefs.PREF} will return the value of gadgets.Prefs.getString('PREF') - see [API]
  • Keys can also map to earlier items in the data pipeline. You can get a viewer and then pass the ID to successive requests. - consider for v.NEXT
  • <os:PeopleRequest> has a special implicit property, "ids", which is used to send along a comma-delimited list of the IDs returned. This is required to enable passing of friends lists. - consider for v.NEXT

Example: 

<os:PeopleRequest key="PagedFriends" userid="@owner" groupid="@friends" startPage="${ViewParams.page}" count="20"/>
<os:DataRequest href="http://developersite.com/api?ids=${PagedFriends.ids}"/>

Issue discussion

<script> tags / type="osml"

There is a strong desire to support a simplified syntax for embedding Data Pipelining and template tags without including <script> tags.

After discussions, proposal is to:

  • Leave <script> tags in for 0.9 and to evaluate removal for 1.0, and
  • Not have separate type="osml"

Reasons were:

  • For the proxied rendering case, the differences between the two possible syntaxes came down to at most 3-4 script tags:
    • The <script> tag is not required for defining the Data Pipeline (see details in "Proxied content" section).
    • There are only three OSML Tags that we anticipate using in proxied content without invoking full templating.
  • <script> tags were seen as beneficial for parsing, both with server side and client side rendering.

So there seemed to be good reasons to keep the tag, and minimal cost for 0.9. As we extend this functionality, we should investigate ways to make the syntax simpler by removing the <script> tag in some cases.

<os:DataRequest> tag name

There was some discussion that this tag is extremely similar to gadgets.io.makeRequest() and therefore should be called <os:MakeRequest>.

However, the expectation that the response format was JSON seemed to be a significant difference.

Also, the ability to optionally support arbitrary methods from JSON-RPC was an additional important difference.

So proposal is for the name to be <os:DataRequest>

Do we need <os:ViewerRequest>/<os:OwnerRequest> ?

Suggestion has been made that <os:PeopleRequest> might be sufficient, and that requests for single users might be redundant.

However, getting the viewer and owner are two of the most common operations, and always having users take the first response item seems to be inviting developer error.

Original proposal had <os:PersonRequest> with an ID - this was covering use cases that weren't as common (getting one user by arbitrary ID). This proposal has been updated to have <os:ViewerRequest> and <os:OwnerRequest> - shorthands for the two most commonly used calls in OpenSocial.

Example with <os:PeopleRequest>: 

<script type="text/os-data">
  <os:PeopleRequest userId="@viewer" key="vwr"/>
</script>
<script type="text/os-template">
  Hello, ${vwr[0].name}
</script>

Same example with <os:ViewerRequest>: 

<script type="text/os-data">
  <os:ViewerRequest key="vwr"/>
</script>
<script type="text/os-template">
  Hello, ${vwr.name}
</script>


Change Log

11/18/08 - Updated in response to feedback from Louis Ryan

  • Added back JS API details (was accidentally left off when moving to Wiki)
  • Clarified that POST data for proxied requests uses JSON-RPC format
  • Changed @filter to @filterBy, @filterOp, @filterValue
  • Clarified that viewer/owner specific information in data pipeline won't be set for proxied requests if opensocial_viewer/opensocial_owner params are not sent to 3rd party server

11/17/08 - Updated with feedback from OpenSocial 0.9 summit and other minor changes:

  • Moved to Wiki
  • changed @userid and @groupid to @userId and @groupId
  • <os:PersonAppDataRequest> removed, obsolete with other 0.9 changes
  • @type="osml" removed (and notes why)
  • Notes on why <script> tags remain
  • <os:DataRequest> defaults to returning JSON, can also return @format="text"
  • <os:DataRequest> can use @method to call optional additional RPC methods
  • <os:PersonRequest> removed, added <os:ViewerRequest> and <os:OwnerRequest> (plus notes as to why)

Resources

Original Proposal

Discussion Thread

See also Declarative Data Definition

Retrieved from "http://wiki.opensocial.org/index.php?title=Data_Pipelining"

Categories: Approved (v0.9) | Has Prototype (v0.9)

Personal tools