This document uses the terms may, must, should as defined in RFC2119

Why you need MetaMate, and what it can do

• MetaMate provides an API for your personal and other interesting data
• Build applications that work with abstract types, your application will work independent of where the data comes from
• Integrate data sources and share them with the world as Docker images

Components

Clients, MetaMate, and Services make up the MetaMate ecosystem.

Clients

A Client connects to a MetaMate instance, and uses the data, the instance can access, to power applications. A Client may need to authenticate itself with the MetaMate instance.

Examples for Clients may be the following:

• Web applications
• Mobile applications
• Command line interfaces

MetaMate

A MetaMate instance facilitates the communication between Clients, and Services with the following:

MetaMate

• protects resources.
• discovers, and looks up Services.
• authenticates to upstream data sources via Services.
• turns Client requests into Service requests, and routes the Service requests to Services.
• merges, filters, and sorts the data, that Service responses return.

Services

Services give MetaMate instances access to upstream data sources. A data source may be, more or less everything, that returns data:

• APIs
• Websites
• Databases
• Files
• Blockchains

Tasks

Deploy & discover services

Integrate data sources

Deployment

Distribution

Security

MetaMate may provide HTTPS, and HTTP Basic authentication for Clients.

HTTPS

If the setup meets the following:

• The metamate.yaml sets host.name.
• The metamate.yaml sets host.https.autoCert.
• The metamate.yaml sets host.https.certPath.
• Let’s Encrypt can reach MetaMate on host.name.

MetaMate

1. Obtains a SSL certificate.
2. Stores the certificate at host.https.certPath.
3. Uses the certificate to provide a secure HTTPS connection.

Authentication

Authentication may happen on the following levels:

• The Client may authenticate to MetaMate with HTTP Basic authentication credentials.
• MetaMate may authenticate to Services with ServiceAccounts.

Client authentication

If the metamate.yaml sets host.basicAuth.user, and host.basicAuth.password, MetaMate adds an internal HTTP Basic authentication middleware. Therefore the Browser, and Client SDKs must authenticate to MetaMate with HTTP Basic authentication credentials.

Service authentication

If Service.endpoints.authenticateServiceAccount is set, MetaMate must authenticate to the Service with a ServiceAccount. Service.endpoints.authenticateServiceAccount.filter specifies what data the ServiceAccount must contain.

The metamate.yaml may set multiple serviceAccounts of the MetaMate internal type ServiceAccountAssignment.

The MetaMate internal type ServiceAccountAssignment holds the following fields:

• serviceAccount of Type ServiceAccount
• serviceId of Type ServiceId, that corresponds to the Service, that the ServiceAccount is designated for

The metamate.yaml must set a subset of the following Fields for every ServiceAccount:

• username
• email
• password
• clientId
• clientSecret
• host

If Service.endpoints.authenticateServiceAccount is set, the following happens before MetaMate sends a ServiceRequest to a Service:

1. MetaMate finds the ServiceAccount, that corresponds to the Service

2. If ServiceAccount.token is not yet set, or if ServiceAccount.token.expiresAt is in the past, the following happens:

   1. MetaMate sends a AuthenticateServiceAccountServiceRequest, that contains the ServiceAccount, to the Service

   2. The Service either obtains an authentication token with

      • the credentials, provided by the ServiceAccount
      • a Refresh Token, provided by ServiceAccount.token.refreshToken

      from the upstream data source. The token may be a Session Cookie, a JSON Web Token (JWT) or any other kind of token.

   3. The Service creates a Token, that

      • must set Token.value to the authentication token
      • may set Token.refreshToken
      • may set Token.expiresAt

   4. The Service sends a AuthenticateServiceAccountServiceResponse, that contains the Token, back to MetaMate

   5. MetaMate copies AuthenticateServiceAccountServiceResponse.output.token to ServiceAccount.token

3. MetaMate injects the ServiceAccount into the ServiceRequest

4. The Service authenticates requests with ServiceAccount.token.value to the upstream data source

Authorization

If the metamate.yaml sets host.basicAuth, every authenticated User has access to all Endpoints of all Services.

If the metamate.yaml doesn’t set host.basicAuth, every User has access to all Endpoints of all Services.

If the metamate.yaml sets endpoints.config.on, MetaMate exposes the metamate.yaml at /metamate.yaml, and therefore all ServiceAccounts set in the metamate.yaml. endpoints.config.on should only be set for debugging in a local environment.

Communication

Clients, MetaMate, and Services use Meta Query Language (MQL) to communicate with one another.

Protocol

The encoding capabilities of MQL are derived as the lowest common denominator of the encoding capabilities of the following serialization technologies:

• JSON
• YAML
• Protocol Buffers
• Apache Thrift

Therefore MQL requests, and MQL responses seamlessly map onto the following transport technologies:

• GraphQL
• REST
• OpenAPI
• gRPC
• Apache Thrift

Special Types

Scalar

Value

Range

Ratio

Unions

A Union is a Type that must hold exactly one value. Every Union has a kind, that indicates, what value the Union holds.

Identifier

MQL identifies Entities by ServiceIds, and GlobalIds.

ServiceId

The Type ServiceId holds the following fields:

• serviceName, that contains the name of the Service, that returned the Entity
• value, a distinct string value, that identifies the Entity within the Service. This value may be one of the following:
  • a row index, in the context of a Service, that integrates a database
  • a username, in the context of a Service, that integrates a social network

A Service must set a ServiceId for every Entity it returns. The Service must only set ServiceId.value. MetaMate sets ServiceId.serviceName for every ServiceId, that is returned by a Service, to Service.name.

GlobalId

The Type GlobalId is a Union, and must hold one the following fields:

• email
• name
• username
• me
• url

A Service may set one more GlobalIds for Entities it returns. A Client may request multiple Entities from multiple Services by providing a GlobalId.

Routing

X = Name of Endpoint

1. Client sends a XClientRequest to MetaMate.
2. MetaMate sends XServiceRequests to Services.
3. Services send XServiceResponses back to MetaMate.
4. MetaMate sends a XClientResponse back to Client.

1. Client sends a XClientRequest to MetaMate.

When MetaMate receives a XClientRequest, MetaMate looks for Services that meet the following 4 conditions:

• Service.endpoints.X is set.
• If XClientRequest.serviceFilter is set, the Service must pass the ServiceFilter.
• If XClientRequest.mode.byServiceId or XClientRequest.mode.byServiceIdRelation is set, the Service must correspond to the ServiceId.
• If Service.endpoints.X.requestFilter is set, the XServiceRequest, that corresponds to the XClientRequest, must pass the RequestFilter.

2. MetaMate sends XServiceRequests to Services.

MetaMate creates a XServiceRequest for every Service.

• If XClientRequest.pages of type []ServicePage is set, and one of the ServicesPages correspond to the Service, MetaMate sets XServiceRequest.page to ServicesPage.page

Methods

MQL provides multiple methods to interact with MetaMate. Requests, and Responses adhere to one of the following methods:

• Get
• Action

Get

X = Name of Type

The Get method provides multiple modes to get Xs. The Client can request

• the whole collection of Xs
• one or more specific Xs by providing an identifier
• Xs that relate to an identifier
• Xs by searching for a term

Pagination

A GetXsServiceResponse, and therefore also the GetXsClientResponse, may only return a paginated subset of the overall available Xs.

The Page union Type

A Page must set one of the following Fields:

• index of Type IndexPage

  • IndexPage.value must set the page index, starting from 0

• offset of Type OffsetPage

  • As defined in wikipedia.org
  • OffsetPage.offset must set the offset
  • OffsetPage.limit must set the amount of Xs to return

• cursor of Type CursorPage

  • As defined in slack.engineering
  • OffsetPage.value must set value of cursor

Get methods modes

The GetXsClientRequest.mode must set one of the following Fields:

Get method specific Request, and Response field

The Client may set the following GetXsClientRequest specific Fields:

• The pages field of Type []ServicePage
• The relations field of Type GetRelations
• The filter field of Type XFilter

MetaMate may set the following GetXsServiceRequest specific Fields:

• The page field of Type Page

A Service may set the following GetXsServiceResponse specific Fields:

• The pagination field of Type ServicePagination

MetaMate may set the following GetXsClientResponse specific Fields:

• The pagination field of Type ClientPagination