Beyond REST maturity levels:

Real life, high-load REST APIs

ConFoo, Montreal - February 26th, 2016

© David Buchmann

You've got a "situation"

• Data in enterprise systems
• Websites take a lot of effort
• Data access is slow
• Data distributed over several systems

Delivering Data

API: Symfony Application

• FOSRest: Routing, Content Negotiation
• ElasticSearch queries
• JMSSerializer: JSON to Model to JSON / XML

API: Versioning

• JMSSerializer feature
• FOSRest handles version detection
  query string / accept header
• Only when not backwards compatible
• Alternatives:
  • Elasticsearch index per version
  • Run old application in parallel

/**
 * The description as HTML
 *
 * @Serializer\Since("2")
 * @Serializer\Type("string")
 */
public $description;

Api-Version: 2
        

/**
 * Plain text version of description.
 *
 * @Serializer\Until("1")
 * @Serializer\Type("string")
 * @Serializer\VirtualProperty
 * @Serializer\SerializedName("description")
 *
 * @return string
 */
public function getDescriptionPlaintext() {
    return strip_tags($this->description);
}

Varnish Cache

• Handle more load
• Super fast responses

Varnish Cache: Access Control

• Basic authentication with htpasswd file
• Symfony dumps the file from the DB
• Varnish reads it
• FOSHttpCache user context: share cache between users with the same set of permissions

Varnish Cache: Routing

• Single point of entry, reroute to other backends based on subpaths
• Our application on hhvm - probe and fallback to PHP 5.5

Varnish Cache: Increase Cache Hits

• Normalize Api-Version and v parameter in URL
• Normalize Accept and format extension in URL
• Normalize Accept-Language
• Normalize Accept-Encoding
• Supress Cookies if possible
• vcl_error with custom status codes to abort early

Get people to use your API

Documentation

• Self-documenting fields is a lazy illusion
• Document relationships
• Tutorials
• Changelog and migration guides

NelmioApiDocBundle

class LabelController {
   /**
    * @ApiDoc(
    *   description="Get a single label",
    *   output={
    *       "class"="App\Model\Label",
    *       "groups"={"api"}
    *   },
    *   section="Products")
    */
   public function getAction($id)

Indexing Architecture

Workflow

Architecture

• Loader (with batch option)
  • SOAP
  • csv
  • XML
  • Doctrine (Oracle)
  • REST APIs
• Persistor
• Indexer

Symfony Commands

• Commands for small tasks
• Meta commands for whole workflows
• Commands as services
• Supervisord for long running workers

Data Quality

• Rules to pick right data
• Rules when to ignore whole record
• Blacklist API to quickly hide problematic records

Monitoring and Diagnostics

• API access to raw data for debugging
• API call to get data status
• Central logging with graylog
• statsd Varnish plugin
• OpsGenie

Elasticsearch

Elasticsearch: Indexes

• one index per language
• de-normalize everything for fast responses
• no joins or parent-child relations

Elasticsearch: Schema Changes

• ES guesses everything
• Not always correctly => manual definitions
• No way to change definition of existing index
• Deploy new code, but not yet online
• Rebuild index
  (from old index or from MySQL)

Elasticsearch: Clustering

• ES runs on several servers
• They communicate among themselves
• Indexes are sharded
• Automatic aggregation and load balancing

Outlook

Beyond REST

• Aggregated information, tailored to actual needs
• Less requests, less bandwith. More server effort.
• Layered Architecture: App API

Alternatives: Stateless, but...

• React and data pull APIs
• Falcor:
  Graph as JSON, specify wanted data fields
• GraphQL:
  Schema for graph data, query language

More Input

Build RESTful APIs easily with Symfony
Sarah Khalil
Friday 11:00 in Room Mont-Royal

Thank you!

@dbu

David Buchmann, Liip SA