Beyond REST maturity levels:

Real life, high-load REST APIs

PHP Benelux, Antwerp - January 27th, 2018

© David Buchmann

You've got a "situation"

• Data in enterprise systems
• Websites take a lot of effort
• Data access is slow
• Data distributed over several systems

Delivering Data

API: Symfony Application

• FOSRest: Routing, Content Negotiation
• ElasticSearch queries
• JMSSerializer: JSON to Model to JSON / XML

API: Versioning

• JMSSerializer feature
• FOSRest handles version detection
  query string / accept header
• Only when not backwards compatible
• Alternatives:
  • Elasticsearch index per version
  • Run old application version in parallel

/**
 * The description as HTML
 *
 * @Serializer\Since("2")
 * @Serializer\Type("string")
 */
public $description;

Api-Version: 2
        

/**
 * Plain text version of description.
 *
 * @Serializer\Until("1")
 * @Serializer\Type("string")
 * @Serializer\VirtualProperty
 * @Serializer\SerializedName("description")
 *
 * @return string
 */
public function getDescriptionPlaintext() {
    return strip_tags($this->description);
}

Varnish Cache

• Handle more load
• Super fast responses

Varnish Cache: Access Control

• Basic authentication with htpasswd file
• Symfony dumps the file from the DB
• Varnish reads it
• FOSHttpCache user context: share cache between users with the same set of permissions

Varnish Cache: Routing

• Single point of entry, reroute to other backends based on subpaths
• Alternate backends, e.g. newest version of PHP and fallback to old

Varnish Cache: To BAN or not to BAN

• BAN: Invalidate regular expression on request
• Every request checked against ban list
• If you ban a lot, this list becomes long
  
  
• API as source of thruth vs maintaining local copy and notifications

Varnish Cache: Increase Cache Hits

• Normalize Api-Version and v parameter in URL
• Normalize Accept and format extension in URL
• Normalize Accept-Language
• Normalize Accept-Encoding
• Supress Cookies if possible
• vcl_error with custom status codes to abort early

Get people to use your API

Documentation

• Self-documenting names is a lazy illusion
• Document relationships
• Tutorials
• Changelog and migration guides

NelmioApiDocBundle

class LabelController {
   /**
    * @Route("/label/{id}", methods={"GET"})
    * @SWG\Response(
    *  description="Returns a single label",
    *  @SWG\Schema(
    *   @Model(type=Label::class, groups={"api"})
    *  )
    * )
    * @SWG\Tag(name="Products")
    */
   public function getAction($id)

Indexing Architecture

Workflow

Architecture

• Loader (with batch option)
  • SOAP, REST APIs
  • csv & XML files
  • Doctrine (Oracle)
  • Message Queue
• Persistor
• Indexer

Symfony Commands

• Commands for small tasks
• Meta commands for whole workflows
• Connected by rabbitmq queues
• Cloud task scheduler for long running workers
  Or: Supervisord

Data Quality

• Rules to pick right data
• Rules when to ignore whole record
• Blacklist API to quickly hide problematic records

Monitoring and Diagnostics

• API access to raw data for debugging
• API call to get status of items
• Central logging with graylog
• Newrelic
• OpsGenie

Elasticsearch

Elasticsearch: Indexes

• one index per language
• de-normalize everything for fast responses
• no joins or parent-child relations

Elasticsearch: Schema Changes

• ES guesses everything
• Not always correctly => manual definitions
• No way to change definition of existing index
• Deploy new code, but not yet online
• _reindex API to copy data from old version

Elasticsearch: Clustering

• ES runs on several servers
• They communicate among themselves
• Indexes are sharded
• Automatic aggregation and load balancing

Outlook

Beyond REST

• Aggregated information, tailored to actual needs
• Less requests, less bandwith. More server effort.
• Layered Architecture: App API, Facade

Alternatives: Stateless, but...

• React and data pull APIs
• Falcor:
  Graph as JSON, specify wanted data fields
• GraphQL:
  Schema for graph data, query language

Thank you!

@dbu

David Buchmann, Liip SA

https://joind.in/