Deploying PHP applications

with zero downtime

ConFoo, Montreal - February 23rd, 2023

© David Buchmann

David Buchmann - david@liip.ch

PHP Engineer, Liip AG, Switzerland

If you really have to, use FTP with TLS: SFTP

Upload your application

git pull / scp / rsync

Replacing files in place on the running system

Replace files in place

• OK to deploy to shared hosting or for small generated pages of static files
• Wordpress or Drupal applications are capable of operating without shell access
• Inconsistent while files are being uploaded
• Not really interruption free, certainly not robust
• Lets look at more powerful options

Goals for a deployment

• Minimize downtime
• Rollback on failure
• Repeatable deployments across environments
• Deploy early, deploy often

Automate everything!

• No manual mistakes
• Nothing gets forgotten
• Each step runs as soon as the previous is finished
• Running code is accurate documentation
• Version control everything, including deployment scripts
• Low effort to do a deployment

Deploy from CI

Have your CI run the deployments

• Guaranteed that no manual steps creep in
• Every team member can trigger a deployment
• Avoids differences between personal setup
• Clean state, no accidental changes deployed.
  Discourages manual hacks.
• Continous deployment
  Feature flags where necessary

Atomic deployment

• Uploading the files takes a time > 0
• Uploading can be interrupted in the middle
• You need your application's code to be consistent

To Cloud or not to Cloud?

Deploying to a server

• Webserver keeps running
• Provide the new PHP files (rsync / archive upload)
• Atomic deployment
  • Webserver serves from symbolic link folder
  • Provide application in new folder
  • Switch symlink: Tiny filesystem operation

Symbolic Link (symlink)

Symbolic Link (symlink)

Consequences

• Need to restart long-running processes
• Clear local caches, even memory caches
• Store content data (user uploaded files, etc) outside the application folder
  =>CDN for images and download files
• Webserver fixed to version, can't preview new version
• Only deploy if CI succeeds

PHP Opcache

• Caches PHP bytecode for big performance gains
• Based on the filenames
• On server usually does not check file timestamp
• Either path needs to change or actively invalidate opcache

Reset PHP Opcache

• Call opcache_reset() in webserver context
• The cli does not share opcache with the webserver
  Web request in the deployment process, e.g. curl
• Or interrupt services by restarting php-fpm

Resolve symlinks

Make webserver use resolved path when talking to php-fpm

• Nginx: $realpath_root not $document_root
• Caddy: flag resolve_root_symlink
• Apache: Options FollowSymlinks
  (otherwise won't work at all)

Zero Downtime?

• Symlink strategy reduces the window for problems
• Requests running exactly while switching symlink can have code changes
• If something is wrong with the new version, the application fails until rollback

Rolling Updates

• Tell your load balancer to remove one host
• Let the host finish all its pending requests
• Do the deployment
• Verify that the new version works correctly
• Tell the load balancer to use that host again
• Repeat for each host

Rolling Updates: Gotchas

• Both versions must use shared resources in compatible fashion
• You want sticky sessions, or users might alternate between old and new version during deployment

Beware of caches!

• If caches persist between deployments, check if you need to purge
• Structure of object changed: cached data broken
• Logic to build object changed: can't detect
• Purge all caches automatically after deployment, or update caches async, or just wait for expiry
• Similar with a HTTP cache, purge after deployment

Client side cache

• Can't purge client cache
• Cache busting: Hash in asset URLs to provoke cache miss after deployment

Migration tasks

• Changes to the database structure
• Or to specific things in the database
• Automate this too!
• Record in database which migrations have been run
• To execute each only once
• Ideally can be re-run without harm

Database changes

• Interesting migration task
• Good time for a backup before deployment starts
• Doctrine Migrations:
  Automatically build changes from your entity classes
• Rolling updates or rollback:
  Database changes must non-breaking

Non-breaking add column

• Add nullable, let code handle if missing
• Rollback: Ignores optional column
• Second deployment:
  • Run task to have column always have value
  • Make column required in database
  • Simplify code

Non-breaking remove table

• Only remove the code, but database unchanged
  (Doctrine: keep entity class to not confuse doctrine:migrations:diff)
• Rollback: Trivial, table still there
• Second deployment:
  • Drop the table
  • (Doctrine: remove the entity code)

Non-breaking remove column

• Stop using it, make it nullable in database
  (Doctrine: Need to remove property to avoid querying for it, this will confuse the diff tool)
• Rollback: Need to fill the column again
• Second deployment:
  • Drop the column

Non-breaking rename column

• Add new optional column, fallback to old if new is empty
• Rollback: Copy data from new to old column
• Second deployment:
  • Copy old column to new
  • Code only look at the new column
• Third deployment:
  • Drop the old column

Security Considerations

• Webserver allowed to read git is an attack vector
• Copy data to server from deployment machine
• Or at least configure to not serve .git folder
• Do not deploy development-only code
• Nor any other files not required
• Less files on the server = lower risk of issues
• Protect development servers with basic auth, especially if they run with debug output enabled

Rollback

Rollback

• Switch symlink back
• Resp. activate previous cloud instance
• If there are database changes: Hope you only have non-breaking changes, or a backup

Into the Clouds

Cloud

• Each deployment is a new server instance
• Instead of switching folders, switch servers
• You can verify that the new server works correctly before switching -> blue-green / stack swapping
• Keep old instances running until they finished serving their current requests
• Old server is destroyed
• Content data needs to be on network storage

Tools for deploying

Deploying

• Deploying means moving files
• and interacting with cli tools
• Automate this!
• bash or a Makefile
• Or use a dedicated deployment tool

Tools for deploying

Currently, the tool of choice seems to be deployer, especially for PHP applications

Many other tools:
Fabric, Capistrano, Jenkins, Apache Ant, ...

Deployer

• Written in PHP, configured in PHP!
• Multiple hosts, multiple environments
• Functions to run tasks locally or on target system
• recipes to organize reusable steps
• Provided recipes to post messages to many chat systems: IRC, Slack, Telegram, ...
• Base recipes for known frameworks: Laravel, Symfony, Drupal, Wordpress, ...
  => Starting point to build your own

Deployer

host('example.org')
    ->set('remote_user', 'deployer')
    ->set('deploy_path', '~/example');
set('shared_dirs', [
    'web/sites/default/files',
]);
set('shared_files', [
    '.env',
    'web/sites/default/settings.php',
]);

Then use dep provision for interactive configuration

Deployer server folders

~/example                      # deploy_path
 |- current -> releases/1      # Symlink current release
 |- releases                   # Dir for all releases
    |- 1                       # Files of release 1
       |- ...
       |- .env -> shared/.env  # Symlink to shared .env file
 |- shared                     # Dir for files to preserve
    |- ...
    |- .env                    # Example: shared .env file
 |- .dep                       # Deployer config files
            

Tasks

task('apt:update', function () {
    run('apt-get update');
})->oncePerNode();
            

Tasks can access per-host configuration

Some Primitives

• upload: Upload files from local to remote host
• cd: Change directory
• get/set: Use and manage configuration
• run: Execute command on remote host
• runLocally: Execute command on local machine
• invoke: Call another deployer task
• writeln: Write message to output
• ask*: Interactive questions

Chaining tasks

$deploy_pre = [             $deploy_post = [
    'deploy:release',           'deploy:shared',
    'build:prepare',            'deploy:symlink',
    'vendors:install',          'build:cleanup',
    'package:build',            'cleanup',
    'package:copy',             'success',
    'package:extract',      ];
];

task('deploy', array_merge(
    $deploy_pre,
    $deploy_post
))->desc('Deploy with tar packaging');
            

Hints

• Avoid deploying local changes:
  Do not deploy a developers working directory =>Run from CI or do fresh checkout of application
• Multiple hosts: use once() for shared resources
• If you use nodejs to build assets, compile files before deploying. No need for nodejs on server.
• Run the deployment in a docker container
• Can deploy to localhost to test the process

Thank you!

@dbu

David Buchmann, Liip SA