A complete guide to modern web applications

Published in

Jeremy Gottfried’s tech blog

21 min readFeb 26, 2020

This is a full guide to web applications for those entering the scene. This guide is for people who want a little taste of every major concept in modern web development. If you read this entire article, you’ll probably be more fluent in web development by the end. This guide assumes a basic knowledge of how the internet works. It has something for every level of software enthusiast. Wikipedia rabbit holes are encouraged. I will start with basic ideas and move to more complex concepts.

Basic software engineering concepts

Most issues in web development can be solved with basic engineering concepts. These concepts are often not prioritized enough, and ignoring them leads to apps that are difficult to debug and build upon.

Separation of concerns

Separation of concerns is probably the most important concept in engineering. It means that you break up code into intuitive blocks of logic that can be reused across multiple locations in your code.

One of the most basic uses of separation of concerns in almost every programming language is the group of math operators:

+ - / * %

If we had to rewrite the computational instructions for addition every time we wanted to add, our code would take way longer to write! Luckily, the writers of our languages have “separated the concern” of mathematical operations. They gave us operators like “+” for adding. This makes it easy to program with math.

Oftentimes when we talk about artful or beautiful code, we mean code in which concerns have been separated in a readable and intuitive way.

Concerns should be separated at every level of app design. Most application design frameworks implement specific layers of concern so that we can easily reuse structures. Here are some common ones:

Backend v. Frontend

Most modern web apps separate their application loosely into a backend and frontend. If it’s a large application, there might be multiple backends and frontends that provide different services. A programmer who works in both areas is “Full Stack”.

Backend

The backend provides data storage, object relational mapping, business logic, and specific services such as language translation. The backend has its own server to run the service and a means of communicating to the frontend, usually with JSON.

Frontend

The frontend is the client-facing part of the application. It’s where a user interacts with the service in a readable format. This part of the application will usually have markup (HTML) and styling (CSS) to render the service. It will usually have a scripting language like Javascript so the interface updates in real time in response to user actions.

Application Programming Interface (API)

Many backend services offer an API to act as a middle man between the client and the data. An API accepts requests for data and responds to the request in a structured format like JSON. A client can also request to update, delete or create entries.

CRUD

Create, Read, Update, Delete, or CRUD, is a helpful concept of the interface provided by most APIs. There are other conceptualizations of APIs out there as well such as command query separation (CQRS) which claims that every interaction with data is either a command (create, delete, update) or a query (read).

API patterns

Most modern applications loosely follow the REST pattern for delivering their API. There are other application patterns that offer alternative ideologies like PRPL and GraphQL. The idea of a Progressive Web App has come to represent an app that makes use of engineering patterns and separation of concerns to create an app-like experience for users on the web.

View

This is where presentational code lies. Usually there is markup involved, like HTML or JSX. This is what the browser interprets to render the application. We can often separate view concerns further with containers, wrappers, templates, generics, etc. Some frameworks like Ruby on Rails implement views by default.

Domain (business concerns)

Most web applications are connected to a business. The business has specific entities that should be mirrored in code. For example, a corner bodega might have entities like “owner”, “employee”, “customer”, “furniture”, “for sale items”, and “finances”. In a web application, these entities and their associated logic are separated into distinct units. This is often referred to with terms like business domain, business logic, and business domain models. Separating these concerns at larger companies can become complex, but doing so will create an intuitive code base.

Object Relational Mapping

In Object Relational Mapping (ORM), the business domain models will directly mirror the tables stored in a database. On the client side, script and view modules may also mirror the business domain models. Patterns like the repository pattern add more layers on to this system.

Application State

Most modern applications store the application state separately from the rest of the app. State is where an application stores the data of user interactions. For example, if a user clicks a like button, a state update would trigger a new color on the clicked button. The application state stores that information. There is often complex business logic involved in application state, so it is usually separated into multiple layers. Oftentimes, the application state sends data to other parts of the application via a publish-subscribe pattern.

Application Services

This layer handles concerns at the top level of an application, dealing directly with the server and physical resources of an application. Things like auto-scaling, performance monitoring, deployment, and cloud services might be handled here. This layer deals directly with allocating memory, processor power, and parallel threads. Some services might be separated into micro-services to optimize resources.

Single Source of Truth

Another basic concept in software engineering is single source of truth. This concept is often ignored, leading to difficult bugs. Single source of truth means that there is one master authority that determines the state of data. For example, in an election, the single source of truth is the paper ballot. If there is any doubt as to the results of an election, the paper ballot can be re-checked. In a web application, the single source of truth is usually the database.

Caching

Single Source of Truth becomes difficult to uphold when other software engineering techniques are implemented, such as performance improvements. Oftentimes, the database is slow, so copies of data are stored in a place that is easier to access. When copies of data are created, there needs to be a way of verifying the truth of that data.

In order to maintain truth across an application, we might have layers of truth that check the database for updates. Usually these layers store their data in a cache, which pulls from RAM instead of storage. Common layers include:

Database query cache
HTTP Cache

Maintaining source of truth:

Expiration date: One simple way of maintaining truth is tag copies of data with an expiration date. When the expiration date passes, a new request will be made directly to the single source of truth (database).
DB updates: if the copy of the data is on the backend, you could have a layer of logic that resets the cache whenever the database is updated.
Polling: if the data copy is on the frontend, you could use polling to check the single source of truth on a looped timer.
Web Sockets: Many modern applications use a sub-pub protocol like Web Sockets or Web Hooks.
Redux stores: The application state is often a source of truth, separate or combined with the backend database. Client-side applications often make use of the redux pattern for handling application state data.

The Black Box

The black box builds off the concepts of “separation of concerns” and “single source of truth”. The black box means a chunk of code that we can rely on without knowing its inner workings.

Example: Snack Machine

If you put money into a snack machine, you probably don’t worry about how the machine processes your money. You just expect that if you put in money, you receive your snack and change. We give it an “input” and expect to receive the correct “output.”

Computer programs built with separation of concerns and single source of truth provide reliable black boxes for clients. Oftentimes, we use concepts like “pure functions”, “immutability”, and a “type system” to provide extra reliability with our black-boxed code.

CSS 3, HTML5, and ES6 Javascript

CSS for styling, HTML for markup, and javascript for dynamic manipulation of HTML and CSS. Every modern app includes a version of each of these components. The latest versions of each include some big changes.

CSS

CSS, of course, is the styling language we use for rendering UIs in a visually appealing way.

Media queries are probably the most important modern CSS tool. A web application needs to work on multiple interfaces, such as a laptop, tablet, and smart phone. Media queries allow you to set different styles based on the size of the viewport.
Import is another useful modern CSS tool. It allows you to import external css files directly into a css file so you can use those styles within your own application.

CSS Preprocessor — SASS and LESS

CSS preprocessors are modern tools that add additional syntax on top of the built in browser syntax. For example, Sass and Less provide reusable css variables and nesting of classes so you can create hierarchies in your stylesheets.

CSS Modules

CSS modules are a tool in applications built with a bundler. They allow you to scope separate css files to different javascript modules. This can help with organization of css.

Javascript ES6 (ECMAScript 2015)

The most notable of the changes in ES6 vs ES5 is the javascript class. This is just syntactic sugar, but it allows us to write visibly cleaner code, similar to what you’d see in other object oriented languages. Javascript is a functional language where everything, including execution contexts and variables, are part of key:value maps. The key:value maps are called objects in Javascript.

Here is an example of a ES6 javascript class:

import ParentClass from './ParentClass'class ES6Class extends ParentClass {
  constructor({ name="anonymous", ...otherData }) {
    super();    this.name = `young ${name}`;
    this.otherData = otherData;
  }  setTrueName = async () => {
    this.name = await this.fetchTrueName();
  }  fetchTrueName() {
    return fetch('/trueName')
    .then((res) => res.json());
  }  set name(newName) {
    this.name = newName; 
  }  get name() {
    return this.name;
  }}export default ES6Classconst jeremy = new ES6Class({ name: "Jeremy", meaning: "young dashing web dev who writes medium articles sometimes"});

I snuck in some other new ES6 features too, including:

Reactive Programming — React and Vue

Reactive programming expands on the publish-subscribe pattern to build modules that “react” to changes in data. ReactJs and Vue.js are two frameworks that make use of reactive programming concepts.

At the core of reactive programming is the idea of declarative vs imperative programming. Reactive programming is declarative. Imperative programming is programming that achieves an end via a specific mechanism. It focuses on “how” a program operates. Declarative programming focuses on the goal of a program. It doesn’t care how a program operates, only that its logic will be satisfied. The how part can be a black box.

Declarative program: Make a grilled cheese and tell me when it’s done!
Imperative Program:

Use a bread knife to cut two slices of bread on a cutting board.
Use a butter knife to spread butter on the bread.
Heat a pan to medium heat with butter.
Slice two bread sized slices of american cheese and put to the side
Put the bread on the pan
When the bottom of the bread is golden brown, flip each piece.
Place one slice of cheese on top of each piece of bread.
When the cheese is melted, flip one piece of bread onto the other, with cheese faced inward.
Tell Jeremy it’s done!

Declarative programming makes use of the concepts of separation of concerns and black box to simplify program logic, so that very little imperative programming is required.

In reactive programming, a component declares which data it cares about and reacts to changes in that data. So a UserList component says give me a list of users and I will render it.

Hypertext markup language (HTML) and the DOM

HTML is the language most often used for markup in modern browsers. You can interact with its data via an API called the Document Object Model (DOM). We usually use javascript to access the DOM, but it is built so that it can be accessed by any language. The DOM has many features built in that make interacting with markup easier —

Select tags, buttons, inputs, and forms: these are examples of html tags that automatically interact with the user. There is no javascript required. You can use pure HTML to set up a form that sends a post request.
Event handlers: the DOM offers built in events like click , change , and scroll . It’s possible to write these event handlers inline in HTML without any javascript. You can use many DOM API methods without JS.

Oftentimes we use javascript combined with the DOM API because javascript provides a lot of features that aren’t included in HTML/DOM without javascript. Javascript can do tasks asynchronously and modularize tasks.

Side note: a library called jQuery used to be widely used. It provided many useful methods for interacting with the DOM and famously used $ as a catchall method. Much of what made jQuery useful is now provided natively by browsers, so most companies are moving away from it. But the $ method is frequently seen in old code at a lot of companies.

JSX

JSX looks like HTML or XML but it is actually an extension of javascript. It allows you to embed javascript directly in markup which dramatically simplifies the logic of scripts interacting with HTML.

Here’s an example in ReactJS:

const NameInput = ({ name }) => (
  <input value={name} />          // this is the JSX
)

In this component, we expect a name to be passed into our function, and then we return an input tag with a value equal to name. The input tag looks a lot like an HTML input. The major difference is we are able to embed the value of name directly into the HTML tag.

JSX is extremely powerful for passing data around an application. It provides an aesthetically pleasing way of visualizing how data is passed. It is usually used in tandem with a reactive programming platform like React or Vue, but you can write your own transpiling script for babel (more on babel later).

Virtual DOM

The Virtual DOM is a representation of the current state of the DOM. It is basically a big javascript object that is constructed similar to a linked list. Both react and vue make use of this concept. Whenever the virtual DOM is updated, those changes are reconciled with the real DOM by a “diffing” process. Diffing just means that the virtual DOM is compared to the real DOM and appropriate changes are made to bring the real DOM up to date. The goal is to provide a black box that maintains a single source of truth efficiently. This allows reactive programmers to write clean, declarative, pure programs.

Babel

Unfortunately, every browser uses different rules for implementing Javascript. Some browsers don’t have certain ES6 Javascript features. Notable among browsers is Internet explorer, because it has not adopted ES6 standards. In order for javascript to be used on all browsers, it needs to be written with older javascript.

Babel takes our sparkly new ES6 code and transforms it into ES5 code.
Babel also transforms JSX markup into pure javascript, since JSX is not supported in any browser.
Babel offers future javascript syntaxes which have not yet been adopted in browsers. For example, private class members:

class UsePrivate {  #privateVariable = "I am private"}

Bundler — Webpack, Rollup, Parcel

A bundler takes all of our javascript code and bundles it into a single file. Webpack, rollup and parcel are three popular bundlers. One of the bundler’s jobs is often to run Babel. It is often referred to as the “uglify, minify” step in deploying an application.

Condenses code to be as small as possible, shortening variable names, removing whitespace, etc.
Reconciles dependencies so that all the required javascript is loaded together
Can also bundle CSS to put modular CSS into a single file.
Builds differently based on environment (development vs production)
Code splitting to create multiple bundles and load dynamically

Typescript

Javascript is a dynamically typed language. This means that we can pass anything into any function and it won’t throw an exception. Typescript is a superset of javascript that adds static typing. This means that when you define a function, variable, or object, you provide it with types that it must satisfy.

For example:

// No Typescriptconst stringToInteger = (str) => {
  return parseInt(str, 10);
}// With Typscriptconst stringToInteger: (x: string) => number = 
function(str: string): number {
  return parseInt(str, 10);
}

Typescript is useful for writing better code. It forces you to examine your expectations when you write code. Its only drawback is it can give you a false sense of security. A lot of bugs have nothing to do with typing.

Accessibility Markup

Many HTML5 elements incorporate accessibility features. For example, image tags include an alt attribute that provides a visually impaired user with text data about the image. The DOM responds to keyboard commands like tab and return to allow visually impaired users to navigate buttons and lists on a page. It is important to use standard markup practices if you want your app to be accessible.

If you are using JSX for markup, a linter like eslint can provide helpful warnings to ensure your markup is accessible.
When you write custom html elements, ARIA standards can be used for accessibility markup if there is no other built in method.
HTML5 includes the ability to customize keyboard accessibility with tabindex.

Polyfills

Babel makes most ES6 javascript usable across browsers, but it doesn’t support more specific browser APIs like getUserMedia. For that you need a specific polyfill. A polyfill is a script that you run globally when you application first loads which provides access to an API that is not available natively on older browsers.

DevOps concepts

DevOps is a set of practices that combines software development (Dev) and information-technology operations (Ops) which aims to shorten the systems development life cycle and provide continuous delivery with high software quality.

DevOps are practices meant to make development easier and more efficient.

Version Management — Git

Git is a version control language. This is important in application development. Oftentimes, we want to add a new feature to an application, but what happens if there are multiple people working on different aspects of the app? Git allows us to merge their work into the main codebase, and even revert changes if they break the code. Git also recognizes conflicts between separate people’s work and gives you an interface to reconcile those conflicts.

Github

Github is a cloud platform that allows teams to interact with git repositories across multiple machines. It is an incredibly powerful tool for teamwork in web development.

Continuous Integration and Deployment

Continuous integration is the process of integrating changes from different developers frequently, ensuring that there are no code conflicts as developers continue to build features.
Continuous delivery builds on continuous integration by checking a series of tests. The tests ensure that code can be deployed at any time.
Continuous deployment builds on continuous delivery by deploying continuous deliveries to a test environment to ensure that the app is always working.

Test Driven Development (TDD)

Test driven development is a software development process where you write tests that the software must pass before writing the actual software. This allows you to prove that that software meets certain standards. There are many different types of tests:

Unit Tests - tests a specific unit of code. For example:

function add(a, b) {
  return a + b;
}test('add adds two parameters', () => {
  expect(add(1, 4)).toBe(5);
})

Integration Test — tests a larger unit of code that may form an entire application. For example:

function startServer() {
  app.start();
}test('logging in redirects you home', async () => {
  await logIn();
  expect(page).toBe(home);
})

End To End / System Tests — tests an aspect of the entire application, with both frontend and backend functioning. Often you will do this by automating actions inside an actual browser process. Selenium w/ WebDriver, Cypress, and Puppeteer w/Jest are all popular libraries for this.
Acceptance Tests — Similar to End To End tests, but they test specific requirements from a user’s perspective, and separate from the implementation details. For example, if you click a submit button in the actual application, does the form submit?

Agile Development

Agile is an approach to software development that incorporates many of the concepts covered so far. It is not strictly DevOps, but it includes many DevOps practices.

TDD, Continuous Integration, and Domain Driven design.
Pair programming and team-driven design
Iterative development and refactoring

SCRUM

A team plans a sprint with a goal of finishing specific features within a set period of time.
The team meets daily for a “scrum” to discuss team progress, roadblocks, etc.
When the sprint is finished, the team meets with product managers to discuss the new feature.

Linting

Oftentimes, a development team will make use of a code linter to catch obvious syntax errors, maintain a code style (tabs vs spaces), and make sure all code is generally readable.

Security

HTTPS

Pretty much every modern website uses HTTPS. This means that communication between client and server (backend and frontend) is encrypted so that it can only be read when it reaches its either end — the server or client. It prevents attackers from eavesdropping on data.

Firewall

A firewall safeguards a web app from unwanted access. Firewalls generally monitor traffic and data and decide whether it is authorized. They protect against a whole litany of attacks, including denial of service, bots, and IP spoofing.

Rate Limiting

Rate limiting sets a maximum rate at which requests can be made to your website from an IP or session. This will help to prevent denial of service attacks, in which an attacker overwhelms your server with many requests so other users can’t use your service. It also helps ward off general abuse of your data, such as a bot scraping your API.

Cross Origin Resource Sharing (CORS)

CORS allows you to safely share resources from one origin to another. For example, if you are storing your photos in the cloud, but want access to them in your web app, you could set up CORS rules to allow access to your web app but prevent other origins from having the same type of access.

Encoding/ Sanitize Input — XSS and SQL Injection

Encoding user input correctly is important for protecting against cross site scripting attacks, where an attacker inserts scripts into an input field. One common protection is to escape all special characters by replacing them with their unicode value. A similar escape of characters can be helpful for common SQL commands. Validations and whitelists are also helpful to prevent certain common XSS attacks. For example, you could validate a link by checking that is uses HTTP or HTTPS. Protected/encrypted cookies can also be used to prevent XSS attacks where a script attempts to access sensitive data stored in cookies. Limiting database permissions is also helpful with SQL injections.

CSRF tokens — cross site resource forgery

CSRF happens when an attacker takes advantage of a trusted user of your app to submit unauthorized commands. It can be prevented with CSRF tokens, which are a unique token stored in a cookie or HTML tag. Since the token can only be accessed when an authentic user accesses the website, it protects from another origin pretending to be that user.

Store Data Securely

Sensitive data like passwords, phone numbers, and social security numbers should be encrypted and hashed, so if there is a data breach, the attacker can’t read the data in plain text. It can also be helpful to hide user input when a user types in a password. One rule of thumb is never store sensitive data in a cookie or form field.

Frontend Optimization

Most modern web apps implement one or more concepts to make the frontend experience feel more fluid and app-like. Here is a list of some of those features.

First Meaningful Paint

First meaningful paint is when the primary content of a page is visible to a user and they can start interacting with the page. In general, you can improve the time to first meaningful paint by keeping the size of the JS/CSS resources and HTML layout as small as possible. You can defer loading expensive resources like images, and asynchronously fetch data, while providing meaningful placeholders.

Server Side Rendering

Server side rendering is a method available in javascript frameworks for rendering in initial page markup before the javascript loads. Usually this is returned as a string and rendered with a special function provided by the framework. This can significantly increase time to First Meaningful Paint. Now puppeteer allows you to do this in headless chrome as well. An additional benefit of server side rendering is the ability to cache commonly used markup for even faster page loads.

Lazy Loading

Lazy loading is a practice of deferring the loading of a resource until it is needed. This prevents the loading of that resource from blocking user interaction with the page. For example, if you have a page with many images, but most of those images are off screen, you could wait to load the images until the user scrolls down.

Pre-fetching

Pre-fetching is when you request data that you expect will be needed before it is required. For example, if a user starts filling out a search form, you might dynamically request search results before the user hits submit.

Render Caching

Render caching is a method of saving the last known state of an app’s HTML in browser localStorage or indexDB . If a user reloads the page, you can use that copy of the HTML to quickly load the page before the server responds. When the server finally responds, you would replace the HTML copy with the updated version. This will also ensure that something gets rendered in the case of a bad internet connection.

Service Workers

Service workers provide optimizations to the client-side app’s interaction with the network. They provide a background thread separate from the web app. They can provide a smooth offline experience, and defer actions that require a solid network connection. They help with serving the page from a cache when the connection is poor. They can also help with things like push notifications.

Timing your scripts

Common sources of bad performance in an app are poorly timed scripts, expensive scripts taking up the main thread, and expensive event handlers. Oftentimes we don’t realize that certain actions cause layout thrashing.

Layout thrashing

Layout thrashing is when a script triggers the browser to recalculate layout or styles. This occurs synchronously, so it blocks the main thread and can make animations stutter or freeze. Here is an extensive list of everything that causes layout thrashing.

requestAnimationFrame

This is a special method provided by most modern browsers that schedules animations (css transitions or manual javascript animation). Using this method will make sure animations occur during the appropriate browser frame, making it more likely that animations render at 60 frames per second.

don’t layout thrash during a requestAnimationFrame callback
use the FLIP method for organizing your scripts during an animation.

Chrome Timeline Profiler

Beyond animations, Chrome’s timeline tool can help you pinpoint expensive tasks that are burdening your application. It will even allow you to track renders in js frameworks like React. Sometimes there are techniques like memoization or delaying an operation which we can use to improve the performance.

Client side router

The browser History API allows web apps to update the URL and browser history without making a new index.html request to the server. Platforms like react can take advantage of this feature to build routers on the client side to connect a specific Javascript module to a URL. This allows for a better offline experience, and faster time to first meaningful paint between pages.

Placeholders and loading icons

Many modern web apps have an animated placeholder that takes up the space of content while it is loading. For example, a list of square-shaped components could be taken by a greyed out square. When the content loads, it replaces the placeholder. Sometimes it makes more sense to use a loading icon like a spinning circle. Placeholders don’t directly improve performance, but they improve the perceived load time of a page because they give users something visual to interact with.

Prevent Memory Leaks

Memory and garbage collection are handled by javascript in the background. You don’t have any direct control over memory allocation in JS. But you do indirectly have control over it. Javascript clears memory when there is no longer a reference to it. In Javascript, a memory leak occurs when there is still a reference to unneeded data. The most common place this occurs is in an asynchronous callback. If we don’t need async data anymore, we should unsubscribe from the async response to prevent a data leak. Here’s an example in pseudo code:

const subscribeToData = (callback) => {
  // subscribe to asynchronous data, and call callback when received  subcribe('data', callback);  // if the component is closed, unsubscribe from the asynchronous data.  this.onClose = () => {
    unsubscribe('data', callback);
  }
}

Use Material Design, Semantic UI, Bootstrap, or another modern design pattern

Design libraries provide intuitive interfaces for creating performant UIs. They have already optimized markup, CSS, and in some cases, javascript. They are helpful for providing basic components for building a UI.

Backend Optimization

API optimization

Database indexes — store an easy access reference to commonly needed data.
JSON serialization — store an entire JSON object in a single entry, so it can all be fetched with a single query.
Query optimization
Cache commonly fetched data (this will be different depending on the language and framework of your backend)
Paginate API responses — separate queries of long lists into multiple pages. (this will also be implemented differently depending on the framework).
Elasticsearch- this is a search interface used for highly performant searches which can returned as lists by an API
GraphQL — creates all-encompassing API endpoints that reduce the total number of necessary endpoints

Compute usage

Send expensive tasks to a background worker or a parallel thread
Auto-scale using a cloud service like Heroku or AWS
Separate services into multiple micro-services that run on different servers for better use of computer resources.
Use WebSockets or WebHooks protocols for real-time communication (messaging apps, push notifications, etc).
Use WebRTC + WebSockets for real time media streaming between users
Use streams to deliver or receive large files or media files to a client.

Search Engine Optimization (SEO)

Search engine optimizations are the techniques that give a web app better placement in search results on sites like Google search.

Crawl accessible — this means that the bots that crawl the internet are able to reach your content. Modern browsers like Chrome understand javascript apps, so usually this is not an issue. It only becomes an issue if crawlers hit exceptions when they attempt to access your site.
Uses keywords that match popular searches. Google’s keyword tool is helpful for finding popular keywords in your business domain. Keywords that are placed more prominently (in the title, meta data) are given more weight.
Good user experience — all the concepts we have covered contribute to optimizing your site for searches
A well linked site. To begin with, every page on a site should be reachable via link from somewhere else on the site. A site map can be helpful for determining the reachability of your site’s pages.
External links are very helpful for SEO. The more your site is referenced by other websites and shares on social media, the higher it ranks with crawlers.
Metadata, meta tags, title, description
Use schema markup for more informative markup

I hope you enjoyed this guide to modern applications! If you have a suggestion, or think something important is missing, please let me know in the comments section!