This Refcard provides an introduction to basic Nosql and Data Scalability terminology and techniques and exhibits in-depth examples of popular Nosql technologies,including architectures,common uses,& more.

Nosql and Data Scalability 2.0 demystifies the latest techniques in high-volume data storage,search,and management by explaining how they work and when to apply them.

Scalable data architectures have evolved to improve overall system efficiency and reduce operational costs. Specific Nosql databases may have different topological requirements,but the general architecture is the same.

In general,Nosql architectures offer:

• Eventual consistency
• High availability with partition tolerance
• Horizontal scalability (optimized for large volume of reads and queries)
• Cloud readiness
• Distributed,structured data storage

Cloud readiness describes the database being used as a service and the ability to deploy the storage grid and cluster manager to a cloud provider.

Nosql describes a horizontally scalable,non-relational database with built-in replication support. Applications interact with the database through a simple API,and the data is stored in a schema-free repository as large files or data blocks. The repository is often a custom file system designed to support Nosql operations with high replication.

Nosql Databases Classification

Database TypesUsesDocumentDocuments,semi-structured dataColumnRead/write raw time series dataGraphNamed entities,semantic queries,associative data setsKey-ValueKey-value pair,where the values can be complex and mixed data structures (e.g. a document)Multi-modelTwo or more database types,including relational databases and the types listed above,with a common database manager for all

While all database types are in common use,document stores are most often associated with Nosql systems due to their pervasiveness in web and mobile content handling applications.

Is Nosql For You?

Does your app design...

• Require high-speed throughput?
• Need to handle high volumes of data?
• Work well with weak data consistency?
• Benefit from direct object-database entity mapping?
• Have to be Always On?

If you checked off four or more items from the list,then Nosql is a good fit for you.

Always On just means that users will have access to complete app functionality at all times. In mobile app and gaming contexts,it can mean access to data that is "a bit behind" the effective system state (i.e. eventual consistency is acceptable).

Nosql Performance and TCO Comparison

Total cost of ownership (TCO) depends on functionality and complexity. A higher TCO may be acceptable when performance (throughput or scalability) is a primary concern.

Document and key-value stores are most popular because of their ease of use,flexibility,and applicability across many problem domains—at a reasonable TCO.

Tip: Graph databases are excellent replacements for complex relational models because relationships between entities (or graph edges) are more efficient and better suited for high-performance applications than using explicit joins and foreign-keys.

Which Data Store Model to Use?

The flowchart in Figure 3 describes how to choose the most appropriate database or store for the application.

Demand-based scaling is an attractive proposition for running Nosql systems on the cloud; it maximizes the advantages of running the application on cloud-based providers like AWS,Azure,or Google Cloud Computing.

• Database-as-a-Service (DBaaS) offers turnkey managed functionality,which delegates all operational responsibilities to the provider.
• Hosted VM databases are provisioned on virtual images,much like they would be on premises,and all operational responsibility belongs to the user.

Tip: Billing overruns are very easy when using a Database-as-a-service. Engage a usage/cost monitoring system to help manage expenses and to avoid nasty surprises.

Many applications require the storage of very large binary data sets. Traditional data stores can't handle them because their size makes it impractical. Enter the High-Volume Data Store (HVDS).

Most very high-volume data applications are used in scientific or financial problem domains. These applications rely on dedicated,optimized binary data formats that allow quick access,manipulation,and data format description within a single scope.

High-Volume Data Characteristics

• Very large data sets
• Multidimensional distribution
• Most data is numerical
• Batch processing (including Hadoop) is optimal
• Strong typing

HVDS Characteristics

• Provided data structures (atoms,groups,arrays)
• Persistence management
• Self-describing mechanisms to store the data,obviating versioning issues
• ACID (Atomicity,Consistency,Isolation,Durability) properties
• Language independence
• No query language—access to structures is application and language/API dependent
• No security model
• Application or access APIs must provide concurrency

Rule of Thumb: High-Volume Data Stores handle very few I/O operations,but each consists of very large amounts of data; Nosql handles lots of I/O operations on small amounts of data.

Data is stored in HVDS prior to initial processing,where the HVDS API provides more efficient access than the file system or a database system. The intermediate or final results move to Nosql or relational stores for end-user reporting and manipulation.

HVDS maps onto local files,like Hadoop's HDFS,allowing volumes to move between files systems (e.g. HFS+ to NTFS) without issues. Most HVDS originates in scientific research organizations,and portability is a primary design concern.

HVDS Workflow

Couchbase Server is a document-based database that bridges the gaps between scalable key-value stores,relational database querying,and robustness capabilities. Its characteristics include:

• Document-oriented storage – data is manipulated as JSON documents
• Querying – uses a robust query language with document handling semantics,N1QL
• Multi-dimensional scalability (MDS) – different components may scale up or out,depending on load and performance required
• MapReduce – built-in,indispensable for querying on non- indexed document attributes
• Caching – integrated across all database services
• Replication – transparent replication across multiple data centers
• Spatial views – handles geometric,geospatial data definitions and allows mapping of attributes to multidimensional indices (e.g. table-like)

Couchbase Server provides datacenter consistency and partition tolerance. The database is based on the independent scaling and replication model shown in Figure 5. Data is handled across 3 different service zones: indexing,querying,and data.

The service zones have different scalability requirements according to their function.

The Couchbase Server software and general documentation is athttp://docs.couchbase.com/admin/admin/Couchbase-intro.html.

Each database service node can replicate data to its peer,and each cluster can replicate to other clusters. Couchbase Server provides facilities for cross datacenter replication (XDCR),simplifying disaster recovery,high availability,and data locality scenarios.

In-Memory Cache == Higher Performance

Couchbase Server performs very well during writes because it uses a memory-first mechanism. Data is written to the in- memory cache with a fast response to the caller. Couchbase Server asynchronously replicates the data to other nodes or clusters,updates the indices,and persists the data to disk. Database clients may override any of these operations to make them synchronous.

A read request is guaranteed to always get the most recent result at the time of the beginning of the request.

Document Format

Couchbase Server handles JSON documents. Being a schemaless database,any valid document can be committed to the database. Couchbase Server assigns two additional attributes to each document upon creation for tracking the document’s unique ID (_id) and revision number (_rev). These attributes are required for all operations other than creation. A typical document and its cross-language representation could be: