AthenaeumAthenaeum
Packages
  • next
  • current
  • v9.x
  • v8.x
  • v7.x
  • v6.x
  • v5.x
  • v4.x
  • v3.x
  • v2.x
  • v1.x
Changelog
GitHub
Packages
  • next
  • current
  • v9.x
  • v8.x
  • v7.x
  • v6.x
  • v5.x
  • v4.x
  • v3.x
  • v2.x
  • v1.x
Changelog
GitHub
  • Version 6.x

    • Release Notes
    • Upgrade Guide
    • New to this...
    • Contribution Guide
    • Security Policy
    • Code of Conduct
    • Origin
  • ACL

    • Introduction
    • How to install
    • Setup
    • Permissions
    • Roles
    • Users
    • Cached Permissions
  • Audit

    • Audit
    • How to install
    • Setup
    • Recording
    • Events
  • Circuits

    • Circuits
    • How to install
    • Setup
    • Usage
    • Events
  • Collections

    • Collections
    • How to install
    • Summation

      • Summation Collection
      • Items Processor
  • Config

    • Configuration Loader
    • How to install
    • Setup
    • Load Configuration Files
    • Custom File Parsers
  • Console

    • Command and Schedule Registration
    • How to install
    • Setup
    • Commands
    • Schedules
  • Container

    • IoC Service Container
    • How to install
    • Container
    • List Resolver
  • Core

    • Athenaeum Core Application
    • How to install
    • Setup
    • Usage

      • Configuration
      • Service Providers
      • Service Container
      • Events
      • Caching
      • Logging
      • Console
      • Task Scheduling
      • Exception Handling
      • Extending Core Application
      • Testing
  • Database

    • Introduction
    • How to install
    • Models

      • Instantiatable
      • Sluggable
    • Query

      • Criteria (Query Filter)
  • Dto

    • Data Transfer Object (DTO)
    • How to install
    • Create Interface
    • Implement DTO
    • How to use
    • Populate
    • Export
    • Json
    • Serialization
    • Nested DTOs
    • Array DTO
  • ETags

    • ETags
    • How to install
    • Setup
    • Usage
    • Generators

      • Default Generator
      • Custom Generator
    • Eloquent Models
    • Macros
  • Events

    • Register Listeners and Subscribers
    • How to install
    • Setup
    • Listeners
    • Subscribers
  • Filters

    • Search Filter Utilities
    • Prerequisites
    • How to install
    • Setup
    • Processor
    • Filters Builder
    • Predefined Resources

      • Search Processor
      • Sorting Processor
      • Constraints Processor
      • Matching Processor
    • Tip: Create a base builder
  • Flysystem

    • Introduction
    • Database Adapter

      • Introduction
      • How to install
      • Setup
      • Data Deduplication
      • MIME-Type Detection
  • Http

    • Api

      • Http API
      • How to install
      • Setup
      • Resources

        • Introduction
        • Timestamps
        • Self-Link
        • Relations
        • Registrar
      • Middleware

        • Introduction
        • Request Must Be Json
        • Capture Fields To Select
    • Clients

      • Http Clients
      • How to install
      • Setup
      • Basic Usage
      • Available Methods

        • Fluent Api
        • Protocol Version
        • Base Uri
        • Http Method and Uri
        • Headers
        • Accept & Content-Type
        • Authentication
        • Http Query
        • Payload Format
        • Payload
        • Attachments
        • Cookies
        • Response Expectations
        • Middleware
        • Conditions
        • Criteria
        • Redirects
        • Timeout
        • Debugging
        • Logging
        • Driver Options
        • Driver
      • Http Query Builder

        • Introduction
        • Select
        • Where
        • Dates
        • Include
        • Pagination
        • Sorting
        • Raw Expressions
        • Custom Grammar
    • Cookies

      • Http Cookies
      • How to install
      • Usage
    • Messages

      • Http Messages
      • How to install
      • Serializers
  • Maintenance

    • Modes

      • Maintenance Modes
      • How to install
      • Setup
      • Basic Usage
      • Available Drivers
  • Mime Types

    • MIME-Types
    • How to install
    • Setup
    • Usage
    • Drivers

      • Available Drivers
      • File Info
  • Properties

    • Properties Overload
    • How to install
    • Usage
    • Naming Convention
    • Properties Visibility
  • Redmine

    • Redmine Api Client
    • How to install
    • Setup
    • General Usage

      • Supported Operations
      • Fetch list of resources
      • Find
      • Fetch
      • Create new record
      • Update existing record
      • Delete existing record
      • Relations
    • Available Resources

      • Predefined Resources
      • Attachments
      • Enumerations
      • Issue Relations
      • Users
      • User Groups
      • Roles
      • Project Memberships
      • Versions (Milestones)
      • Issue Categories
      • Trackers
  • Service

    • Service Registrar
    • How to install
    • How to use
  • Streams

    • Streams
    • How to install
    • Setup
    • How to use

      • Introduction
      • Open and Close
      • Raw Resource
      • Seeking
      • Reading
      • Writing
      • Size
      • Truncate
      • Flush
      • Hash
      • MIME-Type
      • Output
      • Locking
      • Transactions
      • Meta
      • Misc
  • Support

    • Introduction
    • How to install
    • Laravel Aware-of Helpers

      • How to use
      • Enforce Via Interface
      • Custom Default
      • Pros and Cons
      • Available Helpers
    • Aware-of Properties

      • Generator
      • Available Aware-of Helpers
    • Live Templates
  • Testing

    • Introduction
    • How to install
    • Test Cases
    • Testing Aware-of Helpers
  • Utils

    • Introduction
    • How to install
    • Array
    • Duration
    • Json
    • Math
    • Memory
    • Method Helper
    • Invoker
    • Populatable
    • String
    • Version
  • Validation

    • Introduction
    • How to install
    • Setup
    • Rules

      • Alpha-Dash-Dot
      • Semantic Version
You are viewing documentation for an outdated version. It is no longer supported!

Data Deduplication

Imagine that you are building a small application that allows users to upload small text files containing greetings (or other types of files). Chances are good that multiple users will upload the exact same greeting messages. Thus, your storage will eventually contain duplicates.

To avoid this kind of situation, the database adapter makes use of data deduplication: "[...] a technique for eliminating duplicate copies of repeating data [...]" (Wiki) Consider the following:

$content = 'Hi there';

$filesystem->write('home/message.txt', $content);
$filesystem->write('home/other_message.txt', $content);

In the above example, two files are written to the database. However, the adapter ensures that the content is only stored once.

Content Hashing

Data deduplication is achieved by hashing a file's content and checking if that hash already exists. If that is the case, then an internal reference_count is incremented. Content is not inserted if this is the case. However, if the hash does not exist, then content is inserted into the database.

By default, sha256 hashing algorithm is used when hashing contents. You can change this via the setHashAlgorithm():

$adapter->setHashAlgorithm('md5')

Caution

Be careful of what hashing algorithm you choose. Some are fast, but might have a high risk of hash-collision. See wiki for more information.

When files are deleted

The adapter automatically cleans up its file-content records, when a file is requested deleted. In this process, the internal reference_count is decreased when a file is deleted. When the reference counter reaches 0, actual content is automatically removed.

Performance Considerations

Due to this applied deduplication technique, this adapter SHOULD NOT be expected to be very fast. It will get the job done. But, if performance is very important for your application, then you SHOULD choose a different flysystem adapter!

Edit page
Last Updated: 16/02/2023, 09:10
Contributors: alin
Prev
Setup
Next
MIME-Type Detection