Hypercore
Secure distributed append-only log for large datasets and realtime streams.
Hypercore is a secure, distributed append-only log for sharing large datasets and realtime streams. It supports sparse replication, verified reads, optional block encryption, and session-based workflows for building higher-level local-first and peer-to-peer data structures.
Install
npm i hypercoreQuickstart
import Hypercore from 'hypercore'
const core = new Hypercore('./my-first-core', { valueEncoding: 'utf-8' })
await core.ready()
await core.append('hello')
await core.append('from hypercore')
console.log('length:', core.length)
console.log('first block:', await core.get(0))
console.log('second block:', await core.get(1))
for await (const block of core.createReadStream()) {
console.log('streamed:', block)
}
await core.close()API Reference
Constructors and Sessions
new Hypercore(storage, [key], [options])
- Signature:
new Hypercore(storage, [key], [options])(GitHub) - Parameters (GitHub):
storage: a directory path, ahypercore-storageinstance, or a storage value supplied throughoptions.storage.key: optional public key used to reopen an existing core. If omitted, Hypercore loads the key from storage or creates a new writable core.options: optional settings for creation and runtime behavior, including:createIfMissing,overwrite,forcevalueEncoding,encodeBatchkeyPair,writableencryption,onwait,timeout,inflightRange,notDownloadingLinger,allowForkuserData,manifest,preload,storage,key,ongc,onseq
- Returns: a
Hypercoreinstance. Callawait core.ready()before relying on synchronous properties such ascore.key,core.discoveryKey, orcore.length. (GitHub) - Example:
const core = new Hypercore('./data', { valueEncoding: 'json' })
User Data is a local-only key/value store with string keys and string or Buffer values. It is not replicated and is useful for peer-specific metadata such as encryption keys or app state.
The manifest controls how a core is authenticated, including version, hash, quorum, signers, prologue, linked, and manifest-level userData. Changing the manifest changes the resulting Hypercore key.
core.session([options])
- Signature:
core.session([options])(GitHub) - Parameters (GitHub):
options: inherited constructor options plus session-specific fields:weak: close automatically when all non-weak sessions are gone.exclusive: wait for exclusive access before the session becomes ready.checkout: open the core at a specific length.atom: attach the session to a storage atom for atomic changes.name: create a named persisted branch.
- Returns: a new
Hypercoresession that shares the same underlying core state. (GitHub) - Example:
const checkout = core.session({ checkout: 10 })
core.commit(session, opts = {})
- Signature:
core.commit(session, opts = {})(GitHub) - Parameters (GitHub):
session: a session whose blocks should be committed into the default core.opts: optional commit controls:length: expected length after commit.treeLength: expected Merkle tree length before commit.keyPair: key pair used to sign the commit.signature: explicit signature for the committed blocks.
- Returns:
{ byteLength, length }when the commit succeeds, ornullif it fails. (GitHub) - Example:
const committed = core.commit(session)
core.snapshot([options])
- Signature:
core.snapshot([options])(GitHub) - Parameters (GitHub):
options: the same options accepted bycore.session([options]).
- Returns: a snapshot-backed session that does not append or truncate as the underlying core changes. (GitHub)
- Example:
const snap = core.snapshot()
Writing and Local Mutation
await core.append(block, options = {})
- Signature:
await core.append(block, options = {})(GitHub) - Parameters (GitHub):
block: a single block or an array of blocks.options: append controls:writable: ignore the writable guard without changing whether the core is actually writable.maxLength: cap the resulting core length.keyPair: key pair used to sign the appended block or batch.signature: explicit signature for the append.
- Returns:
{ length, byteLength }for the resulting core after the append. (GitHub) - Example:
await core.append(['block 1', 'block 2'])
core.createWriteStream()
- Signature:
core.createWriteStream()(GitHub) - Parameters (GitHub):
- none.
- Returns: a writable stream that appends each written chunk as its own block. (GitHub)
- Example:
core.createWriteStream().end('hello')
await core.clear(start, [end], [options])
- Signature:
await core.clear(start, [end], [options])(GitHub) - Parameters (GitHub):
start: first block index to clear from local storage.end: optional non-inclusive end index.options: optional flags, currentlydiffto return cleared-byte information.
- Returns: metadata about cleared bytes when
diff: true, otherwisenull. (GitHub) - Example:
await core.clear(0, 10, { diff: true })
await core.truncate(newLength, [options])
- Signature:
await core.truncate(newLength, [options])(GitHub) - Parameters (GitHub):
newLength: the shorter length to truncate to.options: a fork number or an options object containing:fork: fork id after truncation.keyPair: key pair used to sign the truncation.signature: explicit truncation signature.
- Returns: a promise that resolves when the truncation is persisted. (GitHub)
- Example:
await core.truncate(4, { fork: core.fork + 1 })
Reading, Streams, and Proofs
await core.get(index, [options])
- Signature:
await core.get(index, [options])(GitHub) - Parameters (GitHub):
index: block index to read.options: read controls:wait: wait for download if the block is missing locally.onwait: callback fired when the read starts waiting.timeout: maximum wait time in milliseconds.activeRequests: advanced replication request handle.valueEncoding: override the core's default encoding.decrypt: automatically decrypt encrypted blocks.raw: return undecoded data.
- Returns: the decoded block, or raw bytes when
raw: true. (GitHub) - Example:
const first = await core.get(0, { wait: false })
await core.has(start, [end])
- Signature:
await core.has(start, [end])(GitHub) - Parameters (GitHub):
start: first block index to check.end: optional non-inclusive end index.
- Returns:
truewhen all requested blocks are available locally. (GitHub) - Example:
const cached = await core.has(0, core.length)
await core.update([options])
- Signature:
await core.update([options])(GitHub) - Parameters (GitHub):
options: optional update controls:wait: wait for a merkle-tree update after peer discovery.activeRequests: advanced replication request handles.force: allow updates even when the core is writable.
- Returns:
trueif the core length changed andfalseotherwise. (GitHub) - Example:
await core.update({ wait: true })
core.update() works especially well with core.findingPeers() or a swarm flush so Hypercore knows when peer discovery is still in progress.
await core.seek(byteOffset, [options])
- Signature:
await core.seek(byteOffset, [options])(GitHub) - Parameters (GitHub):
byteOffset: absolute byte offset in the logical stream.options: optional seek controls:wait: wait for missing data to download.timeout: maximum wait time in milliseconds.activeRequests: advanced replication request handles.
- Returns:
[index, relativeOffset], whereindexis the containing block andrelativeOffsetis the byte offset inside that block. (GitHub) - Example:
const [index, offset] = await core.seek(1024)
core.createReadStream([options])
- Signature:
core.createReadStream([options])(GitHub) - Parameters (GitHub):
options: stream controls:start: first block index, default0.end: non-inclusive end index, defaultcore.length.wait: whether to wait for missing blocks.timeout: maximum wait time.live: keep the stream open for future appends.snapshot: freezeendat open time whentrue.
- Returns: an async-iterable readable stream of decoded blocks. (GitHub)
- Example:
for await (const block of core.createReadStream({ live: true })) {}
core.createByteStream([options])
- Signature:
core.createByteStream([options])(GitHub) - Parameters (GitHub):
options: byte-range controls:byteOffset: first byte to read.byteLength: number of bytes to read.prefetch: number of bytes to download ahead.
- Returns: an async-iterable readable stream over a byte range. (GitHub)
- Example:
for await (const chunk of core.createByteStream({ byteOffset: 0, byteLength: 64 })) {}
await core.treeHash([length])
- Signature:
await core.treeHash([length])(GitHub) - Parameters (GitHub):
length: optional logical length to hash. Defaults to the current core length.
- Returns: the Merkle tree hash for the requested length. (GitHub)
- Example:
const hash = await core.treeHash()
await core.proof(opts)
- Signature:
await core.proof(opts)(GitHub) - Parameters (GitHub):
opts: proof request options:block:{ index, nodes }for a block proof.hash:{ index, nodes }for a hash proof.seek:{ bytes, padding }for a seek proof.upgrade:{ start, length }for an upgrade proof.
- Returns: a
TreeProofinstance for the requested data. (GitHub) - Example:
const proof = await core.proof({ block: { index: 0, nodes: 0 } })
await core.verifyFullyRemote(proof)
- Signature:
await core.verifyFullyRemote(proof)(GitHub) - Parameters (GitHub):
proof: a proof returned by a remote peer.
- Returns: the verified Merkle tree batch, or throws if verification fails. (GitHub)
- Example:
const batch = await core.verifyFullyRemote(proof)
await core.signable([length], [fork])
- Signature:
await core.signable([length], [fork])(GitHub) - Parameters (GitHub):
length: optional logical length to encode.fork: optional fork id to encode.
- Returns: a
Buffercontaining the signable payload for the current key, tree hash, length, and fork. (GitHub) - Example:
const payload = await core.signable(core.length, core.fork)
core.download([range])
- Signature:
core.download([range])(GitHub) - Parameters (GitHub):
range: optional download request:start: first block index.end: non-inclusive end index, or-1for continuous downloading.blocks: explicit list of block indices.linear: download in order instead of randomly.activeRequests: advanced replication request handles.
- Returns: a download handle with
.done()and.destroy()methods. (GitHub) - Example:
await core.download({ start: 0, end: 10 }).done()
Extensions and Replication
core.registerExtension(name, handlers = {})
- Signature:
core.registerExtension(name, handlers = {})(GitHub) - Parameters (GitHub):
name: extension name.handlers: optional extension handlers:encoding: compact encoding for messages.onmessage: callback for messages from peers.
- Returns: an extension handle. (GitHub)
- Example:
const ext = core.registerExtension('chat', { encoding: 'json', onmessage })
This extension API is documented as legacy. For new protocol work, the upstream README recommends creating a Protomux protocol instead.
ext.send(message, peer)
- Signature:
ext.send(message, peer)(GitHub) - Parameters (GitHub):
message: encoded payload for the extension.peer: a specific peer to send to.
- Returns: sends a message over the registered extension channel. (GitHub)
- Example:
ext.send({ type: 'ping' }, peer)
ext.broadcast(message)
- Signature:
ext.broadcast(message)(GitHub) - Parameters (GitHub):
message: encoded payload for all connected peers.
- Returns: broadcasts the message to every peer on the extension. (GitHub)
- Example:
ext.broadcast({ type: 'invalidate' })
ext.destroy()
- Signature:
ext.destroy()(GitHub) - Parameters (GitHub):
- none.
- Returns: unregisters the extension from the core. (GitHub)
- Example:
ext.destroy()
core.replicate(isInitiatorOrReplicationStream, opts = {})
- Signature:
core.replicate(isInitiatorOrReplicationStream, opts = {})(GitHub) - Parameters (GitHub):
isInitiatorOrReplicationStream: either a boolean initiator flag or an existing Hypercore replication stream to multiplex onto.opts: the same options accepted byHypercore.createProtocolStream(), includingondiscoverykey.
- Returns: a replication stream you can pipe to a transport. (GitHub)
- Example:
socket.pipe(core.replicate(true)).pipe(socket)
core.findingPeers()
- Signature:
core.findingPeers()(GitHub) - Parameters (GitHub):
- none.
- Returns: a
donecallback that should be called when the current peer-discovery pass finishes. (GitHub) - Example:
const done = core.findingPeers()
Storage Inspection and Mark-and-Sweep
await core.info([options])
- Signature:
await core.info([options])(GitHub) - Parameters (GitHub):
options: optional info flags. Setstorage: trueto include storage byte estimates.
- Returns: an info object with fields such as
key,discoveryKey,length,contiguousLength,byteLength,fork,padding, andstorage. (GitHub) - Example:
const info = await core.info({ storage: true })
await core.startMarking()
- Signature:
await core.startMarking()(GitHub) - Parameters (GitHub):
- none.
- Returns: enables mark-and-sweep mode and clears any previous markings. (GitHub)
- Example:
await core.startMarking()
await core.markBlock(start, end = start + 1)
- Signature:
await core.markBlock(start, end = start + 1)(GitHub) - Parameters (GitHub):
start: first block index to mark.end: optional non-inclusive end index.
- Returns: marks the requested block range so
core.sweep()retains it. (GitHub) - Example:
await core.markBlock(10, 20)
await core.clearMarkings()
- Signature:
await core.clearMarkings()(GitHub) - Parameters (GitHub):
- none.
- Returns: removes all current mark-and-sweep markings. (GitHub)
- Example:
await core.clearMarkings()
await core.sweep(opts)
- Signature:
await core.sweep(opts)(GitHub) - Parameters (GitHub):
opts: optional sweep controls, currentlybatchSizeto control how often clears flush to storage.
- Returns: clears all unmarked local blocks from storage. (GitHub)
- Example:
await core.sweep({ batchSize: 1000 })
The README describes the full mark-and-sweep flow as startMarking(), read or manually mark the blocks you want to keep, then sweep() to clear everything else.
Lifecycle and Local Configuration
await core.close([{ error }])
- Signature:
await core.close([{ error }])(GitHub) - Parameters (GitHub):
error: optional error object used to reject pending replication requests.
- Returns: fully closes the core and its active session state. (GitHub)
- Example:
await core.close()
await core.ready()
- Signature:
await core.ready()(GitHub) - Parameters (GitHub):
- none.
- Returns: resolves when the core has opened and synchronous properties are populated. (GitHub)
- Example:
await core.ready()
await core.setEncryption(encryption)
- Signature:
await core.setEncryption(encryption)(GitHub) - Parameters (GitHub):
encryption: an object that satisfies the Hypercore encryption interface.
- Returns: updates the encryption implementation used by the core. (GitHub)
- Example:
await core.setEncryption(encryption)
await core.setEncryptionKey(key, [opts])
- Signature:
await core.setEncryptionKey(key, [opts])(GitHub) - Parameters (GitHub):
key: encryption key material.opts: optional flags, includingblockto indicate a block-encryption key.
- Returns: updates the active encryption key. (GitHub)
- Example:
await core.setEncryptionKey(key, { block: true })
core.setKeyPair(keyPair)
- Signature:
core.setKeyPair(keyPair)(GitHub) - Parameters (GitHub):
keyPair:{ publicKey, secretKey }buffers for the core's signer.
- Returns: updates the key pair used internally by the core. (GitHub)
- Example:
core.setKeyPair(keyPair)
core.setActive(active)
- Signature:
core.setActive(active)(GitHub) - Parameters (GitHub):
active: whether the core should stay active for download tracking and lingering replication.
- Returns: toggles activity tracking for the core. (GitHub)
- Example:
core.setActive(false)
await core.setUserData(key, value)
- Signature:
await core.setUserData(key, value)(GitHub) - Parameters (GitHub):
key: string key.value: string orBuffervalue.
- Returns: stores a local-only user-data entry. (GitHub)
- Example:
await core.setUserData('encryption-key', secret)
await core.getUserData(key)
- Signature:
await core.getUserData(key)(GitHub) - Parameters (GitHub):
key: string key to read.
- Returns: the stored user-data value for that key. (GitHub)
- Example:
const secret = await core.getUserData('encryption-key')
Properties
core.writable
- Signature:
core.writable(GitHub) - Parameters (GitHub):
- none.
- Returns:
truewhen the core can append or truncate. Populated afterready. (GitHub) - Example:
if (core.writable) await core.append('next block')
core.readable
- Signature:
core.readable(GitHub) - Parameters (GitHub):
- none.
- Returns:
truewhen the core can be read from. Becomesfalseafter close. (GitHub) - Example:
if (!core.readable) throw new Error('core is closed')
core.id
- Signature:
core.id(GitHub) - Parameters (GitHub):
- none.
- Returns: the z-base-32 string form of the public key. (GitHub)
- Example:
console.log(core.id)
core.key
- Signature:
core.key(GitHub) - Parameters (GitHub):
- none.
- Returns: the public key
Bufferfor the core. (GitHub) - Example:
console.log(core.key)
core.keyPair
- Signature:
core.keyPair(GitHub) - Parameters (GitHub):
- none.
- Returns:
{ publicKey, secretKey }for writable cores afterready. (GitHub) - Example:
console.log(core.keyPair?.publicKey)
core.discoveryKey
- Signature:
core.discoveryKey(GitHub) - Parameters (GitHub):
- none.
- Returns: a derived key used to discover peers without revealing the read capability. (GitHub)
- Example:
swarm.join(core.discoveryKey)
core.length
- Signature:
core.length(GitHub) - Parameters (GitHub):
- none.
- Returns: the number of blocks currently available on the core. (GitHub)
- Example:
console.log(core.length)
core.signedLength
- Signature:
core.signedLength(GitHub) - Parameters (GitHub):
- none.
- Returns: the number of blocks signed by a quorum. For single-signer cores, this equals
core.length. (GitHub) - Example:
console.log(core.signedLength)
core.contiguousLength
- Signature:
core.contiguousLength(GitHub) - Parameters (GitHub):
- none.
- Returns: how many blocks are available contiguously from index
0. (GitHub) - Example:
console.log(core.contiguousLength)
core.remoteContiguousLength
- Signature:
core.remoteContiguousLength(GitHub) - Parameters (GitHub):
- none.
- Returns: the largest contiguous length reported by any known remote. (GitHub)
- Example:
console.log(core.remoteContiguousLength)
core.fork
- Signature:
core.fork(GitHub) - Parameters (GitHub):
- none.
- Returns: the current fork id. (GitHub)
- Example:
console.log(core.fork)
core.padding
- Signature:
core.padding(GitHub) - Parameters (GitHub):
- none.
- Returns: the per-block padding value, usually
0unless block encryption is enabled. (GitHub) - Example:
console.log(core.padding)
core.peers
- Signature:
core.peers(GitHub) - Parameters (GitHub):
- none.
- Returns: the array of peers currently replicating with this core. (GitHub)
- Example:
console.log(core.peers.length)
Events
core.on('close')
- Signature:
core.on('close', listener)(GitHub) - Parameters (GitHub):
listener: callback fired once the core has fully closed.
- Returns: registers a close listener on the core. (GitHub)
- Example:
core.on('close', () => console.log('closed'))
core.on('ready')
- Signature:
core.on('ready', listener)(GitHub) - Parameters (GitHub):
listener: callback fired after the core opens its internal state.
- Returns: registers a ready listener on the core. (GitHub)
- Example:
core.on('ready', () => console.log(core.key))
core.on('append')
- Signature:
core.on('append', listener)(GitHub) - Parameters (GitHub):
listener: callback fired when local or remote appends changelengthorbyteLength.
- Returns: registers an append listener on the core. (GitHub)
- Example:
core.on('append', () => console.log(core.length))
core.on('truncate', ancestors, forkId)
- Signature:
core.on('truncate', listener)(GitHub) - Parameters (GitHub):
listener: callback that receivesancestorsandforkIdwhen the core is truncated.
- Returns: registers a truncate listener on the core. (GitHub)
- Example:
core.on('truncate', (ancestors, forkId) => console.log(ancestors, forkId))
core.on('peer-add')
- Signature:
core.on('peer-add', listener)(GitHub) - Parameters (GitHub):
listener: callback fired when a peer connection is established.
- Returns: registers a peer-add listener on the core. (GitHub)
- Example:
core.on('peer-add', () => console.log(core.peers.length))
core.on('peer-remove')
- Signature:
core.on('peer-remove', listener)(GitHub) - Parameters (GitHub):
listener: callback fired when a peer connection closes.
- Returns: registers a peer-remove listener on the core. (GitHub)
- Example:
core.on('peer-remove', () => console.log(core.peers.length))
core.on('upload', index, byteLength, peer)
- Signature:
core.on('upload', listener)(GitHub) - Parameters (GitHub):
listener: callback receivingindex,byteLength, andpeerfor each uploaded block.
- Returns: registers an upload listener on the core. (GitHub)
- Example:
core.on('upload', (index) => console.log('uploaded', index))
core.on('download', index, byteLength, peer)
- Signature:
core.on('download', listener)(GitHub) - Parameters (GitHub):
listener: callback receivingindex,byteLength, andpeerfor each downloaded block.
- Returns: registers a download listener on the core. (GitHub)
- Example:
core.on('download', (index) => console.log('downloaded', index))
core.on('remote-contiguous-length', length)
- Signature:
core.on('remote-contiguous-length', listener)(GitHub) - Parameters (GitHub):
listener: callback receiving the updated remote contiguous length.
- Returns: registers a listener for
core.remoteContiguousLengthchanges. (GitHub) - Example:
core.on('remote-contiguous-length', (length) => console.log(length))
Static Helpers
Hypercore.MAX_SUGGESTED_BLOCK_SIZE
- Signature:
Hypercore.MAX_SUGGESTED_BLOCK_SIZE(GitHub) - Parameters (GitHub):
- none.
- Returns: the recommended 15 MB maximum block size for smooth replication. (GitHub)
- Example:
if (buf.length > Hypercore.MAX_SUGGESTED_BLOCK_SIZE) throw new Error('split the block')
Hypercore.key(manifest, options = {})
- Signature:
Hypercore.key(manifest, options = {})(GitHub) - Parameters (GitHub):
manifest: a manifest object or a single signer's public key.options: optional derivation flags:compat: treat a single signer public key as the core key.version: manifest version for single-signer input.namespace: signer namespace for single-signer input.
- Returns: the Hypercore key for the supplied manifest. (GitHub)
- Example:
const key = Hypercore.key(manifest)
Hypercore.discoveryKey(key)
- Signature:
Hypercore.discoveryKey(key)(GitHub) - Parameters (GitHub):
key: Hypercore public key.
- Returns: the discovery key derived from
key. (GitHub) - Example:
const discoveryKey = Hypercore.discoveryKey(key)
Hypercore.blockEncryptionKey(key, encryptionKey)
- Signature:
Hypercore.blockEncryptionKey(key, encryptionKey)(GitHub) - Parameters (GitHub):
key: Hypercore public key.encryptionKey: base encryption key material.
- Returns: a block-encryption key derived from the two inputs. (GitHub)
- Example:
const blockKey = Hypercore.blockEncryptionKey(key, encryptionKey)
Hypercore.getProtocolMuxer(stream)
- Signature:
Hypercore.getProtocolMuxer(stream)(GitHub) - Parameters (GitHub):
stream: a Hypercore protocol stream.
- Returns: the attached Protomux instance. (GitHub)
- Example:
const mux = Hypercore.getProtocolMuxer(stream)
Hypercore.createCore(storage, opts)
- Signature:
Hypercore.createCore(storage, opts)(GitHub) - Parameters (GitHub):
storage: path or storage instance for the internal core.opts: internal core options.
- Returns: the internal core object without wrapping it in a full
Hypercoreinstance. (GitHub) - Example:
const internal = Hypercore.createCore('./data', {})
Hypercore.createProtocolStream(isInitiator, opts = {})
- Signature:
Hypercore.createProtocolStream(isInitiator, opts = {})(GitHub) - Parameters (GitHub):
isInitiator: a boolean initiator flag, a framed stream, or an existing Protomux.opts: protocol options, includingondiscoverykeyfor Corestore-style key management.
- Returns: an encrypted Noise protocol stream with Protomux attached. (GitHub)
- Example:
const stream = Hypercore.createProtocolStream(true)
Hypercore.defaultStorage(storage, opts = {})
- Signature:
Hypercore.defaultStorage(storage, opts = {})(GitHub) - Parameters (GitHub):
storage: path or existinghypercore-storageinstance.opts: optional storage configuration passed to the default storage implementation.
- Returns: the existing storage instance or a new default Hypercore storage wrapper. (GitHub)
- Example:
const storage = Hypercore.defaultStorage('./data')
See also
- Replicate and persist with Hypercore — pair a single core with peer discovery and replication.
- Work with many Hypercores using Corestore — recommended pattern when one process manages multiple cores.
- Corestore — helper reference for creating, naming, and co-replicating groups of Hypercores.
- Hyperbee — sorted key/value B-tree built directly on top of a Hypercore.
- Hyperdrive — filesystem abstraction whose metadata and blob stores are Hypercore-backed.
- Compact encoding — binary encoding toolkit used to structure Hypercore block payloads.