• administrators

    Author: @laser

    Proofs Walkthrough

    For infra (Part One):

    What artifacts are built and how are they distributed and consumed?
    What artifacts are accessed at runtime?
    What does "use small sectors" actually do?
    What is the status of PoRep and PoSt?

    For everybody else (Part Two):

    What is a sector base
    What responsibilities & code are in Rust
    Major APIs and how they are used
    Identifiers in use
    Metadata that needs to be kept
    What dials we have on performance of proofs
    What if any big changes are coming down the pipe

    Part One: Stuff Infra Cares About

    Build and Distribution


    When the master branch of rust-proofs-fil builds successfully in CircleCI, a gzipped-tarball gets pushed to GitHub releases ( We build once in CircleCI using an OSX container and once using a Linux container. A release is identified (tagged) by the first 16 characters of the build's Git SHA.

    This tarball contains:

    libfilecoin_proofs.a (a "static library")
    libfilecoin_proofs.h (corresponding C-header file)
    libfilecoin_proofs.pc (a pkg-config manifest, used to specify linker dependencies)
    paramcache (populates Groth parameter-cache - typically /tmp/filecoin-proof-parameters)
    paramfetch (fetches Groth parameters from IPFS gateway into cache)
    parameters.json (contains the most recently-published Groth parameters)

    Groth Parameters

    On occasion, a rust-fil-proof developer will want to publish new Groth parameters. They might do this because we changed some aspect of VDF PoSt or ZigZag PoRep setup or public parameters which obsoletes the previously-published parameters.

    To do this, the developer first runs paramcache, which populates the Groth parameter-directory on their computer with parameters which reflect the state of the rust-fil-proofs proofs code. Then, the developer runs parampublish. The parampublish program facilitates the process of adding the parameters to a local IPFS node and adds the new parameters to the parameters.json file in the root of rust-fil-proofs which is tracked by Git.

    Finally, the developer messages the pinbot with the cid of the Groth parameters which have been pinned at the IPFS gateway.

    Consuming Distributed Artifacts


    go-filecoin depends on rust-fil-proofs as a submodule. At any time, the specific Git SHA of rust-fil-proofs which is consumed by go-filecoin can be determined by running the following command from the root of go-filecoin:

    git ls-tree master proofs/rust-proofs

    When the deps command is run (either in the CircleCI or on a developer computer), the script is run.

    If both of the FILECOIN_USE_PRECOMPILED_RUST_PROOFS and GITHUB_TOKEN environment variables are set, the script attempts to download an OS-appropriate release tarball from rust-fil-proofs (GitHub) releases. The tarball, as previously mentioned, is identified by the first 16 characters of the Git SHA from which it was created. go-filecoin uses the SHA of its rust-fil-proofs submodule to construct a release URL and requests assets via cURL. Once downloaded, the precompiled assets are unpacked to locations hard-coded into the various go-filecoin Go files which consume them (e.g. rustverifier.go, rustsectorbuilder.go).

    If the aforementioned environment variables are not set or no matching release can be found, rust-fil-proofs is built from source from within the submodule directory. Once built, assets are unpacked to locations hard-coded into the various go-filecoin Go files which consume them (e.g. rustverifier.go, rustsectorbuilder.go).

    Groth Parameters

    Groth parameters can be fetched from IPFS using the paramfetch program or can be built using the paramcache program. These programs are part of rust-fil-proofs, and are available to go-filecoin developers who have downloaded a prebuilt rust-fil-proofs build. These programs can alternatively be built from source using the script which is run by the deps subcommand.

    The paramfetch program is provided with a path to the parameters.json which is part of the rust-fil-proofs submodule. Groth parameters in this JSON file are downloaded (via cURL) from the IPFS gateway and placed into the Groth parameter directory (typically /tmp/filecoin-proofs-parameters).

    What Artifacts are Accessed At Runtime?

    A short list:

    Groth parameters are accessed by rust-fil-proofs PoSt and PoRep generation and verification routines at runtime.
    params.out (loaded once, at startup - @laser to research why this exists)
    parameters.json is accessed by paramfetch at runtime.

    What is the status of PoRep and PoSt?


    ZigZag PoRep is used in the nightly and user clusters today. @dignifiedquire is working on an alternative to ZigZag ("boring") PoRep which involves something called accumulators. He and @porcuquine are meeting on Friday to make a go/no-go decision on accumulators. Lots of math involved.


    VDF PoSt is neither complete (from a math-and-science standpoint) nor integrated into go-filecoin. According to @porcuquine, "There are parts of the circuit which haven't been implemented." In short, there are "theory problems" which prevent PoSt from being completed.

    I, Sir @laser, am working to integrate our current, incomplete PoSt implementation with go-filecoin today. There are some challenges which have emerged over the last few weeks which prevent this from being a straightforward process:

    We didn't until recently have a way to sample the blockchain for the randomness required by the challenge seed-generation process.
    Our mechanism for sampling the chain only works from within the state machine.
    A VDF PoSt can be created over a fixed number of sectors (currently 2), which means that I need to modify go-filecoin to call the PoSt generator in a loop (in the case of a miner needing to prove more than 2 sectors in a proving period).

    What Does "Use Small Sectors" Actually Do?

    Every call through CGO from go-filecoin to rust-fil-proofs includes a configuration value. This value is either Live or Test.

    When the FIL_USE_SMALL_SECTORS environment variable is set to true, go-filecoin will pass the Test value on all CGO calls to rust-fil-proofs. Otherwise, Live will be passed.

    When rust-fil-proofs operations run with the Live configuration, sectors are configured to hold 266338304 bytes (254MiB). When Test is provided, sectors are configured to hold 1016 bytes of user data.

    Sealing time is the sum of replication time (scales linearly with sector size) and proof-generation time (grows logarithmically with sector size). So: Environments (and tests) which want to seal in the smallest amount of time possible should use the Test configuration, AKA "small" sectors.

    Each configuration (sector size) uses its own circuit, and each circuit its own Groth parameters. Today, we publish both parameters (in parameters.json) to IPFS.

    Part Two: WTF is a Sector Base?

    What is a Sector Base?

    Today, the "Sector Base" is a grab-bag of functions and classes (traits) scattered across 3 Rust crates in the rust-proofs repository. These crates make up 19% of the of the 32446 lines of Rust in the rust-fil-proofs repository.

    Bundled up in libfilecoin_proofs, these crates are used by go-filecoin to:

    Write (and "preprocess") user piece-bytes to disk
    Schedules seal (proof-of-replication) jobs
    Schedules proof-of-spacetime generation jobs
    Schedules unseal (retrieval) jobs
    Provides an interface to proof-of-spacetime and proof-of-replication-verification tools (called from within the FIL state machine)
    Map between replica commitment and sealed sector-bytes
    Map between piece key (base-58 or base-58BTC-encoded CID of user piece) and sealed sector-bytes

    How does go-filecoin interact with the rust-fil-proofs Sector Base?

    The go-filecoin codebase contains two Go interfaces which are used to interact with the Sector Base:

    type Verifier interface {
    	VerifyPoST(VerifyPoSTRequest) (VerifyPoSTResponse, error)
    	VerifySeal(VerifySealRequest) (VerifySealResponse, error)
    type SectorBuilder interface {
    	AddPiece(ctx context.Context, pi *PieceInfo) (sectorID uint64, err error)
    	ReadPieceFromSealedSector(pieceCid cid.Cid) (io.Reader, error)
    	SealAllStagedSectors(ctx context.Context) error
    	SectorSealResults() <-chan SectorSealResult
    	GetMaxUserBytesPerStagedSector() (uint64, error)
    	GeneratePoST(GeneratePoSTRequest) (GeneratePoSTResponse, error)
    	Close() error

    The Verifier interface is satisfied by the RustVerifier struct, and the SectorBuilder interface is satisfied with the RustSectorBuilder struct.

    Both the RustVerifier and RustSectorBuilder struct use CGO to call Rust code directly (in-process) in the Sector Base. CGO functions like C.CBytes are used to move bytes across CGO from Go to Rust; Go allocates in the Rust heap through CGO and then provides Rust with pointers from which it can reconstitute arrays, structs, and so forth.


    go-filecoin controls the lifecycle of the rust-fil-proofs SectorBuilder. When the node starts, a CGO call is made to Rust in order to construct a rust-fil-proofs SectorBuilder. Rust returns through its FFI system a pointer to the rust-fil-proofs SectorBuilder which go-filecoin holds on to. When go-filecoin is done with its RustSectorBuilder, it calls its destructor method which returns the pointer to Rust, allowing it to be freed.

    Calls through CGO

    Each method on RustSectorBuilder struct has a rust-fil-proofs SectorBuilder equivalent. When a method on the RustSectorBuilder is called, Go makes a CGO call to the Rust equivalent, passing along a pointer to the Rust-allocated, rust-fil-proofs SectorBuilder.

    For example, the go-filecoin SectorBuilder interface contains a method:

    AddPiece(ctx context.Context, pi *PieceInfo) (sectorID uint64, err error)

    which RustSectorBuilder turns into a CGO call to the following function in rust-fil-proofs:

    pub unsafe extern "C" fn add_piece(
        ptr: *mut SectorBuilder,
        piece_key: *const libc::c_char,
        piece_ptr: *const u8,
        piece_len: libc::size_t,
    ) -> *mut responses::AddPieceResponse


    Sealed sector path
    Yet-to-be-sealed ("unsealed") path
    Piece key (base-58 encoded CID of user piece-bytes)


    The rust-fil-proofs SectorBuilder stores some mappings in its metadata:

    base-58(BTC) encoded piece CID to sealed sector
    sealed and yet-to-be-sealed sector to file path
    replica commitment to file path

    Dials on Performance

    For now, go-filecoin has none.


    go-filecoin and rust-fil-proofs must live in same process
    go-filecoin does long polling to learn when sector has been sealed
    Can't scale rust-fil-proofs SectorBuilder to multiple machines
    No easy way for a miner to add more storage to an already in-use rust-fil-proofs SectorBuilder
    All rust-fil-proofs SectorBuilder metadata always buffered into memory
    A zillion other things

    Plans for the Future

    Our short-term plans include:

    Integrate our half-baked PoSt into go-filecoin
    Design PoRep and PoSt configuration
    Allow go-filecoin to make tradeoffs between time/space
    Support multiple sector sizes
    Brian V. involved in convos with @porcuquine
    @porcuquine has design in head but needs spec proposal
    NOTE: more sector sizes means more Groth parameters to be generated

Log in to reply


This domain for sale!


  • X

    开发网 go--filecoin--0.5.7(alphanet)

    0.5 版本的 user-devnet 自上线以来跌宕起伏,十分“精彩”


    #3502: long-range fork attack - 长程分叉攻击:此攻击手法利用0.5.6及以前版本中重量计算方式的缺陷,通过分叉增加个人分叉链的重量进行攻击,此攻击手法在重量算法改进后将不再有效;

    #3523: 同一高度重复出块攻击:此攻击手法利用共识惩罚机制的不完善进行攻击。在惩罚机制中,如果一个矿工在同一高度出多余一个区块,将被判处死刑,即没收全部抵押,取消算力,终止此矿工的永久挖矿资格。在0.5的当前版本中,此惩罚机制还未实现;

    #3502的攻击具有重量上限:总重量不能突破高度的 110 倍,但利用 #3523 的缺陷进行攻击完全没有上限,几乎可以无限增加重量;




    网络目前显示最高区块高度为 43260;

    但,最重区块高度为 23276(同一高度重复出块多达500个);





    ProvingPeriod 更新的问题;

    网络升级以及alphanet 权重函数 (#3536)

    (本周重点Issue)过去一周,多个矿工利用 issue#3523 和issue#3502 对网络进行长程攻击。

    为了应对这种攻击,官方迅速更新网络计算重量的算法。由于当前网络实现了升级功能,官方希望通过升级来实施权重函数的替换,因此这个issue不仅实现了新的权重函数(重构权重函数从根状态中读取数据,升级父块的权重校验方法),还包含了对网络升级的支持。另外,这个issue还引入了alphanet中临时的仲裁高度43000,并由 IsHeavier 函数来判断升级前后的处理。

    由于 issue#3523 尚未修复,攻击者很容易对网络实施长程攻击。尽管当前的开发网络包含了这个更新,但在网络升级的过程中,又遭到了一次更严重的攻击。所以,目前网络处于崩溃状态:部分节点处于升级后的43000+高度,部分节点处于升级前的23000+的高度。


  • X



    正 文


    在7月份,我们重新组合了项目工作组,包括Team Bifrost、包管理器等等!


    js-ipfs 0.37.0和0.38.0发布

    js-libp2p 0.26.0发布
    8月初,js-libp2p也得到了Gossipsub的支持,并有望实现libp2p API。

    go-ipfs 0.4.22发布
    go-ipfs版本0.4.19、0.4.20和go-ipfs 0.4.21发布了非常多的关键回归,这些回归都是在这个最新的补丁版本中修复的。



    现在可以在浏览器中发布JS IPFS HTTP API客户端

    由于技术原因,浏览器中的HTTP API上的pubsub不受支持。但是一旦fetch API支持取消,支持js-ipfs的pubsub就到来了。





    IPFS + ENS无处不在:引入EthDNS


    IPFS Camp



    2019年IPFS Camp深潜会议的故事,是由一位DWeb历史研究员在遥远的未来所讲述的

    IPFS Camp核心课程视频

    IPFS Camp Sci-Fi Fair视频


    管IPFS上的Slate文档, Temporal


    使用IPFS分散化我们的网站,Henrique Dias


    以太坊网格中的IPFS入门,Ethereum Grid



  • X

    我们很高兴地宣布我们的第一个备用 Filecoin 实现
    lotus (

    几个月以来,我们一直在努力构建最初的实现—go-filecoin,同时还与其他团队合作构建 Filecoin 协议规范的其他多个实现。

    这项构建不会改变我们的测试网和主网的时间线,但是会使 Filecoin 主网更加安全和灵活。

    Lotus 是即将推出的众多实现中的第一个。具体来说,今天我们宣布:

    Lotus 代码库的开放

    Lotus 开发网的发布

    正如我们在其他地方提到的,构建一个区块链就像构建一个‘软件火箭’。我们正努力确保我们的协议和软件实施的安全性 -- 我们拥有内部的协议安全团队,并且正在执行内部实现的安全审核和外部安全审核等等。


    拥有可在同一网络上交互操作的多个软件实现,可以大大降低这种风险。尽管每个实现都有自己的一些漏洞,但是所有实现都不太可能具有完全相同的安全问题 - 尤其是在这些实现不共享对于安全性至关重要的软件组件的情况下。

    因此,启动具有多种实现方式的加密货币网络可降低发生灾难性漏洞的可能性,这些漏洞可能会对整个 Filecoin 网络产生不利影响,甚至使整个Filecoin网络陷入瘫痪。


    自从以太坊推出了 Geth 和 Parity 以来,我们已经看到越来越多的加密网络采用多种软件实现来启动。由于多种实现对于网络安全性,弹性和社区参与非常重要,因此我们已定下目标:

    要在2020年3月Filecoin 主网上线时推出具有2种以上可交互操作的协议的实现。

    截止到今天,我们正在开发一系列 Filecoin 的实现 - go-filecoin、lotus 和其他即将正式推出的实现。我们正在寻找更多有才华的团队来构建 Filecoin 实现。


    Lotus是 Filecoin 协议规范的最简且具有实验性的实现,用 Go 语言编写。该代码库尚未完成功能,但是已经实现了几个核心 Filecoin 协议功能,包括:

    挖矿:矿工现在可以在 Lotus 开发网上开始挖矿并开始获得存储能力。Lotus 还实现了多种扇区大小,因此具有许多不同存储配置的矿工可以在网络上进行挖矿。Lotus 集成了 rust-fil-proofs,因此矿工可以按时提交时空证明,以免遭到削减。而且,存储挖矿功能是作为一个单独的模块实现的,因此高级矿工可以根据自己的特定硬件配置优化其挖矿过程。




    除了开放 lotus 代码库外,今天我们还宣布启动第一个公开 lotus 开发网络。

    设置 lotus 节点后,您可以连接到开发网并与其他节点一起开始存储数据或开始挖矿。更多详细信息,请阅读lotus入门:
    (入门 )


    lotus 代码库仍然是 Filecoin 协议的最简且实验性的实现,从现在到我们计划在2019年12月11日发布的测试网之间它将快速发展。

    从现在到测试网启动,lotus 开发网将被多次重置。在2019年12月11日启动测试网之前不要期望网络稳定。

    我们计划在2020年3月启动具有 go-filecoin,lotus和至少一个其他实现的 Filecoin 主网络。

    lotus 开发网和 go-filecoin alphanet 当前不是同一网络。在 go-filecoin 和 lotus 节点可以交互操作之前,它们将继续是不同的网络。此次交互操作将在我们于2020年3月主网启动之前进行,但确切日期仍待定。

    Lotus 入门

    鼓励矿工、客户和开发者从今天开始尝试 lotus。我们希望您的参与将有助于我们在2019年12月11日启动测试网之前增强 lotus 代码库和网络的安全性。

    对于矿工和用户来说,现在主要有 4 种方法可以参与 lotus:

    在本地下载,安装和运行 lotus

    此处说明( )
    连接到 Lotus 开发网

    此处说明( )
    为 lotus 代码库做出贡献

    此处 ( )有未解决的问题 (open issues)
    通过 lotus 开发网检测板 ( )探索网络活动

    具有探索精神的开发者可以开始探索 lotus RPC API,以开始在 lotus 开发网(最终是 testnet)上构建应用程序。请注意,在测试网之前,lotus 将持续发展变化,因此请做好应对变化的准备。


    Lotus 帮助讨论论坛

    社区 中的#fil-lotus 频道

    社区 中的#fil-help 频道

  • X

    It has come to my attention that storage clients wish to obtain the CommD and CommR associated with the sector into which the piece referenced in their storage deal has been sealed. The client can already use the query-deal command to obtain the CID of the commitSector message corresponding to that sector - but message wait doesn't show individual commitSector arguments - it just shows some string encoding of the concatenated arguments' bytes.
    I propose to augment the ProofInfo struct with CommR and CommD such that the storage client can query for their deal and, when available, see the replica and data commitments in the query-storage-deal command. Alternatively, the query-storage-deal code could get deal state from the miner and use the CommitmentMessage CID to look up CommR and CommD (on chain) - but this seems like more work than is really necessary.


Looks like your connection to Filecoin.cn中国爱好者社区 was lost, please wait while we try to reconnect.