# Peer-to-Peer Access to Our Software Heritage - Full Application Content ## Project Overview **Name:** Peer-to-Peer Access to Our Software Heritage (SWH x IPFS) **Tagline:** Access Software Heritage data via IPFS DHT **Fund:** NGI Assure **Period:** 2021-04 — 2023-04 **Grant Agreement No:** 957073 **URL:** https://nlnet.nl/project/SoftwareHeritage-P2P/ **Website:** https://github.com/obsidiansystems/go-ipfs-swh-plugin/ **Source Code:** https://github.com/obsidiansystems/go-ipfs-swh-plugin/ **Run by:** Software Heritage Foundation **Implementation by:** Obsidian Systems --- ## Abstract Peer-to-Peer Access to Our Software Heritage (SWH x IPFS) is a project aimed at supporting Software Heritage's mission to build a universal source code archive and preserve it for future generations by leveraging IPFS's capabilities to share and replicate the archive in a decentralized, peer-to-peer manner. The project will build a bridge between the existing Software Heritage (SWH) API and the IPFS network to transparently serve native IPFS requests for SWH data. In the short term, this allows users using IPFS to form their own Content Distribution Network for SWH data. Longer term, we hope this will serve as a foundation for a decentralized network of copies that, together, ensure that the loss of no one repository, however large, results in the permanent destruction of any part of our heritage. The end product would be a perfect application of IPFS's tools and a step in the direction of a decentralized internet services infrastructure. --- ## Why does this matter? Software Heritage is a non-profit initiative with a mission to preserve open source software code. It houses the largest public archive of software source code in the world. IPFS, developed by Protocol Labs, provides a decentralized, peer-to-peer protocol designed to replace centralized internet infrastructure. These two ecosystems are a natural match for a variety of reasons. They share the goal of preserving access to cultural commons, one by archiving software repositories and the other by distributing them. Behind the scenes, they also share the technical bases of content addressing and Merkle DAGs. Connecting them together is like connecting a warehouse (SWH) and a railroad (IPFS). --- ## Concept In a request-response flow between IPFS and SWH, the bridge node plugin functions as middleware: ### Technical Foundation Both IPFS and SWH understand git hashing, all the way down. This crucial design decision by both parties transforms the process of bridging them from something quite difficult into something so natural the code practically writes itself. - **Software Heritage** stores all files indexed by git hash - **Directories** are stored as git tree objects, referring to children by git hashes - **IPFS** supports git hashing among its content addressing formats ### Identifier Conversion The two content-addressed systems are similar enough that IPFS CIDs and SWHIDs can be converted between one another without seeing the actual content referenced by either ID. Not every IPFS CID can be an SWHID, but every SWHID can become an IPFS CID. ### Data Integrity Because all parties understand the same content addressing format, data integrity is easy. This simplifies communication and removes the need for fussy translation of identifiers. All data is tamper-proof throughout because content-addressed references cannot be fooled without triggering a hash mismatch. --- ## How it works ### Bridge Node Architecture Just one IPFS node with the plugins installed is required to bridge IPFS and SWH. Other IPFS nodes requesting objects from the bridge node do not need the plugins or any other extra modifications. ### Usage Workflow #### Option A: Public Bridge Node Software Heritage deploys a public bridge node: ``` /dns4/ipfs.softwareheritage.org/tcp/4001/p2p/12D3KooWBThWcjxdCsXg2jRWoBnBkLyr8wghDArHotApjJZAyW3T ``` Connect to it: ```bash ipfs swarm connect /dns4/ipfs.softwareheritage.org/tcp/4001/p2p/12D3KooWBThWcjxdCsXg2jRWoBnBkLyr8wghDArHotApjJZAyW3T ``` #### Option B: Self-hosted Bridge Node 1. Build IPFS with plugin using Nix: ```bash $ nix-build ``` 2. Initialize bridge configuration: ```bash $ ./result/bin/ipfs init -e -p swhbridge ``` 3. Add SWH authentication token to config: ```diff --- a/config +++ b/config @@ -11,7 +11,8 @@ "mounts": [ { "child": { - "type": "swhbridge" + "type": "swhbridge", + "auth-token": "" }, "mountpoint": "/blocks", "prefix": "swhbridge.datastore", ``` 4. Start the bridge daemon: ```bash GOLOG_LOG_LEVEL="swh-bridge=info" \ IPFS_PATH=~/.ipfs-bridge/ \ ./result/bin/ipfs daemon ``` ### Client Node Setup 1. Initialize client: ```bash IPFS_PATH=~/.ipfs-client/ \ ./result/bin/ipfs init ``` 2. Adjust ports in config to avoid conflicts 3. Start client daemon: ```bash GOLOG_LOG_LEVEL="swhid-debug" \ IPFS_PATH=~/.ipfs-client \ ./result/bin/ipfs daemon ``` 4. Connect to bridge: ```bash ipfs swarm connect /ip4/127.0.0.1/tcp//p2p/ ``` --- ## Querying the Archive ### Converting SWHIDs to IPFS CIDs Using the `makecid.py` script or by prepending magic bytes: #### For files, directories, revisions, releases: ``` swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ copy this part ^^^ must be one of "cnt", "dir", "rev", or "rel" f0178111494a9ed024d3859793618152ea559a168bbcbb5e2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ paste it here ^^^^^^^^^ prefix so CID is self-describing (means roughly CID v1, "git-raw" codec, SHA-1, base-16) ``` #### For snapshots: ``` swh:1:snp:9dcebebe2bb56cabdd536787886d582b762a0376 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ copy this part ^^^ must be "snp" f01F00311149dcebebe2bb56cabdd536787886d582b762a0376 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ paste it here ^^^^^^^^^^^ prefix so CID is self-describing (means roughly CID v1, "swh-1-snp" codec, SHA-1, base-16) ``` ### Example Queries #### Fetch a file (GPLv3 text): ```bash $ ./result/bin/ipfs dag get \ --output-codec=git-raw \ f0178111494a9ed024d3859793618152ea559a168bbcbb5e2 ``` #### Fetch a directory listing: ```bash $ ./result/bin/ipfs dag get \ f017811141ecc6062e9b02c2396a63d90dfac4d63690e488b | jq ``` #### Fetch a revision: ```bash $ ./result/bin/ipfs dag get \ f017811141a0dd0088247f9d4e403a460f0f6120184af3e15 | jq ``` #### Fetch a snapshot: ```bash $ ./result/bin/ipfs dag get \ f01F00311149dcebebe2bb56cabdd536787886d582b762a0376 | jq ``` #### Recursive queries: ```bash $ ./result/bin/ipfs dag get \ --output-codec=git-raw \ f017811141a0dd0088247f9d4e403a460f0f6120184af3e15/tree/compiler/hash/GHC/hash/Core/hash/Type.hs/hash ``` --- ## Implementation Details ### SWH API Enhancement A new endpoint was added to the SWH API: - `GET /api/1/raw/(swhid)` - Returns raw bytes for any SWH object - Works on all object types (files, directories, commits, tags, snapshots) - Ensures tamper-proof data integrity through content addressing ### IPFS Node Plugins #### Datastore Plugin - Allows IPFS to talk to Software Heritage - Functions as a read-only datastore - Transforms IPFS multihashes into SWHIDs - Issues requests to SWH API and returns responses - Caching built-in (IPFS caches results automatically) #### IPLD Plugin - Parses SWH snapshot objects (non-git type) - Enables recursive querying through Merkle DAG structure - Exposes content addresses of child objects for fetching ### Caching Behavior - Bridge node responds to requests for objects SWH has - Requests for objects SWH doesn't have are ignored - Client nodes cache results for future requests - Deleting cache forces re-fetch from bridge --- ## Future Work ### Performance and Pipelining - Currently: Entire source projects downloaded piece by piece - Problem: High latency due to round trips between client and SWH API - Solution: Implement pipelining for batch requests - Both IPFS and SWH are building out support independently ### Large Objects - IPFS requires chunking (small max atomic object size) - Git doesn't chunk (objects can be arbitrarily large) - Short-term: Cryptography tricks to break up objects without losing content addressing - Long-term: Influence Git's future direction (see git-lfs) --- ## Browsing the Archive Recommended approach: 1. Go to https://archive.softwareheritage.org/browse/ 2. Find a file 3. Click the floating "Permalinks" on the right side 4. Get a `swh:1:cnt:...` SWHID 5. Convert to IPFS CID and fetch --- ## Additional Logging The bridge uses `go-log` like most IPFS software. Enable logging: ```bash GOLOG_LOG_LEVEL="swh-bridge=info" ``` Use `=debug` for more verbose output. --- ## Themes - Data and AI - Decentralised solutions - NGI Assure - Network infrastructure --- ## Funded By This project was funded through the NGI Assure Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 957073.