# NLnet Grant Application Draft — py-ipfs-lite > **Copy the fields below directly into the NLnet proposal form at https://nlnet.nl/propose/** --- ## Contact Information **Your name:** Sumanjeet **Email address:** sumanjeet0012@gmail.com **Phone number:** 91 9576531960 **Organisation:** IPFS-Meshkit **Country:** India --- ## General Project Information **Proposal name:** py-ipfs-lite — A Lightweight Embeddable IPFS Peer for Python **Website / wiki:** https://github.com/IPFS-Meshkit/py-ipfs-lite --- ## Abstract **py-ipfs-lite** is a lightweight, embeddable Python IPFS peer built on top of `py-libp2p`. It provides the core features of IPFS — Blockstore, Bitswap, DHT, IPNI, IPNS, and IPLD DAGs — without the overhead of a heavy Kubo daemon. The project enables Python developers to run a complete IPFS node directly inside their applications and interoperate seamlessly with Kubo over Bitswap. **What is already built (v0.1.1):** - **Core IPFS peer:** The `Peer` class orchestrating five subsystems (Host, Routing, BlockStore, Exchange, DagService) with an adapter pattern for swappability. - **Bitswap protocol:** Block exchange with Kubo-compatible codec negotiation (dag-pb, dag-cbor, dag-json). Upstream improvements merged into `libp2p/py-libp2p` (PR [#1321](https://github.com/libp2p/py-libp2p/pull/1321)). - **KadDHT + IPNI TieredRouting:** Content discovery via Kademlia DHT and Delegated HTTP IPNI (`cid.contact`). - **IPLD DAG support:** `dag-pb`, `dag-cbor`, and `dag-json` codecs for building and traversing content-addressed graphs. - **CAR v1 export/import:** Offline data bundles for archival and portability. - **IPNS:** Publish and resolve mutable pointers with full V2 cryptographic verification. - **FastAPI HTTP daemon:** Kubo-compatible `/api/v0` endpoints for adding files, fetching DAGs, and pinning. - **Prometheus metrics:** Observability for garbage collection, block sizes, and Bitswap traffic. - **CLI:** `py-ipfs-lite daemon` command for standalone operation. - **21 examples:** Covering file operations, DAG construction, CAR files, IPNS, AI agent memory chains, and distributed RAG. - **Documentation:** Architecture guide, AI agents guide, CAR/Filecoin guide, Kubo interop guide. **What this grant will fund (6 months):** 1. **Service layer extraction** — Refactor the HTTP API into a transport-agnostic service layer shared by both the HTTP daemon and the planned MCP server. This is documented in detail in `service-layer-and-mcp-plan.md`. 2. **MCP (Model Context Protocol) server** — Expose py-ipfs-lite as tools for AI agents, enabling programmatic content add, fetch, pin, and verification. This positions py-ipfs-lite as infrastructure for AI applications requiring verifiable, content-addressed storage. 3. **Frontend completion** — Complete the React + TypeScript web UI for file management, DAG browsing, peer connectivity dashboard, and IPNS management. 4. **Protocol hardening** — Improve connection stability, CAR v1 robustness, IPNS V2 edge cases, and Kubo interoperability across new releases. 5. **Documentation & PyPI release** — Expand guides, publish full API docs via MkDocs, and release on PyPI (`pip install py-ipfs-lite`) for easy adoption. **Why this matters:** The IPFS ecosystem has no production-grade, embeddable Python peer. Python is the dominant language in AI, data science, and research — fields where content-addressed storage is increasingly critical for verifiable agent reasoning, distributed RAG, and tamper-evident data pipelines. py-ipfs-lite fills this gap, and the MCP server integration makes it directly usable by AI agent frameworks. --- ## Previous involvement I am a maintainer of **py-libp2p**, the canonical Python implementation of the libp2p networking stack. I have contributed 34+ merged pull requests to the project, including foundational work on peer discovery, Kademlia DHT, Bitswap, and transport protocols. Key contributions include: **Core Protocol Implementations:** - **Kademlia DHT** ([#579](https://github.com/libp2p/py-libp2p/pull/579)) — Full Kademlia DHT implementation including k-bucket routing, iterative lookup, value storage, and provider records. 39 comments of review. - **Bitswap** ([#980](https://github.com/libp2p/py-libp2p/pull/980)) — Implemented the Bitswap block exchange protocol for block-level data exchange between IPFS peers. - **Bitswap Kubo Compatibility** ([#1321](https://github.com/libp2p/py-libp2p/pull/1321)) — Improved Bitswap for interoperability with Kubo daemons including dag-pb/cbor/json codec negotiation. 19 comments of review. Merged into the canonical py-libp2p. - **Bootstrap Module** ([#711](https://github.com/libp2p/py-libp2p/pull/711)) — Implemented the Bootstrap protocol for initial peer discovery. 20 comments. - **Rendezvous Module** ([#916](https://github.com/libp2p/py-libp2p/pull/916)) — Implemented the Rendezvous protocol for peer discovery. 22 comments. - **Multicast DNS** ([#649](https://github.com/libp2p/py-libp2p/pull/649)) — Implemented mDNS for local network peer discovery. 25 comments. - **Random Walk** ([#822](https://github.com/libp2p/py-libp2p/pull/822)) — Implemented random walk for DHT routing table population. 14 comments. **DHT & Peer Discovery Improvements:** - Fallback mechanism in Kademlia DHT for peer lookup ([#1068](https://github.com/libp2p/py-libp2p/pull/1068)) - K-bucket splitting in routing table ([#846](https://github.com/libp2p/py-libp2p/pull/846)) - Custom validators and quorum-based value retrieval ([#1095](https://github.com/libp2p/py-libp2p/pull/1095)) - DHT API migration to string keys ([#1060](https://github.com/libp2p/py-libp2p/pull/1060)) **Transport & Security:** - Multi-transport support (TCP, QUIC, WebSocket) ([#1357](https://github.com/libp2p/py-libp2p/pull/1357)) - QUIC connection lifecycle fixes ([#1393](https://github.com/libp2p/py-libp2p/pull/1393)) - TLS certificate verification hardening ([#1341](https://github.com/libp2p/py-libp2p/pull/1341), [#1346](https://github.com/libp2p/py-libp2p/pull/1346)) - Circuit Relay v2 fix ([#1343](https://github.com/libp2p/py-libp2p/pull/1343)) - PeerRecord signer identity validation ([#1339](https://github.com/libp2p/py-libp2p/pull/1339)) **Interop Testing:** - Fixed interop tests between py-libp2p and rust-libp2p ([#1034](https://github.com/libp2p/py-libp2p/pull/1034)) — 18 comments. - Transport interop testing configuration ([#1042](https://github.com/libp2p/py-libp2p/pull/1042), [#1047](https://github.com/libp2p/py-libp2p/pull/1047)) I am also a maintainer of several **multiformats** libraries (py-multiaddr, py-multihash, py-multibase, py-multicodec) and **IPLD** libraries (py-cid, py-ipld-dag), and maintain **unified-testing** for cross-implementation libp2p interoperability testing. I created **py-ipfs-lite** from scratch, building the complete IPFS peer layer on top of the py-libp2p contributions above. --- ## Requested support **Requested Amount:** $12,000 USD (approximately €11,000) --- ## Budget explanation The requested budget will be used for the following work over 6 months: | Task | % | Amount | Description | |------|---|--------|-------------| | **Core protocol hardening** | 35% | $4,200 | Complete Bitswap streaming, harden CAR v1 import/export, finalize IPNS V2 verification, ensure full Kubo interoperability across all codecs, and improve connection stability. | | **Service layer & MCP server** | 25% | $3,000 | Extract transport-agnostic service layer from the HTTP API. Build MCP server exposing py-ipfs-lite as tools for AI agents (add, fetch, pin, verify content). Enables AI agent integration. | | **Frontend development** | 20% | $2,400 | Complete the React + TypeScript frontend: file upload/download, DAG browser, peer dashboard, IPNS management, CAR import/export UI. | | **Documentation & examples** | 10% | $1,200 | Expand documentation with guides for AI agents, RAG pipelines, Filecoin integration, and Kubo interop. Publish API docs via MkDocs. | | **Testing & PyPI release** | 10% | $1,200 | Comprehensive test suite, interop testing with Kubo, CI/CD pipeline, and PyPI publication for easy installation via `pip install py-ipfs-lite`. | **Rates:** Competitive contractor rate for Python/libp2p development in India. **Other funding sources:** None at this time. --- ## Compare your own project with existing or historical efforts ### Comparison with Kubo (Go-IPFS) Kubo is the reference IPFS implementation and the most widely deployed. However, it is a monolithic Go binary that must run as a separate daemon process. For Python applications, this means: - **Operational overhead:** Managing a separate daemon process, configuring ports, managing the data directory - **No library integration:** Cannot be embedded inside a Python application; must communicate over HTTP - **Language barrier:** Go plugins cannot be used from Python; Python developers cannot extend or customize the daemon - **Deployment friction:** Requires Go toolchain or pre-built binaries; not installable via `pip` py-ipfs-lite addresses these by providing a pure Python implementation that runs as a library inside the application's async runtime. A Python developer can do `from py_ipfs_lite.peer import Peer` and have a full IPFS node running in their application — no daemon, no HTTP overhead, no process management. ### Comparison with other Python IPFS efforts | Project | Status | Limitation | |---------|--------|------------| | **ipfshttpclient** | Maintained | HTTP client only — requires a running Kubo daemon | | **py-ipfs (ipython)** | Unmaintained | Simple wrapper, no native protocol implementation | | **py-libp2p** | Active | Provides libp2p primitives but no IPFS layer (no Bitswap, no DAG, no IPNS) | | **py-ipfs-lite** | **This project** | Full IPFS peer: libp2p + Bitswap + DHT + IPLD + IPNS + CAR, embeddable | ### Comparison with NLnet-funded IPFS projects **ipfs-search.com** (NGI0 Discovery) is a search engine for IPFS content. It solves the discovery problem. py-ipfs-lite addresses a different layer: it provides the IPFS peer itself as an embeddable Python library. py-ipfs-lite could serve as the underlying IPFS peer for Python-based search crawlers. **Oku** (NGI0 Entrust) is a browser integrating IPFS for decentralized web browsing. Oku is an end-user application; py-ipfs-lite is infrastructure that could power the IPFS layer of applications like Oku. **Software Heritage x IPFS** (NGI Assure) bridges SWH with IPFS for archive access. py-ipfs-lite's CAR import/export and Bitswap interop make it a natural fit for similar archival pipelines. **SCION-IPFS** (NGI Zero Core) enhances IPFS performance through path-aware networking. py-ipfs-lite provides the Python-side IPFS peer that could benefit from SCION integration. --- ## What are significant technical challenges you expect to solve during the project, if any? ### 1. Bitswap Kubo Interoperability The Bitswap protocol must handle codec negotiation correctly across dag-pb, dag-cbor, and dag-json to interoperate with Kubo daemons. This requires careful implementation of the WANT/HAVE/BLOCK message exchange and correct CID parsing across different codec versions. The upstream Bitswap improvements were developed as part of this project and merged into `libp2p/py-libp2p` (PR [#1321](https://github.com/libp2p/py-libp2p/pull/1321)), but ongoing compatibility testing with new Kubo releases is essential. ### 2. Concurrent Safety with Trio py-ipfs-lite uses Trio's structured concurrency model, which requires careful handling of cancel scopes and nursery lifecycles. The GC system must use a reader-writer lock to allow concurrent file additions while ensuring exclusive access during cleanup. The `get_file` streaming path must avoid cancel scopes spanning async generator yield boundaries — a subtle Trio constraint that caused nursery corruption in earlier versions. ### 3. IPNS Cryptographic Verification IPNS records require full cryptographic validation: public key hash verification against the PeerID, V2 signature verification over the signed CBOR payload, and expiry checking. The trust model must be clearly documented — IPNI announce-side trust is not verified (a property of the protocol), while DHT record integrity relies on the signature check. ### 4. Service Layer Architecture The current HTTP API mixes transport mechanics, business logic, and error handling in every route. Extracting a transport-agnostic service layer (shared by both the FastAPI HTTP adapter and the MCP server) requires careful separation of concerns while maintaining backward compatibility. ### 5. MCP Server for AI Agents Building an MCP server that exposes py-ipfs-lite as tools for AI agents introduces new challenges: Trio compatibility with the MCP SDK (which runs on anyio), handling large file transfers (inline vs. disk-backed), and ensuring the service layer provides the right abstractions for both HTTP and MCP tool-call paradigms. --- ## Describe the ecosystem of the project, and how you will engage with relevant actors and promote the outcomes? ### Ecosystem Actors 1. **IPFS & Protocol Labs:** py-ipfs-lite interoperates with Kubo over Bitswap and follows IPFS specifications (IPLD, CAR, IPNS). We will contribute Bitswap improvements back to py-libp2p and engage with the IPFS community through the IPFS Forum and weekly calls. 2. **libp2p Python community:** The project builds on py-libp2p and has already contributed upstream improvements (PR [#1321](https://github.com/libp2p/py-libp2p/pull/1321)). We will continue contributing protocol improvements back to the canonical repository. 3. **AI Agent frameworks:** The MCP server integration positions py-ipfs-lite as a tool for AI agents. We will engage with the MCP ecosystem and AI agent frameworks (LangChain, CrewAI, AutoGen) that need verifiable, content-addressed storage. 4. **Filecoin ecosystem:** CAR v1 export/import enables workflows where content stored via py-ipfs-lite can be submitted to Filecoin for long-term storage. 5. **Python developer community:** Through PyPI distribution (`pip install py-ipfs-lite`), comprehensive documentation, and 21+ runnable examples, we will make IPFS accessible to the broader Python community. ### Promotion Strategy - **Open source:** MIT-licensed, published on GitHub under IPFS-Meshkit organization - **PyPI distribution:** Installable via `pip install py-ipfs-lite` - **Documentation:** Full API docs via MkDocs, guides for AI agents, RAG, Filecoin, Kubo interop - **Examples:** 21+ runnable examples covering all features - **Marketplace:** Plan to launch py-ipfs-lite as a managed service (similar to AWS managed IPFS) for easy deployment - **MCP Servers:** Plan to publish MCP servers for AI agent integration ### Open Source Sustainability The adapter pattern for all five subsystems (Host, Routing, BlockStore, Exchange, DagService) allows community members to swap implementations without forking. The service layer architecture is designed for extensibility — new transport adapters (beyond HTTP and MCP) can be added without changing the core peer logic. --- ## Generative AI **Did you use generative AI in writing this proposal?** *(Select the appropriate option based on your actual process.)* --- ## Attachments Suggested attachments to include with the proposal: 1. **Architecture diagram** — from `docs/architecture.md` showing the Peer, five subsystems, and adapter pattern 2. **AI agents and RAG guide** — from `docs/guides/ai-agents-and-rag.md` demonstrating verifiable agent memory chains and distributed RAG 3. **Service layer plan** — from `service-layer-and-mcp-plan.md` showing the planned service layer architecture 4. **Examples index** — from `docs/reference/examples-index.md` listing all 21 examples 5. **Interop test results** — from `TEST_RESULTS.md` showing Kubo interoperability status 6. **CONNECTION_STABILITY_FIXES.md** — documenting connection stability improvements --- *End of proposal draft*