NLnet Grant Application Draft - py-ipfs-lite Copy the fields below directly into the NLnet proposal form at https://nlnet.nl/propose/ ============================================================ CONTACT INFORMATION ============================================================ Your name: Sumanjeet Email address: sumanjeet0012@gmail.com Phone number: 91 9576531960 Organisation: IPFS-Meshkit Country: India ============================================================ GENERAL PROJECT INFORMATION ============================================================ Proposal name: py-ipfs-lite - A Lightweight Embeddable IPFS Peer for Python Website / wiki: https://github.com/IPFS-Meshkit/py-ipfs-lite ============================================================ ABSTRACT ============================================================ py-ipfs-lite is a lightweight, embeddable Python IPFS peer built on py-libp2p. It provides core IPFS features - Bitswap, KadDHT, IPNI, IPNS, IPLD DAGs, and CAR file support - without a heavy Kubo daemon. Python developers can run a full IPFS node inside their applications and interoperate with Kubo over Bitswap. Already built (v0.1.1): Complete IPFS peer with adapter pattern, Bitswap with Kubo codec compatibility (merged into libp2p/py-libp2p), TieredRouting (KadDHT + IPNI), IPLD DAG support, CAR v1 import/export, IPNS with V2 crypto verification, FastAPI HTTP daemon with /api/v0 endpoints, Prometheus metrics, CLI, and 21 examples. This grant will fund (6 months): Service layer extraction, MCP server for AI agent integration, frontend completion, protocol hardening, documentation, and PyPI release. Why it matters: There is no embeddable IPFS peer for Python. Python dominates AI and research where content-addressed storage is critical for verifiable agent memory, distributed RAG, and tamper-evident data. py-ipfs-lite fills this gap. ============================================================ PREVIOUS INVOLVEMENT ============================================================ I am a maintainer of py-libp2p (canonical Python libp2p) with 34+ merged PRs. Key contributions: - Kademlia DHT (PR #579) - Full implementation: k-bucket routing, iterative lookup, provider records. - Bitswap (PR #980, #1321) - Block exchange protocol and Kubo codec compatibility (merged into canonical py-libp2p). - Bootstrap (PR #711), Rendezvous (PR #916), mDNS (PR #649), Random Walk (PR #822) - Peer discovery protocols. - DHT improvements: fallback lookup (PR #1068), k-bucket splitting (PR #846), custom validators (PR #1095). - Transport: multi-transport support TCP/QUIC/WebSocket (PR #1357), QUIC fixes (PR #1393), TLS hardening (PR #1341, #1346), Circuit Relay v2 (PR #1343). - Interop testing: py-libp2p to rust-libp2p (PR #1034), transport config (PR #1042, #1047). Also maintainer of multiformats libraries (py-multiaddr, py-multihash, py-multibase, py-multicodec), IPLD libraries (py-cid, py-ipld-dag), and unified-testing for cross-implementation libp2p interop. I created py-ipfs-lite from scratch on top of these py-libp2p contributions. ============================================================ REQUESTED SUPPORT ============================================================ Requested Amount: $12,000 USD (approximately 11,000 EUR) ============================================================ BUDGET EXPLANATION ============================================================ The requested budget will be used for the following work over 6 months: 1. Core protocol hardening - 35% - $4,200 Complete Bitswap streaming, harden CAR v1 import/export, finalize IPNS V2 verification, ensure full Kubo interoperability across all codecs, and improve connection stability. 2. Service layer and MCP server - 25% - $3,000 Extract transport-agnostic service layer from the HTTP API. Build MCP server exposing py-ipfs-lite as tools for AI agents (add, fetch, pin, verify content). Enables AI agent integration. 3. Frontend development - 20% - $2,400 Complete the React + TypeScript frontend: file upload/download, DAG browser, peer dashboard, IPNS management, CAR import/export UI. 4. Documentation and examples - 10% - $1,200 Expand documentation with guides for AI agents, RAG pipelines, Filecoin integration, and Kubo interop. Publish API docs via MkDocs. 5. Testing and PyPI release - 10% - $1,200 Comprehensive test suite, interop testing with Kubo, CI/CD pipeline, and PyPI publication for easy installation via pip install py-ipfs-lite. Rates: Competitive contractor rate for Python/libp2p development in India. Other funding sources: None at this time. ============================================================ COMPARISON WITH EXISTING OR HISTORICAL EFFORTS ============================================================ Comparison with Kubo (Go-IPFS): Kubo is the dominant IPFS implementation but is a monolithic Go daemon. It cannot be embedded in Python apps, requires separate process management, and is not installable via pip. py-ipfs-lite provides a pure Python peer that runs as a library - no daemon, no HTTP overhead. Comparison with other Python IPFS efforts: - ipfshttpclient: HTTP client only, requires running Kubo daemon. - py-ipfs: Unmaintained, simple wrapper, no native protocol. - py-libp2p: libp2p primitives only, no IPFS layer (no Bitswap, DAG, IPNS). - py-ipfs-lite: Full IPFS peer (libp2p + Bitswap + DHT + IPLD + IPNS + CAR), embeddable. Comparison with NLnet-funded IPFS projects: ipfs-search.com (NGI0 Discovery, 2019-2022) is a search engine for IPFS content. It crawls the DHT, indexes files and directories, and provides a search interface. However, it relies on a running Kubo daemon for IPFS access. py-ipfs-lite could replace that dependency - providing an embeddable Python IPFS peer that the crawler can use directly without managing a separate daemon process. The two projects address different layers: ipfs-search solves discovery, py-ipfs-lite solves the peer itself. Oku (NGI0 Entrust, 2023-2024) is a browser with built-in IPFS support, encrypted data vaults, and onion routing. It is an end-user application. py-ipfs-lite is infrastructure that could power the IPFS layer of applications like Oku. Where Oku focuses on the browser experience, py-ipfs-lite provides the protocol-level peer that any Python application - browser, CLI, library - can embed. Software Heritage x IPFS (NGI Assure, 2021-2023) bridges the Software Heritage archive with IPFS, allowing users to access preserved source code via IPFS CIDs. It uses a custom IPFS plugin to translate SWH identifiers to IPFS CIDs. py-ipfs-lite's CAR v1 export/import and Bitswap interop make it a natural fit for similar archival pipelines - any tool that needs to export content-addressed data bundles for long-term storage or cross-network transfer can use py-ipfs-lite natively. SCION-IPFS (NGI Zero Core, 2023-2026) enhances IPFS performance through SCION's path-aware networking, optimizing DHT lookups and content routing. It works with the Go IPFS stack. py-ipfs-lite provides the Python-side equivalent - a peer that could benefit from SCION integration for path-optimized content routing in Python applications. The two projects are complementary: SCION-IPFS improves the network layer, py-ipfs-lite provides the application layer peer. ============================================================ SIGNIFICANT TECHNICAL CHALLENGES ============================================================ 1. Bitswap Kubo Interoperability The Bitswap protocol must handle codec negotiation correctly across dag-pb, dag-cbor, and dag-json to interoperate with Kubo daemons. This requires careful implementation of the WANT/HAVE/BLOCK message exchange and correct CID parsing across different codec versions. The upstream Bitswap improvements were developed as part of this project and merged into libp2p/py-libp2p (PR #1321), but ongoing compatibility testing with new Kubo releases is essential. 2. Concurrent Safety with Trio py-ipfs-lite uses Trio's structured concurrency model, which requires careful handling of cancel scopes and nursery lifecycles. The GC system must use a reader-writer lock to allow concurrent file additions while ensuring exclusive access during cleanup. The get_file streaming path must avoid cancel scopes spanning async generator yield boundaries - a subtle Trio constraint that caused nursery corruption in earlier versions. 3. IPNS Cryptographic Verification IPNS records require full cryptographic validation: public key hash verification against the PeerID, V2 signature verification over the signed CBOR payload, and expiry checking. The trust model must be clearly documented - IPNI announce-side trust is not verified (a property of the protocol), while DHT record integrity relies on the signature check. 4. Service Layer Architecture The current HTTP API mixes transport mechanics, business logic, and error handling in every route. Extracting a transport-agnostic service layer (shared by both the FastAPI HTTP adapter and the MCP server) requires careful separation of concerns while maintaining backward compatibility. 5. MCP Server for AI Agents Building an MCP server that exposes py-ipfs-lite as tools for AI agents introduces new challenges: Trio compatibility with the MCP SDK (which runs on anyio), handling large file transfers (inline vs. disk-backed), and ensuring the service layer provides the right abstractions for both HTTP and MCP tool-call paradigms. ============================================================ ECOSYSTEM AND ENGAGEMENT ============================================================ Ecosystem Actors: 1. IPFS and Protocol Labs: py-ipfs-lite interoperates with Kubo over Bitswap and follows IPFS specifications (IPLD, CAR, IPNS). We will contribute Bitswap improvements back to py-libp2p and engage with the IPFS community through the IPFS Forum and weekly calls. 2. libp2p Python community: The project builds on py-libp2p and has already contributed upstream improvements (PR #1321). We will continue contributing protocol improvements back to the canonical repository. 3. AI Agent frameworks: The MCP server integration positions py-ipfs-lite as a tool for AI agents. We will engage with the MCP ecosystem and AI agent frameworks (LangChain, CrewAI, AutoGen) that need verifiable, content-addressed storage. 4. Filecoin ecosystem: CAR v1 export/import enables workflows where content stored via py-ipfs-lite can be submitted to Filecoin for long-term storage. 5. Python developer community: Through PyPI distribution (pip install py-ipfs-lite), comprehensive documentation, and 21+ runnable examples, we will make IPFS accessible to the broader Python community. Promotion Strategy: - Open source: MIT-licensed, published on GitHub under IPFS-Meshkit organization - PyPI distribution: Installable via pip install py-ipfs-lite - Documentation: Full API docs via MkDocs, guides for AI agents, RAG, Filecoin, Kubo interop - Examples: 21+ runnable examples covering all features - Marketplace: Plan to launch py-ipfs-lite as a managed service (similar to AWS managed IPFS) for easy deployment - MCP Servers: Plan to publish MCP servers for AI agent integration Open Source Sustainability: The adapter pattern for all five subsystems (Host, Routing, BlockStore, Exchange, DagService) allows community members to swap implementations without forking. The service layer architecture is designed for extensibility - new transport adapters (beyond HTTP and MCP) can be added without changing the core peer logic. ============================================================ GENERATIVE AI ============================================================ Did you use generative AI in writing this proposal? (Select the appropriate option based on your actual process.) ============================================================ ATTACHMENTS ============================================================ Suggested attachments to include with the proposal: 1. Architecture diagram - from docs/architecture.md showing the Peer, five subsystems, and adapter pattern 2. AI agents and RAG guide - from docs/guides/ai-agents-and-rag.md demonstrating verifiable agent memory chains and distributed RAG 3. Service layer plan - from service-layer-and-mcp-plan.md showing the planned service layer architecture 4. Examples index - from docs/reference/examples-index.md listing all 21 examples 5. Interop test results - from TEST_RESULTS.md showing Kubo interoperability status 6. CONNECTION_STABILITY_FIXES.md - documenting connection stability improvements ============================================================ END OF PROPOSAL DRAFT ============================================================