What we changed in Onyx — our FOSSASIA 2026 talk

Onyx is a production-ready open-source stack for enterprise RAG. We piloted it for public-sector search, and here's what we changed to answer real questions from tables, PDFs, and dashboards.

In March our CTO, Dylan Jay, gave a talk at the FOSSASIA Summit in Bangkok about Onyx, an open-source stack for enterprise RAG that's genuinely production-ready, and the customisations we made piloting it for public-sector search. Onyx is a strong base to build on. This is what we had to change to answer the questions that break a naive setup.

The slides are online, and the talk was part of the FOSSASIA 2026 schedule.

What we were actually solving

The brief was a reliable assistant on large public-sector sites: agencies with years of publications, reports, and dashboards behind them. The questions that matter are the ones a naive setup gets wrong. A regulator's users ask how many complaints were upheld in 2024. A health service's users ask for the COVID vaccine information in Arabic. Both need more than fetching a paragraph that looks relevant.

We piloted both: AI search across regulator complaints and multilingual health translation.

Getting it right is a governance question

For a regulator or a health service, a wrong answer isn't a UX glitch; someone acts on it. A confident but incorrect number is worse than no answer at all. So the bar isn't whether an answer sounds plausible, it's whether it's correct and traceable to a source. Most of what follows (computing over tables instead of guessing, careful task-specific prompting, testing against real questions) serves that one idea.

Why Onyx

Onyx is an open-source enterprise RAG platform, MIT-licensed, with the parts you'd otherwise assemble by hand: forty-odd connectors, document-level permissions, hybrid search with reranking, agents, and an admin UI. Among open-source options it's the only turnkey one; the rest are toolkits you still have to build a product on. For a pilot that had to look production-shaped from the first week, that mattered. It's also clean, typed Python with real test coverage, which is what made customising it realistic rather than a fork-and-hope.

The table problem

Ask a RAG system what the total revenue was for 2024 and it hands you a chunk, not an answer. The table got shredded into fragments on the way in, so vector search finds a relevant-looking row and stops. Worse, it might add up a partial column and sound confident about the wrong number. Ask for the latest report and you get a random report chunk, answered at random.

Our fix runs a SQL pathway alongside the normal search. The same question generates a query that computes across the whole table, and the real number goes back to the model next to the semantic results. Total revenue for 2024 was $4.2M, not no documents found. The latest report is today's, not the latest we can see is 2023. Most RAG returns pre-computed answers; tables need computation, so we compute.

PDFs are harder than they look

PDF table extraction is where good demos go to die. Headers and footers, tables that run across pages, bullets that aren't really bullets: layout analysis matters more than OCR, and most PDF-to-markdown tools are tuned for tidy boxed content rather than dense government documents. We made the extraction pipeline pluggable, swapped in a layout-aware analyzer with cross-page table merging, and added a spreadsheet strategy for XLSX that understands named tables. Because Onyx's chunk pipeline doesn't care about the source format, new ones slot in cleanly.

Connectors

Onyx's connector interface is clean enough to extend without fighting it. We added PowerBI dashboard scraping, since that's where a lot of the numbers actually live, plus CSV, XLSX and PPTX table extraction and sitemap improvements that discover linked files. Keeping each connector separate meant we could test them one at a time instead of debugging the whole ingest at once.

Knowing whether it's any good

The lesson we'd pass on: test with real questions, early. There are a lot of knobs (chunking, model choice, reranking) and no tuning them by feel. We built a regression suite for answer quality, scored it with RAGAS for relevancy, faithfulness and factual correctness, and used langfuse to watch real questions, answers, prompts and the chunks behind them, wired into Onyx's own feedback. We also changed the chunking: instead of relating each chunk to a document summary, we relate it to the few chunks before it and the summary, which holds context better.

Scale-to-zero on Fly.io

A pilot shouldn't cost like production. Onyx runs its background work through Celery with named queues, so we split the one monolithic worker into eight specialised ones, each suspending when idle. A watchdog watches the queues and wakes a worker when there's something to do. The lesson there: scale-to-zero needs connection resilience, because different services reconnect differently after a suspend.

Why Onyx is a good base

Pulling it together: clean typed Python, connector and extraction interfaces you can plug into, a composable search pipeline, Celery separation you can scale piece by piece, and enough test coverage to refactor without fear. Upstream stays active and its fixes flow back, and pulling those merges in has been painless so far. That's the difference between customising a product and maintaining a fork.

Dylan enjoyed giving this one, and it was a real pleasure to meet so many of the international guests and speakers who flew in from around the world for the summit.

We're open to collaborations and contracts with teams building AI systems that need better structured-data retrieval: AI search, RAG deployment, table and document extraction, Onyx customisation. There's more on our Onyx work.