
When sensitive information cannot leave the company
Private AI and RAG for businesses, in the cloud or on premises
We deploy open-source AI models on company servers or in a private cloud, connected to internal data. We also develop custom RAG systems.
What is private AI?
Private AI is an artificial intelligence system deployed within infrastructure controlled by the company, instead of sending the information to an external commercial AI such as Claude, ChatGPT, or Gemini. The model runs on a local server or in a private cloud, which makes it possible to keep the information private.
Using commercial models in the same project is not ruled out. Sometimes it makes sense to combine both systems, decide which processes handle sensitive information, and work differently for each type. In any case, in our implementations we leave the applications ready to switch models without modifying the code.
Which projects benefit from private AI?
Mainly when the information cannot leave the organization or when AI is part of a company's own product.
Companies with strict control and regulatory compliance
Healthcare, finance, insurance, law firms, and accounting firms handle special categories of data or data protected by professional secrecy. Reducing potential privacy risks matters to them. With private AI, it is possible to show where the information is and who accesses it.
Companies offering their own product that includes AI
AI is built into the tool to enhance its features and is integrated with the rest of the product through custom development. This way, there is no direct dependence on the prices, terms, or changes of a commercial AI provider. We can decide at any time which model to use and when to update it, bearing in mind that the data does not pass through external services. At Gilsys we have developed SaaS products for our clients, taking into account that the chosen model must have a license that allows commercial use.
Startups
Startups value being able to validate the product they are about to launch with a predictable AI cost. With private AI in the cloud, the GPU is rented, there is no charge per query, and the price depends on the computing capacity. Private AI in the cloud also fits very well in the MVP stages of phased development.
Social organizations and organizations with sensitive data
Case files, social reports, or health data of vulnerable people, which cannot leave the organization. This is the real case described below.
Industry
Plants where the internet connection is not always reliable can install AI locally, so it keeps working even without an internet connection. More detail in AI for industry.
Public administrations
Citizens' data that is preferably stored under the administration's own control, with a predictable cost that can be budgeted.
Differences between private AI and commercial AI
They are solutions that do not compete on the same ground. A commercial model is multimodal and highly capable in many fields, while private AI is sized for what the company needs.
Commercial AI: multimodal
For example, ChatGPT, Gemini, or Claude
They are huge models that run in data centers with a very high infrastructure cost, but that cost is shared among millions of users who pay per use.
- They have very advanced knowledge in many fields.
- They process text, images, audio, and in many cases even video.
- The information sent to them leaves the company. The privacy policies of each contract need to be reviewed periodically.
Private AI: specialized
Smaller, open-source models
Most companies do not have the budget for infrastructure equivalent to that of a commercial model, nor do they need it. A smaller model is chosen, limited as far as possible to the tasks required.
- To process text only, it does not need image processing capabilities.
- To analyze numerical data, it does not need knowledge of history.
- It has fewer parameters (the term used to measure an AI model), but that way it can run on an affordable server and perform well within its scope.
- The information does not leave the company's infrastructure.
What information can be shared with a commercial model?
With a commercial service, GDPR compliance depends on the contract the provider offers. It is necessary to check where it processes the data, whether the data is used to train models, and how long it is stored. With private AI, everything depends only on the company.
Information that can go to a commercial model
Marketing texts, requests to summarize public articles, translations of product sheets or technical data, generating ideas or generic code. None of these tasks identifies people or compromises the company.
Information that should not leave the company without checking the GDPR and internal rules
CVs, case files, customer records, medical records, internal reports with real figures, or any document with special categories of data.
Large providers such as Microsoft or Google offer specialized services for processing documents such as invoices or receipts. They process them in Europe and have retention policies that make it possible to meet privacy requirements. We explain it in confidential document analysis.
Where is private AI installed?
Private AI in the cloud or on a local server
On the company's servers
It makes sense when the company already has the infrastructure to host its own servers and an IT team to maintain them. Apart from the new server, it does not require additional investment in hardware. It is the option with the most control, since the information does not leave the network, not even encrypted.
In a private cloud with a rented GPU
Renting a machine with a graphics card dedicated to the company is affordable and does not require investing in infrastructure. It is the usual choice for a SaaS product and for startups that want to offer their own artificial intelligence services.
Hybrid systems
A company may have different projects and each project may even use several AI models. Each model can run on a local private system, in the cloud, or as a commercial service. In this scenario, functions with sensitive information can be routed to a private model and the rest can use a commercial one, which is usually more capable and cheaper.
RAG: giving AI the context of the company
Once the information has been processed, it can be queried in natural language
The goal is to be able to ask questions in natural language about the company's information (manuals, procedures, technical data sheets, or contracts) and get answers that come from its own documents, not from the model's general knowledge.
There is no need to train a model from scratch: it is not viable, as it requires enormous amounts of data and computing power. Instead, a model is selected (the one considered most suitable for each case) and given access to the company's documentation at the moment of answering. That way, when a piece of data changes, so does the answer.
This type of system is known as RAG (retrieval-augmented generation): the system retrieves the fragments of the documentation that answer the question and the model writes the answer from them. When it is set up on the company's servers, we call it private RAG.
The same vector database can also be used, without writing answers, to search by meaning, relate data, or detect duplicates (product sheets, profiles and skills, technical information) within an application. More details in AI integrated into applications.
How we approach RAG at Gilsys
The six components of a private RAG for internal documentation
From document to answer, everything within the company's infrastructure. The technologies for each component are examples: in each project they are chosen according to the type of documentation, the volume, and the available hardware. They can be replaced when a better alternative appears.
1. Document repository
For example, ownCloud or Nextcloud
The company's document management system stores manuals, procedures, contracts, and case files. The documents stay within the document management system.
2. Chunking and embeddings
For example, bge-m3
Documents are split into chunks, and each one is converted into a vector (embedding) that represents its meaning, not its exact words.
3. Vector database
For example, Qdrant
A specialized database stores those vectors and, given a question, returns the chunks closest to it in meaning, filtered by the permissions of the user asking so that nobody accesses information outside their scope.
4. Language model
For example, gpt-oss, Mistral, or Qwen
A language model writes the answer from those chunks and is configured to stick more or less closely to them, or to say that it does not have specific information when it cannot find it.
5. Model server
For example, Ollama or vLLM
The model and the embeddings run on the server's GPU. Depending on how many people work in parallel, more or less memory and processing speed is required.
6. Query interface
For example, AnythingLLM or a custom interface
A chat interface where each user accesses information limited to their workspace. This interface can be custom-built and extended with advanced features through custom development.
How a private RAG is used: chat, workflows, and APIs
A chat interface
An internal web application or channels such as Telegram, WhatsApp, or Teams, with private memory for each conversation.
Workflows and automations
With n8n or custom development, we automate the continuous ingestion of information: we extract the data, for example from applications and files, process it, and index it in a vector database so that the RAG can query it without manual intervention. More details in AI automation.
A custom API
The RAG can be used as a service of its own, for example with FastAPI or Slim Framework, with one endpoint for indexing documents and another for querying, to which tools such as an ERP or custom applications and services connect.
Real case: private AI at a nonprofit organization
When information cannot leave the organization
A nonprofit organization wanted to query its private documentation with artificial intelligence: case files with highly sensitive data, internal protocols, and codes of conduct, information it considered sensitive and did not want to transfer outside the organization.
In this case, everything runs on local servers at the organization itself. The documentation is centralized in the document management system, the open-source model runs locally, and each department can query the information through a restricted chat, within the workspaces it is authorized to access. No data leaves the organization.
Permissions managed in a single system
User and permission management is centralized in the document management system. Through custom development, our system propagates that access structure to the vector database in real time and automatically reindexes every updated file, so that nobody can query information they do not have access to.
Validation with a consumer GPU
The project was validated with a consumer graphics card, which could run the model, compute the embeddings, and test a few simultaneous conversations. With more memory, the same project will be able to offer more context and retrieve more chunks for each answer.
Automatic indexing
Documents are indexed unattended, without interrupting use of the assistant. In testing, indexing several thousand public sample documents took almost a day of processing.
How we implement private AI in a company
We analyze the specific requirements and offer different technology alternatives, potentially starting from open-source tools with no recurring licenses, which can be extended with custom development.
Before starting: well-organized information
Private AI works with the company's data as it is. Before starting to work with the information, an inventory is needed of the available information, its location, its quality, and the associated permissions. If it is not well organized, the first step is data governance to organize it and, for documents, document management.
Analysis of use and processes
The purpose for which the AI needs to access the information is determined: querying documentation, classifying, or working within agents and automations. Based on the analysis, we recommend an on-premises server, a private cloud, a hybrid architecture, or commercial AI.
Choice of model and infrastructure
Which model fits the use and the language and what hardware it needs to answer in a reasonable time.
Connection to the data, without training the model
The model is not trained with the company's information: it is connected to it. If it has to answer from documentation, the documents are split into chunks, converted into vectors, and indexed by workspace; if it works with internal tools such as an ERP or CRM, the data is queried directly, limited by each user's permissions.
Integration with the company's tools
We develop the query interface and connect it to applications, agents, and workflows. To avoid duplication, users and their permissions are synchronized from a single point.
Testing at three levels
An AI-generated answer is not validated by comparing the text word for word. It is checked at three levels before any change is published. First, automated tests that verify that documents are segmented correctly, that each user's permissions are applied, that sources are cited, and that the engine retrieves the right chunks. Then, a decision AI such as Jev evaluates each answer: it confirms whether what it states is actually found in the documents, whether it answers what was asked, and whether it rules out hallucinations. Finally, between 50 and 100 real questions from the company, with the answers validated manually. Every time the model, the prompt, or the documentation changes, the test suite must be run again to make sure the result does not drift from the set threshold.
Maintenance
A periodic budget with defined tasks can be agreed, although it depends on the capabilities of the client company's IT team. It is worth stressing that every time the model is updated, the test suite must be run again.
Frequently asked questions about private AI
What is local AI and how does it differ from private AI?
Does any regulation require AI to be on private servers?
Can private AI also be used for agents and automations?
What hardware does private AI need?
It depends mainly on two factors: how many people will use the system in parallel and the size of the model (its number of parameters). A model with more parameters is more complete but needs more memory on the graphics card, and each simultaneous user adds load. As a guide:
- Basic tasks or an assistant used by a few people at a time: a 24 or 32 GB consumer graphics card, which can hold a model such as gpt-oss-20b, which according to OpenAI runs with 16 GB.
- More simultaneous users or more capable models: a 48 GB card.
- A large model, such as gpt-oss-120b: an 80 GB card.
In the cloud, on RunPod for example, a 24 GB graphics card costs between about €0.30 and €0.79 per hour and an 80 GB card between about €1.06 and €1.60, as indicative figures. In any case, different hardware alternatives are assessed in each project.
Can the cloud machine be switched off when it is not in use?
How long does it take to implement private AI?
How much does private AI cost?
Deploying private AI involves two costs: implementation and hardware. The work depends on the type of project:
- The model only, for use by applications, agents, or automations. In this case, it is assessed within the budget of each application that requires it. The cost is not in installing the model but in preparing and securing the API that uses it and developing that API or the related application for the project.
- A chat assistant over the company's documentation (RAG): as an order of magnitude, €5,000 to €12,000.
- Broader projects that connect private AI with several company systems or are integrated into tools and applications as services: quoted individually, according to specific requirements.
The server is bought or rented separately, as explained in the question about hardware. These are approximate figures. Requesting a quote is free and with no obligation.

What documentation should the team be able to consult?
With the documentation available and what needs to be asked, we can see what is needed and whether an on-premises server or a private cloud is the better option.




