Same Business. Same AI. Why Could the Kerala Version Cost More to Run?
Imagine two businesses running almost the same AI workflow.
Same use case. Same number of customers. Same AI model.
One mainly serves customers in English. The other mainly serves customers in Malayalam.
Would you expect their AI running cost to be the same?
I probably would have.
Recently, I came across a blog by Santhosh Thottingal about Malayalam tokenization, and it made me think about this differently.
There is a cost variable in AI applications that is easy to miss:
How efficiently does the model tokenize the language your users speak?
Where the difference comes from
LLMs process text as tokens. The tokenizer decides how a sentence is split before it reaches the model.
And different tokenizers can handle Malayalam very differently.
In Santhosh's experiment, a 175-word Malayalam passage used around 335 tokens with a Malayalam-specific tokenizer, but around 2,772 tokens with the GPT-4 tokenizer.
That's more than an 8× difference for the same passage in that particular test.
This doesn't mean Malayalam always costs 8× more. The result depends on the model, tokenizer and text.
But it shows that language can become a real variable in AI infrastructure cost.
Think about a real Kerala business
Take a business running an AI customer-support system through WhatsApp.
Customer
↓
Malayalam message
↓
LLM
↓
Database / RAG
↓
LLM
↓
Malayalam response
Every message going into the model consumes tokens.
Every response consumes tokens.
If you're using RAG, the retrieved documents also consume tokens.
At 100 conversations, the difference might not matter much.
At 100,000 or 1 million conversations, it can become a very different cost calculation.
This is different from normal infrastructure
With a traditional application, we usually estimate:
Server
Database
Storage
Bandwidth
CDN
With an AI application, we also need to estimate:
Messages per user
Average conversation length
Input tokens
Output tokens
Context size
Model pricing
And for a Malayalam-first product:
How many tokens does our chosen model use for Malayalam?
That's something I don't think we normally include when estimating the initial cost of an AI project.
It's not only about the API bill
Token inflation can also affect:
Context: More tokens mean less useful content fits into the same context window.
RAG: Malayalam documents can consume more context, potentially reducing how much information you can send to the model.
Latency: More tokens generally mean more processing.
Scale: A small difference per conversation can become significant at large volumes.
So tokenization isn't just a pricing detail.
It's part of the architecture.
What about the “13×” claim?
You may have seen the claim that Malayalam can require around 13× more tokens than English.
I wouldn't treat that as a fixed number.
Santhosh's benchmark itself shows why. Different tokenizers produced dramatically different results for the same Malayalam text. His Malayalam-specific tokenizer was around 335 tokens, while the GPT-4 tokenizer was around 2,772.
So the better question isn't:
How expensive is Malayalam?
It's:
How efficiently does this particular model tokenize Malayalam?
What should we do as developers?
Before choosing a model for a Malayalam-heavy AI application, take some real sample conversations and benchmark them.
Measure:
Token count → Cost → Latency → Answer quality
Don't just compare the advertised price per million tokens.
A model that looks cheaper on paper may not be cheaper for your actual Malayalam workload.
The good part is that this isn't a fixed limitation. Santhosh's experiment shows that Malayalam-specific tokenization can reduce the token count dramatically. Indic-focused models are also working on improving token efficiency for Indian languages.
The bigger point
We already calculate server cost, database cost and storage cost before launching a product.
With AI, we need to add one more line:
Token cost.
And if the product is mainly used in Malayalam:
Malayalam token cost.
Two businesses can have the same workflow, same AI model and similar traffic but their AI operating costs can still be different because their users communicate in different languages.
That's a cost worth measuring before the product goes to production.
